When Peter remembers his past in Neverland, he is telling tink about it and how she saved him and took him to neverland when he was a baby and it shows how peter went back to see his parents and and they forgot about him, well he was about 5 years old then, and then he went to see Wendy all the time and he was about 12 then. And I thought in Neverland you DIDN'T grow up!! but it seems he did!

Maybe it might be that you never grow up from being a kid. Until Peter leaves neverland he remains a kid, doesn't mature. Or in the indians case, you grow up but you remain a kid at heart. For the pirates they were already mature and never left neverland, or went to neverland after they had grown up.