In the very beginning of the movie, there is a little girl on the ship who is Elizabeth when she was young. Later in the movie, after the pirates invaded Port Royal and Will is looking at the ocean,there is a little girl sweeping the porch at the left side of the screen. If you look closely, you will notice that it is the same little girl as in the beginning of the movie. It is eight years later and Elizabeth has been taken aboard the Black Pearl, so even if Elizabeth magically stayed the same age, she couldn't have been at the port because she was on the Black Pearl.