At the begining of the movie, note the length of the Granfather's hair, and the roundness of his stomach. In the scene where the grandfather is practicing his "Ninga moves" and it is dark, and all you can see is his silhouette, his hair is longer and his stomach is flatter. Later on, he has the same weight and length of hair!
You will also notice in the fight on the boat. At the end of the movie when the grandpa fights the main bad guy the same thing happens. You can clearly see that his "double" is doing the fighting not the actor. Since the stunt double is clearly smaller and thiner then the actor play grandpa.