There is a scene in the movie when two guys are entering Frank's room and are trying to kill him.

He gets away and when he has come down the ladder, a man with a car is trying to run him down.

Now watch when they fight in the car. Frank hits him and his hat falls off his head, but in the next shot, he has his hat back on.

I don't think he would think about getting his hat back on in a fighting situation like this.