When they use the clue where the Liberty Bell tower points to the brick, they would have had to have been there the exact day that it was placed because of the movement of the sun (ok, the tilt of the earth). The shadow would have fallen in a different place on the brick wall depending on the time of year that the clue was written.
it isn’t as tricky as it sounds, any shadow on a given point throughout a year, using a daily photograph, it would appear to move left to right, a brick is about 6 inches long, and say the shadow is 2 feet wide, the range of the shadow for a year could be as much as 4 feet 6 inches. (2feet either side, and the 6 inches of the brick) and since the brick was marked, the shadow, needed to be close enough to get an approximate place to look
It's not so much that they would have had to have been there, but to really be able to determine which was the correct brick they would have had to know the date and time that the position of the shadow was determined. Given the state of timekeeping in the US (and the rest of the world, for that matter) at that time this would have been tricky. There was no such thing as Standard Time - local time was set according to local noon. With a good set of ephemeris tables, and some geometry, they could still have made a good stab at it, but obtaining the precision to pick out the correct brick would be chancy.
Interestingly enough, the time on the Independence Hall clock on a $100 bill is actually 4:10. If the hands are the same length, 4:10 looks just like 2:22, but they're not; the hour hand is shorter. So it turns out they were nearly two hours early.
Interesting point. I want to comment that yes, the shadow is cast over at different times of the year. However, in the winter, it probably would snow heavily and be too cold to allow anyone to place the clue there. But in the summer it would be ideal. Also notice the clue said "as the timely shadow crosses in front of the house of Pass and Stow" The correct time was two twenty-two. So wherever the shadow was at two twenty-two, that's where the clue is. I believe our Founding Fathers in real life would of specified the exact time of year.
At any time of year, the shadow would have been pointing in the right direction at 2:22 or thereabouts, indicating which wall to search. Finding the brick wasn't so hard once the wall it was in was indicated; it did have an arc and compass on it.