There are other ways to the Cliffs of Insanity. At one point, Vezzini says: "only Fezzik is strong enough to go up our way; he'll [the man in black will] have to sail around looking for another port."
The prince is there so quick not because he is legitimately the hunter/tracker of his reputation, but because he knew where Vezzini was headed, having hired him to do the job. There must be land routes to Gilder, not all of which go through the fire swamp. Vezzini avoided those because they would be patrolled upon the Princess's disappearance.
I'd bet the VERY long-winded book goes into it, but I've never made through to be sure.
I made it through the book, theoretically, and although I don't remember it mentioning how the prince got there; your answer seems the most intelligent conclusion.
First off, it is Fezzik, not Phisyic, secondly, the book (which I have read numerous times) explains that the prince did find a harbor and the reason he got there so fast was that he was riding his "four, white horses" and they are the "fastest horses in the known world."