In the episode Author, Author (season 1) Marty says that he doesn't have a brother. But in the episode Beware of Greeks (season 5), the whole plot is around Frasier's cousin on his father's side.
That seems like the logical explanation, but in the season five episode, Martin expresses that he hasn't spoken to his brother in years, and it's awkward when they reunite, and his overbearing sister-in-law is proudly of a different culture and heritage to Martin.