Before the Joker shoots Bruce Wayne in Vicki Vale's apartment, he delivers the line "Ever dance with the devil by the pale moon light?" He tells Bruce that he asks this of all his prey. By that point in the movie he had killed at least four people and never once said that line.
While he may have killed 4 people and not uttered the phrase, there is a big difference between being a casualty and being prey. Prey usually indicates a stalking situation. So, perhaps he only 'dances with the devil' when he's got a specific target in mind.
Actually prey doesn't mean the people he has killed, it means the people he hunts, he didn't hunt the four other people, they just seemed to be in the wrong place at the wrong time, or they betrayed him, therefore, batman (Bruce Wayne) was his only PREY.
Why would Bruce be Joker's prey, he's only sat in the wrong place (Viki Vale's apartment) at the wrong time (when the Joker decides to pay a visit).
Surely this would make him an unfortunate casualty rather than prey.
If you remember that Jack had been stalking his parents the night they were killed and he had shot them but with the way the scene was done we wouldnt have noticed the line being said. Jack then said it to tiny bruce and then he had to run. and the other killings were done at angles where we wouldnt have been able to hear him say it (except for when he killed Eckhart)