What I've gleaned from reading many reviews and user opinions:
The 70-200 f/4L IS is the lightest and extremely sharp (good "walk-about" tele-photo if you don't want as much weight).
The 70-200 f/2.8L non-IS is extremely sharp and stop faster than the f/4 IS for about the same price, but larger and heavier. Quite popular with sports shooters I believe, since for sports IS is superfluous but an extra stop can really help.
The 70-200 f/2.8L IS (mark 1) isn't quite as sharp as either, but it's f/2.8 with IS.
Also try a forum search, there are plenty of threads already discussing these lenses.