I switched over recently because I wanted that Greasemonkey plugin to use people's icons in the replies in Flickr.
For me, the first thing I noticed is that Firefox seems to draw the images on and although it's still quick I think IE preloads an image first before showing it so it "seems" quicker. Although website designs can also determine if the image is preloaded or not.
Also, two things I've noticed which annoy me, although there may be a setting I can change, is that if you are half way down a gallery then click on a photo then use the back button it takes you back to the top of the page and not back to where you were in the gallery unlike IE. The other thing is streaming videos in Media Player. IE would let it stream but Firefox seems to download it all first before playing. As said, there may be settings to change this but I've not really looked yet.