I went from a 300D to a 30D, the photos look just the same but I now get 20% in the bin instead of 20% keepers
LOL - I went from a 300D to a 40D and had to readjust my shooting as I was anticipating and compensating for the 300D's shutter lag.
I have recently upgraded from a 40D to a 50D and have a 400D as backup/son's camera. I mostly shoot sport and had to use the 400D for two months when my 40D broke.
I can understand the 'toy' comment. With the 100-400 lens on a 400D it felt 'wrong' just too small for such a big lens. It still produced the shots though.
The size and weight is also the reason I bought it though as I wanted a light camera for my son. The 40D/50D is around 25% heavier than the 400D.
So useful things to have 40D/50D over the 400D.
Bigger screen - it really makes a difference, especially when showing someone the shot.
Live preview - I've used it a few times when it was invaluable (and should have a couple of times but forgot about it). It's improved on the 50D.
Higher ISO on the 40D/50D. I've used the really high ISO on the 50D in caves as a test. Noisy but gets the shot. The ISO performance of the 40D is better than the 400D in my opinion, the 50D is better still. That may be down to the Digic processor, digic2 in the 400D, digic3 in the 40D and digic4 in the 50D.
Battery life - I can shoot over 900-1000 raw images on my 1500mah batteries in my 40D/50D, but only around 350-400 on the 750mah 400D batteries. Smaller batteries so lighter, but caught me out a couple of times as I wasn't expecting it so soon on the 400D.
Other bit's are very useful once you get used to them, such as the joystick, big wheel etc. I struggle sometimes to find the settings I want on the 400D as I'm used to them being available.
Write speed to CF. 50D is blisteringly fast compared to the 400D, however you need bigger, or more, CF cards as it's a much bigger raw image.