I suppose it all depends on what you are looking for. Before I got my 450D last year, I had a Panasonic Bridge camera. After having previously owned film SLRs before, I wanted to have something that bit better than the bridge camera.
What I really liked about the Panasonic, however, was the fact it was small and light. Whilst the 450D is heavier and larger than the bridge camera, I chose not to go for the 40D as the 450D also had good reviews, I didn't know whether my photograhpy would justify a 40D, and, as I travel regularly, both in the UK and abroad, and also tend to travel with hand baggage only, I didn't want to go for something larger or heavier like the 40D.
I think it's fine to have a larger heavier camera if you're driving somewhere and don't have to lug all your kit around, or if you're staying in the one place, but if you're travelling and not driving, and have to carry around clothing, washbag and camera equipment, compromises need to be made somewhere on the weight side.
I am flying to Norway soon with Ryanair to see the Northern lights and I also have a budget internal flight over there so I need to think carefully about what I will take with me as it will cost over £40 extra if I exceed my hand luggage limit, which is 10kg with Ryanair. The overall size of my camera bag is another factor to consider.
For that reason, I will not be taking my 70-300 or my kit lens but have chosen to take my new 10-22 and borrow a mates 28-135 to give me a decent range; I have figured that as this is a city break/scenery trip, I will take a gamble that 135 will be long enough for any telephoto stuff.
As with all things, you have to balance out what is right for you. I'm not denying the 40D is a good camera, by my 450D gives me the quality I want and I am quite happy with the smaller size.