I've noticed this a lot in my own photos, but I mainly use my 16-35 so I suppose the downside of going wide with most of my vista type shots. I prefer my main subject to be a part of the bigger picture, rather than just filling the whole picture, so I suppose I don't have much of a choice there.
In the fact the other day in Brighton, I half surprised myself that the West Pier was bigger than I remember it as I'm forever taking shots of it with a wide making it look a lot smaller than it is.