Depends on the holiday and what I think I can get away with.
Earlier this year we spent a week in Scotland (Loch Lomond, Glen Coe and Cairngorms), so I took my 6D, 16-35, 24-105, 70-300 and 50mm, plus filters, tripod and EOS M10 backup camera
My argument with this was that I didn't know when I would next get back to Scotland so I wanted some good images. My OH was pretty good about it, even allowing me nearly an hour at the waterfall in Glen Coe, while the kids played in the water.
I didn't use the 50mm and barely touched the EOS M10.
In the summer we went to Devon and I cut it down (slightly), I took the 6D, 16-35, 24-105 and 50mm, filters, tripod and EOS M10 again.
This time I varied what I carried around during the day according to what we were doing.
Both were family camping holidays, but the Devon trip had the in-laws as well, so there was less time to spend on photography and less time for individual pursuits on the whole.
Like Andrew, it's hard to get out for sunrise or sunset if you're camping.
My thought is that it's my holiday too and I like to enjoy a trip by taking photographs... so long as everyone else gets to do something they want.