IIRC, the nests are used for just the one year so it will be abandoned fairly soon anyway. Once cold weather sets in,
Its been my experiance that unless its a truly harsh winter the queen will hibernate / over winter in the nest.
And start all over again in the spring. Of course they don't always survive, if its an especilly wet spring, for example they'll drown foraging for food and wood.
Each queen builds a nest reaching about football size through the year, but obviously the "huge" nests you hear about take more than one season to construct.
TBH In my 30 years as a pestie I've only come across a handful of truly large nests.
A couple that stand out in my memory, Wasp, I couldn't find the damned nest, although I knew it had to be in the loft of this particular house.
It eventually dawned on me, the nest was about 6 inches or so thick but covering the the whole of the gable end!
Hornets, There was a nest in a small property, ( one of these BT type out stations converted into a bed sit type thing) they were going under the roof tiles so the nest had to be in the loft, I lifted the hatch, to get a better look, and tore through the center of the nest.
I was attacked by ONE African bee AKA Killer bee and even after stinging me in the forehead it chased and kamikaze into my face for a very long distance.
I've been chased too, You just need to move away from the nest fairly quickly
They usually give up after a 100 yards or so.