I guess to fully answer the question, you have to ask why you pan in the first place.
Usually it's because you want to use a slower shutter speed to introduce the feeling of movement. The way you introduce that feeling is to move or "pan" the camera in conjunction with the moving subject so the subject remains clear despite it's speed.
Just cos you're panning doesn't mean you have to rely on a background...
If it were me I would be still panning in your situation Neil so that the slower shutter speed blurred the blades on the propeller, i.e. the only real indication you have against a clear sky that the subject is actually moving.