My 100-400L did that. It's got an older IS system without tripod sensing and you can get a feedback loop betwen the gyro and the motors when there's no movement actually happening.
You can see it in live view on max magnification. The image stays still most of the time, then it sometimes drifts very slowly before coming to its senses again.
I used still used to leave the IS on a lot of the time, as it's very effective at killing mirror slap at those critical speeds between about 1/20sec and 1/8sec or so that show it most. But longer than that you have to balance the reducing benefit with slap against the increasing likelihood of IS interference.
As a guess, I would say longer than 1/8sec, turn IS off for sure. Even better, turn it off anyway and use mirror lock-up, or live view, or some other form of damping the mirror vibration such as a weighty bean-bag resting on top, or a very careful hand just to kill the vibes. Tricky business.