I usually stick to ISO200 which is the native base ISO on my Nikon(s). The trouble with dropping below that is that you have to make it up by opening the aperture.
If your light is f/8 bright you can probably afford to open the aperture as shifting to f/5.6 is unlikely to affect your DoF. But for those with dimmer light sources already shooting at, say, f/5, there's not enough latitude to open the aperture without losing focal sharpness.