If you're using a tracking mount it's not a problem - you'll just need a longer exposure than you would with a faster lens to collect the same amount of light. I use a Sigma 150-600 on a driven equatorial mount, and that's rather smaller than f4 when it's wide open. I use f2.8 for wide field stuff, but that's because I happen to have the Tamron 17-50 f2.8.