If you can get away without the ND filter using a low ISO, smaller aperture (diffraction aside) then go without.
If anything, adding a filter infront of the lens could increase the likelihood of glare, reflections, vignetting, colour casts and reduction in sharpness.