I've never shot Polo but I would be trying to as high a shutter speed as I can; when I have done show jumping I am normally way higher than 1/640. The higher the better to freeze action; of course if the light conditions are poor then you get as much as you can get out of it under the conditions; if the light is sunny you will be getting way more.
Its all about managing the ISO at a level you are happy with for the image quality. I personally with my 7D rarely go over 6,400 ISO, if I really cant get a shutter speed I am happy with I will go to 12,800 rather than miss the shot but then I am dealing with a lot of work to reduce the noise.