Years ago when I got my first loupe, I was advised by Linhof and Studio that 4-6 times was the most useful; I bought a 4 times and never regretted it, although I now use a 6x. On another forum, someone, answering the same question, said that they had both 4x and 6x, and used the 4x on 5x4 and the 6x on their 10x8 camera. So, I suppose format size may play a part (I now prefer 5x7 to 5x4, so I have moved up in size).
My own reasoning runs that if I view a 5x4 screen at 4x and the image is sharp, it should be sharp if I enlarge the negative to produce a print 4 times larger - which covers me for 20x16 or A2 prints from a 5x4 negative.