The clue is in the ground. The little chap is riding a rigid framed bike over cobbles...he is bouncing up and down with every stone he goes over. Not a hope of stopping that verticle movement while you pan.
Panning is all about removing ONE plane of movement whilst retaining the other (nOT always side to side!).
To start with pan sideways, and swing with the subject. The faster the subject travels across the film plane, the easier it is to get a smooth swing. Within reason. A low flying jet and you wouldn't keep up, so lets keep this in context.
The way panning works is the subject is kept RELATIVE in the same place on the film, the background and foreground don't, they sweep across the film. The shutter speed required (other than for correct exposure) depends upon the focal length of the lens, the distance of the subject from the lens and the speed the subject is traversing your shooting arc.
Longer lens = faster shutter speed
Further away requires a much slower shutter speed (but it is then more difficult to swing it cleanly)
RELATIVE speed of subject. A jet airliner doing 500mph at 40,000 feet looks like an ant crawling across the sky - it takes a relatively long time to cross your shooting arc.
A bike doing 10mph 10 feet away wooshes through much faster.
You have to balance these things. You find out by trial and error....but, you do need to help yourself by taking out the bumps the bike is going over. That poor little chap is having his teeth chattered.
Try exactly the same settings on smooth tarmac and see what happens. Also I would say pre-focus on manual focus and concentrate on GENTLY pressing the shutter button just before the bike gets opposite you. At the kind of shutter speed (for a 50mm I would start at least as low as 1/30th, no faster) you are talking about yu are going to have all the depth of field you need, f11? or so.
PRESS THE SHUTTER GENTLY - you can introduce camera shake to your pans simply by stabbing the shutter button. Don't change your settings, but change your timing first, and how far away the subject is,with a 50mm even a couple feet will make a big difference. So have a swing, then move forward a foot and do it again at same settings, go forward again to fill the frame and shoot another one. Then try going backwards. KEEP ALL OF THEM SO YOU CAN ANALYSE THE RESULTS....by using the same setting you are only introducing ONE variable, the distance of the subject. From that you can work out how to do it with different things and different lenses and different subject distances.
This shot is one I did the other day. I was in a small 5.5m RIB, I would have like somethng a bti bigger to give a higher shooting position, but had to make do. I was stood on the console with my toes between the screen and grab rail to get some purchase. I also had a hand to stay put in the wash as we rolled about from a hand grabbing my trouser belt!
70-200, polariser, f6.3 at somethink like 1/500th (I can't remember exactly) probably about 110mmlength. he came by much closer on subsequent passes - speed of boat was about 85 knots (close to 100mph). So subject was passing, but also bouncing a little over the waves about, we were swinging around wildly and I was perched precariously...I needed some help to keep things sensible, a bigger boat would have allowed me to go much slower because i would have had a steadier platform.