Normally it is held in tight with only a small movement of the plate. When you open the plate, (I can see a small brass pin) did you press that in to move the plate fully back?
The other thing to check is on the underside of the plate, there is a screw thread which secures the pivot plate, you will need an allen key to tighten it - if this is loose it can allow the plate to swivel.