'OK' depends on you! Trespass is not a criminal offence (other than on certain places - railways, airfields, nuclear, military installations, etc), but is a civil offence. If anyone finds you somewhere, then they can ask you to leave and may even call the Police. I have heard of some 'over aggresive' security operatives / site owners out there though.
Hypothetically, there's nothing to stop the Police accusing you, however, it's rare (but not unknown) that Police get called out, it's more likely to be private Security. If they found anything on you that could be used to gain entry, or found evidence of fresh damage then there would potentially be grounds for throwing the book at you.
I'm somewhat out of touch with the urbex community of late, so I wouldn't take this as gospel - have a read of the urbex forums and see if there's any 'stickies' on there for more definitive guidance.