I went from 5DIII to Fuji four or five years back, and while objectively it made for a lot smaller kit, with minimal difference to the pictures, and I was genuinely happy with the output at the time, when I would look back over older pics, I could't help but feel there was something missing, so much so I recently switched back to FF with Sony.
My head tells me there's no real difference between the images I took on my Fuji XH-1 and 56 f/1.2 compared to my Sony A7III and 84 f/1.4 for example (strict 'equivalence values' aside), but it was a nagging doubt, so my heart would tell me something was missing. I don't regret my 5 years with Fuji at all, but the niggle would not go away, and so I'm back! It could be just a itch to try a new system, but rightly or wrongly, the sensor size played a part in that decision.
One thing my previous switch from Canon to Fuji did cement in me was the move to mirrorless (hence the Sony). Nothing to do with size (it's not the camera that's heavy it's the lenses, and they are the same for FF regardless of which system), more to do with the utility and focus accuracy etc.