You're not mad, I seriously contemplated the move from a two system set up (D750 and Olympus EM1) to a Sony A7RIII. Several things stopped me, the first and main one being the ergonomics of the Sony as I found the grip too cramped in terms of space between the grip and lens (even the EM1 has noticeably more room). The next thing was the slight EVF lag when panning, I know I had all the optimal settings as there was a Sony rep on hand to change all the settings. The next thing was the balance with heavy lenses such as the 70-200mm f2.8, not too bad with the camera in landscape orientation but as soon as I turned it to the portrait orientation it didn't feel right. And lastly the cost. I chose to buy a D850 instead to replace my D750
On launch of the Nikon Z7 I contemplated the switch again and whilst the Nikon feels much better ergonomically (both in terms of grip room and also balance), the AF performance was noticeably worse than my D850. Not bad by any stretch, but for the cost to change I would not be willing to take a hit on performance. It also had the same minor EVF lag, fps with exposure change is low, and then there's the lack of native lenses and all current native lenses are extremely pricey.
If you only shoot landscapes and portraits etc then these mirrorless will be fantastic and probably better than DSLR due to live view and the extra bells and whistles, but if you shoot sports and pan a lot then they might not be for you, unless you look at the A9.