I ditched my Canon 50D with L glass to pick up the X-E1 due to weight/back issues. I was worried about losing the 70-200 f2.8 and having nothing to replace it with (at the time).
I ended up buying the X-E1 + kit zoom as well as the 35mm and the 14mm. I intend to pick up the 55-200 at some point as well as the 23mm when it lands. That 14mm though is just forcing me to take landscape shots all the time. I'm actually enjoying myself pressing the shutter. Even the "kit" zoom works well.
Macro-wise, the 60mm is pretty good, but the general consensus (based on what I've read) is that because AF isn't massively important, using legacy glass and an adapter will do the trick just fine. You can read more about this on the Fuji x forum
here. Consider a Nikon adapter and just keep your 105?
There are a LOT of threads from DSLR users who've downgraded on the forum linked above. Most are happy with their decision.
Ian.