Where would the 35mm one come in handy then?
Dave's simple guide to portrait lenses
Headshots (head & shoulders or even tighter on the head)
The closer you get to a head the more likely you will distort someone's face as the nose becomes larger relative to ears simply because its closer to you. To keep things in better perspective its best to shoot from further away, hence a longer focal length. Traditionally the 85mm was THE go-to focal length for this, but it has other benefits too in that the DoF can be shallow, meaning its easier to place focus on the eyes and also to lose any annoying background
Waist and up
Here you're shooting wider and hence the distortion issue isn't as bad, you can comfortably get away with shooting a 50mm for this
Full length & small groups
Again the distortion issue is becoming less & less important, so now (and for family groups too) a 35mm would be fine
Overall though
You are using a DX body which inherently gives a greater DoF at any aperture than FX does, so f4 on an FX body is more like F2.8 on your camera. Shooting family groups with a 35mm even at f1.8 will mean you have a surprising amount of any background still in focus enough to be a distraction. Its use is that you are close and hence can chat without raising your voice, but the lack of ability to throw the background well OoF goes against it for me
If I was buying just one lens it'd be the 85mm - great for headshots, fine for waist & up (you're just further away than the 50mm) and FAB for family groups as you will be able to throw the background nicely OoF compared to the 35mm, but now you'll be quite some distance away - which can also be good as they'll feel less awkward as there's no obvious camera in their faces
For what you're wanting to do I might in time add the 35mm to the 85mm to give a fuller range of options (you don't always want to throw backgrounds OoF), but first off it'd be the 85 all the way
HTH
Dave