I have not had issues yet but when I fly in the neighborhood I am at almost ceiling limit so as to not bring a lot of attention to my drone. As far as airspace above ones house.......it is up to interpretation to a degree. What I mean by that is what one poster stated about usable space above the what we commonly refer to as our "property". If you fly directly over ones home at about lets say 200-400 feet I would say there is no area for gray area or interpretation for that is a distance high enough not to entangle any legal battle.
Bottom line is what my fellow LEO "UAVdude" stated, there are a lot of miserable people in life that just want to complain on anything......even the presence of the sun every morning through their window! Now I am not advocating pissing off your neighbor by buzzing all the time over his house, but you have a right to the air space as long as the FAA has no issue with the flight and it's pattern.
Hopefully the neighbor will fade away. Curious however, to the OP, what was your altitude over this guys house that made him upset.....do you have a any clue as to your altitude?