I just had a debate with a now former client about exactly this. I have a firm 20 MPH cut-off...doesn't matter if it's sustained or gusts.
It's not that I'm worried that the aircraft can't handle the wind; it can. I'm thinking ahead to a GPS failure and subsequent fly away. I'd rather not have to address that sort of issue on a windy day.
My experience is that still photos start to suffer with ~15 MPH wind. Certain maneuvers (i.e. an orbit) also become less precise. So, I stick to my 20 MPH policy, even if it is arguably conservative.