I suppose it depends where the nest is situated, but I had one in my garden a few years back.The 'bee-line' was right across where SWMBO frequented, using the washing line. So we had the nest removed. The guy who did the job, advised to leave them be, if they are not causing problems. In the autumn, they desert the nest, and that's when you can dispose of it. I haven't had a problem since, but should I have another nest, then all things being equal, I am leaving well alone. As an aside, the nests these critters build are marvel of natural engineering.