Clearly I'm doing it wrong because I use my impact driver for almost everything
It's convenience as much as anything - my combi drill is for drilling and the impact is for driving, saves switching bits so much. I also like the smaller form factor of the impact, you can get it into tighter spaces than a combi.
As many have said they can be a bit aggressive and wreck screws, but you soon get used to how much pressure to put on the trigger. For smaller screws and anything that shows, I prefer to drive it most of the way and then finish off by hand. Slotted screws are always done entirely by hand!
Just for reference my impact is a Makita 18v lxt - I think it was the cheapest they sell so definitely nothing fancy, I bought it for building my workshop but I've found myself using it for pretty much everything since.
I must admit though, I'm very tempted by the Bosch 12v flexiclick drill/driver with the interchangeable heads, so I can easily see that replacing the impact for the finer work.