If the application means the ferrule can be forced on the tenon doesn't need to be perfect so it's usually just a whittle or paring job for me. For a very nice fit I leave the tenon large, then fine-tune with a file. Although if you're good with a knife or chisel they are all you need I find you can sneak up on a perfect fit faster and more easily by filing. Also a godsend where the grain is being a pain.
If you do go too far and end up with a rattling fit (not that I'm speaking from personal experience, no sir) you can glue a shaving around the wood to pack it out, or glue the ferrule in place with a firm mix of epoxy and sanding dust.