You wont get a clean hole with a brad point or butterfly twist drill as you are going in at 90 then moving to your angle, you end up pulling one side of the bit out of the dowel so the lower edge of the hole rips (as it is doing). As your jig didn't work, I'd be tempted to try a piece of tape over where you want to drill, it should help a little bit with tear out. the jig is still the right idea, but you might need to make it from something other than a old piece of fast grown pine.
crack on fella, if your not painting it then find yourself some wood filler that's colour matched or use the old trick of sawdust and woodglue to make your own, apply a thin smear under and around each branch let it dry then do your sanding for top coat, if you've got a thin enough fill, it shouldn't show on the finished project.