Hi shed9, could you give an example of the sort of inaccuracies you are getting please?
As I see it the possibilities are:-
Hole diameter, or deviation from the correct circular form,
Angle of bored hole to surface,
Exact position of hole,
or Straightness of bored hole.
What accuracy do you need to achieve?
One of the most difficult things about drilling a hole is starting it in the right place. With wood which has a great difference between the hardness of summer and winter growth often a hole centred on a 'hard bit' will fall into the nearest soft bit. In this case any play at all in drillbit support, be it shaft bearings or chuck carriage bearings/slides will make matters worse. Whatever guide system is used. The use of a centre punch will help but even using those is difficult.
A standard twist drill is inherently blunt, try pushing by hand into timber, and will readily move to the nearest soft bit. An auger, with sharp point, should be better, but try starting on the edge of some open grain, in say Ash or Oak to find out how difficult that is.
The firmest reference, in most workshops, is probably the router, if it can be positioned correctly. Drill holes in the precise location, correct diameter of course, using a suitable bit in the router. When you have run out of drilling depth change to an auger. That way at least the hole should start in the right position.
xy