I agree. Here's my take on BCO (but based on one instance of personal experience on a two year project).
BCO's are understaffed. They have targets to meet. They may be spread very thinly over the area. Mine in Herefordshire have a huge area to cover. So they have to be pragmatic. Over a period of time they will get to know many of the regular builders on their patch. They will develop a working relationship with them. If the builder is deemed 'trustworthy' then for a run-of-the-mill project the BCO is, like as not, going to let it go by with only a nod in the direction of proper inspections. And builders being builders, eventually they will cut corners because they know the chances of it being picked up are slim. So they won't necessarily call in the BCO as they know 'they'll get away with it' as they are trustees. For example, bet you won't get the BCO coming along and monitoring the pressure test on your soil pipes.
Equally, BCO's have their own 'pet' hotspots. One of my BCO's (you are getting different BCO's aren't you?) is keen on fire prevention etc and twice I caught him out trying to get me to do something that I didn't need to. When I challenged him, he backed down.
Thinking this through a bit more I really think that you ought to consider some pre-emptive action. After all, you don't want builder and BCO to disappear and then two years later start getting damp through? If it was me then I would be writing to Building Control outlining your concerns, their missed inspections being contrary to their website etc. But then I am confident that I can argue the toss with them if they tried to ******** me - as they have done - since I used to take the Building Regs book to bed with me (yup..saddo). In your case I do suggest that you investigate the cost of getting an independent view from a chartered surveyor (from out of area as suggested earlier).