It might not be critical to the millimeter, but more than a very few mm out of co-planar can start to give you tracking problems. Having said that, I've only ever come across one BS where the wheels were seriously non-co-planar.
Even if you do successfully eliminate drift (as opposed to just compensating for it with the fence, which is poor practice, if a common one), you can get problems with stopped cuts such as tenons. If the wheels are out you can end up with a blade that leans backwards, for example, so that when you cut your tenon cheek and stop cleanly at your shoulder on top, when you turn it over you find that you have cut further than you wanted. Not too much of a problem on a softwood door that is going to be painted, but not what you want for the rails of your dining chairs.
1 or 2 mm, no problem - 5mm, go to the trouble of fixing it, it shouldn't be difficult.