It shouldn't be a big issue. The wheel's shaft pivots to adjust the blade tracking. On mine (SIP, not Record) there's a hinge about 3" up from the shaft, and a bolt that pushes against the shaft, so it both tilts and moves in-out at the same time. The tensioner lifts this entire assembly on a screwthread. And I've always had some float as you describe - probably not quite as much, say 0.3mm, but it's definitely there.
The bearings settle in a pair of locations on the shaft (there's a pair of bearings for each wheel, I think to keep costs down) and are kept there by friction caused by the blade tension. Down at the table, the back guides (top+bottom) hold the blade against the pressure of the stock, so there is very little effective pressure to make the wheel move on the shaft, and anyway the blade would have to climb up the crown of the wheels to be a problem - I'd expect you to set it so that the meat of the blade goes over the crown and the set teeth ride off the crown to the front (obviously it's different for very small blades).
I had an annoying drift issue over the w/e, but almost always I find it's not actually drift (the blade not wanting to cut straight) but something else, for example if I've damaged a blade by dulling one side more by cutting circles.
In this case, I'm pretty certain it was the fence twisting in use (off square, vertically), and an awkward bit of design on the table, which often gives problems. That was exacerbated by me ripping fluffy, damp wood with teeth that were a bit too small (still for ripping, but too-small gullets probably), and a ZCI that kept blocking with tearout splinters that were pushing the stock around.
I should know by now: don't skimp on the setup! With the wrong blade, or an asymmetrically-worn blade, or the wrong ZCI, or in my case also not properly checking the table "halves" were level with each other properly, or not checking the fence is secure and square, I get bad behaviour. If I'm systematic and religiously careful, one pass with a hand plane after the saw is usually enough for spot-on dimension (even that is unnecessary for tenon cheeks). It's an old saw, with bits modded, replaced, and repaired, but I still get good results.
I think you're worrying unnecessarily, unless you're getting obvious issues with new middle-sized blades of good quality.
E.