I think to avoid vibration, the bit moving, and general safety and accuracy, it's always recommended to tighten. perhaps for a light edge treatment I might not, however 5cm deep 8mm wide mortises in 19mm oak boards? I am tightening it.
when changing height, I might use a gauge on my workpiece...