Another little tip.
Put the tool in the holder and let hold it so that the tool hangs vertically and do the nut up, this way your not fighting gravity. Give the handle a wriggle to check that it is seated properly and re-tighten the nut if necessary.
If the jig is set accurately by checking the bevel with a black marker pen, the tool should only need to be swung from one wing to the other once to sharpen it taking a few seconds. In fact for small 6mm spindle gouge I don't switch the grinder on, just pull the wheel round by hand and with a well dressed wheel that is sufficient to sharpen it.
With the holder coming out of the socket, try only holding the holder and tool shaft and not the handle. If you are holding down the handle it's very easy to inadvertently out of the socket and also apply too much pressure against the wheel.