When using a honing stone or moving a chisel back and forth along a piece of sandpaper, not only do you have to move the chisel, you also have to move your arms back and forth. You usually have to accelerate your arms, hands and the chisel, then decelerate it and accelerate in the reverse direction, then decelerate to the start point, and so on. You have to provide the energy to move both chisel and arms. Arms are heavy, and need a finite amount of energy to move them. Putting it another way, the whole moving 'system' consists of chisel, hands and arms.
With the hand-crank grinder, only one arm needs to move. There's a further efficiency gain because the arm and chisel does not need to be constantly accelerated and decelerated; once the system (arm, hand, crank handle, gears, stone) has reached it's working speed, only sufficient energy needs to be input to maintain wheel speed. (An analogy would be the greater efficiency of a gas turbine over a reciprocating engine - you're not using energy to constantly slow, stop, start, accelerate the other way, then decelerate and stop, start again ..... and so on.)
Thus, the hand-crank grinder is more efficient than the back-and-forth rub stone. This is very noticable in practice, as Nabs and Bugbear commented.