I'm attempting to give the benefit of the doubt that there could be some actual and tangible advantage during use. All of the explanations are perfectly plausible and I tend to agree with them. Yet, I still wonder...
Otherwise one wonders, too, why wheels gave them so much heartburn. If you're building a temple by hand, you have time to transport a grinding wheel to the jobsite. The time to build a lot of those things was measured in decades I'm sure. Erecting shop fixtures, shelter for the building materials, etc. would be but a speck of time in the total time to build a Japanese temple (have you seen some of these things? they didn't go up overnight). In other words, "we don't have time, energy, manpower, etc. to move a wheel to the site" makes virtually no sense at all. I imagine some craftsmen lived out their entire working lives on no more than one or two projects. Think about it.
They probably built shelters in order to live at the worksite, and certainly other structures, shelters, shanties, and shacks to service the project and its materials, but no time to install a wheel? Frankly, a half dozen or more of them were probably in order.
Sump'n don't gee-haw boys and girls.
Could it be that the rotating grindstone simply never occurred to them? I'm pretty sure there's sandstone in Japan.