I no longer own a morticer, so can't go look at it, but...
... mine was the Axminster "little" one with the two-axis sliding table, and very handy it was too. I think the Rutlands one is very similar.
The size you quoted is the socket to accept the top of the square chisels into the head of the machine, just below the chuck that powers the auger.
The Axminster one had a removable sleeve, so would take two sizes of chisel. Yours may well be the same (you won't know until you unpack it, probably). You'll have to look and to measure.
I can't believe that Rutlands sell chisels that don't match their machines. Those ought to give you an idea as to what can be used (at least I'd hope so).
There's no trick to getting good performance, in my limited experience, pretty much just the same sort of stuff that applies to hand tools:
- Smooth/polish the square sides of the chisel. This helps it cut, and reduces the friction pulling it out of the wood, and, like flattening the back of a hand chisel, helps you get a really keen edge on it. It's pretty much the same process, too - flatten on a goodly flat stone or grades of wet+dry on glass or MDF - if you can get a good polish even better, but near the cutting edge it needs to be flat square and smooth. My chisels all had a slight taper so the cutting edges were the widest part. This is to help prevent jamming in the workpiece (especially that first plunge).
- De-burr the chip ejection slot(s) in the chisel. The ones I had, from various sources, were all pretty roughly machined. The chips need to come out easily and not get trapped and to thus build up and cause a jam. Smooth their path, with a fine file (I used a diamond needle file set), and break the sharp corners inside the chisel, if you can get to them - just the edge of the ejection slot, not the bottom cone though.
- Sharpen the chisel by polishing its internal cone. I bought a sharpening kit and it worked well with a battery drill. Axminster sell them. Use cutting compound and consider running the drill backwards at the end, so as to polish rather than cut the surface. You might be able to use a good quality metal countersink, but check the angle is correct before doing anything dramatic!
- Check the augers. They often have roughness on the spiral, which can cause chips to catch. It needs to move the chips away from the cutting edges as fast as possible, so smooth off anything that might snag them. I sprayed mine with a PTFE aerosol coating (Freud, IIRC), because it's heatproof and slippery. It seemed to work.
- In use, face the ejection slot away from the direction you'll cut the mortice. This is so the chips have somewhere to go after the first plunge. It reduces the work the motor has to do and the chisel stays a bit cooler.
- The instructions should say how to get the right gap between chisel and auger but two things might help: There's a shoulder round the top end of the chisel, to locate it in the machine. Get a 2p piece: fit the chisel with the coin jammed between the shoulder and the collar the chisel fits in. just nip it up there. Now fit the auger, and snug that up into the chisel completely & tighten the chuck up. Loosen off the chisel again, remove the coin, and snug it up properly (tightly against the collar).
That gets you a 2p gap between the auger and the chisel, so chips can squeeze past up inside them. I find that's good for 3/8" and 1/2" chisels in softwood. For hardwoods and smaller chisels, use something thinner as a spacer (and a bigger spacer for larger chisels!). It's a bit fiddly to do this, but put a pad of wood under the chisel+ auger so when you (inevitably) drop either of them, they don't get damaged on the cast table.
Running or not, never ever pass your unprotected hand under the chisel, unless you want stitches...
Finally, if you have the machine with a sideways parallel clamp for the stock, glue some fairly fine sandpaper or wet+dry to the face of the clamp, to give good grip. I think I just used Spray Mount or similar - it worked well, reduced the clamping force necessary by quite a lot. On mine the L-shaped table+back fence was a weak point (it's been redesigned since), and using grippy paper meant far less force was necessary.
Hope all that helps. I miss mine now it's gone!
E.