As big as you can go!
As a comparison, I've got a 6x6 workshop with lathe, mitre saw, table saw with outriggers, planer/thicknesses, bandsaw, workbench. That fills the space up pretty completely.
For what you describe I reckon 5x5 will be fine, however I'd echo the other sentiments here to...