Silly question, but do you have it switched off when not in use?
Some of them have a little lever on the top. When it is upright, it is off (or on). When you want the oil to flow you rotate it 90 degrees sideways which lifts (or drops) the needle. I cannot remember which is which, but it has two settings one of which should stop the flow.
Old oilers and cheap replacement oilers all leak. Fill them with just enough oil to suit your present turning session when you commence. Oil is cheap. Spindle bearings are not.
If you are keen, you can dismantle and lap the needle to its seat.