Yes, I would say safety glasses and that would be it. No need for hearing protection just for the saw. Some would recommend a dust mask but the amount of dust you get from a scroll saw is truly minimal on account of the narrow kerf.
One thing to expressly avoid is gloves of any kind - you will frequently work at quite close quarters to the blade and you want to avoid the risk of them snagging on it.
Dust mask is essential for me. The dust blower seems designed to blow any dust straight at me. I need to bodge something that blows the dust from the side.