I've not had a TV for over 8 years. I don't think that my life is lacking anything as a result. Indeed, when I go over a my friend's house and the telly is on it really annoys me. I don't like it.
You do not need a TV license just because you have a television set in the house.
You DO need a licence if you want to watch the BBC (Live or iPlayer, but not legacy stuff on YouTube), or any other terrestrial channel (ITV, Channel4 etc) live. Catchup for those is still OK, I think, unless they have changed the rules in the last few years.
When the TLM* knock on your door, you are not obliged to let them in. You can, of course, if you wish, but they do not have a legal right of entry. If you want them to go away, the phrase you need is, "I withdraw your Implied Right of Access". The Implied Right of Access is because you have a letterbox, and so the implication is that anyone can come up your drive to deliver something. But it is Implied, it is not a legal right, so you can withdraw that implied right.
That means that they are now trespassing if they come up your garden path. They must not come closer than the pavement outside your gate.
Of course, if you really are watching BBC or anything else live, they you should pay up, but if you don't, don't be bullied by them .
S
*Television Licensing Mafia