Let's start at the beginning: there s no "need to be changed". The Conservation Officer CANNOT order you do do anything to an existing building, whether listed or not, except in the case of a Dereliction Order. I'm assuming the house is not so derelict as to be the subject of that.
Once you've got that clear, and explained to the Conservation Officer the limits to her powers, you can discuss the door.
Yes, replacing it will need Listed Building Consent - as will anything else external [if it's Grade 2, anything internal should be outwith LBC control]. But it should not need to be oak. Most Georgian doors were painted, and made of Baltic Pine. So it's important to make a distinction between what the CO wants, and what she has the right to rule on. Conservation Officers are frequently maximalists, who try to impose their peculiar views on everything. If in doubt, find out who your Ward Councillor is, and get him/her involved. Ultimately, if you cannot get a reasonable response from the CO, you can ask your Councillor to have the LBC Application "called-in" to the Council's Planning Committee, for the elected Councillors to decide. And you get your chance to stand up in committe and convince them.