It isn't solely 'your' chimney though, even if though, it is for all practical use and intruding on your internal space. The chimney is integral with the party wall for which there is equal shared responsibility. As the picture shows the centreline of the party wall indicates that about 3/4 of the chimney stack is on 'your side' and a 1/4 on his side (including the flashing on 'his side').
So given that you have fulfilled your responsibility to the party agreement by maintaining 'your side', so should he to his. Unless of course he is willing to pay you the extra to get it done on his behalf. If I were you I would spin it round by pointing out the potential cost saving to him by paying you (or your builder directly) for his side now rather than him having to pay more later on for his own builder.
Sadly there must be many, many more neighbours like yours around the country. Not enough commonsense to go round anymore!
cheers,
ike