Is it possible that the tapers are rusted together?
It may help to firstly hit the flange as though to tighten it to loosen the grip, then hit the flange on both sides to release it. I also think a heavy duty slide hammer attached to the flange is the answer.