I've said before [incl. on this forum] that offence is often in the eye of the beholder.
Racism is poor thinking. A natural result of the fact that the brains [incl. human] react to contrasts and differences, but flawed none the less.
I reserve the right to use any taboo word, in front of those targeted by the word, especially if those targeted are 'allowed' to do so.
I don't use taboo words unless being deliberately provocative.
I reserve the right to offend someone if I feel like it. They'll get over it, maybe retalliate, maybe offend me back. We'll both get over it.
Sticks and stones etc.
The surest way to perpetuate racism is to keep making a fuss every time you think of it, especially if it's not there.