If the fence is of any height and happens to be draught proof, then the slightest window against it can deliver quite a force to break the wall. Use fence posts tight against the wall and fix the fence to those. That way neither the fence nor the wall are dependant upon each other.
xy