We had an European Tree Bee nest in one of our birdboxes a couple of years ago. They aren't dangerous and are "solitary" (well, not as numerous as wasps anyway!). It seriously p***ed off the Blue-**** (who apparently don't eat Tree Bees), and I only discovered it when I took the box down.
The nest material is chewed up plywood (from the box walls), and it's very tough and springy. You can see where the wood pulp came from - a lot of parallel grooves like yours above. Oddly, the evicted Blue ****' nest was quite intact underneath, at the bottom of the box.
Tree Bees don't nest in the same place twice, and we were given another nestbox as a Christmas present, so problem solved (pretty much).
So I think yours is damage left over from either a bees' or wasps' nest - but the less common sorts, not the usual "Jaspers" or honey bees.