That happened to my wife's ancestral castle, Mor Graig.I recall reading something about the Great Wall of China being in danger because of people taking stones from the walls. It went so far in some areas that people were taking truckloads to build patios. LOL
The whole thing (about 95%) was removed for houses, a bridge, etc. except for a few ground-level walls.
I took a walnut-sized stone. Just to "keep it in the family".