I have similar memories. I remember as kids we used to play in an old castle by the beach, its just open and anyone can go and wander around it. I travel around the UK with my kids and these days everything wants to charge an entry fee, even other castles in similar states of decay/disrepair.
Easily my favourite castle https://en.wikipedia.org/wiki/Llansteffan_Castle