It's because the town has one out of the set of all castles.
You can also say "Our town has the castle", which... now there it gets weird. I think it means it doesn't have much, but it has that. Well I guess it's specific in that it is one of the few (specific) things it has going for it.
You can also say "Our town has the castle", which... now there it gets weird. I think it means it doesn't have much, but it has that. Well I guess it's specific in that it is one of the few (specific) things it has going for it.