I would take "Our town has the castle" to imply that there's a group of towns that collectively only have one castle, and your town is the one that has it.
Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.
Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.