Does Mastodon have something to the effect of robots.txt (or is robots.txt already robust enough to express an answer to the question "This site administrator is willing to have these toots indexed for search?")
At the end of the day, it's a federation protocol. The protocol's creator can have an opinion on whether things should be indexed for search, but it's up to users of the protocol, not the protocol's creator, how it's used.
(... and I have to comment on this bit from the article. "The problem would be a public search engine that Gamergaters and Kiwifarmers use to hunt down vulnerable targets." This heavily implies that Gamergaters and Kiwifarmers can't just build that for their own purposes, which would, of course, ignore robots.txt settings).
At the end of the day, it's a federation protocol. The protocol's creator can have an opinion on whether things should be indexed for search, but it's up to users of the protocol, not the protocol's creator, how it's used.
(... and I have to comment on this bit from the article. "The problem would be a public search engine that Gamergaters and Kiwifarmers use to hunt down vulnerable targets." This heavily implies that Gamergaters and Kiwifarmers can't just build that for their own purposes, which would, of course, ignore robots.txt settings).