Comment by kevinemoore

Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.