kevinemoore

Joined 11 years ago

Karma

Submissions

Member since

Sep 20

Recent submissions

The search is powered by Elasticsearch. We want to support the full power of elastic’s query language, but so far haven’t found a great way to prevent overly intense (or malicious) queries and still allow arbitrary queries—love to hear from Elastic experts out there. Exact match on multiple words seems possible though.

Comment · 7 years ago

We'll look into that for sure!

Comment · 7 years ago

+1 for everything Aneesh said, but I also wanted to add that the public cloud offers opportunities in data sharing that academia hasn't yet provided, specifically the ability for collaborators to bring their code to the data. I posted a quote from Jed Sundwall, Global Open Data Lead at AWS in another thread. I think he really nails it when he says that the cloud "completely changes the dynamic for sharing data."

There certainly have been efforts in academia to provide shared computing resources. Cyverse (https://www.cyverse.org/about) comes to mind. At Wisconsin many researchers shared clusters using Condor. But, none to my knowledge come close to the scale, reliability and features of AWS and the other major cloud providers.

Comment · 7 years ago

There's no doubt that local processing is a lot cheaper than the cloud for a lot of workloads.

That's a very interesting pattern--publishing "fake" (perhaps safe or anonymized) data online along with code to spur research and development then running the enhanced code locally on private (e.g., PII data) on local compute resources.

We hope Quilt packages can play a role to make that easier. The package serves as an interface and layer of abstraction between the code and the data so the same code can be run against the safe or private data.

Comment · 7 years ago

The list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?

Comment · 7 years ago

Thanks! That's really interesting feedback. We hadn't thought of that.

Comment · 7 years ago

Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.

Comment · 7 years ago

Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?

Comment · 7 years ago

In general, a data publisher simply creates an S3 bucket in their account and sets the access control to allow public read. In the specific case of the AWS Registry of Open Data, data providers create a clean AWS account to hold the data (the account will have only S3 and won't run any compute services). Once the dataset is accepted into RODA, AWS will cover the costs (S3 storage and egress bandwidth) for the clean account.

Comment · 7 years ago

Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc. Looking forward to checking out Open Images.

Comment · 7 years ago

I just want to give a plug for sharing data in the public cloud and S3 in particular. Jed Sundwall (AWS Global Open Data Lead) sums it up really well: "The cloud completely changes the dynamic for sharing data. When data is shared in the cloud, researchers no longer have to worry about downloading or copying data before getting to work. Instead, they can deploy compute resources on-demand in the cloud, where a single copy of the data is made available. It is much more efficient to move algorithms to where the data is, than to move the data to where the algorithms are, and this makes it cheaper for researchers to ask more questions and experiment often." See the full whitepaper here: https://s3-us-west-2.amazonaws.com/opendata.aws/AWS_Sharing_...

Comment · 7 years ago

Sirix sounds like a very interesting system! Is it similar in its internal structure to noms (https://github.com/attic-labs/noms) or Dolt (https://github.com/liquidata-inc/dolt)?

I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS?

I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.

Comment · 7 years ago

Aneesh's co-founder here. I just want to add a word of thanks to Jed Sundwall and the AWS Registry of Open Data. The support of AWS makes publishing data at this scale possible. I also want to thank Jackson Brown and everyone else who worked so hard to compile, document and annotate these large and extremely valuable datasets.

Comment · 8 years ago

Quilt co-founder here, no, Quilt doesn’t use tar or zip. Each package version has a manifest that specifies the set of items it contains. Each item in the collection is stored and transported separately. Items are identified by their hash and stored once even if used in more than one package version.

Comment · 9 years ago

We definitely imagine saving edits as creating new versions of the same package. The most common pattern we've heard is adding more dataframes or files to a package as new results are generated. But, we can certainly imagine other transformations.