outpan

Outpan here,

We use a combination of polite massive-scale web crawling and user contribution. We had an in-house web crawler for years until recently when we released it as a stand-alone service.

Semantic web is too good of an idea not too iterate on constantly :)

haha. We should use this as the project mantra.

Would you be open to having a little chat via email? [email protected]

This is great! thanks. I will look into adding the dbpedia.org data.

"a journey of a thousand miles begins with a single step"

As for your question about real world applications: Outpan was only a product database up until last week. It is used in over a hundred apps, some with more than a million users.

I expect this to work on a larger key space as well. It is interesting to see how the expansion works out in terms of usage patterns.

I think graph overlap is what actually determines what data is "accurate". There are currently bots writing to the database by people who are not connected and I have yet to see the overlap (since the key space is too large for the number of bots right now). I'm excited to see how that plays out.

I will send you an email :)

I was driving so couldn't stay on the phone. The number of keys for different concepts is just very large (if not infinite for the sake of avoiding philosophical debates). It will take a some time to have enough data to cover even popular keys. Fortunately I have a lot of time :D

Up until last week, outpan was strictly used for gathering data on product barcodes. We are in the process of adding data in other categories.

Twitter url seems like a strong key since it is referenced in many other contexts.

As for curation, it is intended to provide examples of what key, attr and values are regardless of their content. This is an experimental feature and might be removed/tweaked...

Common Crawl is great! however, some use cases require larger crawls with a higher frequency.

Thanks a lot! This sounds reasonable. Did you guys look into professional services for this?