Comment by chatmasta

Cool project, and a mammoth undertaking in terms of scraping and data processing. Would you be able to share any details on what your ingestion infrastructure looks like?