portInit

We certainly have, it is admittedly quite new to both of us, so we have been exploring the best way to introduce something like this, as well as tying up technical bits. The open core model with commercial features is certainly appealing.

Open to any perspectives on this.

Thank you! Will make this more clear.

I'm having trouble replicating. Although I did get a prompt from Safari to allow downloads from crul.com. The screenshot looks like it is redirecting from the button, is that what you are seeing - rather than the button triggering a download?

Yeah exactly. Living up to your username :) nice find! Note this is a global default, unlike domain policies which are associated with a domain.

That's unfortunate. A quick look suggests ISP but maybe not with the hotspot. Will do a little more digging.

Thank you! Docker image has everything for functionality, there's just an update check that pings our end to check for updates.

Sorry to hear that - we do need to think about this. It's our first pass at product tiers and features and we may need to adjust.

Scheduling and Domain Policies were the main features we chose to gate initially as they don't affect core functionality other than performance and deployment.

We'll have to see how close our current postgres integration is! Would like to understand more and will reach out.

Thanks for this, we're still trying to figure out these details ourselves.

Related to the quote, we've seen interest for API data wrangling, where prepackaged data feeds can be cumbersome to edit, or other implementation details become challenging like credential management, domain throttling, scheduling, checkpointing, export, etc.

It's also interesting for webpage data when you need to use a particular page as an index of links to filter and crawl. We've tried to build an abstraction layer around that.

Initially, we were mainly focused on webpages, and wanted to bypass the visuals of the browser and use a headless browser to fulfill network requests, render js, etc. then convert the page to flat table of enriched elements. With the APIs as another data set there is some work for us to do around language.

We're now trying to figure out what workflows are most relevant for crul to optimize around, as well, honestly - we just built what we thought was cool. Some features/workflows will certainly be more straightforward with existing tools and software - especially for a technically savvy user.