nakovet
Joined 15 years ago
Joined 15 years ago
I attempted to scrape a one page grid with 800 items and also ended up asking for the Javascript look with document query selector instead of the result as I was hitting all sort of limits, context, or the LLM would do the wrong capture, print it out and get worse responses on next prompt.
For us this is resulted in a big weak point on our architecture because when the service reboots both job pushing and job pulling stops, with the pushing being on the API side bringing the API down. With containers we could have multiple of them running at the same time, but the shared reading/writing of the abstract Redis locks itself.
We are considering BullMQ, because the architecture is sane: * job push: API writes to Redis * job pull: Worker reads from Redis then writes the completion.
How do you see this issue for Bunqueue? What happens when it goes down for 5 minutes, can the jobs be enqueued? Can you run multiple instances of it, failover?
Our throughput (jobs/sec) is small we do have 100k+ scheduled jobs anywhere from minutes to months from now.
How do you do background job processing without Celery? Every app/website I ever developed required some sort of background job processing, for Python I use Celery, Rails I use Sidekiq, Node.js I use Faktory. One of the biggest drawbacks (at least until a while ago) was that setting this up had to be a hacky webhook call to your own app that allowed up to N seconds request/response.
One thing that was obvious to me is that if you want to focus on a specific topic the $15/month subscription is too cheap, you can pay, take the course, cancel because you are done with the course, all within the month.
The other thing that is challenging is that the majority of programmer is either too cheap or "too broke", $15 USD is a meaningful amount of money in other countries with weaker currency, and in countries with strong currency I've met people paying thousands for a university diploma, doing a well paid internship that "could not afford exercism". That's though, I hope I never have to build a business for software developers.
All of that shared experience comes from 3-4 years ago, so prices, models might have changed since that, but if they go under it's sad, their UI and their UX was so clean, fun, interactive, pleasant, etc, it was a joy to study using their material.