cgenuity
Joined 8 years ago
Joined 8 years ago
The Node.js ecosystem doesn't seem to have a good primitive for app-level, per-event scheduling (things like "expire this abandoned cart in 24h" or "debounce 50 profile updates into one reindex"). Cron is too broad and you still have write the polling/scheduler code yourself. Job queues optimize for throughput and usually bolt `run_at` on as an afterthought. Workflow engines are overkill for a simple "do this thing later, tied to this user" and want you to adopt their runtime.
DelayKit aims to be the in-between. It is backed by Postgres and uses keys like `dk.schedule("expire-cart", { key: cartID, delay: "24h" })`. Handlers get the key, not a payload, and fetch fresh state at fire time. This way DelayKit is only responsible for the "hey remember you have to do this thing for cart X" part.
I'm working through making it production-ready at the moment, the initial passes were more around figuring out the API and general architecture. Thoughts and comments greatly appreciated!
> How much would it cost to do 3 million requests per month?
If you want to use Posthook to schedule 3 million requests a month please email support [at] posthook.io and we can work out a special plan and support contract. Do keep in mind that background jobs and scheduled jobs are two different things and what I aim for Posthook to solve are scheduled jobs. I suspect a big part of those 3 million were background jobs.
> Also how would you set up custom retry policies?
The retry policy is fixed at the moment but it seems like this a common request so I want to roll that feature out soon.
If the outage is on the customer's side, all hooks that were attempted during the outage would be marked as failed. I plan on adding a feature that will allow the developer to fire off again all failed hooks in a given time period.
Right now the retry logic is just one retry 5 seconds after the first failure. At which point the hook gets set to a failed status and failure notifications get sent out. Retries are tricky because depending on how the job is implemented they can cause more harm than good. So I plan to refine that more based on customer feedback.
I've found that with scheduling tasks for the future it's important to do a final check before fulfilling the action, whether it's sending an email, push notification, etc. You don't want to send a reminder for an event that has been deleted, for example. That is why I have decided to keep the scope small and let developers make the final decision there.
Reports and retries are definitely things that I think add value though, and I have plans to expand on that.
It can be an alternative to running your own systems like Sidekiq, Celery, or Quartz and the operational overhead that comes along with them. Cron jobs and cloud provider tools like CloudWatch Events are also used for job scheduling but lack observability and may force you to frequently make expensive queries to your data store just to see if there is any work to do.
Questions and feedback are greatly appreciated :)