Comment

Comment by LennyWhiteJr

A couple ways. If the need is not real-time and analytical, you feed the data from multiple services into a separate BI database which can do slower and more complex joins across data from multiple data sources. Or if the need is real-time, you build a paginated API with a page limit that can always be processed within the API SLA. Then you build workflows on top of the paginated API to operate on that data.

Generally, unbounded operations have to be broken up at some point. It just depends on how big the data set is.

No comments yet

Be the first to share your thoughts.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton