destel
Joined 1 year ago
Joined 1 year ago
Don’t know what’s the reason: my bad prompting or these models being tuned to work with photos/illustrations only
Typically function "f" does two things. 1. Performs calculations (this can be parallelized). 2. Writes results somewhere (this must happen sequentially and in the correct order).
Here's the typical OrderedLoop usage example from the article:
OrderedLoop(in, out, n, func(a A, canWrite <-chan struct{}) {
// [Do processing here]
// Everything above this line is executed concurrently,
// everything below it is executed sequentially and in order
<-canWrite
// [Write results somewhere]
})
Without the "canWrite" the entire body of the "f" will be executed either concurrently or sequentially. With it - we can have this dual behavior, similar to critical sections protected by mutexes.It's worth mentioning that OrderedLoop is a low-level primitive. User-level functions like OrderedMap, OrderedFilter and others do not need the "canWrite" channel to be exposed.
I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.
Your first example definitely gives me merge-sort vibes - a really clean way to keep things ordered across multiple sources. The second and third scenarios are a bit beyond what I’ve tackled so far, but super interesting to read about.
This also reminded me of a WIP PR I drafted for rill (probably too niche, so I’m not sure I’ll ever merge it). It implements a channel buffer that behaves like a heap - basically a fixed-size priority queue where re-prioritization only happens for items that pile up due to backpressure. Maybe some of that code could be useful for your future use cases: https://github.com/destel/rill/pull/50
Your snippet looks good and concise.
One thing I haven’t emphasized enough in the article is that all algorithms there are designed to work with potentially infinite streams