pyper-dev

Your third point intrigues me a lot. I imagine for the majority of cases, it's generally more useful to work with async functions in the structure of async syntax, but I suppose it's possible to run async functions in a synchronous pipeline.

Even though there's currently no built-in support for this, a workaround could be to just define synchronous helper functions to handle running your async logic in an event loop.

We came across this at one point and thought it was a very innovative and interesting package!

The important design point we're differing on is that Pyper implements 'pipelines' as functions, whereas pypeln seems to implement 'pipelines' as iterable objects.

Great feedback, thank you. We'll certainly be working on adding more examples to illustrate more complex use cases.

One thing I'd mention is that we don't really imagine Pyper as a whole observability and orchestration platform. It's really a package for writing Python functions and executing them concurrently, in a flexible pattern that can be integrated with other tools.

For example, I'm personally a fan of Prefect as an observability platform-- you could define pipelines in Pyper then wrap it in a Prefect flow for orchestration logic.

Exception handling and logging can also be handled by orchestration tools (or in the business logic if appropriate, literally using try... except...)

For a simple progress bar, tqdm is probably the first thing to try. As it wraps anything iterable, applying it to a pipeline might look like:

  import time
  from pyper import task
  from tqdm import tqdm


  @task(branch=True)
  def func(limit: int):
      for i in range(limit):
          time.sleep(0.1)
          yield i


  def main():
      for _ in tqdm(func(limit=20), total=20):
          pass


  if __name__ == "__main__":
      main()