Comment by whilenot-dev

Concurrency based on coroutines is making use of cooperative multitasking, leaving the concurrent execution of tasks up to the runtime. To elaborate a bit on what that implies, let me just ask you the following question: Is there something less cooperative than a task that doesn't yield its control back to the main thread?

Regarding the fire-and-forget pattern I can think of at least two issues, let me illustrate them with the following example:

  import asyncio
  from typing import Any, Coroutine

  _background_tasks = set[asyncio.Task[None]]()

  def _fire_and_forget(coro: Coroutine[Any, Any, None]) -> None:
    task = asyncio.create_task(coro)
    _background_tasks.add(task)
    task.add_done_callback(_background_tasks.discard)

  async def _run_a() -> None:  # to illustrate issue A
    raise RuntimeError()

  async def _run_b() -> None:  # to illustrate issue B
    await asyncio.sleep(1)
    print('done')

  async def main() -> None:
    # issue A: Exceptions can't be caught
    try:
      _fire_and_forget(_run_a())
    except RuntimeError as exc:
      print(f'Exception caught: {exc}')

    # issue B: Task won't complete
    _fire_and_forget(_run_b())

  if __name__ == '__main__':
    asyncio.run(main())

Feel free to comment out either task in the main function to observe the resulting behaviors individually.

For issue A: Any raised error in the background task can't be caught and will crash the running main thread (the process)

For issue B: Background tasks won't be completed if the main thread comes to a halt

With the decision for the fire-and-forget pattern you'll make a deliberate choice to leave any control of the runtime up to blind chance. So from an engineering POV that pattern isn't a solution-pattern to some real problem, it's rather a problem-pattern that demands a reworked solution.

> How do you want it to be structured

Take a look at the caveats for FastAPI/Starlette Background Tasks: https://fastapi.tiangolo.com/tutorial/background-tasks/#cave...

Losing control of a background task (and therefor the runtime) might be fine for some demo project, but I think you'll want to notice raised errors in any serious production system, especially for any work that takes several minutes to complete.