Comment by cmdalsanto

There's some secret sauce here, but since we intercept each chunk as the LLM pushes them out, we can perform evaluations on them and decide what gets sent back to the client if we detect a fault.