JackHopkins

Can't you replay pending requests? How can you mitigate the issue of differing side effects generated by the new / old versions?

Cool! why is it crucial to keep old versions of code, and what are the risks of running outdated code?

Thanks - what is your sweet spot use-case do you think? I've built a few GraphQL projects in the past, so I'm curious where you fit into the ecosystem?

This is really cool, nice work! How does this differ from Apollo Federation? I'm a bit confused here, because you have integrations with AF too - is this competitive, or is this a more vertically-integrated solution? Cheers!

What is the order of the cost of the workflow function?

This is really cool! Am I right in thinking that the cost for running this program is equivalent to all of the dependency execution durations? i.e no busy waiting?

Hey everyone,

I'm a main contributor of Tanuki (formerly MonkeyPatch).

The purpose of Tanuki is to reduce the time to ship your LLM projects, so you can focus on building what your users want instead of MLOps.

You define patched functions in Python using a decorator, and the execution of the function is delegated to an LLM, with type-coercion on the response.

Automatic distillation is performed in the background, which can reduce the cost and latency of your functions by up to 10x without compromising accuracy.

The real magic feature, however, is how you can implement alignment-as-code, in which you can use Python's `assert` syntax to declare the desired behaviour of your LLM functions. As this is managed in code, and is subject to code-review and the standard software-lifecycle, it becomes much clearer to understand how an LLM feature is meant to behave.

Any feedback is much appreciated! Thanks.

I understand the point. I would ideally like an association with monkey-patching something as that is relevant to the behaviour of the package. However, not so similar that it shadows the technique of monkey-patching!

Yeah this is fair. I’m not attached to a simian theme if we’re ditching specific association to monkey-patching something. Or indeed, a ‘patching’ theme for that matter.

A new name is definitely in order. I will think about it over the weekend.

Thanks for the feedback, I appreciate it.

I do get the point and the difference from the classical monkey-patching. I like the stub ideas though!

Good to know, we'll make it more clear in the docs! To answer regarding these 2 areas,

1) The data for finetuning currently is saved on disk for low latency reading and writing. Both test statements and datapoints from the function execution are saved to the dataset. We also are aware that saving to disk is not the best option and limits many use-cases so we're currently working on creating persistence layers to allow communication with S3 / Redis / Cloudflare as the external data storage.

2) Currently starting the fine-tuning job happens after the dataset has at least 200 datapoints from GPT-4 executions and align statements. Once the finetuning is completed, the execution model for the function is automatically switched to the finetuned GPT 3.5 turbo model. Whenever the finetuned model breaks the constraints, the teacher (GPT4) is called upon to fix the datapoint and this datapoint will be saved back to the dataset for future iterative finetuning and improvements. We are also working on adding in ways for the user to include a "test-set" which could be used to evaluate if the finetuned model achieves the required performance before switching it as the primary executor of the function

Hope this makes it more clear, if you have any additional questions, let me know!

Do any other names jump out at you as preferable?

Thoughts on something like PyMonkeyPatch? GorillaPatch?

Don't get me wrong, I do appreciate the criticism of the current naming! It does seem to create some unwanted friction of using or talking about the library, I was just trying to explain the thought process and ideate on top of it but we will have a second look regarding the name and how to make using and talking about the library as unconfusing as possible