hellovai
Joined 12 years ago
Karma
154
Submissions
46
Member since
Mar 5
Recent submissions
Comment · 10 months ago
(on of the creators of BAML here) yep! exactly!
that workaround we've found works quite well, but the problem is that its not sufficient to just retry in the case of failed schema matches (its both inefficient and also imo incorrect).
Take these two scenarios for example:
Scenario 1. My system is designed to output receipts, but the user does something malicious and gives me an invoice. during step 2, it fails to fit the schema, but then you try with step 3, and now you have a receipt! Its close, but your business logic is not expecting that. Often when schema alignment fails, its usually because the schema was ambiguous or the input was not valid.
Scenario 2. I ask the LLM to produce this schema:
class Person {
name string
past_jobs string[]
}
thats the algorithm we created: schema-aligned parsing. More here if you're interested: https://boundaryml.com/blog/schema-aligned-parsing
Benchmark wise, when we tested last, it seems to help on top of every model (especially the smaller ones) https://www.reddit.com/r/LocalLLaMA/comments/1esd9xc/beating...
Hope this helps with some of the ambiguities in the post :)
Comment · 1 year ago
if you haven't tried the research -> plan -> implementation approach here, you are missing out on how good LLMs are. it completely changed my perspective.
the key part was really just explicitly thinking about different levels of abstraction at different levels of vibecoding. I was doing it before, but not explicitly in discrete steps and that was where i got into messes. The prior approach made check pointing / reverting very difficult.
When i think of everything in phases, i do similar stuff w/ my git commits at "phase" levels, which makes design decision easier to make.
I also do spend ~4-5 hours cleaning up the code at the very very end once everything works. But its still way faster than writing hard features myself.
Comment · 1 year ago
Have you tried techniques that don’t require modifying the LLM and the sampling strategy for structure outputs? For example, schema aligned passing, where you build error tolerance into the parser instead of coercing to a grammar.
Comment · 1 year ago
if you share your prompt with me on promptfiddle.com i can play around with it and see how i can make it better!
Comment · 1 year ago
its a bit more nuanced than applicative lifting. parts of of SAP is that, but there's also supporting strings that don't have quotation marks, supporting recursive types, supporting unescaped quotes like: `"hi i wanted to say "hi""`, supporting markdown blocks inside of things that look like "json", etc.
but applicative lifting is a big part of it as well!
gloochat.notion.site/benefits-of-baml
Comment · 1 year ago
appreciate you tyring it. the reason it dropped the day was due to your type system not being understood by the LLM you're using.
the model replied with
{
"Text": "coffee liqueur",
"Type": "Liqueur",
"Liquor_type": "Liqueur",
"Name_brand": null,
"Unit_of_measure": "ounce",
"Measurement_or_unit_count": "3/4"
},
there's no way to cast `Liqueur` -> `IngredientType`. but since the the data model is a `Ingredient[]` we attempted to give you as many ingredients as possible.
The model itself being wrong isn't something we can do much about. that depends on 2 things (the capabilities of the model, and the prompt you pass in).
If you wanted to capture all of the items with more rigor you could write it in this way:
class Recipe {
name string
ingredients Ingredient[]
num_ingredients int
...
// add a constraint on the type
@@assert(counts_match, {{ this.ingredients|length == this.num_ingredients }})
}
{{ ctx.output_format }}
No quotes around strings
Comment · 1 year ago
have you tried schema-aligned parsing yet?
the idea is that instead of using JSON.parse, we create a custom Type.parse for each type you define.
so if you want a:
class Job { company: string[] }
{ "company": "Amazon" }
https://www.boundaryml.com/blog/schema-aligned-parsing
and since its only post processing, the technique will work on every model :)
for example, on BFCL benchmarks, we got SAP + GPT3.5 to beat out GPT4o ( https://www.boundaryml.com/blog/sota-function-calling )
Comment · 1 year ago
really cool to see BAML on here :) 100% align on so much of what you've said here. its really about treating LLMs as functions.
Comment · 1 year ago
yea! even deepseek. Calling an external function / tool calling is really just a data extraction problem.
say you have a tool:
def calculator(expr: str) -> float
then the model just needs to say:
{ "function": "calculator", "args": { "expr": "5 + 10" } }
then in your code you can easily pass that to the "calculator" function and get the result, then hand the result back to the model. Making it feel like the model can "call" an external function.
deep seek can also do this: https://www.boundaryml.com/blog/deepseek-r1-function-calling
Structured LLM parsing that works 100% of the time
Email is: vbv [dot] boundaryml [dot] com