Why this metric. "-X% tokens removed!" is a marketing number if it ignores whether the answer was right. A tool that saves 90% of tokens but makes 20% more mistakes has worse cost-per-correct.
I found that token savings is really not worth looking at. Benchmark a baseline (without novgraph) vs novgraph on a set of tasks. Often with such tools, the agent requires more turns and cache hits are fewer so the actual cost ends up similar, but the agent actually takes more time to complete.
Build your own benchmark or try mine: https://github.com/jahala/copeca .. Haven't touched it for 3 months, so not sure if it's still up to scratch.
cost_per_correct = total_spend / correct_count
Why this metric. "-X% tokens removed!" is a marketing number if it ignores whether the answer was right. A tool that saves 90% of tokens but makes 20% more mistakes has worse cost-per-correct.
I found that token savings is really not worth looking at. Benchmark a baseline (without novgraph) vs novgraph on a set of tasks. Often with such tools, the agent requires more turns and cache hits are fewer so the actual cost ends up similar, but the agent actually takes more time to complete.
Build your own benchmark or try mine: https://github.com/jahala/copeca .. Haven't touched it for 3 months, so not sure if it's still up to scratch.