andreybavt

Hi, thanks for trying it! 19m for ~20 tables is slower that I'd expect, I assume you activated historic queries, right? As Luca said we're focusing on performance improvements right now. 99% of the latency is LLM calls, so we're optimizing them.

As for the descriptions: `pg_description` should be used already, we store raw pg descriptions in the semantic-layer/<connection>/_schema/*.yaml files under descriptions.db. there's also an AI generated description next to it. AI is the combination of the raw DB description + bunch of metadata we manage to collect about a table/column.

could you run `ktx sl` and see if the descriptions appear in the CLI ?

also feel free to join our community Slack - ktx.sh/slack We'll be happy to help with other questions

good point! yes, there's a feedback mcp tool that agents are instructed to use when they want to save/update ktx "memories". This is a way to keep the wiki+semantic layer up to date beyond the initial ingestion. Plus every time there's a new ingestion the reconciliation mechanism will gather potentially drifting context in one place.

We don't have a manual compression yet, it's part of our other product that ktx used to be a part of before it became open-source. But we decided to move that feature to OSS too, so expect it to come out in the next releases !

Hopefully more agents and fewer PowerPoints and McKinsey folks in the future :)

yes, exactly

When you install ktx, you'll have to initialize the project directory and ktx initializes a git repo there to assure the version controls.

ktx project directory is self-contained. The main 2 parts inside are :

- wiki: a collection of .md files

- semantic-layer: a collection of .yaml files

typically all these files are created/edited automatically during the ingestion, but you can also edit them by hand or even sync with a remote git.

ktx treats these files as sources and builds internal indexes in a sqlite db, located in projectDir/.ktx

The main way to use ktx is to start an mcp server by calling `ktx mcp start` it'll start an http server and multiple people will be able to connect it to their agents.

Hope this helps, happy to answer any other questions!

Thanks! ktx maintainer here, related to the queries sandbox question, we're attacking this problem from a different angle: instead of catching bad queries at runtime we try to make it impossible to generate a bad query. ktx allows agents to generate semantic queries (declarative json) and takes care of translating it to the correct SQL with the help of the join graph that it builds during the data source ingestion (dijkstra path finding + fan-out handling etc.). So the agent doesn't need to think "how" it needs to query the data, it just says "what" it wants and ktx builds queries. Also the join graph allows you combining raw tables and high level ktx metrics.

On nao: it's solving an adjacent but different problem, it's a full analytics app, whereas we believe that the general purpose agents (e.g. Claude Desktop) are getting better at the representation layer generating widgets and that's where people work anyway. What these agents need is a solid foundation - this is what we're focusing on. ktx isn't an agent or UI, it's an executable institutional knowledge that keeps any analyst (nao, Claude, Codex, your own agent) from guessing which table is canonical or which join is safe.

Thanks! yes, exactly. But what you described is just half of what ktx does, we call it "wiki" - self organizing collection of ingested markdown files.

The second (equally important) part of ktx is similarly self-organizing executable semantic layer.

It allows agents to send simple declarative "I want X sliced by Y" requests instead of imperative "X sliced by Y that has to be queried this way and joined the other way ..."

To answer your questions:

> who writes the wiki / business rules most is create after `ktx ingest` pulls raw data from your data stack, it can be edited/created manually, but it's rarely necessary. Also if you plug ktx to your agent it allows that agent to send extra memories for ingestion to keep the context up to date

> Can they be reverse engineered from existing query stack Yes, that's the case when we analyze historic SQL queries or scan Looker or Metabase dashboards for example

yep, happy to share more details with you @modus-tollens if you're interested

Great questions! we're currently working on Spider 2 submission, hope to have first results soon. It's true that we took file-first approach and not a graph DB. Main reason is that the ktx wiki and semantic layer entities while being written in plain text files (md or yaml) still contain links to each other. This allows an agent to find the right entry point (with the help of lexical and semantic searches merged with RRF) and then traverse these links to collect enough context.

As for the business context evolution - that's exactly the reason we have ingestion reconciliation and git versioning. The idea is to give ingestion agent a way to deduplicate/consolidate knowledge during the ingestion and leave complex conflicts to humans to resolve