flutetornado

I was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?

450 with PTQ_01 and 900 with the other PQ2_0.

GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.

Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.

Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.

I ended up creating a personal vim plugin for merges one night because of a frustrating merge experience and never being able to remember what is what. It presents just two diff panes at top to reduce the cognitive load and a navigation list in a third split below to switch between diffs or final buffer (local/remote, base/local, base/remote and final). The list has branch names next to local/remote so you always know what is what. And most of the time the local/remote diff is what I am interested in so that’s what it shows first.

My experience with qwen3.5 9b has not been the same. It’s definitely good at agentic responses but it hallucinates a lot. 30%-50% of the content it generated for a research task (local code repo exploration) turned out to be plain wrong to the extent of made up file names and function names. I ran its output through KimiK2 and asked it to verify its output - which found out that much of what it had figured out after agentic exploration was plain wrong. So use smaller models but be very cautious how much you depend on their output.

There are several useful ways of engineering the context used by LLMs for different use cases.

MCP allows anybody to extend their own LLM application's context and capabilities using pre-built *third party* tools.

Agent Skills allows you to let the LLM enrich and narrow down it's own context based on the nature of the task it's doing.

I have been using a home grown version of Agent Skills for months now with Claude in VSCode, using skill files and extra tools in folders for the LLM to use. Once you have enough experience writing code with LLMs, you will realize this is a natural direction to take for engineering the context of LLMs. Very helpful in pruning unnecessary parts from "general instruction files" when working on specific tasks - all orchestrated by the LLM itself. And external tools for specific tasks (such as finding out which cell in a jupyter notebook contains the code that the LLM is trying to edit, for example) make LLMs a lot more accurate and efficient, efficient because they are not burning through precious tokens to do the same and accurate because the tools are not stochastic.

With Claude Skills now I don't need to maintain my home grown contraption. This is a welcome addition!

I prefer it because it forces distillation to core ideas, consumable quickly. Busy people have too little time to read too much verbiage.

Everyone in my entire team - best of engineering as well as every manager left. Underpaying and over subscribing people has become a hallmark over there - it's just a body shop now. Engineers are just numbers on a sheet, to be exploited, chewed and cast aside when they eventually burnout. Upper management has no vision and everyone's constantly firefighting and struggling to catch up with competitors who had long term vision to invest in engineering teams, tooling and infrastructure to scale up the products and people. They want to do in 2 years what took Google and Amazon a couple of decades. Result post-HPE: poor quality, unscalable, cobbled together, barely functional codebase. Before, the startup I worked for had a well balanced rare combination of high performance, modular and well architected codebase. Later the constant push to ship as fast as possible to catch up with competition, completely destroyed the whole thing - teams, codebase and infrastructure. All because they only know how to react and have no idea how to stay ahead of the curve. Buying startups has become their only means of survival as talent stays away from their brand and the only way to justify value to shareholders is to jump from one rock to another, hoping the new one will rocket them away from the black hole they are spiraling into - all they manage to do is stick to the new rock and pull it with them as fast as they were going into the hole they will eventually vaporize in.

I’d do that every time I get a chance! Ex-HPE black label on my resume from a startup I used to work in that they bought. That company is a complete horror show.

I think the performance in terminal apps could come from the fact that terminals can use pre rendered, pre cached in GPU, text glyphs covering a fixed size small grid (of glyph sized blocks instead of pixels.) At least that’s what I have wondered - because the terminal UI experience has been always a lot more slick compared to heavy GUI based programs. Feel free to correct me if someone has done actual performance analysis.

GPU workloads are either compute bound (floating point operations) or memory bound (bytes being transferred across memory hierarchy.)

Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.

Altair is superb. Have used it a lot and it has become my default visualization library. Works in VSCode and Jupyter Lab. The author has a great workshop video on youtube for people interested in altair. I especially like the ability to connect plots with each other so that things such as selecting a range in one plot changes the visualization in the connected plot.

One possible downside is that it embeds the entire chart data as json in the notebook itself, unless you are using server side data tooling, which is possible with additional data servers, although I have not used it, so cannot say how effective it is.

For simple plots its pretty easy to get started and you could do pretty sophisticated inter plot visualizations with it as you get better with it and understand its nuances.