dsmmcken

The claude provided skill-creator provides a decent jumping off point. It is easy enough to start with, but unless the skill is really simple I found it best to consider it a scaffold for building more tailored evals and reports.

The report leaves out a lot of detail. Several changes I found useful were: Pair with/without on same screen as left/right for easier viewing, token count for skill consumed, token used per run, time, pass rate, estimated cost, detailed aggregate stats, a parsed version of the conversation log (capturing the jsonl with each run, sometimes reading the log is the only way to find out why it's screwing up), work output logging (in my case screenshots and outputted script code), better formatting (syntax highlighting, log formatting).

Finally, I think the most useful thing was adding a self-reflection pass. After an eval is done, another agent looks at everything from that eval and tries to evaluate what went wrong along the way and what should be added to the skill, and conversely, from the without skill run what was in the skill that didn't need to be. It produces a skill change recommendation file for each eval. A further summary agent aggregates up all those recommendations in a way I can feed back to an agent.

fwiw, I just tried running the agent-skill they provide for fun to migrate an app-router based next 15 site and the end result is it entirely failed to start.

Vite just hangs when running vinext dev, with no output in logs whatsoever beyond printing`vinext dev (Vite 7.3.1)`.

Just curious, about how long did this project take you? I don't see that mentioned in the article.

Level 48 is the last level, and you get a pdf certificate proving you are human.

The tool we use for our docs AI answers lets you mine that data for feature requests. It generates a report of what it didn't have answers for and summarizes them as potential feature gaps. (Or at least what it is aware it didn't have answers for).

People seem more willing to ask an AI about certain things then be judged by asking the same question of a human, so in that regard it does seem to surface slightly different feature requests then we hear when talking to customers directly.

We use inkeep.com (not affiliated, just a customer).

I feel like I should be writing with the goal that the end reader is actually an LLM. The LLM will be the one spitting out the answers to the actual users via things like co-pilot, but I am not sure how that should change my approach to structure or level of detail in docs. Heavier on the number of code examples?

Link is gated behind a facebook/meta account.

> align parts across different grids

Maybe a CSS `subgrid` would help? Subgrid is newly available. (I know nothing about music notation, and basing this entirely off that statement).

I have the reverse problem with a bank atm in my city on google maps. I've marked it permanently closed several times. It will get marked permanently closed, then months later appear open again. The first time that happened, I actually went back thinking maybe they re-opened it, but nope still just a vacant wall with a bunch of 1 star reviews.