arsenide
Joined 13 years ago
Joined 13 years ago
Telling AI to "do TDD" is a shortcut that gets me over 80% there by itself in aggregate with little effort. The last 20% is painful. More painful than the usual "last 20%" in game design and development. I think this is part of the human vs. LLM gap.
If you revert the code to before the feature was implemented, then called out the way not to do it (the failed version you're describing) - what would happen? Maybe that "80% there" turns into "90% there", or maybe it's a bit better. Or maybe a bit worse. Working with LLMs vs. humans, does the gap shrink further if you gave other people the same requirements you gave the LLM? I think it depends on who you're working with - other people have their own opinions and don't necessarily just do exactly what they're told without taking their experiences into account.
In a way I see it like playing a game of telephone. I have my understanding of the system, I translate it into words, the LLM interprets it and does what it does based on that. For some reason, it feels like this is a telephone game with 3 people: A to B to C where the translation layer of my thoughts into words is an inserted "B". In contrast, if instead I'm expressing my thoughts to another person it's A to B. This extra connection here is lossy in the same way a game of telephone is lossy from player A to player C.
quackwave.com
Hasn't really been stress tested yet (this is the first time I'm mentioning it publicly), but it's been fun playing with friends & family and iterating.
Party game platform, think "jackbox but no host screen" - just need a web browser open on phone/tablet/PC.
Your comment matches my experience: it has to do what can and cannot be verified. As an example, it was much easier to have AI write a large e2e test using Playwright (then add test cases and expand) than to assume it'll correctly fix bugs without guidelines like this. Also, the human loop is still important in things like screenshot verification - but the frontier models are getting even better here so I'm not sure for how much longer this will be true. The ratio of test code to production code is a bit over 2:1 right now.
Email in profile - would love to connect.
Calculations for slot machine mechanics and payouts have been in Excel for a long time. There can be a LOT of complexity in these workbooks. Sometimes it’s tricky to debug - but what’s the alternative? Code is often hard to debug too.
Simulating results (Monte Carlo) is nice but having two sets of data for validation/checking against each other is nice.
I am not aware of any alternatives.
Hard problems or domains are unknowns. Working towards solving hard problems involves thinking through unknowns, which may or may not lead to understanding. An aversion to hard problems is an aversion to the unknown.
In the usual Excel Solver case, for a simplified example, I want to create a biased coin where getting heads doubles your money and tails loses your money, with expected payout 90% in the long run. The parameters to change are heads/tails coin weighting, to target a value 90%. A fair coin gives a long run payout of 100%. It turns out that if a biased coin hits heads 45% of the time and tails 55% of the time, we end up with this 90% payout. Excel solver can come up with these two values.
In this example, we can come up with these 45% and 55% values theoretically. With more complex systems, we may want to see the effects of changing a subset of parameters that have an indirect effect on payout.
For example, if we extend the “biased coin game” to instead be “win your money back, 2, or 3 times your wager each with some probability” on a heads result, there are more parameters (and many solutions!) to get to 90% payout. Changing the heads probability has a further, second-order effect on the payout. Using Excel solver it’s straightforward to fix some parameters (heads/tails probability) and allow others to change (1x/2x/3x odds) to get desired results (90% payout).
If we further extend this methodology for a biased coin game, we can end up with a coin game that pays with distribution exactly like a slot machine in a casino, though it would not look like one!
It’s great, but you have to be able to put your problem into Excel to use it which does not work for all use cases.
In cases where I’m trying to optimise something that doesn’t quite fit into Excel cleanly I’ll usually do some sort of hand-rolled Monte Carlo optimisation. This generally works for me in the types of problems I most often solve.