owickstrom

That sounds like a pretty ideal case for Bombadil. Yeah you can write custom generators for exactly this reason!

OK. Yeah so I made the decision early to go one level below Playwright and target CDP directly, because I believe I need the tight connection with Chrome/Chromium to make it as fast as possible. Also I really value the distribution aspects of writing this in Rust and compiling to a single executable with everything in it.

Do you mean test runners like JUnit, pytest, etc? Or browser test runners specifically?

Thanks! I don't know much about it, but from what I can tell from their site, it does differential testing based on screenshots and behavior, and mocks out network IO, focusing on the client side. I'm not sure what the AI part is, if it's for extracting information from the page, for generating actions, for smarter diffing, or some combination of those.

Bombadil doesn't mock the network (although I'm working with a contributor to add something like Playwright's Route system for manipulating responses). It runs on your full stack (if you want) and if you need determinism for perfect bug reproducibility, you should run it all in Antithesis.

On the AI front, my view is that Bombadil should be optimized for using LLMs to generate and refine specs over time (properties and action generators), but that it should be an offline process. It's not like a "browser testing agent". It's analogous to having the LLM generate a program, rather than having the LLM itself perform actions. For some problems, the latter makes more sense, but for many problems the former is better, and I believe that PBT (and things like proof systems, but that's not my domain) are suited to the former strategy.

I have some sketches on how the refinement-with-human-in-the-loop thing should work with Bombadil and LLMs, but it's only a roadmap item for now.

One smartass way to answer that for the general case is: you don't. And I mean that in the same sense as "you don't write example based tests using a PBT framework". You supply independent actions through generators and have it apply them "chaotically", not strictly in sensible sequences.

That said, for things like auth, you need to do some fixed steps to get anywhere at all. I haven't added support for this yet, but as I'm envisioning it, you'd either:

a) authenticate the browser using headers or cookies that you inject into the Bombadil browser, or b) use a "custom action" (essentially a stringifed JS function) that'd perform your particular steps

Or what you could do now: add preconditions to your actions so that they'd necessarily fill out the form before anything else could be done.

Another thing we might need in Bombadil is secrets, but for now I think it's fine to inject test credentials that are plain visible text.

Please let me know if you try it out and if you need any help!

I'll quote my lobste.rs reply to this question:

> I think his jolly chaotic good energy, along with how he's uncorruptible and sort of outside the normal world, fits a fuzzer in a spiritual sense. But it might all just be a good excuse to use one of my favorite names from my favorite book.

It's also my favorite part of the book, and definitely not something I'd skip. I know some people would object, but I think of it as an homage rather than stealing.

Cool. Is this publicly available?

Go for it! I've been meaning to do an "architecture of Bombadil" blog post that'd likely answer this question. It's not super advanced by any means, but it's a mindset shift to how you might think about browser testing coming from the mainstream frameworks like Playwright.

Yes, and in fact, capturing "prior" values with bindings and closing over them in temporal operator thunks is how you talk about some relation between s and s' in a Bombadil formula (not having that particular syntax though). It's a deliberate way of embedding this LTL flavor in JS/TS in the most natural and ergonomic way I could think of. I didn't want a deep EDSL or even a new bespoke spec language that people (and LLMs) would have to learn, and to have to write tools for. Now you can write Bombadil specs with a good LSP and be able to import packages off of npm or whathaveyou. Most web devs will probably be comfy with JS or TS, so that's why I chose that style.

I hope that makes sense?

Thanks for the nice feedback!

Hey, yeah, so actions are indeed independent and you can't express fixed sequences of them (e.g. fill out thus form by doing X, then Y, then Z). If you want to enforce certain sequentiality you'd need to precondition your generators. This is admittedly limited right now. I do want to add QoL things like `filter` on the generators to make that more ergonomic.

Also, as you note, you can't implement custom actions. And that's just something that I haven't gotten to yet. It'd be quite straightforward to add a plain function (toStringed) as one of the variants of the Action type and send that to the browser for eval. (Btw, we're taking contributions!)

Weighting is also important right now. There's no smart exploration in Bombadil yet, only blind random, so manual weighting becomes pretty crucial to have more effective tests (i.e. unlikely to run in circles). I'd like to both make the Bombadil "fuzzer" better at this, but eventually you might want to run Bombadil inside Antithesis instead to get a much better exploration, even in a single campaign.

The Until operator is also one of those things that I haven't gotten to yet. I actually didn't expect someone to hit this as a major limitation of the tool so quickly, which is why I've focused on other things. Surprising!

To add some more context, Bombadil is an OSS project with one developer (but we're hiring!) and it's 4 months old and marked experimental. I'm sorry you were disappointed by its current state, but it's very early days, so expect a lot of these things to improve. And your feedback will be taken into account. Thanks!

Fair enough! I'm probably doing a demo video that'd help with this. Maybe can condense that in text form into the readme as well.