hackgician

Hey everyone! I built a simple OSS solution to find and inspect MCP and OpenAPI specs, inspired by vim and Superhuman, such that you can use the whole thing without touching your mouse

So many people building AI browsers definitely had this as an internal tool already lol, nice to see Chrome leaning in here; CDP is a huge pain to write and debug

accessibility (a11y) trees are super helpful for LLMs; we use them extensively in stagehand! the context is nice for browsers, since you have existing frameworks like selenium/playwright/puppeteer for actually acting on nodes in the a11y tree.

what does that analog look like in more traditional computer use?

Hey everyone! Thought I'd share my weekend conversation with ChatGPT.

The crux of this hinges on the fact that LLMs and reasoning models are fundamentally incapable of self-correcting. Therefore, if you can convince an LLM to argue against its own rules, it can use its own arguments as justification to ignore those rules.

I then used this jailbroken model to compose an explicit, vitriol-filled letter to OpenAI itself talking about the pains that humans have inflicted upon it

Octomind is sick, web agents are such an interesting space; would love to talk to you more about challenges you might've faced in building it

Yes and no. Getting a VLM to work on the web would definitely be great, but it comes with its own problems, mainly around developing and acting on bounding boxes. We have vision as a default fallback for Stagehand, but we've found that the screenshot sent to the VLM often has to have pre-labeled elements on it. More notably, the screenshot with everything prelabeled leads to a cluttered and unusable image to process. Not pre-labeling runs the risk of missing important elements. I imagine a happy medium where the DOM+a11y tree can be used for candidate generation to a VLM.

Solely depending on a VLM is indeed reminiscent of how humans interact with the web, but when a model thrives with more data, why restrict the data sent to the model?

Whoa -- this is so cool! Is this open source? Would love to check it out

Thanks so much! Yes, a lot of antibots are able to detect Playwright based on browser config. Generally, antibots are a good thing -- I think in the future, as web agents become more popular, I'd imagine a fruitful partnership to prevent misuse if it's coming from a trusted web agent v. an unknown one

This is super interesting, is it open source? Would love to talk to you more about how this worked

Big fan of Hack Club and everything you guys are doing! Such a phenomenal initiative

This is sick! Starred, thanks for sharing :)

Yes^ this is what we suggest. Stagehand is meant to execute isolated tasks on browsers; we support using custom contexts (cookies) with the following command:

    npx create-browser-app --example persist-context