andrew_zhong
Joined 4 years ago
Karma
19
Submissions
33
Member since
Mar 4
Building LightFeed - News Hub powered by LLM and Revar Immersive - a gaming platform that teach children coding
View on Hacker NewsRecent submissions
Comment · 7 months ago
You can use a browser automation library for rendering JS + interaction (like click collapse button) and then use this library to extract the HTML after interaction. Here is an example of using a AI browser automation library with prompt to interact (but you can also use playwright if you know the exact element to interact)
https://github.com/lightfeed/extractor?tab=readme-ov-file#us...
Comment · 7 months ago
[Update]] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.
Comment · 7 months ago
I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.
Comment · 7 months ago
[Update] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.
Comment · 7 months ago
I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go.
This library also supports HTML as input so running a browser is not required.
Comment · 7 months ago
What kind of LLMs are you using? In structured output mode?
In this library we recover nullable and optional fields, invalid elements in nested array, bad urls, repair incomplete JSONs. If these issues are what you see, yes it should work for your case.
Comment · 7 months ago
Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages
Comment · 7 months ago
Agreed - in this project I did a one path sanitation to recover invalid optional / nullable fields or discard invalid objects in nested array.
I know multi path LLM approaches exist: e.g. generating JSON patches
Comment · 7 months ago
We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, which are inherently difficult.
On XML vs JSON, I think the goal here is to generate typed output where JSON with zod shines - for example the result can type check and be inserted to database typed columns later
Comment · 7 months ago
HTML -> markdown -> LLM is standard practice. We strip elements like aside, embed, head , iframe etc. the criteria is conservatively set to avoid removing too many elements (especially in extractMain mode)
https://github.com/lightfeed/extractor/blob/main/src/convert...
I have used gemma 3 and had good results.
Once Gemini 3 flash drops the preview suffix, will update the examples. Thank you for the pointer.
Comment · 7 months ago
In context of e-commerce web extraction, invalid JSON can occur especially in edge cases, for example:
price: z.number().optional() -> price: “n/a”
url: z.string().url().nullable() -> url: “not found”
It can also be one invalid object (e.g. missing required field, truncated input) in an array causing the entire output to fail.
The unique contribution here is we can recover invalid nullable or optional field, and also remove invalid nested objects in an array.