sunshadow
Joined 3 years ago
Karma
86
Submissions
42
Member since
Apr 12
Recent submissions
Comment · 1 year ago
Too late, we all went to cursor.
Comment · 1 year ago
I can make this in couple of weeks, but I need an incentive other than just having fun
Comment · 1 year ago
What made you use caddy in front of tiny-http? I guess tiny-http should be still fine on its own. genuinely curious
Comment · 1 year ago
They made 11B$ last year. It has incredible amount of ads. If you haven't noticed, then that means they did a great job. (tip: look for the custom logo pins in the map. Its printing money)
Comment · 2 years ago
According to the license, this repo is not that different than me just extracting the source code from the binaries. Pass.
Comment · 2 years ago
There is no difference between this and a well known bot prevention mechanism, from the scraper perspective.
Comment · 2 years ago
I don't see any difference than browserless?
Comment · 2 years ago
If you're looking for open-source function calling: Checkout Functionary: https://github.com/MeetKai/functionary/
It's MIT licensed. Supports both CPU and GPU
Comment · 2 years ago
Yes, Functionary: https://github.com/MeetKai/functionary/
MIT Licensed
Its actively maintained as open-source. You can even run it with llama.cpp
Comment · 3 years ago
Original project: https://github.com/xenova/transformers.js
Comment · 3 years ago
Unless you start a war against scrapers, you don't need to worry about that as I'll always find a way to scrape your site as long as its valuable to 'me'. Even if it requires Real browser + OCR :)
Comment · 3 years ago
Crul looks nice, though, you cannot imagine how many startups that I've seen failed doing a very similar thing as Crul. Wouldn't rely on it.
The problem is complex: Humans generating messy pages
Comment · 3 years ago
You don't use XPath&CSS selectors at all (Except if you dont have choice). You rely on more generic stuff, e.g, "the button that has 'Sign in' on it":
await page.getByRole('button', { name: 'Sign in' }).click();
Comment · 3 years ago
These days, I'm not even using Go for scraping that much, as the webpage changes makes me crazy and JS code evaluation is a lifesaver, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary).
Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: (https://github.com/geziyor/geziyor).
My favorite stack as of 2023: TypeScript+Playwright+Crawlee(Optional) If you're serious in scraping, you should learn javascript, thus, playwright should be good.
Note: There are niche cases where lower-level language would be required (C++, Go etc), but probably only <%5
Comment · 3 years ago
It would be nice to see it open source. As im concerned about the privacy policy as its a really valuable information