JohannesKauf
Joined 2 years ago
Karma
144
Submissions
12
Member since
Jun 16
Developer from Berlin
View on Hacker NewsRecent submissions
Comment · 2 years ago
Sorry, I mixed up a few topics here:
- Moved everything to a VPS - way better value for money. Extra TB of traffic only costs €1-10 with Hetzner/DigitalOcean compared to 400€ with Vercel's old pricing.
- Put Cloudflare in front - gives me an extra layer of control (if I ever need it)
- Built a proper REST API - now there's an official way to use the converter programmatically
- Made email registration mandatory for API keys - lets me reach out before having to block anyone
That other server was probably running a scraper and then converting the html-websites to markdown. After about 2 weeks they noticed that I was just returning garbage and it stopped :)
Comment · 2 years ago
The problem was: Doing rate limiting on the application level was not enough. Once the request hit my backend the incoming bandwidth was already consumed — and I was charged for it.
I contacted Vercel's Support to block that specific IP address but unfortunately they weren't helpful.
Comment · 2 years ago
Yeah good point, that's actually difficult. They use many `<span>` html tags to color individual words and syntax.
But I wrote logic to handle that. It probably needs to be adapted at some point, but works surprisingly well. Have a look at the testdata files ("code.in.html" and "code.out.md" files [1]).
Feel free to give it a try & let me know if you notice any edge cases!
[1] https://github.com/JohannesKaufmann/html-to-markdown/blob/ma...
Comment · 2 years ago
My final university project was about a clean-up-approach on the HTML nodes before sending it to the html-to-markdown converter. But that was extremely difficult and dependent on some heuristics that had to be tweaked.
Your idea of comparing multiple pages would be a great approach. It would be amazing if you build something like this! This would enable so many more use cases... For example a better “send to kindle” (see other comment from rty32 [1]).
Comment · 2 years ago
There are two parts to it:
1) convert html to markdown
This is what my library specifically addresses, and I believe it handles this task robustly. There was a lot of testing involved. For example, I used the CommonCrawl Dataset to automatically catch edge cases.
2) Identify article content
This is the more challenging aspect. You need to identify and extract the main content while removing peripheral elements (navigation bars, sidebars, ads, etc.)
For example, the top of the markdown document will have lots of links from the navbar otherwise.
Mozilla's "Readability" project (and its various ports) is the most used solution in this space. However, it relies on heuristic rules that need adjustments to work on every website.
---
The html-to-markdown project in combination with some heuristic would be great match! There is actually a comment below [1] about this topic. Feel free to contact me if you start this project, would be happy to help!
Comment · 2 years ago
Thanks, Jina actually looks quite nice for use in LLMs.
I also provide a REST API [1] that you can use for free (within limits). However you have get an API Key by registering with Github (see reason below).
---
The demo was previously hosted on Vercel. Someone misused the demo and send ~5 million requests per day. And would not stop — which quickly brought me over the bandwidth limits of Vercel. And bandwidth is really really expensive!
So that is the reason for requiring API Keys and hosting it on a VPS… Lessons learned!
Comment · 2 years ago
Pandoc is amazing. Especially because they support so many formats.
And their html to markdown converter is (in my opinion) still the best right now.
But html-to-markdown is getting close. My goal is to cover more edge cases than pandoc...
Comment · 2 years ago
Oh very nice to hear, thank you very much!
That’s actually a great idea!
I personally use Raindrop for bookmarking articles. But I can’t find stuff in the search.
The other day “Embeddings are underrated” was on HN. That would actually be a good approach for finding stuff later on. Using webhooks, converting to markdown, generating embedding and then enjoying a better search. You just gave me the idea for my next weekend project :-)
Comment · 2 years ago
That is unfortunately out of scope. I like the philosophy of doing one thing really well.
But nowadays—with Playwright and Puppeteer—there are great choices for Browser automation.
Comment · 2 years ago
> built something similar for the Elixir
We interact with the web so much that it’s worth having such a library in every language... Great that you took the time and wrote one for the Elixir community!
Feel free to contact me if you want to ping-pong some ideas!
> feeding LLMs web page data
Exactly, that one use case that got quite popular. There is also the feature of keeping specific HTML tags (e.g. <article> and <footer>) to give the LLM a bit more context about the page.
Comment · 2 years ago
Cool to see another library in this space!
I see that you took the test cases from Turndown. However Turndown isn’t actually that accurate. This is especially noticeable when converting entires websites.
The best comparison would be against Pandoc. That is (in my opinion) the best html to markdown converter right now.
Although it is extremely difficult to handle every edge case. As an example, this usually causes problems:
<p>nitty<em>-gritty-</em>details</p>