Comment Comment by alexey-salmin We do have that in Exa search:* Extraction of main content from HTMLs, the index stores markdown representations (free of headers/footers/sidebars/menus etc)* Serving highlights picking the most relevant part for each result to reduce token usage downstream* Also serving dynamic highlights where we summarize all the sources at once reducing the token counts even furthersee here: https://exa.ai/docs/search/highlights and https://exa.ai/docs/contents/quickstart parent alexey-salmin 27 days ago
* Extraction of main content from HTMLs, the index stores markdown representations (free of headers/footers/sidebars/menus etc)
* Serving highlights picking the most relevant part for each result to reduce token usage downstream
* Also serving dynamic highlights where we summarize all the sources at once reducing the token counts even further
see here: https://exa.ai/docs/search/highlights and https://exa.ai/docs/contents/quickstart