whatl3y

try:

$ curl x.tty.news

Smart search engine for the Epstein files from the DOJ built on a RAG pipeline that indexes and makes them searchable via natural language (i.e. LLM style). Documents (PDFs, HTML, images) go through text extraction with OCR as a fallback for scanned pages, then get sliced into ~500-token chunks with 50-token overlap. Each chunk gets a metadata prefix baked in (document title, source section) before being embedded into 1536-dimensional vectors through OpenAI's text-embedding-3-small. Those vectors live in PostgreSQL with pgvector, sitting behind an HNSW index with ef_search cranked to 400 (default is 40, which misses too much).

Queries hit the same embedding model, and the system pulls the top-K most similar chunks by cosine distance. There's a hybrid search mode too: it over-fetches 5x candidates from the vector index in parallel with keyword search (full-text search via a GIN-indexed tsvector column, falling back to trigram ILIKE when FTS returns few results). Results are merged using a slot reservation system: 60% of the final top-K comes from vector results ranked by cosine similarity, with up to 40% reserved for keyword-only matches that the vector search missed. Retrieved chunks get stuffed into a prompt with source metadata and sent to Claude Sonnet or GPT-4o with instructions to cite sources in bracket notation.

On the backend, pub-sub workers handle the indexing pipeline: text extraction, chunking, batch embedding in groups of 100, and firing off face detection through AWS Rekognition on images pulled from PDFs (very good in some cases, not so much in others). The query endpoint is free with some rate limiting, but also sits behind x402 micropayments ($0.50) that bypass rate limits when valid (it's not cheap to run these queries as of now). There's also an MCP server so AI agents can query directly as a tool.

Built with the help of Claude, so some of the tech (RAG via LLM, pgvector, etc.) is newish to me. Was a fun exercise!

I did not know this and appreciate the insight! Will definitely store this back of mind and try and remember to use block comments when needing to explicitly return a promise.

While this is fair and I don't disagree that changing a column name or possibly other schema changes using an ORM and it's subsequent migration scripts is potentially going to be a faster exercise than without one, changing a column name (or any schema change for that matter) in a database in my experience is rare. If you choose to use an ORM for reasons like it might (emphasis on might) be a bit faster and easier to change a column name than without one instead of for other more meaningful people or efficiency-oriented reasons of the day-to-day developer workflow is probably a poor approach.

Keep in mind I've installed and used an ORM in projects where the ORM is used only for migrations, but not used in application code and this is absolutely a fine reason to use one imo. But adopting an ORM for migration purposes and forcing or using in application code simply because it's installed isn't necessarily a good approach.

While some people are hard lined enough on their anti-ORM stance that it sort of becomes weird, I agree with you that my beef with ORMs comes from being burned before a couple of times by a super inefficient aggregation ActiveRecord did on GROUP BY queries that it 1. not only took a really long time to figure out why a particular page in our app was loading slow but 2. we ended up having to write raw SQL to fix it.

I think the answer of whether to use one depends on the type and load/volume of app you're working with combined with the dynamics, size, and skill level of your team(s). I'm extremely comfortable writing, profiling, query planning, and debugging SQL queries. Others aren't, and therefore having an ORM to query data in the DB with the syntax of the language you're using in your projects makes way more sense, if nothing other in order to speed your team up.

While some people are hard lined enough on this that I agree it sort of becomes weird, I can tell you my beef with ORMs comes from being burned before a couple of times by a super inefficient aggregation ActiveRecord did on GROUP BY queries that it 1. not only took a really long time to figure out why a particular page in our app was loading slow but 2. we ended up having to write raw SQL to fix it.

I think the answer depends on the type and load/volume of app you're working with combined with the dynamics, size, and skill level of your team(s). I'm extremely comfortable writing, profiling, query planning, and debugging SQL queries. Others aren't, and therefore having an ORM to query data in the DB with the syntax of the language you're using in your projects makes way more sense, if nothing other in order to speed your team up.

1. Don't rename your columns, but if you have to

2. project search and replace on $COLUMN_NAME

If your column name is a common keyword, variable name, etc. in your code base and it's difficult to find using project search, that's unfortunate, but we organize our backend code and tests in a logical enough way that it's never taken longer than an hour to create a PR to create a migration to rename a column and update all places in code that reference it.

While this is true, I've found over time it's easier for me to quickly see in my code if I need to `await` on something if the function is an `async` function and not just that it returns a Promise.

I prefer

async function foo() { return await new Promise(...) }

as opposed to

function foo() { return new Promise(...) }

They're the same thing for the most part but the latter I have to potentially dig deeper into the function to confirm it returns a promise compared to the former.

I think it's necessary to put emphasis on the "get-rich-quick schemes" part, and not the "blockchain/cryptocurrencies" part.

I hope you're not implying blockchain tech and all cryptocurrencies don't/can't provide real utility value, but if so I beg to differ substantially. I built a hook to the Stellar blockchain to store SHA256 hashes of audit evidence that can be validated against at a later time as tamper-proofing evidence. This is just one extremely simple use case, not to mention the entire Defi ecosystem that's being developed as we speak.