Paul-E

Corporations are made of humans. An airline may be a corporation, and it may take people's lives into it's hands, but when a plane crashes the pilots are the first one on the scene.

> the onus is STILL on you the individual engineer. You are mistaken if you think that's changed.

In a lot of shops right now, this leads to an accountablility-authority gap. Where AI changes are merged in quickly without review and without my input, how can I be responsible for understanding the system? I can't. Its the same with reliability.

A lot of good engineers take it a a personal duty to understand the system and keep it up. The way a lot of businesses are using AI makes that impossible. And your job might just be cranking out features, with regard to little else.

I think this is what is stressing a lot of engineers out. It might be worth having a conversation with your boss about what you are actually accountable for. If your boss agrees you are not responsible for uptime, reliability, security, or even understanding the system you might find yourself much happier. If you your boss wants you to be accountable for those things, then you should feel empowered to ask for authority over the things that give you control of the outcome.

Obsidian seems like a perfect candidate for a WASM/WASI based plugin system that would properly sandbox plugin code.

When queries get complicated enough, you end up writing a custom query builder to build your SQL.

Ceci n'est pas une pipe

> And, IMO, making dynamic queries harder is preferable. Dynamic queries are inherently unsafe. Sometimes necessary, however you have to start considering things like sql injection attacks with dynamic queries.

Depends on what you mean by "dynamic query". You are dealing with injection attacks as soon as you start taking user input. Most useful user facing applications take user input.

In a simple case it might be "SELECT * FROM posts WHERE title LIKE '%hello world%', where "hello world" is a user specified string. This is easy with sqlx. Where things get more difficult is if you want to optionally add filters for things like date posted, score of the post, author, etc... That makes the query dynamic in a way that can't be solved by simply including a bind.

That's where sea-orm shines over sqlx IMO. sqlx will force you to do something like

```

let mut my_query = "SELECT * FROM posts WHERE title LIKE '%' + $1 + '%'";

let mut my_binds = vec![args.keyword];

if let Some(date) = args.date {

  my_query = format("{my_query} AND date = $2");

  my_binds.push(date);

}

...

```

Your building a string and tracking binds. It gets messy. A good query builder like seaorm has lets you do something this:

```

let mut query = Posts::find().filter(Column::title::like(args.keyword));

if let Some(date) = args.date {

  query = query.filter(column::Date::eq(date));

}

```

This pays off as your queries get more complicated. It pushes the string manipulation and bookkeeping into a library, which can be more thoroughly tested.

It also lets you pass around typed partial queries, eg in the example above query might be returned from a function, which helps you build more modular code.

I want to address this one point:

> Similar thing can be said about writing SQL. I was really happy with using sqlx, which is a crate for compile-time checked SQL queries. By relying on macros in Rust, sqlx would execute the query against a real database instance in order to make sure that your query is valid, and the mappings are correct. However, writing dynamic queries with sqlx is a PITA, as you can’t build a dynamic string and make sure it’s checked during compilation, so you have to resort to using non-checked SQL queries. And honestly, with kysely in Node.js, I can get a similar result, without the need to have a connection to the DB, while having ergonomic query builder to build dynamic queries, without the overhead of compilation time.

I've used sqlx, and its alright, but I've found things much easier after switching to sea-orm. Sea-orm has a wonderful query builder that makes it feel like you are writing SQL. Whereas with sqlx you end up writing Rust that generates SQL strings, ie re-inventing query builders.

You also get type checking; define your table schema as a struct, and sea-orm knows what types your columns are. No active connection required. This approach lets you use Rust types for fields, eg Email from the email crate or Url from the url crate, which lets you constrain fields even further than what is easy to do at the DB layer.

ORMs tend to get a bad reputation for how some ORMs implement the active record pattern. For example, you might forget something is an active record and write something like "len(posts)" in sqlalchemy and suddenly you are counting records by pulling them from the DB in one by one. I haven't had this issue with sea-orm, because it is very clear about what is an active record and what is not, and it is very clear when you are making a request out to the DB. For me, it turns out 90% of the value of an ORM is the query builder.

I haven't tested that, so I'm not sure if it would work. The import only inserts rows, it doesn't delete, so I don't think that is the cause of fragmentation. I suspect this line in the vacuum docs:

> The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.

means SQLite does something to organize by rowid and that this is doing most of the work.

Reddit post/comment IDs are 1:1 with integers, though expressed in a different base that is more friendly to URLs. I map decoded post/comment IDs to INTEGER PRIMARY KEYs on their respective tables. I suspect the vacuum operation sorts the tables by their reddit post ID and something about this sorting improves tables scans, which in turn helps building indices quickly after standing up the DB.

I'm curious if you have tried SeaORM? I've used it a little bit (not too extensively) and really like it. It's like sqlalchemy in that you can declare your tables and have a type checked query builder, which is a big win IMO. It's nice to add/change a field and have the compiler tell you everywhere you need to fix things.

I've definitely had issues when using sqlalchemy where some REST API type returns an ORM object that ends up performing many queries to pull in a bunch of unnecessary data. I think this is harder to do accidentally with SeaORM because the Rust type system makes hiding queries and connections harder.

Most of my usage of SeaORM has been as a type query builder, which is really what I want from an ORM. I don't want to have to deal with lining my "?" or "$1" binds or manually manipulate strings to build a query. IMO a good query builder moves the experience closer to writing actual SQL, without a query builder I find myself writing "scripts" to write SQL.

This is what a number of startups, such as Yurts.ai and Vannevar Labs, are racing to build for organizations. I wouldn't be surprised if, in 5-10 years, most large corps and government agencies had these sort of LLM/RAGs over their internal documents.

Has anyone tried to decentralize curation?

Here is straw man proposal, similar to cert chains and webs of trust: Say I'm a "curator". I say on HN/Reddit/Discord "here is my key hash 'p2pcuration:185da2bc59167692f596404fd83235f9bcb4e107b041f2e6e8d972da6dba00b7'". Any user that clicks the link or copies it into the search app adds the key to the trusted user list. With my private key I can sign torrents after I download them myself, which would mark the torrent as "good". When anyone who has added my key searches, the system searches for a corresponding signature from me as well. If a signature is found, the UI can chose to elevate that result.

The system could be extended so that signers could also sign other keys, expanding the trust network.

This system doesn't need to be run or maintained by each user. It could be served through a webui that can be run locally or shared with a small community. Migrating the interface to a new host would just require moving the config and keys.

Thank you for making this point. I didn't articulate it, but this is part of why I felt I had to say something.