penciltwirler

Joined 5 years ago

Karma

Submissions

Member since

May 2

View on Hacker News

Recent submissions

Comment · 1 month ago

I think it's useful for implementing Sagas https://temporal.io/blog/saga-pattern-made-easy

Basically, a multi-step event handling system, where the data could be spread across multiple databases/systems, so there's no way to rollback a transaction atomically across all the databases. Instead, you explicitly codify "compensations" to undo your previous commits so that you eventually end up in a consistent state.

Temporal is the orchestrator/framework/library to implement the above in an easier way.

Comment · 2 months ago

Hugging Face's internal storage system is more sophisticated now after they acquired XetHub. They have a clever way to chunk and content dedupe the data that makes it a lot more efficient to version control large files.

Comment · 3 months ago

Dumb question, but how do they know if a codebase is actually written by an LLM? I know that sometimes code agents write "Co-authored by" in the commit messages, but this is something you can over-write.

Comment · 4 months ago

The premise is that Git-LFS sucks, so we need to build a new data versioning system (in Rust, from scratch). While I mostly agree with this premise, but there are already lots of existing (mature) data versioning systems with the same tricks under the hood:

- Pachyderm (Go): https://github.com/pachyderm/pachyderm

- XetHub (acquired by HuggingFace): https://huggingface.co/blog/xethub-joins-hf

- LakeFS (Go): https://github.com/treeverse/lakeFS

- Oxen (Rust): https://github.com/Oxen-AI/Oxen

I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days...

But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system that can version control large binary files at scale. I think there's lots of opportunities to share ideas here, but perhaps the lack of idea sharing (currently) creates opportunity!

Comment · 5 months ago

blatant copyright infringement of https://rosalind.info/problems/locations/

Comment · 5 months ago

"Chromebook" had a better ring to it.

Comment · 3 years ago

I think in Picasa's case, they learned from that and turned it into Google Photos. I don't think there's any shared code, but at least they learned from it from a product perspective. I would say Google Photos turned out great.

Comment · 3 years ago

There's lots of tools built on top of K8s to accomplish this tho. For example, Argo, Tekton, Flyte etc.

Comment · 3 years ago

Does this affect Hacker News?

Comment · 3 years ago

Have a look at ULIDs

Comment · 3 years ago

Lots of comments about Avro Arrow here lool. As a Canadian, I'm both proud and saddened by the fact that although we have elite engineering talent, that talent tends to be drained to the US.

Side note, just a couple decades after the Arrow, Canadian researchers once again lead the world, but this time in the field of AI, specifically deep learning. However, what happened after that? Google hired Geoffrey Hinton, and the industrial might of the US took over.

Perhaps this is inevitable given that we are much weaker economically compared to the US.

However, one last thought is that, aerospace vs software is not the same. For aerospace, yea I think you need a giant super power of a nation to keep it alive. However, software can be built by one guy in a garage. I'm not sure why Canada just let its dominance in AI "slip away".

Also side side note about Avro Arrow, is it just a weird naming coincidence that the two Apache projects exists: Avro and Arrow? I think Avro is named after the original British Avro company. However, I don't think Arrow has anything to do with aviation.

Comment · 3 years ago

just use dbdiagram.io

Comment · 3 years ago

Haha a lot of funny comments here. I think overall it's neither here nor there. You should be proud that the "elites" at Google copied your code ;)

Comment · 3 years ago

Just read the original paper "Attention Is All You Need". Cohere's founder was the guy who wrote that paper in the first place.

Comment · 3 years ago

Hold on... aren't there a bunch of incredily successful "open source" / "source available" companies? Databricks, Cockroach Labs, Hashicorp, MongoDB, Elasticsearch etc?