adrienconrath
Joined 8 years ago
Karma
5
Submissions
5
Member since
Sep 12
ex senior software engineer at Meta Superintelligence Lab
Creator of Omnesis, an open source context layer for AI agents
View on Hacker NewsRecent submissions
Comment · 1 month ago
I built this with https://omnesis.dev
It can index all your data (emails, whatsapp/imessage, gdrive, finance, health, etc) and notably also allows you to install a chrome extension to also capture and index pages you see on the web.
It builds a search index and connected graph of all this data.
I have just open sourced: https://github.com/omnesis-dev/Omnesis
Comment · 1 month ago
Let me know if you have any feedback! :)
Comment · 8 years ago
> Because of non-deterministic data placement, it is very resilient to failures in terms of write availability.
I am happy to expand more on this point.
We have this concept of "node set" of a log which is the set of storage nodes available to receive record copies sent by the sequencer. It is typically made of 20-30 nodes in typical deployments at Facebook. Write availability is maintained as long as enough storage nodes in the node set are available to accept copies. When storage node failures are detected, the sequencer can just exclude these nodes from the list of potential recipients for new records. It does not need to update a view that needs to be synchronized with readers, which is a heavy-weight operation. This model allows preserving high write availability even if many nodes in the node set are unhealthy.
Additionally, this record copy placement flexibility allows the sequencer to quickly route around latency spikes on individual storage nodes, which helps guarantee low append latency.
Comment · 8 years ago
LogDevice has this concept of "node set", which is the set of storage nodes that can be selected by the sequencer as recipients for a record or a block of records. A typical node set size is around 20-30 in our deployments. Each storage node in the node set contains a subset of the records (or blocks of records) of the log, we call that subset a log strand. The amount of IO capacity available to append records to a log or read records from a log scales with the size of the node set.
All of this is done while preserving the total ordering guarantee thanks the separation of sequencing and storage.
The operator could for example set a bigger node set size for logs that are known to have multiple consumers and require more IO capacity.
At facebook, we have use cases where a single consumer will need to replay a backlog of records in a log, sometimes hours or days worth of data to rebuild its state. We call this a backfill. Node sets allow the IO to be spread across multiple disks which improves backfill speed and helps reduce hotspots.
-- Adrien from the LogDevice team.