lichtenberger
Joined 7 years ago
Karma
1,848
Submissions
626
Member since
Dec 27
Recent submissions
Comment · 18 days ago
I hope you're fine
Comment · 3 months ago
Oh, and Sirix optionally already stores a rolling merkle hash over the data.
Comment · 3 months ago
It's first and foremost a document store / so for JSON or XML currently, that's why it's not SQL compliant. I think other query languages as JSONiq are much more tailored to this, albeit a niche of course (based on XQuery).
Regarding tamper proof audit logs not much is missing. Cryptographic hashes instead of XXH3, a commit hash chain and signed commits. Actually, I think that's a great addition with minimal changes needed.
What you can audit currently is "who changed what" for instance.
Comment · 3 months ago
Besides, documents are split into fine granular nodes, so we have "no" upper limit besides running out of 48bit nodeKeys/node identifiers at some point maybe.
Furthermore, a (versioned, as always) path summary keeps track of all distinct paths, which is a key ingredient for the optional secondary indexes, which can index paths or paths and content among other stuff as simply indexing fields. Optionally you can also add indexes to speed up aggregate queries (which are basically column projections).
Furthermore, the whole storage can self-validate through checksums stored in parent pages up to the root as in ZFS for instance.
DeweyIDs can be optionally stored for each node, which can speed up the comparison of subtrees between revisions (in order to detect simply which changesets belong to a certain subtree without having to traverse ancestor chains). They lend themselves well for compression.
We even have importers which can identify based on heuristics minimal edit operations to import existing revisions of XML or JSON documents and to commit these with hopefully a minimal or close to minimal set of update operations, but it depends heavily on the data.
I'm working on an evolutionary, accumulate-only database system called SirixDB written in Java (and a module in Kotlin) in my spare time and in my day to day job I'm also working as a software engineer (embedded; besides I'm dancing a lot -- swing dances, mainly Lindy Hop and travel to festivals in Europe often times in my holidays) :-)
Furthermore, I'm working on Brackit (http://brackit.io), a query engine for JSON and XML using a JSONiq like query language. It can be used as ad-hoc in-memory query processor as the query processor of a data store. We envision, that it encapsulates common proven optimizations whereas the individual data stores can add further stages to rewrite the queries for index matching and physical optimizations.
SirixDB (https://sirix.io or https://github.com/sirixdb/sirix) is all about efficient versioning of your data. That is on the one hand it reduces the storage cost of storing a new revision during each transaction-commit while balancing read- and write-performance through a novel sliding snapshot algorithm and dynamic page compression. On the other hand Sirix supports easy query capabilities for instance to open a specific revision by a timestamp or several revisions by a given timespan, to navigate to future or past versions of nodes in the tree-structure and so on. It basically never overwrites data and is heavily inspired by ZFS and Git and borrows some ideas and puts these to test on the sub-file level.
In stark contrast to other approaches SirixDB combines copy-on-write semantics with node-level versioning and does not require a write-ahead-log for consistency.
It all started around 2006 as a university / Ph.D. project of Marc Kramis and I worked on the project since 2007 and already did my Bachelor's Thesis, Master's Thesis as well as several HiWi-Jobs on topics regarding the project and I'm still more eager than ever to put forth the idea of a versioned, analytics plattform to perform analytical tasks based on current as well as the history of the data.
e-mail address: [email protected]