Comment

Comment by jandrewrogers

This looks like a very nice implementation of a graph database. However a 6-machine cluster barely qualifies as "distributed" for the purposes of a graph database. You will experience almost no sub-linearity at this scale no matter how poorly the algorithms distribute. I am not intimately familiar with the transactional implementation but the graph processing algorithms described in Titan are not significantly distributable.

For graph queries, linearity can be definitely achieved on parallel systems with tens of thousands of compute nodes using the state-of-the-art algorithms. However, Titan does not use any of those algorithms and will be very lucky to exhibit linearity at a couple dozen nodes. Not to take away from the implementation but people looking for a scalable graph database will not be satisfied with Titan.

As an aside, the best algorithms and data structures for doing massively parallel graph operations are not published but obviously exist. The fastest system in the Graph500 benchmark uses a half million cores on a trillion edge graph. That is a several order of magnitude gap between what the best open source systems can do and what systems developed by closed research organizations can do as published in public benchmarks.

(Disclosure: I invented one of the aforementioned families of massively parallel graph processing algorithms in 2009, and not the first. The published literature has not even caught up with the first generation of such algorithms. A significant portion of advanced computer science research is no longer published.)

parent jandrewrogers 14 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton