skafoi

Not sure if I got your comment correctly, but this is rather not about existing drives (though it could be applied to them), but to have a dedicated drive running within WSL that has a native performance. WSL handles existing drives (those that are readable under windows and linux) as a network drive using the 9p protocol. And, frankly, the performance sucks. E.g. I only reached write speeds of ~125MB/s when using 9p (while the disk is NTFS formatted and encrypted with Bitlocker), while the native LUKS encryption + ext4 reached 460MB/s. Using this approach transfer between Windows and Linux still would be slow though.

Thinking about that...I should probably have mentioned this in the article.

I am one of the developers (and a co-founder) of TiloRes. We are not using a graph database for that purpose, because they would be way to slow for huge entities. Instead we're using a combination of AWS serverless services, for storing the data mainly DynamoDB and S3.

That's looking pretty. Little bit lagging though in my browser as long as the "Real-Time Pull Requests" are not paused.

Do you also offer some kind of abstraction libraries when e.g. working with DynamoDB on AWS and then switching to GCP? Or would I still have to prepare my code for that?

The idea for that is, that typical enterprise features like authorization for certain records or even attributes are not publicly available. Also e.g. encryption of the data in S3 and other parts may be an enterprise only feature. Other things, like API authorization, preventing public access to S3 and therelike must be included in the OSS version for the same reasons you mentioned.

Thanks for your input.

1k hops is also not something we see on a regular basis in our old business, which is much about people moving houses and transactional data from payment service providers. Ppl with money issues seem to move a lot more often and also fraud cases often have a lot of hops.

Regarding the performance on neo4j: the challenge for an honest and fair test towards this would be about how to properly compare a server-based solution vs. a serverless solution. TiloDB automatically scales up and down without any further interaction due to using Lambdas for all calculations. So would you compare it with a relatively small neo4j instance or with a large cluster? I honestly don't know. When we started doing this internally for our previous company, we obviously did test the edge cases. After all, we didn't want to create our own solution. For graph databases, the issue here are either a lot of edges leaving from one node or a long chain of edges. The first scenario was still handled okish: response times of around 6 seconds for 1.000 nodes if I remember correctly. The second scenario was a total fail. The problem for the later one lies in the transitivity as a graph database has to jump from one node to the next one and so on. To be fair though: when it comes to dynamic entities, so choosing which rules are relevant, graph databases might be the better choice - especially when response times don't matter.

The response times provided in the article are for the whole process of searching and returning the entity. The indexes themself are obviously a lot faster - to be precice we are using DynamoDB for storing the indexes, which most times return results in <10ms. Compared to other databases this may still sound slow, but we know that we won't run into scaling issues in this way and that's kind of what matters currently most for us.

Hope that somehow makes sense what I wrote.

Thanks a lot. Sounds similar to the issues we had in our previous company. We provide some basic ETL functionalities to tackle these. Will be interessting to see where this will eventually end up and how much our solution does there or where it is better to use the probably in those companies existing ETL tools.

That sounds mostly like deduplication which is often used in marketing contexts. There are indeed some good solutions out there, but from our experience they have difficulties handling huge amounts of data (>1 billion data sets) and they are often batch based, so your data is always outdated, whereas we constantly add new data in near real-time.

I would really love to hear more about your experience regarding the client customizations. So far the two things I can see are domain model customization and rule customization. Obviously with the rule customizations being the more challenging one.