kyotov
Joined 5 years ago
Karma
79
Submissions
14
Member since
Sep 19
Recent submissions
Comment · 5 years ago
check out https://kyall.notion.site/kyall/KySync-v1-1-dd9931f330f24146...
Comment · 5 years ago
I can see your point. We really need to run some experiments in S3 to satisfy this point rigorously.
Intuitively, the more similar the files are, the more 3x-10x+ will be representative in the real world. As the files become more and more different, one is of course at the mercy of the bandwidth between the computers. If you need to transfer 1 GiB differences over 1 MiB/s connection, it will take ~1000 seconds -- no magical way around it.
The comparison so far is practically on the computational cost of the sync, which can be significant as differences pile up.
Comment · 5 years ago
We have released a v1.1 with improved performance!
In summary, we see:
- 10%-100%+ performance improvement with v1.1 on 2 GiB data and 16 threads;
- A significantly improved single thread performance, now on par with Zsync for 2GiB data.
Checkout the whole write-up at https://kyall.notion.site/KySync-v1-1-dd9931f330f241469d3e60...
Comment · 5 years ago
Yup! Will be looking at that one of the following nights.
It is so convenient to edit in Notion, but I share the frustration about paging and navigation for reading!
Comment · 5 years ago
Instead of an explanation, we decided to address some of the reasons :)
https://kyall.notion.site/KySync-v1-1-dd9931f330f241469d3e60...
Enjoy!
Comment · 5 years ago
I should have made my intentions clearer from the start :)
KySync started as a pet project to brush up on my C++ skills which I had not used for 20 years and was going to need for a job that I started last week! I had spent my fair share of time optimizing single threaded performance back in grad school, and was looking for different experiences.
That said, and as we mentioned in the write up, we had some improvements cooking that we did not put in v1.0. I spent some time tonight to merge them in and run some performance experiments.
Single threaded performance is now much improved and practically on par with Zsync. I released it as v1.1.
https://kyall.notion.site/KySync-v1-1-dd9931f330f241469d3e60...
We will of course keep looking for more performance, but if you have any suggestions for further improvements, please share -- or join us!
Best, Kamen
P.S. Both of these are contributed by Chaim Mintz, whom I worked with in a previous life.
Comment · 5 years ago
Comment · 5 years ago
Indeed... but thanks for the feedback.
Since the content of the page is pretty stable now, I can probably host it in more normal way than Notion.
Comment · 5 years ago
Thanks for the pointer -- I will look into comparing against bita!
Comment · 5 years ago
If there are specific scenarios that need to be optimized for the use case, please get in touch!
Comment · 5 years ago
It's not really apples-to-apples as rsync does work on both the client and the server.
Zsync has a comparison with rsync in isolated cases where it makes sense and it is very close in performance!
Comment · 5 years ago
We should make a clarification... The intent of KySync (as well as Zsync) is to use across systems, not on a single system. KySync supports HTTP (like Zsync) as well as HTTPS (which Zsync does not).
The primary reason to do the performance comparison on a single system, is so that the results are easy to replicate with as little setup as possible. Because we do this for both KySync and Zsync it is apples to apples.
HTTP bandwidth and storage cost money, and this is self funded project, so I can't afford to put up test files of data publicly visible to the world.
One thing we can look into is leverage AWS/S3 to upload some data and use it for a performance experiment, but that will need some logistics for the developer to set their AWS account properly. Will look into it.
Of course, the more similar the files are, the closer the remote results will be to this first set we published.
Comment · 5 years ago
[Intro](https://kyall.notion.site/KySync-v1-0-29eaad446308449a8c9a23...)
KySync is an efficient way to distribute new data which makes use of older (but similar) data that you may already have present locally. KySync supports HTTP v1.1, but can easily be extended to support any server protocol which supports range queries.
KySync is [released](https://github.com/kyotov/kysync/releases) under the MIT Open Source License (see [COPYING](https://github.com/kyotov/ksync/blob/master/COPYING) in root of repository).
KySync is a full rewrite of [Zsync](http://zsync.moria.org.uk/) in modern C++. While no code was reused from Zsync, the awesome [Zsync technical paper](http://zsync.moria.org.uk/paper200503/) was the major resource used for the implementation of KySync.
The value proposition of KySync over Zsync is that it takes advantage of modern architecture features (multi-core multi-CPU systems as well as exceedingly fast IO subsystems, e.g. NVMe SSDs). KySync is 3x-10x (or more) faster than Zsync on such commonly available modern hardware. We have not spent much time optimizing KySync single-thread performance, so there are cases where with sufficiently high similarity, Zsync is faster when less then 4 threads are used in KySync.