AntonFriberg
Joined 8 years ago
Karma
11
Submissions
10
Member since
Nov 8
https://www.linkedin.com/in/antonfriberg/
https://github.com/AntonFriberg
View on Hacker NewsRecent submissions
Comment · 1 month ago
I would like to see some sources on your claim about it being Russian. It is incorporated in the US with most developers in Amsterdam.
After the invasion of Ukraine they stayed silent for a while but that was because they needed to allow there developers to get out of Russia. Many of them are Ukrainians including the CTO and founder. As soon as it was safe for there team they took a very firm stance against Russia with Ukrainian flags on the website and written statement from the team.
I am not aware of any Russian influence currently.
Comment · 6 months ago
For Nix Home-Manager on non-nixos you need to fix the driver integration.
https://nix-community.github.io/home-manager/index.xhtml#sec...
Nowadays it is quite easy https://github.com/AntonFriberg/dotfiles/blob/master/modules...
Note that the way this works is that after you activate your home manager generation it outputs a script path that you need to run manually as root which installs a Systemd service which ensures that the drivers are linked correctly.
Comment · 8 months ago
Came across this at work as well during the early days of Kubernetes. At that time all VM storage was entirely backed by NFS or Network attached iSCSI, there was no local disk at all. We noticed intermittent issues that caused kube API to stop responding but nothing serious.
Then all the sudden there was a longer lasting outage where the ETCD did not really recover on its own. The kube API buffered requests but eventually crashed due to OOM.
The issue was due to the Kubernetes distro we picked (rancher) which runs a separate cluster in front of the deployed ones for the control plane. Large changes happening in the underlying clusters needed to be sent to the control plane. After we hit the latency threshold it started failing at regular intervals and the control plane started drifting more and more and needing more and more changes at start up. Until it could not recover.
Solving it took some time and confidence. Manually cleaning up large unused data in the underlying ETCD instances so that they did not cause upstream errors.
It was later during post-mortem investigation that I understood the RAFT algorithm and the storage latency issue. Convincing the company to install local disks took some time but I could correlate kube API issues to disk latency by setting up robust monitoring. Fun times!
The requirements are well documented nowadays! https://etcd.io/docs/v3.1/op-guide/performance/
Comment · 2 years ago
I am the author of the first link.
We successfully replaced Elasticsearch with ClickHouse within a Year. We went from having difficulties managing 3 months of data to storing 5 year+ of data. We also had difficulties onboarding users into Kibana and Elasticsearch world, this was not such a big problem with ClickHouse since most developers feel comfortable with SQL.
This change together with Apache Superset for the BI layer made a huge impact in the amount of internal users that could extract value from the data collected. Went from around 150 to 800 internal users.
We have not yet managed to bring everything into a wide event table but as long as the logs can be joined with the metrics under an interface that the developers feel comfortable with and the solution is cost-effective enough to allow all relevant context to be collected you will get far imo.
Comment · 2 years ago
Thanks Randal! I hope to be able to complete a follow-up post on how to automate most of the schema changes by utilizing a form of data contract with the data owner of each metric table. It is very easy to roll out incorrect changes when working with the materialized views in a sharded ClickHouse table due to the need to having to toggle the materialized view and distributed tables when changing the underlying long-term storage tables.
Comment · 2 years ago
Author here, thanks! Feel free to share it on other platforms. Let me know if you have any questions.
Comment · 3 years ago
As a non-native English speaker I was confused by the need to capitalize so many words (but not all) in headers. I make frequent use of https://capitalizemytitle.com/ to make sure that I follow some type of standard.