pacmansyyu

Joined 9 years ago

Karma
77
Submissions
28
Member since
Feb 23
DevOps @ Zerodha
View on Hacker News

Recent submissions

Comment · 1 year ago
Comment · 1 year ago
Comment · 1 year ago
Yes, SOPS does have `exec-env` which does the same thing, kind of. From one of the issues, it currently lacks support for the POSIX-semantic way to run commands: https://github.com/getsops/sops/issues/1469, where you cannot add a `--` to tell sops that everything after it is supposed to be a command, so you end up having to quote everything. Other things that I found lacking were that with SOPS, adding a new team member means manually updating .sops.yaml, re-encrypting all files, and managing PGP/age keys. With kiln, you just add their SSH key to kiln.toml and run `kiln rekey`.

kiln also lets you have different access controls per environment file (devs get staging, only ops get production) without separate .sops.yaml configs, automatically discovers keys from SSH agent/~/.kiln/, and has built-in template rendering and export formats for different tools. You could definitely build similar workflows with SOPS + scripts, or any other tool, but kiln packages these common patterns into a single tool with better UX for teams.

Think of kiln as "opinionated SOPS", focused specifically on environment variables rather than general file encryption.

Comment · 1 year ago
Well, technically SOPS/age are both encryption tools first. Both of them are excellent, mind you. But they lack the user experience, specifically SOPS, with handling keys in a multi-user environment, and subsequently with the overall developer workflow. They do offer a lot more than just accessing environment variables securely though, something that kiln is trying to solve.

At first, I did consider using them instead of building my own tool on top of age. But our requirements were far beyond just encrypting and decrypting files in a single environment.

What kiln adds here is the role-based access control, so you can define multiple files, and users/groups who should be able to access them. It also adds to the developer workflow where you can directly run commands through kiln with the variables injected in the command's shell environment. You can also render templates for all the kiln-encrypted files you have access to.

You can say it's a wrapper over age, but adds functionality that allows seamless sharing of developer workflows, and environments, all from a single place. It's git-friendly, and primarily aims for your secrets to travel along with the code so all deployments can be done offline (as an alternative to something like Infiscal, or Vault). I've tried to make it as simple as possible to adopt for anyone in the team.

The only other best way for me to put it is that you should try it out, and I'm sure it'll be helpful in a lot of ways. If you have any more questions, I'm happy to answer them!

Comment · 1 year ago
I'm working on Damon[1], a Nomad Events stream operator that automates cluster operations and eliminates repetitive DevOps tasks. It's a lightweight Go binary that monitors the Nomad events stream and triggers actions based on configurable providers.

A few examples of what it can currently do:

- Automated data backup: Listens for Nomad job events and spawns auxiliary jobs to back up data from services like PostgreSQL or Redis to your storage backend based on job meta tags. The provider for this is not limited to backups, as it allows users to define their custom job and ACL templates, and expected tags. So it can potentially run anything based on the job registration and de-registration events.

- Cross-namespace service discovery: Provides a lightweight DNS server that acts as a single source of truth for services across all namespaces, solving Nomad's limitation of namespace-bound services. Works as a drop-in resolver for HAProxy, Nginx, etc.

- Event-driven task execution: Allows defining custom actions triggered by specific Nomad events; perfect for file transfers, notifications, or kicking off dependent processes without manual intervention. This provider takes in a user-defined shell script and executes it as a nomad job based on any nomad event trigger the user defines in the configuration.

Damon uses a provider-based architecture, making it extensible for different use cases. You can define your own providers with custom tags, job templates, and event triggers. There's also go-plugin support (though not recommended for production) for runtime extension.

I built this to eliminate the mundane operational tasks our team kept putting off. It's already saving us significant time and reducing gruntwork in our clusters.

Check out the repository[1] if you're interested in automating your Nomad operations. I'd love to hear your thoughts or answer any questions about implementation or potential use cases!

[1]: https://github.com/Thunderbottom/damon

Comment · 1 year ago
No, unfortunately our company’s and external regulatory compliance policies require us to host all data within the country itself, alongside it being required to be run on an infrastructure that is easily auditable. So as a policy within the company, all our internal services are open source and self hosted.
Comment · 1 year ago
We had separate database and app instances, the DB instance had 32 cores and 64GB memory, which we doubled to keep up with our requirements. We have tens of millions of visits daily, and our database was close to ~300GB within the first few months.

For plausible I believe that since it runs on Postgres, scaling should not be a problem as long as you scale the resources with it.

Comment · 1 year ago
We had huge servers, with the database and the application itself running on separate instances. IIRC, we had a 32 core, 64GB instance just for the DB itself which we doubled when we started adding more sites to our configuration and it still wasn’t enough. As for the numbers, our site(s) get heavy traffic everyday, in millions daily, since we are a stock broker.

You’re right about MySQL performing alright for 10-20m rows, but from our perspective those numbers are not that big for a company this size.

Comment · 1 year ago
In all honesty, these optimizations are quite basic. We already used MariaDB instead of MySQL itself. Other things listed in the post are something that we have standard across all our databases, well, except for deleting the data to speed up the database.
Comment · 1 year ago
Comment · 2 years ago
Cheers! The issue has been fixed now and is live. Thanks a lot for your feedback :)
Comment · 2 years ago

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton