hassy
Joined 19 years ago
Karma
1,235
Submissions
257
Member since
Sep 6
Recent submissions
Comment · 2 years ago
This is a great blog post! just taking the opportunity here to comment on this:
> Finally for full scale high fidelity load tests there are relatively few tools out there for browser based load testing.
It exists as of a few months ago and it's fully open source: https://github.com/artilleryio/artillery (I'm the lead dev). You write a Playwright script, then run it in your own AWS account on serverless Fargate and scale it out horizontally as you see fit. Artillery takes care of spinning up and down all of the infra. It will also automatically grab and report Core Web Vitals for you from all those browser sessions, and we just released support for tracing so you can dig into the details of each session if you want to (OpenTelemetry based so works with most vendors- Datadago APM, New Relic etc)
Comment · 2 years ago
> I'm very biased
Comment · 2 years ago
Don't write your own load testing tool other than as a fun little exercise. At least not without understanding coordinated omission and thinking about workload modeling (open? closed? hybrid? all of the above?) [1]. Get this wrong and the results produced by your tool will be worthless.
Once you've got that out of the way, don't forget that you'll want a distribution story. It does not matter how efficient your tool might be on a single machine - you'll want to distribute your tests across multiple clients for real-world testing.
"Sure it's easy" you might say, "I know UNIX. Give me pssh and a few VMs on EC2". Well, now you've got 2 problems: aggregating metrics from multiple hosts and merging them accurately (especially those pesky percentiles. Your tool IS reporting percentiles rather than averages already, right?!), and a developer experience problem - no one wants to wrangle infra just to run a load test, how are you going to make it easier?
And, this developer experience problem is much bigger than just sorting out infra... you'll probably want to send the metrics produced by your tool to external observability systems. So now you've got some plugins to write (along with a plugin API). The list goes on.
I'm very biased, but it's 2024. Don't waste your time and just use https://www.artillery.io/
1. https://www.artillery.io/blog/load-testing-workload-models
Comment · 3 years ago
curl is fantastic. There's also HTTPStat which provides a waterfall visualization on top of curl timings: https://github.com/reorx/httpstat
There's also Skytrace (made by yours truly), which provides timing info as a waterfall visualization inspired by HTTPStat + lots more (syntax highlighting for responses, built-in JMESPath support, command-line assertions and checks etc) - https://github.com/artilleryio/artillery/tree/main/packages/...
Comment · 3 years ago
The main thing about the advice comes immediately before the bit you’re zooming in on. Want to talk to people who have seen hundreds of companies deal with what you’re dealing? YC is great for that.
Comment · 3 years ago
Did YC S21 which was an all remote batch. my 2c.
YC is 100% what you make of it. It's not a lean back experience.
I did not meet most of the companies in my batch but I've gotten to know many founders through YC that I would not have otherwise. Founders that have been source of advice and support.
Network of clients - yep don't go into YC expecting to sell to other YC cos. It is easier to get warm intros through the network though.
YC advice, office hours specifically - it's what you make of it too. Expecting a group partner to know your space in great detail is unreasonable but if you recognize that they've seen hundreds of companies with similar problems and make use of that pattern matching, you can get very valuable advice. Some of the advice I did not take and did the opposite and it was the right decision. And some advice that I did not take was exactly right, but I only saw it in retrospect months later.
Fundraising - being a YC company definitely opens doors, and also helps protects you from bad actors who have to think twice before fucking with a YC co. Very valuable for any first-time founder. You have someone to sanity-check everything, terms you're not sure about etc. The bump in valuation is real too.
I'd do YC again in a heartbeat.
Comment · 4 years ago
Clickable links:
Blog post with demo: https://www.artillery.io/blog/this-load-test-cost-us-how-muc...
Project on GitHub: https://github.com/artilleryio/artillery
Comment · 4 years ago
This lets you load test with Playwright (very similar to Puppeteer):
Comment · 4 years ago
Yes indeed but wrk2 or Vegeta is still better for this particular use case (unless k6 has support for setting a constant RPS rate, afaik it does not), as otherwise the overhead of establishing a new TCP connection for a single HTTP request will dominate the benchmark.
Comment · 4 years ago
Yep, k6 suffers from coordinated omission [1] with its default settings.
A tool that can send a request at a constant rate i.e. wrk2 or Vegeta [2] is a much better fit for this type of a performance test.
1. https://www.scylladb.com/2021/04/22/on-coordinated-omission/
Comment · 4 years ago
Bombardier is cool, but serves a very different use case.
I mentioned elsewhere in the comments that we have a Docker image, and are working on other methods of installing the CLI to alleviate some of these dependency-related concerns.
Getting side tracked here, but there seems to be a common sentiment when it comes to Node.js that it's uniquely insecure. Node.js has indeed had some unfortunate press when it comes to supply-chain security, but every other runtime is susceptible to those attacks (PiPy, Gems, Maven, Rust Crates). Ultimately of course, if you choose to avoid using any software built on top of those stacks, that's your choice.
Artillery specifically is no different to any other Node.js-based project in how large the dependency tree is. VSCode for instance is used by millions of developers has 1.6k dependencies [1].
Comment · 4 years ago
I haven’t come across Hurl before. It looks cool!
Comment · 4 years ago
Yes, those dependency trees can be large. Yes, supply chain attacks are a real threat. But Node isn’t that different than Python or Ruby in that regard. How far down the stack do you personally choose to go? I trust you’re familiar with that famous paper published by a certain mr Thompson in the mid-80s?
The world is a big place. There’s a lot of software out there written in Node.js, used happily and productively by millions of developers, many of them in corporate environments.
Given the opinions you expressed elsewhere in the thread here I think it’s clear that this tool is not for you. I hope no one is forcing you to use it.
Artillery - test everything you build & run in the cloud.
We're hiring! join us to help build the future of testing for DevOps.
https://www.artillery.io
Twitter: https://twitter.com/hveldstra
Email: [email protected]