dikaiosune
Joined 12 years ago
Karma
4,199
Submissions
743
Member since
Dec 10
http://blog.adamperry.me
View on Hacker NewsRecent submissions
Comment · 8 years ago
Well this interpretation certainly explains some of the reactions that word has gotten in my use of it!
Comment · 8 years ago
I talked about this a little bit in the meetup talk I linked and I intend to write a bit more about this, but I'll try to summarize.
There are kind of three prongs here:
First, using criterion.rs does a ton for giving us more stable metrics. It handles things like warmups, accounting for obvious statistical outliers in the sample runs, postprocessing the raw data to provide more meaningful statistics, etc. I'm currently using a fork of the library which additionally does this recording and processing of a variety of metrics we get from `perf_event_open` on Linux but which I assume you could get through ETW or Intel/AMD's userspace PMC libraries.
Second, I try to provide a stable environment so that results over long time deltas are comparable and we can store the data for offline analysis rather than having to checkout recent prior commits and compare the current PR/nightly/etc against them. Prior to the current deployment I was using cgroups to move potentially competing processes off of the benchmark cores which produced some nice results. However I had some issues with the version of the cpuset utility I installed on the debian machines and I haven't sorted it out yet.
Third, we do a few things with the time-series-esque data we get from measuring multiple toolchains to try and only surface relevant results. Those are mostly in src/analysis.rs if you want to poke around. It basically boils down to calculating the Kernel Density Estimate of the current toolchain's value being from the same population (I hope these terms are halfway correct) as all prior toolchains' value.
I hope that with a few extensions to the above we can get close to being reliable enough to include in early PR feedback, but I think the likely best case scenario is a manually invoked bot on PRs followed by me and a few other people triaging the regressions surfaced by the tool after something merges.
Here are a few issues that I think will help improve this too:
https://github.com/anp/lolbench/issues/20
Comment · 8 years ago
I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.
Comment · 8 years ago
great idea!
Comment · 8 years ago
I currently do something pretty similar by using the perf subsystem in the Linux kernel to track the behavior of each benchmark function. In my early measurements I found concurrent benchmarking to introduce unacceptable noise even with this measurement tool and with cgroups/cpusets used to pin the different processes to their own cores. Instead of trying to tune the system to account for this, I chose to build tooling for managing a single runner per small cheap machine.
Comment · 8 years ago
Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them when I ran some phoronix benches to compare.
I go into a little bit of detail on this in the talk I link to towards the bottom of the post, here's a direct link for convenience: https://www.youtube.com/watch?v=gSFTbJKScU0.
Comment · 8 years ago
Thanks!
Not yet, but I am tracking this as a desired feature: https://github.com/anp/lolbench/issues/9. The benchmark plan generation, storage keys, and results presentation will at a minimum need to be extended to support a matrix of inputs to each benchmark function. Right now there are a number of implicit assumptions that each benchmark function is tracked as a single series of results.
Comment · 8 years ago
hi! author here if you want to ask questions or (nicely pls) let me know where I've made mistakes!
Comment · 8 years ago
Do you have numbers for how much of the web avoids variable hoisting? How much of the web has more than one top-level fn per file? Do these numbers shift significantly between different toolchains or methods of delivering JS?
Comment · 8 years ago
How is partial execution baked into JS? IIUC, my whole JS file is fully parsed an resolved before it is evaluated. If you're referring to the use of multiple files for parallelism here, then the obstacle there is not wasm itself but the work that's still in progress to define a dynamic linking story for it.
I'd point out that the need for better dynamic linking to enable lower TTI is also a problem faced in the bundling/tooling heavy JS community.
Comment · 8 years ago
It would be interesting to try for some apples-to-apples comparisons for the parse and initial exec time of js vs wasm. The latter was designed with the goal of streaming validation and compilation, js needs to have a lot of work done before it can load.
Which strongly suggests to me that intuitions about KBs-of-JS vs KBs-of-Wasm are likely to be wrong, especially as the state of wasm implementations improves.
Comment · 8 years ago
Many types in the standard library do not consider it sound behavior to have their allocations freed by functions other than their own destructors:
https://doc.rust-lang.org/std/boxed/struct.Box.html#method.i...
After calling this function, the caller is responsible for the memory
previously managed by the Box. In particular, the caller should properly
destroy T and release the memory. The proper way to do so is to convert
the NonNull<T> pointer into a raw pointer and back into a Box with the
Box::from_raw function.
Comment · 8 years ago
Re: non-thread-safe items in lazy_static, you can also wrap them in a mutex within just the lazy_static so the struct only uses synchronization when it actually needs to be thread-safe.