wrhn

Recent submissions

Comment · 4 years ago
You're assuming that Firefox is eating all your commit space, but it's not, far from it, I can see it from the crash reports. You've got other applications running and they are also fighting for it, leaving large chunks unused. Without some swap space available Windows is leaving 10+ GiB of physical memory free which it cannot use for anything else.

This is how Windows behaves by design, there's nothing we can do about it. Windows without a swap file sucks.

Comment · 4 years ago
From the looks of it you have a minuscule page file (16MiB), that's why you're running out of commit space. I suggest setting it to auto-resize (the Windows default) which should provide a much better experience even with other applications, unless you need to keep it small because of storage concerns. Generally speaking Windows behaves very poorly with small swap files because of its inability to overcommit memory.
Comment · 4 years ago
Could you open a bug on our tracker, and point to those crash reports? I'm very interested in analyzing real-world scenarios where this happens to figure out where those committed areas are coming from. Our best guess is that they come from the graphics stack because we account all the memory we commit in Firefox and there's nothing missing, so wherever this is coming from it's outside of our direct control.
Comment · 4 years ago
Yes it does, we've disabled tab unloading on Linux because low-memory detection just didn't work. We will re-enabled it sometimes down the line but only to avoid swap, not crashes. The OOM killer does already a good job with this change and we feel no need to do anything further (besides optimizing memory usage, but that's system-agnostic).
Comment · 4 years ago
But is it? I wanted it to be tongue-in-cheek but the title is actually a rather accurate description of the contents of the article. This is a ~20 lines weird hack that we threw at the wall to see if it would stick... and it massively improved Firefox stability. I can't stress it hard enough, it's the largest, most visible stability improvement in the past decade.

More specifically - on the memory front - we spent lots of engineer/years working on improvements which sometimes would barely register, only for some weird trick make OOM crashes fall off the radar entirely.

Also I'd like to point out that we have no way to tell if this is working because we give Windows time to resize the swap file, if it's because other processes in Firefox die, or it's because other unrelated processes die, or a mix of all the above. It's pure speculation on our part.

Comment · 4 years ago
Here's the thing: they're not! The reason those users where crashing was because something, somewhere (possibly in the graphics stack) was reserving ton of space without using it. We had crashes on file with 20+ GiB of free physical memory which was the reason why I started looking into why a machine with so much free memory could suffer a crash.

I hope I described how this whole things work because Windows memory management is not well known and some things about it are counter-intuitive; especially if you're coming from Linux.

Comment · 4 years ago
I'm glad you noticed! We did indeed roll an improvement on Linux too, see: https://bugzilla.mozilla.org/show_bug.cgi?id=1771712

In a nutshell we're directing the OOM killer towards less interesting processes within Firefox so that when the system is low on memory they'll get killed first. This not only makes it more stable overall but it plays nicer with other applications too. Web pages that leak memory in the background are particularly likely to be killed by this mechanism and that alone is a huge improvement in overall stability.

Comment · 4 years ago
They do and so does jemalloc in Firefox, however that lowers contention, it doesn't let you do away with synchronization. Consider this simple scenario: a thread allocates a chunk of memory from its per-thread pool, but the object is later released by another thread. You can't do that w/o synchronization around the pool. Per-CPU pools are even trickier because a thread might be moved around between different CPUs by the scheduler while it's allocating memory. So you need special facilities to implement those in user-space like restartable sequences: https://lwn.net/Articles/650333/
Comment · 4 years ago
You're confusing general purpose locks with special-purpose ones used in the memory allocator. Firefox has its own GP locks, you can find them here: https://searchfox.org/mozilla-central/search?q=&path=xpcom%2...*

However those are not suitable for use in the memory allocator, and neither are WebKit's. WebKit uses os_unfair_lock within its memory allocator:

https://github.com/WebKit/WebKit/blob/520379e30f3b2b6d4de995...

And so does Chromium:

https://source.chromium.org/chromium/chromium/src/+/main:bas...

Comment · 4 years ago
No specific hardware support is needed to implement a lock. The atomic operations are all accessible from user-space - they are platform dependent though. The problem with locks is that they interact with scheduling and that's where the kernel comes in, that's why you need an OS-specific component too.
Comment · 4 years ago
In a word: no. Efficient locks can only be implemented with significant involvement from the kernel. The best locks that can be implemented entirely in user-space will still suffer from the problems described in the article. Locks have complex interactions with scheduling and the only place where informed decisions can be made about them is within the kernel.

That doesn't mean that good locks don't need a good user-space component - they do - but it's only one side of the coin.

Comment · 4 years ago
If you could grab a profile of the problem with the Firefox profiler and file a bug it would be greatly appreciated. From the sounds of it it's probably an edge-case we're not aware of where Firefox performs very poorly. The problem with this kind of things in Mozilla is that we're often blind until someone brings up the issue and notifies us.
Comment · 4 years ago
Post author here, `__ulock_wait2` is used under the hood by `os_unfair_lock()`. I considered it but it would have required more scaffolding, especially to support all the versions of macOS we care about. We might use it in the future though.
Comment · 5 years ago
Recent Linux kernels allow you to measure memory consumption from a QoS point-of-view via pressure stall information: https://www.kernel.org/doc/html/latest/accounting/psi.html

This isn't the same thing as detecting how much memory is available because Linux overcommits by default. PSI will tell you when the kernel is swapping too much (depending on the thresholds you set) and from that you can infer the user is running low on memory.

Android's lowmemkiller had an explicit free memory threshold userspace could set to be notified. Starting with Android 11 however it's now using PSI too IIRC.

Comment · 5 years ago
Unfortunately those crashes do not appear to be related to low memory scenarios. They seem to be genuine bugs in the screen reader. I think the right bug tracking them is this one: https://bugzilla.mozilla.org/show_bug.cgi?id=1691928

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton