quadruplebond

What do you mean bare metal? Because all of the big DOE computers are running Linux. Users will probably use hardware specific libraries (like CUDA/ROCm) and occasionally write some hardware specific asm, but none of the big computers are running without a POSIX OS.

They got back to me after about 5.5 weeks (it was a no, but nicely worded). I think they are just really busy.

If dod research labs are like NNSA research labs part of why 0.4 FTEs can be considered significant investment is because overheads are quite high. My experience is that for every dollar a FTE makes their project is paying an additional 2-3 dollars in overhead. That means your $130K/yr FTE is using 50% of your $1M/yr budget.

I'm not sure what the typical cost for an FTE are in industry, but I believe they are lower than 3-4x base salary.

Not all positions require a clearance, but most citizens eventually get one. Life as a foreign national can be difficult if you are at SNL. And if you are from a sensitive country it is even more difficult.

send me an email at <my-user-name>@gmail.com if you want to chat more,

I am in a software role, focused on future architectures and HPC, WLB is good in our department.

Job security is really good IFF you are an FTE, if you are a postdoc or an LTE then meh.

Changing departments requires some work, but at least on the open science side it's easy to switch projects. You keep the same manager, but just work on different stuff. If you are in a super specific mission area it might be harder to change work.

If you are interested in HPC at Sandia just email me and lets chat more, HPC is where I currently sit and know the most people.

Happy to answer specific questions if you have them.

It is going to depend on your org and some other stuff. How much do you want to work on the high side etc.

Nvidia was probably under contract for this before mining blew up to where it is today. Provisioning for these machines starts 5-10 years before they ever come online.

Partly because we don’t want to be locked into using Intel for both of DOE’s open science machines. In the lead up to the exascale or near exascale machines (2012-now) I don’t remember hearing much about AMD, so presumably Intel and IBM were the only vendors supplying general purpose CPUs.

Yes, but rumor has it that Intel will still be the supplier for Argonne’s next computer. Which if everything goes to plan will be the first exascale machine.

Sort of, but Blue Gene tech was always an Argonne thing, Oak Ridge has been the one doing GPUs for awhile now. Unless something weird happens Argonne and Oak Ridge will always be running different types of computer since we don’t want to be locked into one specific vendor or hardware configuration.

Most machines will allow for back filling if your jobs are short enough and small enough. But that will still leave some of the machine under utilized.

Unfortunately you would be disappointed. top is only going to display what's running on your current node, not the whole machine. There is probably some sort of global top, but just logging in and running top in your shell will only run top on the head node. That head node might still be beefy, but only as beefy as a large shared memory machine can be.

Think of it as a much higher bandwidth PCIE. You still have to think about data locality, the same way you do with cpu cache levels. It just makes the problem of not having data on the gpu less painful than before.

Depends on what the computer is designed for. Imagine you want to parallelize up to about 16 nodes or 1000-2000 cores of a new xeon. And your code is not optimized for accelerators. Well you might build a large machine with hundreds of nodes, but with the intent to run many 16 node jobs at a time. The largest machines though mostly are aiming for jobs that use a significant fraction of the total machine and are chasing peak performance so accelerators it is. Also I didn't look too closely at the link, but some machines that aren't technically accelerators like K computer or the old IBM Bluegene computers are closer to general purpose accelerators than they are to fat xeon nodes.