apl

Neural architecture search (NAS) is a thing! But it's almost exclusively based on meta-gradients. Again, wouldn't put my money on GAs ever outperforming gradient-based methods again.

Several hints here are severely outdated.

For instance, never train a model in end-to-end FP16. Use mixed precision, either via native TF/PyTorch or as a freebie when using TF32 on A100s. This’ll ensure that only suitable ops are run with lower precision; no need to fiddle with anything. Also, PyTorch DDP in multi-node regimes hasn’t been slower or less efficient than Horovod in ages.

Finally, buying a local cluster of TITAN Xs is an outright weird recommendation for massive models. VRAM limitations alone make this a losing proposition.

Hard disagree. V100s are a perfectly valid comparison point. They're usually what's available at scale (on AWS, in private clusters, etc.) because nobody's rolled out enough A100s at this point. If you look at any paper from OpenAI et al. (basically: not Google), you'll see performance numbers for large V100 clusters.

> gradient descent no longer has to be written by hand

Nobody's been writing derivatives by hand for 5+ years. All major frameworks (PyTorch, Tensorflow, MXNet, autodiff, Chainer, Theano, etc.) have decent to great automatic differentiation.

The differences and improvements are more subtle (easy parallelization/vectorization, higher-order gradients, good XLA support).

Mainly because it is genuinely exhausting for any medical practitioner. That lots of patients "enjoy" googling symptoms and coming up with far-fetched self-diagnoses is a given. But couple that with the perceived intellectual superiority of (software) engineers and you get a recipe for disaster. It's the equivalent of a doctor leaning over your shoulder while you're coding and telling you to remove random keywords.

For this particular problem, Mask R-CNN would have been the way to go -- it spits out instances as opposed to just deciding, for each pixel, to which class it belongs. Or an SSD (if we don't care about the mask at all).

Yeah, that's right. Doing too little proofreading with HN comments...

Their applications are tabular data (for which MLPs have never been the method of choice) and MNIST (which I could classify at 85% with a rusty nail), so it's not super impressive.

NNs and the associated toolkit shine with structured high-dimensional data where CNNs, RNNs, or modern shenanigans like Transformer networks excel. I sincerely doubt that these networks turn out to be reducible to polynomial regression in any practically useful sense of the notion. But who knows.

This represents the absolute worst kind of science journalism, completely devoid of context and domain knowledge. Virtually every definitive statement in here is wrong. Their explanation of spiking alone is a complete disaster.

Out of all the modalities, vision is easily the one we know the most about. And we do so at a fairly deep level. The discussed work seems fine but it's not the groundbreaking insight it's made out to be. Great PR work from the involved scientists (or their enterprising university marketing department).

Running Windows is perfectly fine; the major libraries for GPU-accelerated autodiff and networks (CUDNN with Pytorch or Tensorflow) have great support nowadays. It's the AMD GPU that remains essentially useless, as of 2019. If you want to get into the game, I'd recommend buying a middle-of-the-road NVIDIA GPU like the RTX2060.

For toying with autodiff and basic CNNs, CPU works just fine by the way...

That's not quite how the process works. These papers go through multiple (> 2) revisions. At any iteration, there's ample opportunity for updating references. This applies double given how long the Cueva/Wei work has been available (preprint & CCN'17 (?) contribution).