I question this meme that deep learning has less mathematical elegance or interpretability than other machine learning model. It's simply not true. At this point, it reads like a mindless copypasta with the possible political intention to reduce AI funding and use, such as the EU regulation which requires all algorithms to "explain" their reasoning. Either that, or the author is just more comfortable with familiar old-school methods which makes them think that those methods are more elegant. This push towards elegance is unscientific and counterproductive, much like the mistaken push Einstein and co. made for deterministic interpretations of quantum mechanics. We can now understand that its motivation was simply due to a biased notion of elegance due to familiarity with classical mechanics.
Examples of elegance in deep learning: The gating mechanisms of LSTMs (also adds to interpretability); structure of convolutions, i.e. weight sharing, translation invariance; the policy gradient theorem (Sutton 1999); dropout, its relationship to biology and variational inference; Generative Adversarial Networks, and connections to game theory, split-brain; the reparametrization trick; the log-derivative trick; connectionist temporal classification. These examples are only the surface. Depending on your specialization inside deep learning, you'd find many more.
Examples of interpretability: Nguyen et al.'s Synthesizing preferred inputs to hidden neurons, Zeiler et al. convnet visualization, Guillaume Alain's linear probes of hidden layers, attention readouts in attentional models for machine translation or speech synthesis, and many more. Ultimately, deep learning methods are probabilistic, and decent deep learning engineers would be able to tell you why a model is doing what it's doing by printing probabilities and activation statistics, much like any other probabilistic machine learning model.
Examples of elegance in deep learning: The gating mechanisms of LSTMs (also adds to interpretability); structure of convolutions, i.e. weight sharing, translation invariance; the policy gradient theorem (Sutton 1999); dropout, its relationship to biology and variational inference; Generative Adversarial Networks, and connections to game theory, split-brain; the reparametrization trick; the log-derivative trick; connectionist temporal classification. These examples are only the surface. Depending on your specialization inside deep learning, you'd find many more.
Examples of interpretability: Nguyen et al.'s Synthesizing preferred inputs to hidden neurons, Zeiler et al. convnet visualization, Guillaume Alain's linear probes of hidden layers, attention readouts in attentional models for machine translation or speech synthesis, and many more. Ultimately, deep learning methods are probabilistic, and decent deep learning engineers would be able to tell you why a model is doing what it's doing by printing probabilities and activation statistics, much like any other probabilistic machine learning model.