This canard keeps coming up. I've come up with a few explanations. 1) It's to protect the reputation of people who make their living copy/pasting examples from the SK-learn gallery
2) The authors fully understand the tools they're using, but feel there's an element of unexplainable meaning in them that I wager isn't actually there.
3) I'm ignorant and I think this is explainable when I just don't know enough to know it's not.
I've done machine learning projects, and if asked to explain to a "non ML trained" person, I'll try like this:
You know how if you get a scatter plot, you can draw a line through the middle of the points and make predictions off the y=mx+b function of that line?
Well, that's what we're doing here, except the function isn't a line, it's a shape with as many dimensions as we have features.
Let's pretend the only features we've got are lat and long. Imagine you asked for us to create a classifier to predict the depth at a given lat and long over the grand canyon. Each neuron is a function, I think of each neuron like it's casting a "line" through a scatter plot along its particular axis. Then we combine all the lines together and weave a mesh. The act of training simply adjusts the details of all the simple functions, trying to seek out the sweet spot in that point cloud, so our mesh is close enough to the grand canyon to be useful. Too many neurons, and the mesh wont be flexible enough to make useful predictions from. It'll be overfit!
Now, we might be able to get decent performance out of a model built out of x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models perform better. Lots of the "unexplainable magic" comes from these hunches being tested and providing benefits to accuracy. I'd wager without seeing any data that if we add two new features: normal direction of the rock face and speed of the Colorado river at that point -- we can make the model much better for negligibly additional training time.
The more compute time and data we've got, the more features we can throw into the model to see if they help the usefulness of the mesh.
I've got a metaphor to explain why some features drop out during training that involves gimbal lock here if people are still asking questions. It requires non "trained" folks to have seen Apollo 13. Fortunately no non-technical stakeholder ever cares enough to make it there though. :D Shipping a product before the sun burns out makes this a decent time to introduce PCA/SVD or whatever dimensionality reduction techniques I used this time around.
If the solution ends up being a fever-dream abortion of an LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a heavy night drinking -- I just say I came up with the model in an ad hoc manner, and the cross validated R^2 scores speak for themselves.
2) The authors fully understand the tools they're using, but feel there's an element of unexplainable meaning in them that I wager isn't actually there.
3) I'm ignorant and I think this is explainable when I just don't know enough to know it's not.
I've done machine learning projects, and if asked to explain to a "non ML trained" person, I'll try like this:
You know how if you get a scatter plot, you can draw a line through the middle of the points and make predictions off the y=mx+b function of that line?
Well, that's what we're doing here, except the function isn't a line, it's a shape with as many dimensions as we have features.
Let's pretend the only features we've got are lat and long. Imagine you asked for us to create a classifier to predict the depth at a given lat and long over the grand canyon. Each neuron is a function, I think of each neuron like it's casting a "line" through a scatter plot along its particular axis. Then we combine all the lines together and weave a mesh. The act of training simply adjusts the details of all the simple functions, trying to seek out the sweet spot in that point cloud, so our mesh is close enough to the grand canyon to be useful. Too many neurons, and the mesh wont be flexible enough to make useful predictions from. It'll be overfit!
Now, we might be able to get decent performance out of a model built out of x,y->z -- but some thoughtfulness and domain knowledge TENDS to make models perform better. Lots of the "unexplainable magic" comes from these hunches being tested and providing benefits to accuracy. I'd wager without seeing any data that if we add two new features: normal direction of the rock face and speed of the Colorado river at that point -- we can make the model much better for negligibly additional training time.
The more compute time and data we've got, the more features we can throw into the model to see if they help the usefulness of the mesh.
I've got a metaphor to explain why some features drop out during training that involves gimbal lock here if people are still asking questions. It requires non "trained" folks to have seen Apollo 13. Fortunately no non-technical stakeholder ever cares enough to make it there though. :D Shipping a product before the sun burns out makes this a decent time to introduce PCA/SVD or whatever dimensionality reduction techniques I used this time around.
If the solution ends up being a fever-dream abortion of an LSTM/Perceptron/Markov chain that I came up with while sleep deprived after a heavy night drinking -- I just say I came up with the model in an ad hoc manner, and the cross validated R^2 scores speak for themselves.