jimfleming

I'm aware of that and I've done quite a bit of work on both spiking neural networks and modern deep learning. My point is that those complexities are not required to implement many important functional aspects of the brain: most basically "learning" and more specifically, attention, memory, etc. Consciousness may fall into the list of things we can get functional without all of the incidental complexities that evolution brought along the way. It may also critically depend on complexities like multi-channel chemical receptors but since we don't know we can't say either way.

It's a tired analogy but we can understand quite a lot about flight and even build a plane without first birthing a bird.

> That needs to work before moving to more complexity.

It really depends on what level of abstraction you care to simulate. OpenWorm is working at the physics and cellular level, far below the concept level as in most deep learning research looking to apply neuroscience discoveries, for example. It’s likely easier to get the concepts of a functional nematode model working or a functional model of memory, attention, or consciousness than a full cellular model of these.

More specifically, a thousand cells sounds small in comparison to a thousand layer ResNet with millions of functional units but the mechanics of those cells are significantly more complex than a ReLU unit. Yet the simple ReLU units are functionally very useful and can do much more complex things that we still can’t simulate with spiking neurons.

The concepts of receptive fields, cortical columns, local inhibition, winner-take-all, functional modules and how they communicate / are organized may all be relevant and applicable learnings from mapping an organism even if we can’t fully simulate every detail.

I've noticed that many JAX libraries (including those from Google) seem to adopt an object-oriented style more similar to Torch/Keras rather than JAX's functional style demonstrated in modules like jax.experimental.stax. This is disappointing since stax is quite clean and these libraries seem to use a lot of hacks to make OO work with JAX. Is there an effort to implement and maintain more full-featured functional libraries in the jax/stax style?

I think your conclusions are accurate. For many problems LightGBM or xgboost can often yield decent results in short amounts of time and for many problems that’s sufficient. A lot of the work we do is about pushing the results as far as we can take them and the business case justifies the extra time it can take to get there. For those types of problems, today, we would probably choose a neural network because then we have a lot more knobs as you mentioned.

Just like the rest of ML, whether neural networks are the right choice still depends on the problem at hand and the team implementing the solution. It definitely impacts where the performance / time curves intersect. If we just need something decent fast, or we’re working with another team that doesn’t have the same background, we tend to focus on approaches with fewer moving pieces. If we need the best possible performance, have a qualified team to get there, and have the time to iterate on development then the curves would favor neural networks.

It’s really very problem dependent. I allude to a few low-hanging things in my post above: e.g. feature engineering. Just because neural networks have an easier time learning non-linear feature transformations doesn’t mean its good to ignore feature engineering entirely.

Possibly more important is to focus on the process for how you derive and apply model changes. You get some model performance and then what? Rather than throwing something else at the model in a “guess-and-check” fashion, be methodical about what you try next. Have analysis and a hypothesis going in to each change that you make and why it’s worth spending time on and why it will help. Back that hypothesis by research, when possible, to save yourself some time verifying something that someone else has already done the legwork on. Then verify the hypothesis with empirical results and further analysis to understand the impact of the change. This sounds obvious (it’s just the scientific method) but in my experience ML practitioners and data scientists tend to forget or were never taught the “science” part. (I’m not accusing you of this; it just tends to be my experience.)

Random search, AutoML, hyperparameter searches, etc. are incredibly inefficient at model development so they’ll rarely land you in a better place unless a lot of upfront work has been put in. For us, they’re useful for two things: analysis and finalization. For analysis the search should be heavily constrained since you’re trying to understand something specific. For finalization of a model before going into production, a search on only the most sensitive parameters identified during development usually yields additional gains.

It takes time and a lot of hands-on experience. Many ML teams tend to work on one or just a few tightly coupled project for years. By contrast, we’ve worked on a lot of unique projects with real-world constraints so it gives us a different perspective. An important part has been developing a rigorous process; sort of a framework for applying the “art”. As you mention, this often isn’t covered in ML or data science education, which tends to focus on (important) fundamentals.

More anecdata: we consistently outperform lightgbm, xgboost, random forests, linear models, etc. using neural networks even on smaller datasets. This applies whether we implemented the other algorithms ourselves or simply compared to someone else’s results with them. In my experience it really comes down to how many “tricks” you know for each algorithm and how well can you apply and combine these “tricks”. The difference is that neural networks have many more of these tricks and a broader coverage of research detailing the interactions between them.

I call them “tricks” but really they’re just design decisions based on what current research indicates about certain problems. This is largely where the “art” part of neural networks comes from that many people refer. The search space is simply too big to try everything and hope for the best. Therefore, how a problem is approached and how solutions are narrowed and applied really matter. Even simple things like which optimizer you use, how you leverage learning rate schedules, how the loss function is formulated, how weights are updated, feature engineering (often neglected in neural networks), and architectural priors make a big difference on both sample efficiency and overall performance. Most people, if they’re not just fine-tuning an existing model, simply load up a neural network framework, stack some layers together and throw data at it expecting better results than other approaches. But there’s a huge spectrum from that naive approach to architecting a custom model.

This is why neural networks are so powerful and why we tend to favor it (though not for every problem). It’s much easier to design a model from the ground up with neural networks than it is for e.g. xgboost because not only are the components more easily composable thanks to the available frameworks but there’s a ton more research on the specific interactions between those components.

That doesn’t mean than every problem is appropriate for neural networks. I completely agree with you that no matter what the problem is you should never jump to an approach just because its popular. Neural networks are a tool and for many problems you need to be comfortable with every one of those decision points to get the best results and even if you’re comfortable it can take time and that isn’t always appropriate for every problem. My other point is that I wouldn’t draw too many conclusions about a particular algorithm being better or worse than another. I’m not saying that was the intention with your comment but I know many people in the ML industry tend to take a similar position. It really depends on current experience with the applied algorithms, not just experience with ML in general.

> Seems indie frameworks in AI can't survive?

AI frameworks are enormously complex pieces of software—a mixed bag of GPU acceleration, math utilities, low and high-level implementations of state-of-the-art components, and a (hopefully, eventually, but not usually) friendly interface to tie it all together.

While Keras is certainly useful, it was only possible because of the underlying libraries (i.e. Theano, TensorFlow, etc.) However, maintaining useful frameworks on a shifting landscape of underlying libraries often results in mismatches which leads to weird incompatibilities, performance regressions, numerical instabilities, etc. Even the big players deal with these issues internal to their own AI frameworks.

Moreover, due to performance and deployment requirements, and the rapid development of the field, these underlying libraries are often tightly coupled or have poorly defined API interfaces for third-parties outside of "model development".

It's not impossible to have well-defined abstractions but "AI" is too new, too varied and too ill-defined itself to have developed those abstractions sufficiently for _large_ indie frameworks to be successful to a broader community. It would be similar to trying to design a high-level framework for "software development". Again, not impossible—Keras did much better than most—but exceedingly difficult for independent developers. Therefore, we're left with only the biggest players.

It's not all bad news, though. There's certainly room for independent development, but outside of hobby projects or thought experiments, I don't think these contributions are effective at the broadest level of scope. The best way for independent developers to contribute and keep their sanity is on implementations of individual components, utilities, or abstractions for specific use cases which don't try to be everything to everyone.

The article's demonstration of a counting model is horribly inaccurate to the point where I'm not sure why it was included. Most people see "AI" as being either good at something or not. There's little nuance such as some models being better than others. This kind of demonstration just weakens the reader's confidence in more fully developed results or different approaches. I'm not sure this brief statement offsets the prominent visuals:

> On the day of the protest, Mr. Yip and the A.I. team used technology that is much more advanced. They spent weeks training their program to improve its accuracy in analyzing crowd imagery.

Setting aside the presentation, from the photos the researchers appear to be using object detection rather than density estimation. This choice is problematic given the quantities involved and the need for temporal consistency.

I'm also skeptical of using human volunteers and surveys to calibrate the model. Humans are terrible at counting large numbers of people in real-time. That's a central point of the article with different groups of people providing wildly different counts.

To those that are upvoting this and previous Swift + TF announcements: What are you excited about, specifically? Why Swift? Why not Julia? Is it the syntax? Types? Compilation? Performance? Community?

I like Swift and all but our ML/DL/RL/DS tools and libraries are in Python (and occasionally R). Most are missing for Swift without an awkward Python compatibility layer and I don't see a compelling reason to adopt it.

This is sort of true. Birds are significantly more energy efficient than planes for some kinds of flight. Birds are also able to perform maneuvers that planes cannot such as landing on a branch. For transportation we just don't care about these features.

I agree that CV does not need to mimick everything about human vision but we still have a long way to go before having vision as robust and efficient as humans. Moreover there are benefits in interpretability and explainability to mimicking human vision more closely. If CV perceived the world similarly it can fail in more similar and predictable ways.