brosco

One reason is that it would be like hanging a picture using a sledgehammer. If you're just studying various ways of unwrapping a sphere, the (very deep) theory of manifolds is not necessary. I'm not a cartographer but I would assume they care mostly about how space is distorted in the projection, and have developed appropriate ways of dealing with that already.

Another is that when working with manifolds, you usually don't get a set of global coordinates. Manifolds are defined by various local coordinate charts. A smooth manifold just means that you can change coordinates in a smooth (differentiable) way, but that doesn't mean two people on opposite sides of the manifold will agree on their coordinate system. On a sphere or circle, you can get an "almost global" coordinate system by removing the line or point where the coordinates would be ambiguous.

I'm not very well versed in the history, but the study of cartography certainly predates the modern idea of an abstract manifold. In fact, the modern view was born in an effort to unify a lot of classical ideas from the study of calculus on spheres etc.

Good point! I used to be guilty of this myself, so now I'm pretty sensitive about other people doing it. I am now one of the more senior students in an academic research group, and some of the younger members would benefit from this advice. I think it's a symptom of sophomorism, and hopefully most will grow out of it.

I agree it's especially frustrating when they don't even get it right. That crosses the line for me, and I will admonish them to let me finish what I am saying.

I'm not saying it's a learning method. And I don't see how anyone could mistake this for science, so why would it be pseudoscience? It's not really about effort either.

It's just a trick that helps me pay attention in lectures, which a lot of people struggle with. Certainly you have to put the work outside of the classroom as well.

I have a tip for following lectures (or any technical talk, really) that I've been meaning to write about for a while.

As you follow along with the speaker, try to predict what they will say next. These can be either local or global predictions. Guess what they will write next, or what will be on the next slide. With some practice (and exposure to the subject area) you can usually get it right. Also try to keep track of how things fit into the big picture. For example in a math class, there may be a big theorem that they're working towards using lots of smaller lemmas. How will it all come together?

When you get it right, it will feel like you are figuring out the material on your own, rather than having it explained to you. This is the most important part.

If you can manage to stay one step ahead of the lecturer, it will keep you way more engaged than trying to write everything down. Writing puts you one step behind what the speaker is saying. Because of this, I usually don't take any notes at all. It obviously works better when lecture notes are made available, but you can always look at the textbook.

People often assume that I have read the material or otherwise prepared for lectures, seminars, etc., because of how closely I follow what the speaker is saying. But really most talks are quite logical, and if you stay engaged it's easy to follow along. The key is to not zone out or break your concentration, and I find this method helps me immensely.

There is indeed a lot of crossover, and a lot of neural networks can be written in a state space form. The optimal control problem should be equivalent to training the weights, as you mention.

However, from what I have seen, this isn't really a useful way of reframing the problem. The optimal control problem is at least as hard, if not harder, than the original problem of training the neural network, and the latter has mature and performant software for doing it efficiently. That's not to say there isn't good software for optimal control, but it's a more general problem and therefore off-the-shelf solvers can't leverage the network structure very well.

Some researchers have made interesting theoretical connections like in neural ODEs, but even there the practicality is limited.

Very cool. Analysis I was the first "real" math textbook that I (an engineer, not a mathematician) felt like I could completely follow and work through, after a few attempts to get through others like Rudin. Hopefully the Lean companion will help make it even more accessible to people who are familiar with math and programming and looking to learn things rigorously.

Compared to Matlab (and to some extent Julia), my complaints about numpy are summed up in these two paragraphs:

> Some functions have axes arguments. Some have different versions with different names. Some have Conventions. Some have Conventions and axes arguments. And some don’t provide any vectorized version at all.

> But the biggest flaw of NumPy is this: Say you create a function that solves some problem with arrays of some given shape. Now, how do you apply it to particular dimensions of some larger arrays? The answer is: You re-write your function from scratch in a much more complex way. The basic principle of programming is abstraction—solving simple problems and then using the solutions as building blocks for more complex problems. NumPy doesn’t let you do that.

Usually when I write Matlab code, the vectorized version just works, and if there are any changes needed, they're pretty minor and intuitive. With numpy I feel like I have to look up the documentation for every single function, transposing and reshaping the array into whatever shape that particular function expects. It's not very consistent.

This looks great! I've been needing something like this for a while, for a project which is quite compute-heavy and uses lots of threads and recursion. I've been using valgrind to profile small test examples, but that's definitely the nuclear option since it slows down the execution so much. I'm going to try this out right away.

That's a good observation, and it is indeed true for many Markov chains. But your counterexample of the identity matrix is not quite right; every vector is an eigenvector of the identity, so there is no "realignment" needed.

More generally speaking, you're asking when the iteration `x_+ = Ax` converges to a fixed point which is an eigenvector of A. This can happen a few different ways. The obvious way is that A has an eigenvector `v` with eigenvalue 1, and all other eigenvalues with magnitude < 1. Then those other components will die out with repeated application of A, leaving only `v` in the limit.

For Markov chains, we can get this exact property from the Perron-Frobenius theorem, which applies to non-negative irreducible matrices. Irreducible means that the transition graph of the Markov chain is strongly connected. If that's the case, then there is a unique eigenvector called the stationary distribution (with eigenvalue 1), and all initial conditions will converge to it.

In case A is not irreducible, you may have different connected components, and the stationary distribution may depend on which component your initial condition is in. Going back to the n x n identity matrix, it has n connected components (it's a completely disconnected graph with all the self-transition probabilities = 1). So every initial condition is stationary, because you can't change anything after the initial step.

It's absolutely being explored. There is a lot of active research into using ML to learn solutions of PDEs (Navier-Stokes in this case). It's not my field so I don't know much about the specifics.

The works that I've read train an NN on numerical solutions for different geometries and boundary conditions. Then they try to infer the solutions for configurations outside the training set, which should be much faster than recomputing the numerical solution.

If you're interested to learn more about aerodynamics I would highly suggest learning a bit of classical aerodynamics. It will not be software oriented, since most of the theory deals with approximating very complicated behavior with simple analytical models.

It could be interesting to do a comparison with finite volume methods to see when/how those approximations break down.

Thanks for clarifying the motivation, that makes a lot of sense.

I guess this will probably come up in the reviews but the presentation of the Kalman filter is lacking. I know it's not the point of the paper, but getting these details wrong in a paper about Kalman filters is not encouraging.

The statement that the Kalman filter is mean-square optimal because it generates a correct estimate in expectation is false. In fact, any gain L will generate an estimate whose expected value is x_k, as long as w_k and v_k are zero mean. The Kalman gain is a specific choice of L that is optimal in the mean square sense only when the disturbances are Gaussian. The Kalman gain is also time-varying and depends on the evolution of the estimate covariance, although it will converge to a steady-state value.

What's being described here is more properly called a Luenberger observer, but I guess that name doesn't get the same recognition outside the control community.

I'm also wondering why they chose to include H past estimates and measurements in the transformer. They're already embedding the Kalman gain into the weights of the transformer, so taking just one past estimate/measurement should exactly recover the Kalman filter. Going further into the past just makes the estimate worse, because of the softmax.