E&M is simple in the sense that the equations that govern the relationship between the charge distribution and the electromagnetic field are quite simple. And the equations governing how charges move in an electromagnetic field are quite simple. I think this is what Dyson has in mind. Understanding the implications and limitations of those equations is not simple at all.
An even more blatant difficulty with E&M, related to your third point, is that it is schizophrenic. Given the trajectories of charges it will tell you what the electromagnetic field will be, and given the electromagnetic field it will tell you how charges will move. Unfortunately, these two parts of the theory seem to be incompatible, and the theory will not tell you how fields + charges will evolve in time.
An even more blatant difficulty with E&M, related to your third point, is that it is schizophrenic. Given the trajectories of charges it will tell you what the electromagnetic field will be, and given the electromagnetic field it will tell you how charges will move. Unfortunately, these two parts of the theory seem to be incompatible, and the theory will not tell you how fields + charges will evolve in time.