vinn124

for a good introduction to order and complexity (including intelligence) arising from nothing, read "the origins of order" by stuart kauffman.

who cares?

> At the core RL is just updating a table of values, and then using function approximation (aka, machine learning) for more complex cases.

this seems to be a common assertion about ml. other refrains include "ml is just matrix multiplication" and "ml is just affine transformations followed by nonlinearity".

while technically true, it is an unhelpful comment as it doesnt shed any light on the salient questions, such as:

* how do you reduce bias/variance, since youre sampling a minuscule slice of the state/action space? * how do you construct a value function (or something like it) in a sparse reward environment, with possibly thousands of time steps? * how do you know your policy network is exploring new states?

it is the equivalent of learning how to draw an owl: draw a few circles, then draw the fucking owl.

particularly when most microsoft investors value microsoft stock for its earnings potential, as opposed to growth/revenue potential.

> Increased accuracy per unit wallclock time is what you want.

yah, especially for a framework for distributed learning!

> 3) Test that the solution works on toy examples, like MNIST, simple block worlds, simulated data, etc.

youre right: mnist, imagenet, etc are toy examples that do not extend into the real world. but the point of reproducible research is to experiment on agreed upon, existing benchmarks.

> What are the use cases for adding yet another layer to the stack?

in my limited experience with horovod, horovod is most useful when youre running large clusters of workers/ps. in those situations, you typically have to manually find the appropriate balance of workers/ps. (otherwise youd run into blocking or network saturation issues.) horovod addresses this issue with their ring allreduce implementation.

having said all of that, im sticking with distributed tf for now.

> Usually you throw everything and see what sticks.

most practitioners start with the simplest possible learner, then gradually, and thoughtfully, increase model complexity while paying attention to bias/variance. this is far from a "kitchen sink" approach.

> Many losses which don't seem differentiable can be reformulated as such...

agreed, especially with policy gradients.

> If the dimensionality is small, second-order methods (or approximations thereof) can do dramatically better yet.

i have not seen second order derivatives in practice, presumably due to memory limitations. can you point me to examples?

the quality and preciseness of this answer is why i read hn every day.

> Why should we expect there to be any mathematical foundation to this stuff?

i would be surprised that "this stuff" would be exception to the unreasonable effectiveness of mathematics. mathematics underpins virtually every observed phenomenon, including theoretical physics, computer science, economics. in fact, the mathematical structure of any physical theory often points the way to further advances in that theory and even to empirical predictions.

to not expect that "this stuff" should not have any mathematical foundation is a fantastically naive view.

youve missed the point.

the point is: anything complex can be dismissed as "just x,y,z" if you dont appreciate the massive body of work behind it.

i made that point because OP observed that "ml is just affine transformations" or something to that effect. yes, that's one way to frame it - if youre okay overlooking roughly 30 years of research.

> Then my next question will be how can I trust something that I don't understand.

ive always found this mentality silly. do you understand aerodynamic principles? laws of electromagnetism? mendelian genetics? or information theory?