moshe
Joined 19 years ago
Karma
229
Submissions
20
Member since
Aug 16
https://www.linkedin.com/in/moshelooks/
View on Hacker NewsRecent submissions
Comment · 9 years ago
I've never used PyTorch or Chainer, so maybe I'm wildly wrong, but here goes..
Dynamic batching requires you to predefine a set of "batch-wise" operations (node types for computation graphs). This works fine for eager execution as well, because the eager code is inside a function definition. Of course if you want to do dynamic batching you need to introduce a layer of abstraction, e.g. instead of saying a+b you create a dynamic batching operation for addition and add edges to an execution graph, this is not different than in our TF implementation.
Comment · 9 years ago
Thanks, insightful questions.
You're absolutely right about combinators and long-range dependencies. We have a special block type in the high-level API (https://github.com/tensorflow/fold/blob/master/tensorflow_fo...) for accumulating results here without having to explicitly shuttle them around; not perfect but very handy in many cases.
Regarding your second question, the equivalent of padding in dynamic batching is "pass through" do-nothing ops, which are introduced transparently but worth being aware of if you want to understand the machinery. The worst case scenario here is chains of wildly varying lengths, where we need to add pass-throughs to the shorter chains to match the length of the longest chain.
Comment · 9 years ago
Please note that the GitHub page says "not an official Google product", rather than "project". An official Google product would be something like gmail.
Comment · 9 years ago
TensorFlow Fold provides a TensorFlow implementation of the dynamic batching algorithm (described in detail in our paper [1]). Dynamic batching is an execution strategy for computation graphs, you could also implement it in PyTorch or Chainer or any other framework.
Our particular implementation of dynamic batching uses the TF while loop, which means that you don't need to make run-time modification to the actual TF computation graph. At runtime, we essentially encode the computation graph for (let's say) a parse tree as a serialized protocol buffer (tf.string), so instead of varying the computation graph itself we vary the input to a static computation graph instead. This particular implementation strategy is very much a byproduct of how TensorFlow works (static computation graph, heavy lifting happens in ops implemented in C++).
[1] DEEP LEARNING WITH DYNAMIC COMPUTATION GRAPHS, https://openreview.net/pdf?id=ryrGawqex
Comment · 18 years ago
I have gotten plenty of great applicants - thanks guys! Please stop sending me your application info at this point...
Comment · 18 years ago
It is, actually.
Comment · 18 years ago
"Currently pursuing a BS, MS or PhD in computer science or a related technical field." is verbatim from the intern job listing (http://www.google.com/support/jobs/bin/answer.py?answer=1262...), and I suppose it is there because the target audience for Google Engineering internships is students who are aiming to work some place like Google Engineering after graduating - not exactly the same category as "technically qualified students". So if you'd consider working at some place Google Engineering post graduation, then you should definitely apply.
For the record, I would consider economics to be a related technical field (cf. John von Neumann, cf. "Evolution of Cooperative Problem-Solving in an Artificial Economy" - http://www.whatisthought.com/hayek32000.pdf)...
Comment · 18 years ago
> Intelligence is general and is capable of coping with
> novel problems. AI therefore invovles building machines
> which have that level of generality.
>
There is a bias when you think of "programs" to think of programs that a human would actually write or consider useful. But in fact, this is only a tiny (and nontrivial to specify) fraction of the space of "possible programs" - see for example _Foundations of Genetic Programming_, by Langdon and Poli, for some next explorations of general program space. Most possible programs are completely useless and uninteresting to humans (same with proofs, grammars, etc. etc.).
To do AI you have to realize that in the sentences: "Humans have general intelligence and can solve problems in many domains" and "Breadth-first search is a problem-general technique that will always find a solution if one exists" the word "general" does not mean the same thing at all! (cf. the huge literature on inductive bias in human cognition).
For a nice list of program induction problems solvable by search (with a system called ADATE), see http://www-ia.hiof.no/~rolando/Examples/index.html .
Comment · 18 years ago
a) Not everyone in Google Research has a PhD, though of course there is a bit of a correlation (also, plenty of "research" takes place in general engineering)
b) You do generally have to be very familiar with one of the Big 4 to get hired (C++ in my case)
c) Plop is kind of a weird edge case insofar as it is very long-term research, and really cries out to be implemented in a language where you have the same features at compile-time and run-time, and can conveniently represent and manipulate code as data (i.e. a Lisp). For doing plop in any other language, your first task would be to implement these features, which is rather a lot of work (believe me, I've tried ;->). For most anything else one of the Big 4 would be used, for the reasons Steve Yegge outlines...
Comment · 18 years ago
Well, the difficulty with this is that program induction is in fact too general to be very interesting - to create a competition you would need to publish some test problems, or describe a special distribution of "natural programs" that you especially cared about...
Comment · 18 years ago
Well, sort of. Genetic programming with probabilistic modeling and sampling instead of crossover/mutation (i.e. it an estimation-of-distribution algorithm), where programs are represented in reduced (normal) form and which variables to use in the probabilistic models is determined adaptively and changes as the search progresses... ;->
Comment · 18 years ago
Thanks for the pointer. I was actually just looking at ACL the other day - I have a friend here at Google who is investigating integrating it with plop as a 20% project. I'm very excited that your work combines theorem-proving with learning, kudos! Do you do anything probabilistic to decide which generalizations to try?