I wonder why gradient descent (gating all compute/memory operations with parameters, gradient updates to parameter vectors along sample loss gradient) ended up dominant over genetic programming (algorithms as parse trees, mutate/recombine to minimize sample loss). Both seem equally theoretically unjustified, although I guess SGD is more friendly for gpus.
Also I wonder if all programming should just be done by showing a sgd-solver example i/o. Of course this notion has been around since the 80s with prolog but apparently representing "concepts" as 1000 dimension feature vectors is more robust than single boolean variables.
Also I wonder if all programming should just be done by showing a sgd-solver example i/o. Of course this notion has been around since the 80s with prolog but apparently representing "concepts" as 1000 dimension feature vectors is more robust than single boolean variables.