Comment by TheCog

I think one of the general takeaways about ML is that barring a few experts, its really challenging to reason about your system. Like, yes, you might expect that a convolutional layer will behave in a specific way under ideal conditions, but the way that behavior manifests is often wildly hard to predict during the early days.

I agree that step 1 for most beginner projects should be to start with something that works and then tweak.