Comment by salty_biscuits

I'm sure there is a way of interpreting a relu as a sparsity prior on the layer.