Comment by beckhamc

Learnable parameters on activations do exist, look up parametric activation functions.