Great post, William. I really appreciated your exposition on both the challenges you folks were facing and your solution to the problem.
As a few others have pointed out, sending a lambda function through NumPy is almost always the last thing you want to do. Unfortunately, you were in a situation where you were either going to have to do something really painful like using einsum: https://stackoverflow.com/questions/29989059/matrix-multipli... or writing your own ufunc:https://docs.scipy.org/doc/numpy-dev/user/c-info.ufunc-tutor...
I suspect that your primary limitation was the Google Compute Engine infrastructure. I'm not familiar with the limitations there, but a quick search on Google turns up a fairly limited set of libraries indeed.
I thought it would be interesting to adapt your code slightly to use Numba acceleration. Here's what it looks like:
from numba import jit
def avg_transform(image):
m, n, c = image.shape
for i in range(m):
xi = image[i]
for j in range(n):
avg = xi[j].sum()/3
xi[j][:] = avg
return image
fast_avg_transform = jit(avg_transform, nopython=True)
I observed 25ms per image
https://gist.github.com/ahmadia/c1f8be119f3cb2d2b8e5 processing times on my laptop on 1280x720 pixels.
Re-reading your post, I suspect that einsum might actually be your cup of tea, but I really enjoy the simplicity and performance of using Numba for these sort of tasks.
As a few others have pointed out, sending a lambda function through NumPy is almost always the last thing you want to do. Unfortunately, you were in a situation where you were either going to have to do something really painful like using einsum: https://stackoverflow.com/questions/29989059/matrix-multipli... or writing your own ufunc:https://docs.scipy.org/doc/numpy-dev/user/c-info.ufunc-tutor...
I suspect that your primary limitation was the Google Compute Engine infrastructure. I'm not familiar with the limitations there, but a quick search on Google turns up a fairly limited set of libraries indeed.
I thought it would be interesting to adapt your code slightly to use Numba acceleration. Here's what it looks like:
I observed 25ms per image https://gist.github.com/ahmadia/c1f8be119f3cb2d2b8e5 processing times on my laptop on 1280x720 pixels.Re-reading your post, I suspect that einsum might actually be your cup of tea, but I really enjoy the simplicity and performance of using Numba for these sort of tasks.