Very interesting, I recently put together a simple caffe static compiler of my own on the path to FPGA deployment of CNN inference pipelines. I looked at halide[0] as a possible intermediate representation for CPU, GPU, and FPGA, (as a step above LLVM IR), but Likely seems like an interesting option.
[0]: http://halide-lang.org/