This is where GStreamer normally steps in. A lot of hardware manufacturers provide a gstreamer plugin for their module. I’ve had experience with NVIDIA and atmel SoCs and that seemed to be the default path.
Good luck with the gstreamer pipeline learning curve however!
Good luck with the gstreamer pipeline learning curve however!