Comment by colonCapitalDee

You should check out HLS and DASH. If you're already familiar and you're not using them because they don't meet your requirements, then apologies for the foolish recommendation. If not, this could solve your problem.