Comment by c0deR3D

Your scenario makes me suspecting that it is Xilinx flavored Yocto causing the problem. I think that removing some unused Xilinx-specific layers/recipes can reduce the prologue and epilogue execution time.