This was exactly my experience trying to build a RAG pipeline that answers complex questions by generating sub-questions [1]. The hidden prompts in the complex libraries are quite difficult to get to and then start tweaking, whereas writing my own pipeline boiled down the code to 4 LLM calls. +1 for using the instructor library for quick schema definitions!
[1] https://github.com/pchunduri6/rag-demystified