Comment by avibhu

Have you tried few shot prompting? Something on the lines of:

User: Extract x from the given scanned document. <sample_img_1>

Assistant: <sample_img_1_output>

User: Extract x from the given scanned document. <sample_img_2>

Assistant: <sample_img_2_output>

User: Extract x from the given scanned document. <query_image>

In my experience, this seems to make the model significantly more consistent.