I reckon this type of task is crossing capabilities in a way that the LLMs (very quietly, mind) don't support.
An LLM can describe a photo fine, but it cannot tell you the exact coordinates of what it's describing. When I last tested this, you could sometimes get it to lie about the coordinates, but asking it to draw an approximate bounding box around a given feature - total fail, sometimes laughably so.
An LLM can describe a photo fine, but it cannot tell you the exact coordinates of what it's describing. When I last tested this, you could sometimes get it to lie about the coordinates, but asking it to draw an approximate bounding box around a given feature - total fail, sometimes laughably so.