Developer of the model here. We built this model in the form of an LLM precisely to address this problem - to be able to utilize the textual data that accompanies the image such as the order history or clinical background e.g. patient demographics. Images and text are both embedded into the conversation, meaning the LLM can in theory respond using both.
Of course, there are lots of remaining challenges around integration and actually getting access to these data sources e.g. the EMR systems, when trying to use this in practice.
Of course, there are lots of remaining challenges around integration and actually getting access to these data sources e.g. the EMR systems, when trying to use this in practice.