jarrelscy
Joined 2 years ago
Karma
13
Submissions
7
Member since
Sep 10
Recent submissions
Comment · 2 years ago
No problem! Thanks for picking it up.
Comment · 2 years ago
Thanks for the comments. I’m well aware as I’m also a practicing radiologist! Some hospitals in Australia where I work do a good job of enforcing that radiology orders are sent with the appropriate metadata but I agree that is not the case around the world. Integration, as always, remains the hardest step.
PS genuinely appreciate the engagement and don’t see it as undermining.
Comment · 2 years ago
Agreed, updated at https://harrison.ai/harrison-rad-1/
Comment · 2 years ago
Developer of the model here. We built this model in the form of an LLM precisely to address this problem - to be able to utilize the textual data that accompanies the image such as the order history or clinical background e.g. patient demographics. Images and text are both embedded into the conversation, meaning the LLM can in theory respond using both.
Of course, there are lots of remaining challenges around integration and actually getting access to these data sources e.g. the EMR systems, when trying to use this in practice.
Comment · 2 years ago
Developer here - its a good point that most of the models were not specifically trained on diagnostic imaging, with the exception of Llava-Med. We would love to compare against other models trained on diagnostic imaging if anyone can grant us access!
Comparison against human experts is the gold standard but information on human performance in the FRCR 2B Rapids examination is hard to come by - we've provided a reference (1) which shows comparable (at least numerically) performance of human radiologists.
To your point around people just out of training (keeping in mind that training for the FRCR takes 5 years, while doing practicing medicine in a real clinical setting) taking their first test - the reference shows that after passing the FRCR 2B Rapids the first time, their performance actually declines (at least in the first year), so I'm not sure if experts would do better over time.
1. https://www.bmj.com/content/bmj/379/bmj-2022-072826.full.pdf
Comment · 2 years ago
This current model is radiographs alone. The FRCR 2B Rapids exam is based on only radiographs.
Comment · 2 years ago
One of the developers here. The paper links to an earlier model from a different group that could only interpret X-rays of specific body parts. Our model does not have such limitation.
However, the actual FRCR 2B Rapids exam question bank is not publicly available and the FRCR is unlikely to agree to release them as this would compromise the integrity of their examination in the future- so the test used are mock examinations, none of which have been provided to the model during training.