To be done right, such vision benchmark should consider the peculiarities of the vision tower, and was it quantized and optimized. So a lot of success/loss of quality may not be due to the transformer.
Second of all, the "find coordinates" of something is super difficult task of any model, so you tried to test a small quantized buddy with a tri-star challenge. Not sure what expectations were set.
disclaimer: myself do large volume VLM work daily, including in production, for more than 1.5 years now.
Second of all, the "find coordinates" of something is super difficult task of any model, so you tried to test a small quantized buddy with a tri-star challenge. Not sure what expectations were set.
disclaimer: myself do large volume VLM work daily, including in production, for more than 1.5 years now.