Pelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.