It would be purely up-and-down (due to the Earth's 23 degree orbital tilt) if the Earth was on a perfectly circular orbit. But because the orbit is slightly elliptical, and because its orbital speed thus varies throughout its orbit[0], the alignment of the Sun on the sky at "noon" drifts back and forth according to when the Earth's orbit "outpaces" its daily rotation or when the daily rotation outpaces the orbital motion.
[0] https://en.wikipedia.org/wiki/Kepler's_laws_of_planetary_mot...