I don't know about physical laborers being fatter, but our bodies are horribly/wonderfully efficient when it comes to work. The amount eaten will be the larger factor.
See:
http://en.wikipedia.org/wiki/Basal_metabolic_rate#Biochemist.... About 70% of a human's total energy expenditure is due to the basal life processes within the organs of the body (see table). About 20% of one's energy expenditure comes from physical activity and another 10% from thermogenesis, or digestion of food (postprandial thermogenesis).
See:
http://en.wikipedia.org/wiki/Basal_metabolic_rate#Biochemist.... About 70% of a human's total energy expenditure is due to the basal life processes within the organs of the body (see table). About 20% of one's energy expenditure comes from physical activity and another 10% from thermogenesis, or digestion of food (postprandial thermogenesis).