Comment by rgejman

Data cleaning/munging is a surprisingly pervasive problem in my field ('big data' cancer research). I spend a good fraction of my time hunting down bugs caused by dirty sources upstream of my scripts. One way to fix this is to become more rigorous about using tools and libraries that check and enforce data correctness and consistency... but it's hard to do this on an individual level. Even if I'm careful about ensuring the correctness of my data, other researchers who send me data may not be as scrupulous. As with any challenge that requires teamwork and constant vigilance (e.g. unit testing), perhaps the data science community needs to invest in tools and computational frameworks that constantly monitor data for correctness and consistency using DDLs and the like.