Comment by chaps

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago.

Also, here's a tweet-fitting one liner I wrote that downloads all Socrata CSV datasets across all gov instances:

  mkfifo f;echo a>f&xargs -E, -a<(cat f) -I{} bash -c "wget http://api.us.socrata.com/api/catalog/v1?scroll_id={} -O-|jq -r '.results[]|.resource.id+"\","\"+.metadata.domain+"\"/api/views/\""+.resource.id+"\"/rows.csv\""'|tee <(cut -d, -f2|xargs wget -P{} --content-disposition)|tail -1"|tee f