wsprague

My thoughts about Tk:

1. I think buttons with relief are easier to use, but other than that I am always surprised at how worked up people get about theming (but then I prefer a terminal based emacs...).

2. Tcl/Tk has some GREAT ideas that are not implemented anywhere else: the built-in event handler, trivial to write callbacks, and the everything is a string philosophy (which is strangely like a self-evaling lisp approach)

3. Tcl/Tk had some really bad luck. This weekend at opensqlcamp, someone asked Selena Deckleman why PostgreSQL has not had a tenth the market share as MySQL. She said (paraphrasing): in 1999 a lot of the core devs thought the internet was just a fad -- whoops... Same mistakes maybe happened with Tcl/Tk -- it was there just a little too early.

4. Tcl/Tk demands C to be a really full platform, and C is scary to a lot of the new breed of scripters.

"Murray and Strout don't understand human nature. People -- especially teenagers -- don't like following pointless rules."

Au Contraire -- Murray and Strout understand fully, but as employees of the public school system, Murray and Strout are paid to teach their students to follow pointless rules with docility and even joy (or at least the feeling that they are doing the right thing).

(Edit -- I consider it a compliment that I get upvoted and downvoted so often. Teenage rebel gives me a point... humorless school principal takes it away...)

Interesting point, though I more or less disagree about the level of curation necessary to make a minimally useful product. I think data.gov is FAR below that level, and is just noise so far. But these are practical questions, and I think we agree in principal about a lot.

I moved my reply to my top comment and screwed up the reply tree here. So this is for the comment that follows this one:

data.gov is fundamentally flawed, and won't be anything but annoying until it is reworked into something along the lines of what I suggest. Or so I think...

If you need hierarchies, use JSON, IMHO, or keys that reference between tables (the census PUMS data does this with persons nested within households, using two tables).

OK, we disagree. Except that #4 IS sort of redundant, though I want to make the point that data is almost impossible for a layperson, and still really hard for a practiced analyst.

I mean that for all practicing data analysts that I know, XML is a pain in the ass (parsers, xpaths, etc, all to get it into a csv that you can import), while nice ascii text is easy to work with.

If you want metadata, a well written narrative paragraph along with a code book is INFINITELY better than embedding the metadata in the data.

Furthermore, a lot of supposed metadata in XML is just dross like "<column>blah</column>".

Finally, all the crap in XML way ups the signal/ noise ratio; if you do need something that maps to a complex data structure use JSON or something rational. Such needs are not very common in data analysis, in contrast to web applications; data analysts use multiple tables and are usually pretty close to relational databases and SQL (even if they don't call it that).

The problem with data.gov and sites like it is that they are built on faulty premises about data:

1. Fiction: Data doesn't require lots of work to make it useable, so we can just upload whatever we have and it will be useful to somebody. Fact: the big useable datasets (census, ipums, nlsy, all the private marketing datasets) have armies of people cleaning and integrating them. It costs money, it takes time, and it is easy to screw up.

2. Fiction: Links are worth something. Fact: links are worthless.

3. Fiction: XML adds values. Fact: aascii tab delimited in consistent formats add value, while XML SUBTRACTS value.

4. Fiction: a good dataset is easy to use. Fact: even a good dataset (google IPUMS for an example) takes a lot of work to get to know how to manipulate, presuming one can use some sort of statistical programming language in the first place.

5. Fiction: simple summaries of common data data are useful. Fact: everybody has already done the simple summaries. (This is just a bonus item, and doesn't apply to data.gov, but does apply to faulty thinking about data in general.)

6. Fiction: Federated data is just fine. Fact: Data that is curated, cleaned, and integrated into one big monolithic package is FAR better, because an analyst can then learn the conventions and names and such in one piece, and parallel categories are more likely to align.

7. Fiction: Good data is easy for a layperson to use. Fact: good data still requires a lot of skill. Well, maybe in nations with decent public schools a layperson can do something with data, but not in the US.

What I WOULD like is the following (taken from another post, now deleted):

An ideal data.gov would have a lot of staff who put together a few integrated and curated datasets from the agencies. These would be hierarchies of data in a few formats (shp, txt, raster, SQL text dump, and ...?), along with well written codebooks and narrative READMEs. They would be distributed using git or subversion. The staff would have the expertise to make such nice data packages for you and me, and they would have the political oomph to demand that the agencies release the data to them. The staff would also give classes on how to use the data along in some open source statistical packages to do useful work. Good examples of curated data that I know are IPUMS and the Portland Metro's RLIS (both google-able).

The only thing I HATE about Python is the mandatory indentation. The problems with it are:

1. It mixes format with logic. Don't we all know this is one of the greatest evils one can commit?

2. Indentation is a bitch to parse compared with braces. Violates occam's razor of software of engineering -- simpler is better. Unnecessary complexity is the second greatest evil in software engineering.

3. Indentation is neither symmetrical nor logically unique. Therefore you can't delete all the formatting from a python program and expect a formatter to get it right. With symmetric braces a formatter always knows exactly what logical level it should be on by counting the number of closed braces; not possible with python's goofy system.

4. All that tiresome debate and worry over tabs versus spaces.

5. One can't really say "braces are redundant not indents" or "indents are redundant not braces". Redudency is commutative. I would say braces are for computers, indents are from human; it is simple to go from braces to indents but not so in the other direction.

Just a note -- I will always pick Python over any other scripting language. I just think the indentation is a serious mistake; there are so few other bad choices it tends to irk me that much more.

I would like to see this supported (or not, methinks) by empirical evidence. A priori arguments without evidence aren't worth a damn, in my estimation.

I upvoted you because it is a good point, but ... who is advancing the most quickly in manufacturing these days? That would be China....

Plus, I would hesitate to naturalize the idea of intellectual property as if it were a "thing" just like a cheese sandwich. Some of your argument seems to rest on protecting IP which (somehow) exists; a lot of people would doubt that IP is a valid concept or exists at all.

In python, it seems like it would be trivial to add a __test__() method to a class that was run, well, whenever.

I liked the way he applies moderately useful statistics to a real world application, mostly just as an example of how we all could be using the standard deviation along with the mean.

I know people who work in social services who have noticed a change in the last generation. Before, you would think twice before telling a working class parent that the child misbehaved because the parent might discipline too harshly. Today, the parents always assume the kids are right when they come home whining about how mean the teacher was. Don't know which is better or worse....