frig

For all the thought you've put into your comment you clearly have a lot of your ego -- probably too much -- invested in being the guy who's not afraid to publicly point out it might be worth making scientific inquiry into racial differences.

Let me point out exactly how you're being stupid here.

The article is about future ramifications of genetic sequencing technology finally becoming economical enough that it's possible to cheaply sequence a person's complete DNA.

You're right that this will allow research to be done of the form: grab a DB with 100k different peoples' sequences; discover strong statistical associations between such-and-such genes and such-and-such amount of alcohol intolerance, and notice that those genes are particularly prevalent amongst people of asian descent.

Where you go awry is your conceptualization of how this information will likely be used; it's a failure to take a premise you've accepted in one part of your scenario -- the premise that DNA sequencing will be cheap-and-thorough enough to allow for research like just sketched -- and failing to apply it to another part of your scenario -- the doctor-patient interaction, and so on.

Given a world in which DNA sequencing is super-cheap and super-affordable, which doctor-patient interaction is likelier:

- doctor says to patient: "you clearly are asian just by looking at you. asians have been shown to have lower alcohol tolerance on average. be careful with how much alcohol you imbibe."

- doctor says to patient: "your DNA sequence contains 7 of the 10 major genes associated with higher levels of alcohol intolerance. this isn't surprising given your family history, but it's nice to know concretely what you've got. be careful with how much alcohol you imbibe."

...which is why you're being stupid: race is a proxy measure for ancestry that's "field-performable" (just look!) and cost-effective and is not an entirely useless proxy measure for ancestry; it is indeed the case that were political bugaboos different than they are that some useful research could be done to characterize racial differences that could be put to general use.

It's also a terribly inaccurate proxy and given the ability to do direct DNA sequencing would be dropped like hotcakes; why use the inaccurate proxy when you can make a direct measurement?

So even in your hypothetical example of looking for a wife: when this research is available, would you not be better served testing for specific genes in a particular individual rather than cutting it off at the level of race? You might still want to move to Ankara if you discovered that "Turks" are likelier than Brits to carry the genes you want your kids to have, but would you really not take it further and check the specific woman's DNA directly (the way that some jews test for tay-sachs at some point in the courtship process)? If you take the research seriously enough to move to Ankara to stack the deck why stop there?

You're essentially making the same kind of mental error you saw in the early dot-com days where people understood that "in the future, you'll buy stuff at home over the internet" -- so were semi-prescient -- but couldn't discard what they already knew about "shopping"; the consequence being that they'd write articles about how you'd sign into some 3d virtual world and then visit an online bookstore and browse the shelves in virtual reality (transporting outdated ideas into a world of new possibilities).

The specific example of 'da Bears' was a weak attempt at humor. I can't easily clarify what I'm getting at and keep this short, and I've got limited time so I'll do the best I can.

I see a lot of people using "informal" bayesian reasoning (meaning a lot of talk about priors and updating and reference to theorems but never any use of actual distributions beyond super-super-cursory examples applied to trivial situations like the boy/girl thing here or stuff like the monty hall problem).

I don't have any problem at all with bayesian analysis applied in a rigorous setting to a rigorously specified problem (like spam detection and so on).

In an informal setting I'm extremely skeptical of the uses I tend to see b/c there's no careful attempt to clearly delineate which informally-statable hypotheses are valid and which are "invalid" "meta-hypotheses" like the optimism thing.

What you've described here is a way in which someone reasonably smart would eliminate the meta hypothesis, which is fine. In general I wouldn't expect it to be feasible to take a full mental inventory, do a topological sort on your beliefs, and then apply the same procedure; most people most of the time will be running around holding partially-inconsistent beliefs (where "hold" means if you were to ask them to give an estimate of, say, what beliefs they had about what # of their beliefs were likely to wind up revealed to be significantly off in the future, or to give an estimate of what they believe about the frequency with which they'd encounter evidence leading to significant revisions of their beliefs, they'd have an answer on offer which would still have "work to do", the way the unexamined belief that "I'm too optimistic about the bears" really has work to be done).

What I'm curious about is if there's either a clearly-specifiable criteria for which types of beliefs or hypotheses are workable and which are "too meta to work", or there's some kind of theorem guaranteeing that starting out with "inconsistent" beliefs -- in the sense of "meta-hypotheses" like with da bears -- you can apply this algorithm to process evidence and over time you'll converge on beliefs that're at least more consistent than you started with.

It's hard to say much more without getting formal and I'm out of time for now; since I'm mainly concerned with informal use of "bayesian" metaphors it's not hugely critical to formalize this stuff but later I could give it a proper whack.

Say a proper Bayesian who's a fan of 'da Bears' just saw the bears lose again, badly. How is he supposed to update these two beliefs:

- this season 'da Bears' will tear shit up every Sunday (fwiw assume prior on this has it as likely)

- I am usually over-optimistic about 'da Bears''s performance (fwiw assume prior on this also has it as likely)

...in light of the new evidence?

(Side note: I'm specifically not talking about bayesian analysis or statistics in a formal setting with a prespecified, finite list of allowed hypotheses (like spam or ham).

If I wrote this out in extreme detail I'd get nothing done today, I think you can charitably fill in the gaps and missing pieces of what I'm getting at; if not I'll fill them out in a few.)

Just something I had a hard time figuring out and seems poorly documented:

    --type-set=scala=.scala # works in .ackrc
    --type-set scala=.scala # doesn't work in .ackrc

...despite the version with one equals being correct command-line sequence. Only seems to matter if you want each flag on a single line in your .ackrc file. Might not be true for everyone.

I like the look of it. Am I correct that the order the statements are in translates into the order things are evaluated in when the code is called?

If eg I edited it to be:

    do { l <- locations;
         w <- widgets;
         guard (l `hasInStock` w);
         s <- sprockets;
         guard (l `hasInStock` s);
         guard (w `isUsableWith` s);
         return (w,s,l); }

Does that force it to go through "location-first" and only check the w and s of (w,s,l) for compatibility if it has already ascertained that w and s are in stock @ l?

I'm partially defending list comprehensions here and partially challenging you to consider the possibility that there are higher levels of abstraction out there than those employed by functional programming primitives like map and filter.

Level 0: for-loops with an explicit accumulator

Level 1: map + filter (!)

Level 2: ??? arguably an atemporal set-theoretic approach

In practice in python list comprehensions are a superior syntax for computing with multiple source collections.

(!) Really all you need is reduce

    map = lambda f,l: reduce(lambda h,t: h + f(t), l, [])
    filter = lambda f,l: reduce(lambda h,t: h + t if f(t) else h, l, [])

You'd be silly to implement them that way of course but know your tools.

I have a longer comment here that argues the opposite: list-comprehensions are only a syntactic pun or two away from set-builder notation, which is a higher level-of-abstraction (it declaratively states what it is) than map+filter (which specify a procedure to generate it, albeit at higher level of abstraction than a for-loop).

If you can put together a nontrivial usage of map+filter with at least three source collections that's more concise than the equivalent list comprehension I'll (figuratively) eat my hat.

It depends slightly on how you came to functional programming.

You can arguably trace Python's list comprehension syntax all the way back to setl, a "set-theoretic programming language", and the resemblance to mathematical set-builder notation is intentional; compare

- let B = { g(a) s.t. | a \in A and f(a) holds}

- B = [g(a) for a in A if f(a)]

If you're used to thinking in sets having to decompose into "maps" and "filters" is a speedbump; easy to do but nice to avoid.

Where list comprehensions really start to shine is making it comparatively trivial to pull from multiple source collections without a lot of ugly machinery:

    [{'widget':w,'sprocket':s,'location':l} for w in widgets for s in sprockets for l in locations if l.hasInStock(w) and l.hasInStock(s) and w.isUsableWith(s)]

...which is about where explicit map + filter start to become annoying. You can use:

    map(lambda i: {'widget':i[0], 'sprocket':i[1], 'location':i[2]}, filter(lambda i: i[2].hasInStock(i[0]) and i[2].hasInStock(i[1]) and i[0].isUsableWith(i[1]), itertools.product(widgets,sprockets,locations)))

...but to my eyes that is not only very ugly but just going by character count the # of characters given over to keywords (map, lambda, filter) instead of "what i'm doing here" is huge. Additionally use of itertools forces use of tuples for your intermediate values and thus the lambdas are gobbledegook until you get to the tail end of the statement and see that i[0] == widget, i[1] == sprocket, and i[2] == location. I could define some constants (WIDGET = 0, SPROCKET = 1, LOCATION = 2, etc) but now it's even longer.

It gets even worse if you try to be clever with the sequence of operations.

You might look at that definition and say lo! I can pre-filter out stuff not in stock at each location and make things more efficient. Naively you'd wind up with:

  map(lambda d: {'location':d[0], 'widget':d[1], 'sprocket':d[2]}, flatten(map(lambda l: list(filter(lambda p: p[1].isUsableWith(p[2]), itertools.product([l],filter(lambda w: l.hasInStock(w), widgets), filter(lambda s: l.hasInStock(s), sprockets)))), locations))) #nb must-supply-you-own-flatten-method

Under some circumstances that might be substantially faster than the previous approach. But compare the equivalent but you could have instead gone with:

    [{'widget':w,'sprocket':s,'location':l} for l in locations for w in [widget for widget in widgets if l.hasInStock(widget)] for s in [sprocket for sprocket in sprockets if l.hasInStock(sprocket)] if w.isUsableWith(s)]

So you lose a little flexibility in simple cases at in exchange for increasing the scope of what you can get away with as "readable" one-liners.

Yes, you did miscommunicate; even allowing for the miscommunication you're making an extremely poorly-considered analogy.

The kind of peer review that matters for purposes of scientific integrity is review by outsiders; eg, a paper is "peer-reviewed" when experts in the field not involved in writing it or conducting its research give it a going-over and see if it appears to hold up.

The kind of transparency that matters for purposes of scientific integrity is making data available as-is to outsiders, so that they can meaningfully replicate your results ab initio (or, perhaps, not!).

Neither Google nor Amazon conducts meaningful amounts of peer review in the scientific sense nor are they transparent in the scientific sense (nor should they be, the last thing I want is any old anyone seeing the raw data backing someone else's gmail account or search history).

So you're making a useless assertion in the context of the issue at hand: neither Google nor Amazon does much "peer review" (to my knowledge Microsoft in fact does to some limited extent with shared source and hiring 3rd party auditors for some important code chunks); neither is "transparent".

In this thread you'll find multiple people speaking from positions of actual experience working on large-scale endeavors of scientific computation who've commented at length upon their internal practices.

I'm not going to repeat what they've typed but if you read those comments you will see that at least those commenting here were in fact engaging in what you're calling "peer review" and "transparency" wrt their development practices; their reports match my experience concerning scientific computation with large annual budgets but I won't claim any authority for my anecdotal experiences.

I will close out by posting a protip here; you might benefit from it but it's not specifically aimed at you.

If a blog with a name like "chicagoboyz" makes a bold, sweeping, and somewhat shocking assertion about an entire area of human endeavor -- that it itself has no plausible claim to expertise in (it's an ec blog, not a large-scale computation blog, and no claim of direct experience was made that I saw) -- and you find yourself nodding your head and thinking "yeah, that sounds plausible", proceed to do the following:

- slow down, step away from the computer, and count to 20 backwards in greek

- ask yourself: do I have any concrete knowledge, at all, about the area this claim is being made? Any involvement in a project in that area, or business involving that area, etc.? (In this case: do you have any direct experience with large-scale computational efforts in science? do you know anyone who's been involved in such a project? anything beyond the flamebait du jour?)

- if you do have any concrete knowledge: great, you have at least some nonzero evidence base from which your initial "yeah, that's right feeling" may or may not be substantiated. Think carefully about what you already know and see if the "yeah that's right" feeling holds up.

- if you don't have any concrete knowledge: you've given yourself an awesome opportunity for self-discovery and personal growth. Clearly there's something that makes you want to uncritically believe this specific sweeping claim about some area about which you literally know nothing concrete; we generally consider who believe sweeping claims without evidence suckers, and we've found an area where your preexisting biases leave you a sucker, and therefore at the mercy of others. You might still have the right intuition about the sweeping claim, but at least take the opportunity to de-suckerify yourself on this front before drawing your conclusion.

In what meaningful sense do Google or Amazon have peer-reviewed and transparent data sets?

Please be specific.

Yeah I've not studied that interpreter so if that's your specific interest no luck sadly; rank is an easy idea but wouldn't surprise me if there's some trick to it in an implementation that short.

I think it'd be easier to untangle if you already had a working knowledge of the language but given the line-noisy look of J I can't blame you for trying the shortcut.

The interactive J labs are very good. You might also find the J for C programmers discussion of rank particularly helpful if you are looking at learning by understanding an implementation.

I figure the array languages will make a minor comeback soon what with gpus / larrabee / etc showing up on the horizon and what with more-versatile input devices showing up (making it easier to go back to funky symbols instead of line noise).

Please take this not as an insult but as invitation to offer help.

Rank shouldn't be something to get confused about; if it is you're probably overthinking it.

What did you find confusing about it?

Another point of experience: out of the box it is pretty poor if you work on the macintosh; by default there's nothing mapped to "command" and there're not enough "keys" on it to remap anything to command without giving up something else you probably need (like shift or alt or control).

You can work around this with a footswitch but it's another point to keep in mind.

Mine suffered a similar fate to yours: in a box, awaiting some future period where I have enough downtime to ride out a month or two of reduced typing speed to train myself.