Comment

Comment by e12e

Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before:

  (cat ngrams.tsv \
    |parallel --pipe awk -f map.awk \
    |awk -f reduce.awk )
  max_key: 2006 sum: 22569013

This now runs in 17 to 18 seconds... :-/

  $ cat map.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      print "ignore", i, a[i]
    }
  }

  $ cat reduce.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      if (a[i] > max) {
        maxk = i
        max=a[i]
      }
    }
    print "max_key:", maxk, "sum:", max
  }

[ed: However, there are faster awks than gawk:

  $ time mawk -f sum.awk ngrams.tsv
  max_key: 2006 sum: 22569013

  real    0m2.826s
  user    0m2.391s
  sys     0m0.422s

mawk is (a little) faster than pypy on my machine.

]

parent e12e 9 years ago

Comment by e12e

Discussion