Comment by Radim

Well pandas iterates through every row anyway, just in native-land rather than pure Python. That's not the difference.

The trouble with this code was the choice of the quadratic algorithm (which the idiomatic pandas version avoids). You could achieve a similar speed-up as OP for the pure Python code too, by dropping the doubly-nested loop.

The point stands as usual: better algo > clever tricks :) Personally, I find the code brevity that pandas offers does not outweigh the added complexity and cognitive load of its (IMO hard to remember and reason about) API.

EDIT: pure Python version

  def gen_stats_python2(dataset_python):
      aggr = {}
      for product_id, product_order_num, quantity, price in dataset_python:
          item = aggr.get(product_id, [0, []])
          item[0] += quantity
          item[1].append(price)
          aggr[product_id] = item
      return [
          (int(product_id), len(prices), int(quantity), round(float(sum(prices)) / len(prices), 2))
          for product_id, (quantity, prices) in aggr.items()
      ]

761x speedup!

  %timeit gen_stats_python(dataset_python)
  1 loops, best of 3: 36.3 s per loop

  %timeit gen_stats_python2(dataset_python)
  10 loops, best of 3: 47.7 ms per loop