westandskif

supports aggregations, joins + now optional collection items

Just to add my previous answer: the trivial native python equivalent doesn't have the same functionality, because it consumes data iterator 3 times in your case, while convtools would consume it only once.

In terms of the implementation, it's kind of trivial to implement the "magic", but it would be both confusing and inconsistent, see below.

e.g. imagine a case where you'd want to call datetime.strptime, partially initializing it at the moment of conversion definition. at the moment it is:

  c.call_func(
      datetime.strptime,
      c.item("updated"),
      "%Y-%m-%d"
  )

but it's unclear to me how would the "magic" approach deal with the case above.

We would need to bring the whole python into it :/

Pandas are great, but I had a few cases where I had frustrating experience -- dealing with Decimal & float columns is a pain (missing data without any signs when using both in calculations).

However this was not the reason why I needed to build convtools, I needed to process reports, touching only some columns (without failing if an unrelated column is no longer processable). So I needed to reuse and combine python expressions across multiple procedures.

There are no benchmarks at the moment, you can just pass debug=True to the gen_converter method to see the generated code and judge whether it's optimal for your use case. This is a python library which generates simple python code: - without unnecessary conditions and loops - without keeping all items of iterable in memory to aggregate (it leverages reducers) - making no use of C-extensions.