Comment by britmob

You only need a large amount of system RAM for the initial dataset encoding. You can do this once per dataset on a high memory config on AWS for pennies.