We have smart batching in Dip where we deduplicate work received in what we call a "batched" request. We use BTreeMap to deduplicate work in the preprocessing "scatter" phase.
This deduplication in smart batching is the slowest phase on most bulk write workloads (200k+ inserts in a single request) and takes sometimes up to 3x the time taken for the combined fanout + db writes + db commits + gather.
We have smart batching in Dip where we deduplicate work received in what we call a "batched" request. We use BTreeMap to deduplicate work in the preprocessing "scatter" phase.
This deduplication in smart batching is the slowest phase on most bulk write workloads (200k+ inserts in a single request) and takes sometimes up to 3x the time taken for the combined fanout + db writes + db commits + gather.
Will try out with Bbtree and see how it goes.