Comment by kenoph

You can do a lot with just bash + pipes + unix tools. It can get messy as your pipeline grows though, and there are a lot of edge cases.

Relevant: "bashML: Why Spark when you can Bash?" (https://rev.ng/blog/bashml/post.html), aka how to deduplicate git repositories using `comm` + `awk`.