Hi HN, author here. Corral is my attempt at a performant, easy-to-deploy MapReduce. Unlike traditional frameworks like Hadoop, it uses AWS Lambda for execution and is “serverless” as a result. It was initially kicked-off by AWS adding Lambda support for Go, but draws on experience I’ve had using Lambda tools like Zappa[1] and Serverless[2] in the past.
I think there’s a lot of interesting applications for using function-as-a-service platforms as executors in data processing frameworks such as this.
I think there’s a lot of interesting applications for using function-as-a-service platforms as executors in data processing frameworks such as this.
If you’re interested more in the development/internals of this project, I wrote a blog post with more details: https://benjamincongdon.me/blog/2018/05/02/Introducing-Corra...
[1]: https://github.com/Miserlou/Zappa [2]: https://serverless.com/