willpeter

Easy-to-use tool to quickly sync or transfer large amounts of data based on Flink or Spark

Apache DolphinScheduler is a cloud-native platform with powerful visual interfaces. Compared with other open source orchestration platform, it has lower latency, higher concurrency and more stability, which has been produced and used by more than 1000 companies, including IBM, Accenture, Walmart, VMware, SheIn, Cisco.

The key features for DolphinScheduler are as follows:

- Easy to deploy, we provide 4 ways to deploy, such as Standalone deployment,Cluster deployment,Docker / Kubernetes deployment and Rainbond deployment

- Easy to use, there are 3 ways to create workflows:

  - Visually, create tasks by dragging and dropping tasks

  - Creating workflows by PyDolphinScheduler(Python way)

  - Creating workflows through Open API

- Highly Reliable, DolphinScheduler uses a decentralized multi-master and multi-worker architecture, which naturally supports horizontal scaling and high availability

- High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day

- Supports multi-tenancy

- Supports various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, Hive, Spark SQL), Python, Procedure, Sub_Workflow, Http, K8s, Jupyter, MLflow, SageMaker, DVC, Pytorch, Amazon EMR, etc