Ask HN: Best tools for real-time API proxying/transformation?
I am a data engineer and do a lot of open source work writing API proxies over scientific APIs to present them as standardized formats [0] with better discoverability. I don't have access to the underlying db so I can't just create a new API directly.
When writing data pipelines for generating data assets, I have really benefited from using a scheduler framework like Dagster. It gives my projects a quick way to get a nice UI and common helpers (cron schedules, asset graphs, partitioned data, concurrency) with relatively little code.
However, it doesn't seem like an analogous framework exists for real time proxying. (i.e. something with config as code, a nice UI for tracing requests, concurrency/caching helpers for dealing with messy transformations). Am I missing anything?
When writing data pipelines for generating data assets, I have really benefited from using a scheduler framework like Dagster. It gives my projects a quick way to get a nice UI and common helpers (cron schedules, asset graphs, partitioned data, concurrency) with relatively little code.
However, it doesn't seem like an analogous framework exists for real time proxying. (i.e. something with config as code, a nice UI for tracing requests, concurrency/caching helpers for dealing with messy transformations). Am I missing anything?
[0] https://ogcapi.ogc.org/