dalailambda

SQL has definitely become the defacto tool for a lot of data processing. This model of working is generally referred to as ELT as opposed to ETL.

For small/medium scale environments Fivetran/Stitch with Snowflake/BigQuery using getdbt.com for modelling is an insanely productive way to build an analytics stack. I consider this the default way of building a new data stack unless there's a very good reason not to.

For larger scales Facebook has Presto, Google has Dremel/Procella/others, and a lot of data processing is done using SQL as opposed to writing code.

The only downside really is that it tends to be fairly focussed on batch pipelines (which are fine for 95% of workloads). But even that is becoming less of an issue with Beam/Spark so you can use SQL for both batch and streaming.

Source: Solution Architect at an analytics consultancy.

Lambda will create a function instance to handle concurrent requests, i.e. if 100 requests happen at the same time there will be 100 function instances. It will then however keep them "alive" for a few minutes allowing it to reuse already running instances. Additionally, a function instance won't handle multiple requests at a time.

Given this, using beam is a bit of a waste in terms of individual instance scalability. That being said however, you might be able to use shared actor pools (e.g. caches, etc.) across all your functions. I want to emphasise the might as rapidly adding and removing nodes from the beam cluster might not work well or at all.

At the end of the day, the lambda model kind of supplants the actor model as your unit of messaging and concurrency, and so trying to mix the two isn't the best idea. If you want to use the beam on AWS I'd recommend sticking to ECS/Fargate/EKS. That being said, Elixir might be a nice match due to developer ergonomics, just don't expect to be able to drag and drop actor reliant features.

In my experience the reason for a JSON object per line is because a tool can then split the entire file by newlines and have a list of objects to start parsing/processing in parallel, which avoids having to parse the entire file up front to get usable data, and lets the tool start processing things while some data is still being parsed.

One of the major reasons that would influence a language like Jai is that once you do that, even if your generated C/++ is fairly optimal, you still have to go through another slower compiler. Jai's compiler is fairly optimised for speed and I think that would be an unacceptable trade off.

On the target platform side, a video game tends to have a fairly well known set of platforms it will run on (new PC's, PS4, Xbox), so I think this is an acceptable trade off.

And just a final note, the generated C/++ would be very unreadable anyway, unless it was a direct transpiler, at which point you lose a lot of the benefits of having a new language.

The app can still use environment variables and be completely independent (12 factor), but the Dockerfile/Kubernetes config still needs to provide those things, so there is a clear distinction between ops and dev, they're just more mingled than previously.

There are generally two levels of autoscaling involved with kubernetes.

Firstly, kubernetes is able to create multiple instances of your app as load increases and scale them out over all the nodes.

Secondly, the nodes your kubernetes cluster is running on can also autoscale. With terraform you can set up an AWS autoscaling group for example to automatically increase the size of your cluster as load increases.

There doesn't seem to be any part of that package that requires ES6 to use. You can use it as an ES6 module, but the commonjs version should work fine as well.

Building distributed systems makes you aware of how unreliable things are at the large scale, e.g. the network. The parent comment implies that Google's network is so fast and reliable that it becomes tempting to ignore best practices and work as if it's a non distributed system.

A web page, for example, should not have direct access to the database.

In regards to Purescript/Typescript, they're both statically typed and that results in friction when trying to integrate with the existing JavaScript ecosystem/libraries. Erlang/OTP might be different, but there will probably be situations where the type system is either incompatible with a certain library, or the type system is made less strict (e.g. an any type).

While I agree that the OTP perhaps is not as easily statically typed, since it was built with Erlang in mind, I do think that static typing adds a layer of robustness to distributed systems, especially if you design it that way upfront. In my experience the problem comes when you try to apply static typing to a dynamic system.