Twey

The programming model and the implementation are not 1:1. This thread is mostly about the model; we can talk about implementation below :)

Message-passing doesn't necessarily involve naïvely copying the data around. If you have access to shared memory then you can pass a pointer; the only thing you really need to ‘pass’ is the capability to access the message data. This is how seL4 does it, for example.

Of course if you don't have shared memory then you have to copy anyway :) And because of CPU caching architecture the benefits of having an owned copy of the data can sometimes outweigh the cost of copying it, even if shared memory is available.

I think at the end of this long road lies algebraic effects. You can write your computations with typed effects and, once you've handled every effect (or all but a finite subset, if you want to leave some to the runtime), you can drive your computation generically. Because the effects are typed, you get to narrow the types of the effect arguments at each yield, so you don't have to handle every possible effect at every yield (on either side of the channel).

> What if I want to change some state in-between two state transitions (i.e. between two `await` points)?

I think as phrased what you said is an _advantage_ of the Rust model: the state of your state machine is entirely contained within the dedicated state-machine type and you know its behaviour can't depend on surprise outside state.

But I think the spirit of your criticism is correct: you do really want to be able to type processes with intermediate inputs and outputs, in general, and Rust makes this hard (though not impossible: see Munksgaard's work on session types in Rust, for example). I would not have had a problem if the original article had decided that Rust's futures are unfit for its purposes, and it would have been an interesting case study in what exactly real-world code needs from the API; my complaint is just that they get categorized as a type of I/O, which couldn't be further from the truth.

Your comment is phrased as a rebuttal, but I actually agree with it wholeheartedly, with one minor quibble.

> Blocking threads are a much more natural way of thinking about concurrency.

The (ideal) programming model for writing a thread vs an async task is identical, and indeed the most natural way of writing sequential flows that may incidentally be temporally separated. A blocking operation and an await are equivalent (in the absence of shared memory; in its presence they differ in that an async task on a single thread gets to assume atomicity of the code between awaits, but that's not too relevant here). I think what Joe is arguing against here is exactly splitting up your business flows into a bunch of explicit state machine transitions.

The difference is only in the caller. Callers of async tasks (à la Rust) have the right and responsibility of controlling how (and whether) the task they create resumes, while threads/BEAM processes (and ‘stackful’ futures à la Go, JavaScript, etc.) are answerable only to the great global event loop in the sky (sometimes known as the OS scheduler or the BEAM multiplexer, etc.). This makes it easier to avoid bugs that are about forgetting to drive a task, but harder to avoid bugs that are about forgetting to stop a task, or stopping a task wrongly; see structured concurrency. It also hurts composability a bit, because by construction there can be only one event loop and there's no way to decompose it.

You can see Erlang (or I suppose we should talk about BEAM more generally now people like Elixir) in terms of OS processes as an analogy, but it's just as valid and perhaps a little more accurate to think about it as a pool of async tasks with a global event loop, à la Go, where all blocking-looking code is actually async. It shares a common problem of such languages that because it can't interrupt real blocking code, e.g. FFI, you have to be very careful not to accidentally starve your other processes with such a call. This is made more obvious (but not really fixed) by languages that colour async functions differently, because the async colour acts as a promise (sadly not enforced by the language) that the function will be a good neighbour and not block for too long (whatever that means).

> Function coloring is about (the annoyance when) composing interfaces designed around those different runtimes/thread that might use the same sans-io library internally.

No, it's about the difficulty of calling async functions (or functions with other typed effects that can't be handled locally) from pure functions (or more generally functions that aren't themselves typed with that effect).

> Yes async compiles down to a statemachine, but we're talking about the elegance of the language syntax/composition. Most languages require _a lot_ of ritual (or overhead) if you want to use the async-statemachine as a pure statemachines and sidestep the build in runtime-side-effect channels.

That's true, which I would consider a limitation of those languages. Thankfully Rust futures are exactly state machines; there are some rough edges on their API still, but I'd have preferred a discussion about those to throwing the baby out with the bathwater and claiming that those state machines are a type of I/O.

> This is incorrect. You can easily call that flow from a blocking function, just call it in a loop.

Sure, the same can be said of an async library — so long as you're okay with none of your other flows being able to make progress in the meantime. You can just poll in a loop.

> It's the best of both world but as always you have a tradeoff, and in this case it's the ability to express the state machine as if it was a straightforward function.

‘The best of both worlds’ means that it shares the advantages of both, but the main (arguably only) advantage of async/await is that you get to write your temporally separated flows sequentially. (And maybe also that they look a bit like functions, I guess.) That's not true here.

> A Sans-IO library needs none of these.

To be clear I'm not by any means arguing that ‘sans-IO’ (or, as we used to call it, pure functional programming) is a bad thing. I'm very in favour of it. I'm just saying that a ‘sans-IO’ design shouldn't need to eschew async/await, which you can write in terms of pure functions.

(As a sibling comment points out, there are languages where that is not true, i.e. async/await inherently introduces some kind of side effect around concurrency or a global event loop. Thankfully that's not the case in Rust.)

It was all going great until I read

> The design also avoids problems like function coloring and plays well with Rust's ownership rules.

Async is not I/O. It can be used for I/O, but it's just a way of writing pure, deterministic state machines that keeps a logical flow together instead of scattered over a bunch of transitions.

Even if you choose to manually compile your async flows into explicit state machines, you still have to deal with ‘function colouring’ (a.k.a. typing): a flow that crosses several state transitions (and may have other disjoint parts of the state updated in the meantime, via other transitions) now has to be rewritten as a (series of) state transitions, and can't be called from a blocking function. If anything, rewriting all your state machines without `async` makes the function colouring ‘problem’ worse, because multi-transition flows are even more different from normal functions.

I think that used to be true when systems were simpler and expectations were lower. But the expected standard of modern languages has come a long way. And I'm not just talking about libraries: performance, safety (strong static type systems are one can of worms, GC and other runtime checks are another), and even syntax design are things that state-of-the-art languages spend years iterating on before they're considered ‘usable’.

Some other more recent data points:

Go was designed at Google (one of the biggest software development entities of our times, with input from existing PL luminaries) in 2007, released in 2009, and officially considered 1.0 in 2012, five years later;

Clojure was started in 2005 (there were two other attempts to build similar languages dating back to 2003, but let's charitably omit them) and hit 1.0 in 2009;

Zig has been going for about 11 years and hasn't hit 1.0 yet.

Rust's nine-year record (2006–2015) is on the longer side, but not a huge outlier.

Of course, this is just the time to 1.0; I think in practice languages tending to iterate a lot before becoming mainstream (which is a weak proxy for general usability, but at least should work in one direction). Rust today has come a long way from 1.0, for example, as has Go. Definitely the ecosystems at 1.0 releases tend to remain noticeably underpowered, and even if you want to ignore libraries for a language it's harder to justify excluding other tools that make it easier to work with the language on your own or with other people to build real-world software, like editor support, debuggers, documentation, or package managers.

Famously, as with many other things in computer science, writing a toy programming language that can compute Fibonacci is doable in a couple of hours, but writing a usable programming language that needs to stand up to real programs for users is typically somewhere between a decade and a lifetime of work.

If he'd announced the language twelve years ago and then not touched it again until a few months ago I'd believe you. But there's plenty of documented evidence that he's been spending a bunch of time on it, and (allegedly) even using it in his commercial projects.

Not to say that he's been working on it full-time for 12 years, of course, but even allowing for part-time work that's far beyond the initial ‘easier than The Witness’ premise.

> that is probably 10% of a PL if you want it to do something people will actually use

I think this is really understating the challenge of generating machine code that is competitive in terms of size, performance, etc. with state-of-the-art compilers. I'll be the first to agree that it's far from the only (or even the biggest) difficulty in PL, but it remains a huge chunk of engineering work. The interesting part of this experiment is how much of that work, as a percentage of the work of building a programming language, is now given to you for free, for a language that is indeed mostly ‘just a syntax wrapper over C-like functionality’.

(Of course, a big part of ‘people will actually use’ is nothing to do with the language at all, but the next twenty years of building an ecosystem and community buy-in.)

I'd agree more except that as far as I know (I'll admit I've not been following the development that closely) he's been significantly building it in public on Twitch. So he's happy to spend time sharing it with the public, and has been from the start; it's just taken a long time to get it to a usable state.

> If you are fine with getting there 80% his claim is correct. That he's using the language for his games proves it to me. Of course, if you require it to be both open and popular to be a success you kind of reach a stalemate. In a world where everyone can quickly whip up a language I would expect most to be closed to the public anyway.

I think we're firmly in agreement here, except that IMO the default for software that's not for public consumption is to throw it over the wall, put the source up on GitHub or something, and not bother with documentation, explanation, or interaction. A carefully curated closed beta is kind of the opposite of that. If anything he's spending more time promoting it than I would expect.

In fact reading between the lines of that video a bit, I think he might believe that general-purpose languages are poorly suited to his use because they're focused on solving other people's problems. But that's just me extrapolating.