lukehoban

https://github.blog/ai-and-ml/generative-ai/migrating-the-gi...

* Succinct syntax and operators tailored to AI model definition

* Fully statically typed tensors, including generic functions over tensor dimension and batch dimensions (...)

* First-class hyper-parameters, model parameters and model arguments for explicit model specification

* Einops-style reshaping and reductions - tensor dimensions are explicit not implicit

https://github.com/lukehoban/ten

Ten is a statically typed tensor programming language for defining AI models.

Ten has the following features: (1) Succint syntax and operators tailored to AI model definition (2) Fully statically typed tensors, including generic functions over tensor dimension and batch dimensions (...) (3) First-class hyper-parameters, model parameters and model arguments for explicit model specification (4) EinOps-style reshaping and reductions - tensor dimensions are explicit not implicit

Example (a functional GPT2 implementation):

    Gelu(x: {...}) -> {...}:
        return 0.5 * x * (1 + Tanh(0.7978845608 * x + 0.044715 * x**3))    

    SoftMax[N](x: {...,N}) -> {...,N}:
        exp_x = Exp(x - Max(x))
        return exp_x / Sum(exp_x)    

    LayerNorm[S,E]|g:{E},b:{E}|(x:{S,E}) -> {S,E}:
        mean = Mean(x)
        variance = Var(x)
        return g * (x - mean) / Sqrt(variance + 1e-5) + b    

    Linear[N,K]|w:{N,K},b:{K}|(x:{...,N}) -> {...K}:
        return x@w + b    

    FFN[S,E]|c_fc, c_proj|(x:{S,E}) -> {S,E}:
        a = Gelu(Linear[E,E*4]|c_fc|(x))
        return Linear[E*4,E]|c_proj|(a)    

    Attention[Q,K,N,V](q:{...,Q,K}, k:{...,N,K}, v:{...,N,V}, mask:{Q,N}) -> {...,Q,V}:
        return Softmax[N](q @ Transpose[N,K](k) / Sqrt(K) + mask) @ v    

    MHA[H,S,E,K]|c_attn, c_proj|(x:{S,E}) -> {S,E}:
        q, k, v = Linear[E,E*3]|c_attn|(x) {S,(3,H,K) -> 3,H,S,K}
        causal_mask = (Tri[S]() - 1) * 1e10
        out = Attention[S,K,S,K](q, k, v, causal_mask) {H,S,K -> S,(H,K)}   
        return Linear[E,E]|c_proj|(out)    

    Transformer[H,S,E]|mlp, attn, ln_1, ln_2|(x:{S,E}) -> {S, E}:
        y = x + MHA[H,S,E,E/H]|attn|(LayerNorm[S,E]|ln_1|(x))
        return y + FFN[S,E]|mlp|(LayerNorm[S,E]|ln_2|(y))    

    GPT2[H,S,E,B,V]|wte, wpe, blocks|(inputs:{S}) -> {S,V}:
        x = wte.[inputs] + wpe.[Range[S]()]
        z = for i in 0...B: x, y -> Transformer[H,S,E]|blocks.[i]|(y)
        return LayerNorm[S,E]|ln_f|(z) @ Transpose[V,E](wte)

From https://medium.com/hackernoon/the-first-typescript-demo-905e...:

> There were many options already available, but none seemed to be resonating well with a broad enough section of the market. Internally at Microsoft, Script# was being used by some large teams. It let them use C# directly instead of JavaScript, but as a result, suffered from the kind of impedance mismatch you get when trying to stand at arms length from the runtime model you are really programming against. And there was Google’s Closure Compiler, which offered a rich type system embedded in comments inside JavaScript code to guide some advanced minification processes (and along the way, caught and reported type-related errors). And finally, this was the timeframe of a rapid ascendancy of CoffeeScript within the JavaScript ecosystem — becoming the first heavily used transpiled-to-JavaScript language and paving the way for transpilers in the JavaScript development workflow. (Aside — I often explained TypeScript in the early days using an analogy “CoffeeScript : TypeScript :: Ruby : C#/Java/C++”, often adding — “and there are 50x more C#/Java/C++ developers than Ruby developers :-)”)

> What we quickly discovered we wanted to offer was a “best of all worlds” at the intersection of these three — a language as close as possible to JavaScript semantics (like CoffeeScript) and syntax (like Closure Compiler) but able to offer typechecking and rich tooling (like Script#).