femto

In the One Electron Theory the positron is just the One Electron moving backwards in time. No separate positron!

"interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).

As used in LDPC decoders?

My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.

Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"

From memory the entire filter was updated in one symbol period (ie. global)

The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.

Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?

Most professional engineers, especially those in electrical or communications, have the tools to understand AI.

I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.

This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.

In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.

I write this because many electrical/communications engineers probably know how AI works, but don't realise it.

My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:

1) Arranging the layers and the connections between them.

2) Having access to the right training data

3) Having the computing power to perform the gradient search (aka training)

My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.

Deep Learning With Python, Third Edition by Francois Chollet and Matthew Watson

Recently read books by Anthony Doerr (Shell Collector, About Grace, Memory Wall, All the Light We Cannot See) and Simon Singh (Code Book, Fermat's Last Theorem, Trick or Treatment, The Simpsons and Their Mathematical Secrets).

> These things can get messy fast.

Yep. I was on a fixed term contract, which had expired, even though I was still working on the project well after its expiration. It would have been interesting to argue that any IP assignment expired with the contract. As you said: messy. As I judged: not worth it.

I was in on the ground floor of WiFi, in that around 1995 I did a significant chuck of the work that proved that WiFi works, and my work at a university lead to the formation of a company that put WiFi into the market (Radiata). The IP situation was murky. A patent holder made about $1b and the company sold for $560m (in 2000). I gather the university complained and got a good chunk of "go away" money. I could have thrown my hat in the ring: maybe I would have got something, maybe I wouldn't have. Either way, I walked away, as I judged it wasn't worth the non-financial cost. 30 years layer I still think I made the right decision. I might have been living in a nicer house, but I wouldn't have the life I have today, which I am happy with.

I don't know your situation in life, but if your experience is anything like mine I reckon you've probably made the right decision.