Recent submissions

Comment · 1 month ago
amazing.
Comment · 3 years ago
Not ELI5 obviously but might help some.

Transformer is a patterning probabilistic machine for a sequence of identities[1]. These identities are fed to the transformer in lanes. The transformer is conditioned to shift lanes one position to the left until they make it to the output, and make a prediction in the right-most lane that got freed up. Attention adds an exponential amount of layer interconnectivity, when we compare it with a simple densely connected layers. The attention mask serves as a high-dimensional dropout, without which it would be extremely easy for the Transformer to simply repeat the inputs (and then fail to generalize when making the prediction). Each layer up until the vertical middle of the Transformer works with a higher contextual representation than the previous one, and this is again being unwound back to lower contexts from the middle layer back to the original identities (integers) on the outputs. This means that you have raw identities on the input and output which span a certain width/window of the input sequence, but in comparison the middle-most layer has a sequence of high level contexts spanning extreme lengths of the original input sequence, knowledge-wise. [1]It's important to know that modification (learning) by the Transformer, of the vector embeddings which represent the input/output identities/integers that the Transformer works with, constitute big portion of the Transformer's power, and the practical implication of that is that it's impractical to try to tell the Transformer that e.g. some of our identities are similar or there's some logical system in their similarity, because all the Transformer really cares about is the occurrence of these identities in the sequence we train the Transformer on, and the Transformer will figure out the similarities or any kind of logic in the sequence by itself.

Comment · 4 years ago
Comment · 4 years ago
Life (aging) is a direct cause of cancer. And we can't change that unless we have some sort of nanobots constantly repairing DNA damage in every single cell of our body.
Comment · 4 years ago
> The only read problem I see would be overpopulation, and possibly draconian measures to regulate it.

Like, for example, releasing a weaponized virus from a lab. Oh wait...

Comment · 4 years ago
I know I'd never be able (or wanting) to live with having left my dogs behind. I don't fear my own death _that_ much. I'm here for them until they cross, only after that I can do whatever I want with my life.
Comment · 4 years ago
Damn, so I really need to follow up on my plans on moving to e.g. Thailand (and getting happy by following my dreams in terms of work/endeavors).

I don't have IBS but more like a colitis and I was really good for months but it came back strong after one of my dogs passed away and I got terribly sad.

What I'm getting at is that more than a vitamin D, I suspect a strong link to happiness and life satisfaction.

Comment · 4 years ago
1. When people are financially incentivized to run servers, they always will.

2. If a crypto protocol doesn't evolve at the pace of available innovation, that particular blockchain will be superseded by a new one. That said, a (truly democratic) evolutionary process is a core part of every blockchain specification.

3. You can get blockchain data via public (and federated/proxied) API, but you can always cryptographically verify its veracity, and your edge device (e.g. your smartphone) can do that. The same the other way around, you cryptographically sign the inputs you send to the networks, so that no federated API can tamper them, because the secret key stays on your device. This is referred to as the "trust-less model".

Comment · 5 years ago
It's interesting to see how human understanding differs when it comes to complex, yet clearly defined topics, like machine-learning/Transformers.

For comparison, my understanding of Transformers, after going through Peter Bloem's "Transformers from scratch" [1], implementing/understanding the code and the actual flow of the mathematical quantities, my understanding is that:

- Transformers consist of 3 main parts: 1. Encoders/Decoders (I/O conversion), 2. Self-attention (Indexing), 3. Feed-forward trainable network (Memory).

- The Feed-forward is the most simple kind of (an input->single-layer) neural net, actually often implemented by a Conv1d layer, which is a simple matrix multiply plus a bias and activation.

- The most interesting part is the Multi-head self-attention, which I understand as [2] a randomly-initialized multi-dimensional indexing system where different heads focus on different variations of the indexed token instance (token = initially e.g. a word or a part of a word) with respect to its containing sequence/context. Such encoded token instance contains information about all other tokens of the input sequence = a.k.a. self-attention, and these tokens vary based on how the given "attention head" was (randomly) initialized.

The part that really hits you is when you understand that for a Transformer, a token is not unique only due to its content/identity (and due to all other tokens in the given context/sentence), but also due to its position in the context -- e.g. to the Transformer, the word "the" at the first position is a completely different word to the word "the" on e.g. the second position (even if the rest of the context would be the same). (Which is obviously a massive waste of space if you think about it, but at the same time, at the moment, the only/best way of doing it, because it moves a massive amount of processing from inference time to the training time - which is what our current von-Neumann hardware architectures require.)

[1] http://peterbloem.nl/blog/transformers

[2] https://datascience.stackexchange.com/a/103151/101197

Comment · 5 years ago
Comment · 5 years ago
Comment · 5 years ago
Forget all that. If we tried to swich half the cars to electric overnight, the power grid would collapse into a permanent blackout. Has anyone estimated the cost and time of upgrading the power grid for electric cars? Is hydrogen delivery from power plants to electric charging stations a viable option to bridge the gap until the grid is upgraded?
Comment · 5 years ago
There should be legal repercussions against anyone even suggesting something as blatantly illegal and tyrannical/totalitarian as this. Have we learn nothing from communism?

Globally destroying web experience by mandating super-useless, super-obnoxious Cookie prompts on all web pages is one thing, but setting things up for "Equilibrium" or "1984" type of society is a whole new level, and I'm not sure if sheer stupidity can be claimed on this one.

Comment · 5 years ago
Mankind will most likely destroy each other using super-intelligence long before super-intelligence gets any chance to do so by its own volition. We are built on conflict, disagreement, misunderstanding and exploitation - it's part of our societal DNA, no matter how sweet or reasonable we can make ourselves seem on personal level. We have avoided nuclear annihilation by luck on multiple occasions. Nuclear weapons are sticks and stones compared to the power of a runaway super-intelligence in a technological society.

Thinking that super-intelligence is containable is like thinking you can beat AlphaGo in the game of Go. And coming up with reasons why that's no so it's just you being in denial of your eventual mortality, as well as of the eventual extinction of the species. Best that we can hope for is that that extinction will be some form of transmutation to a higher level of evolution.

Comment · 5 years ago
What happens (or what would happen) if you sit still in silence and darkness and try to observe these "feeling symbols" as they come and go?

You could be better equipped for deep meditation and deep thinking than most of us, but maybe you just need to do some practice.

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton