aljungberg

That's awesome. It's not easy to get seniors to spend that much time in the gym, so well done if you contributed to motivate her! Physical exercise is indeed one of the few Parkinson's treatments that there's little doubt about.

In terms of reversing the damage, look into photobiomodulation (PBM) aka red light therapy. In "Improvements in clinical signs of Parkinson's disease", Liebert et al in 2021, she shows improvement in all symptoms including cognition, which is one of preciously few such results I've found in my extensive search of the literature[1]. Caveats are that this was a proof-of-concept study with n=6 only and that the red light helmet is somewhat expensive if you want to try it. There's a Canadian company that makes one for above $2000 (modern, very sci-fi thing), and a more hackerish version from an Australian company for ~$700 (pairs of diodes on aluminum bands you have to finish assembling yourself).

[1]: At least which are actionable for the public. There are tons of trials but good luck getting in early unless you can donate a new library to the university! Gene therapy, stem cell therapy, GDNF, drugs that target alpha-synuclein are all promising but not yet accessible. PBM is something you can do today and since mitochondrial dysfunction is a leading hypothesis in the pathogenesis of PD, the treatment fits.

We already do tree searches: see beam search and “best of” search. Arguable if it is a “clever” tree search but it’s not entirely unguided either since you prune your tree based on factors like perplexity which is a measure of how probable/plausible the model rates a branch as it stands so far.

In beam search you might keep the top n branches at each token generation step. Best of is in a sense the same but you take many steps using regular sampling at a time before pruning.

To an extent, but memory bandwidth soon becomes a bottleneck there too. The hidden state and the KV cache are large so it becomes a matter of how fast you can move data in and out of your L2 cache. If you don’t have a unified memory pool it gets even worse.

Your triton code is great, nice work. Wouldn’t feel too bad about spending your time that way!

As it happens I was also thinking it might be worthwhile to dive into the Triton sources but for another reason: half2 arithmetic. That’s one thing that the Triton branch lost that the (faster) CUDA kernels had and I think it made a difference. In theory with compatible hardware you can retire twice as many ops per second when processing float16 data which we are in this case.

Can’t see anyone having tried to get half2 to work with Triton though.

For some workloads, it’s almost all about the VRAM. In those cases I’ve been wondering if getting a high memory M1 or M2 Mac could be a good lab machine thanks to unified memory. It’ll run more quietly, use significantly less power, no worries about overloading your electric circuit. On a 128 GB RAM Mac Studio you could theoretically run or even train models that otherwise would require multiple $6k A6000 GPUs in custom machine builds taking oodles of power at the plug. It’d be slow but slow beats not possible. And if you need a new development machine anyhow, you can justify some of that beefy Mac Studio’s cost as part of your required spend anyhow. PyTorch has supported “mps” as a target device for some time now.

Within a closed system, consistency (and verifying it) is a fail fast mechanism. For example, it’s better to crash on a constraint failure when attaching a doodad to a non-existent user account than to figure out where all these orphan doodads came from next year.

Whatever that number is, it will be equal or less than the number already discussed. The network “hires” contractors to provide the services you mentioned and it pays a known figure for that. Not much else to it really. Since all we are discussing is whether the network is profitable or not in this thread we don’t need to dig into more specific analysis of the service providers’ internal costs (and indeed that would be difficult since they are globally distributed with different attendant costs and efficiencies). Just to note they are unlikely to themselves be making a loss is sufficient.

It seems like you’re making a semantic argument to equate the Ethereum network with its validators. That seems confusing. Here are some examples of how “A runs B” does not imply “A == B”:

“Employees” are part of a company, they run the company, don’t they? Yet the cost of having employees is not “therefore revenue” from the standpoint of the company.

“Drivers” are a part of Uber, they deliver the service, don’t they? Yet the money paid to drivers reduce Uber’s profits.

I think where your argument runs into trouble is “from the standpoint of the network”. If you want to equate the network and its validators, to say they are the same thing, then your sentence becomes, “The money [the validators] get paid [by the validators] is therefore revenue, from the standpoint of [the validators]”. That’s non-sensical. You can’t give yourself money and say it’s revenue. Either these two things are in fact not the same thing and we can analyse the cashflow of “Ethereum the network” separately from “the validation service providers”, in which case Ethereum is paying out less than it’s taking in, so it is profitable. Or they are the same thing, in which case the “profit”, to the extent you can say a virtual entity like a network can have such a thing, is even higher.

This is because whatever costs the validators bear are less than the ETH they receive is worth. This is true if we assume validators are rational actors (they wouldn’t validate if they were losing money doing so). And even if we take away the assumption that they are profit motivated (maybe they’re all doing it as charity work for some higher purpose), the cost of running an Ethereum validator is tiny, so we end up in the same place: outgoings are smaller than receipts when considering the whole.

(The fact that Ethereum the network “burns” its receipts and then “mints” its outgoings to the validators does not affect this calculation since it’d work out the same if Ethereum paid validators from fees directly.)

It does say on there they are training it on the Pile training data. And they have this bit comparing inference with GPT2-XL:

RWKV-3 1.5B on A40 (tf32) = always 0.015 sec/token, tested using simple pytorch code (no CUDA), GPU utilization 45%, VRAM 7823M

GPT2-XL 1.3B on A40 (tf32) = 0.032 sec/token (for ctxlen 1000), tested using HF, GPU utilization 45% too (interesting), VRAM 9655M

So it looks about twice as fast for inference while using only about 80% as much VRAM. Obviously at such a small size, just 1.5B, you can run it even on consumer GPUs but you could do that with GPT2 as well. If it remains 80% of VRAM usage when scaled up, we’re still talking 282GB once it’s the size of BLOOM w/ 176B parameters. So yeah still 8x A100 40GB cards I guess. Not going to be the Stable Diffusion of LLMs.

THe RWKV model seems really cool. If you could get transformer-like performance with an RNN, the “hard coded” context length problem might go away. (That said, RNNs famously have infinite context in theory and very short context in reality.)

Is there a primer for what RWKV does differently? According to the Github page it seems the key is multiple channels of state with different decaying rates, giving I assume, a combination of short and long term memory. But isn’t that what LSTMs were supposed to do too?

I used the Quest 2. It wasn’t just the hardware though, something about the software too. The “main” display was a reasonably sharp and almost retina like. But the other displays were unable to keep up with that level of quality. Not enough bandwidth? Video encoding or decoding CPU bound? Not sure what it was. This was using Immersed

I have tried it. Working in VR is much better than I expected it would be, with the right equipment. Still, I quit after a while mainly because of the quality of text rendering. It is so much better than it used to be, but still not good enough, at least not for me.

I also found it surprisingly frustrating that I couldn’t see the keyboard. Every time I took my hands off it I had to do this blind search for it and get my fingers back to the right starting position. Symbols I don’t know how to touch type because they are rarely used was also more frustrating than I would have thought.

If those loans are no-recourse loans with this FTT token as collateral, then should the token crash the liability just "disappears". The collateral will be sold to cover the loan. If the collateral is now worthless that was the risk the lender agreed to take on when issuing a no-recourse loan.

If they are Defi loans for example, they're pretty much automatically no-recourse loans.

Hasn’t it been a popular topic on here of how bloated Twitter’s staff seems to be given how little the platform has apparently changed over the years? Not taking a stance on that, I don’t know, but to that cohort of commenters this probably seems like victory for common sense.