theonlybutlet

Backpropagation Through Time (BPTT) is an adaptation of backpropagation used for training recurrent neural networks (RNNs), which are designed to process sequences of data and have internal memory. Because the output at a given time step might depend on inputs from previous time steps, the forward pass involves unfolding the RNN through time, which essentially converts it into a deep feedforward neural network with shared weights across the time steps. The error for each time step is computed, and then BPTT is used to calculate the gradients across the entire unfolded sequence, propagating the error not just backward through the layers but also backward through the time steps. Updates are then made to the network weights in a way that should minimize errors for all time steps. This is computationally more involved than standard backpropagation and has its own challenges such as exploding or vanishing gradients"

Comment · 2 years ago

I personally know my attention span will not be 100% driving for 3 hours straight, a short break is necessary and parking, ordering a coffee can take a bit longer than you'd think.

Comment · 2 years ago

It seems that is exactly what they've done just looking at the prop image in the video.

Comment · 2 years ago

Hah, if only you've seen me in my ICE car. Plenty of hoping and praying the glimmering fuel light doesn't die out and that I'll make it to the fuel station. No peace mind of mind.

Comment · 2 years ago

FT do a nice overview of the top reasons in the below article:

https://www.ft.com/content/02228292-b88f-11e6-961e-a1acd97f6...

Comment · 2 years ago

Thanks for your reply, you raise a very good point, transformer models are a lot more complex. I'd argue conceptually they're the same, just the data and process is more abstracted. Autoencoded data implies using efficient representations, basically semantically abstracted data and opting for measures like back propagation through time.

Comment · 2 years ago

They'll keep depressing down other employees wages to continue increasing their wages at the top to "aTRact THe bESt taLeNt".

Comment · 2 years ago

If you read their account, they actually believe they would've made it there without stopping to charge, also they used a fast charging station on the trip which would've charged to 80% in 30 minutes (a much more usual break on a long 3 hour trip). Many people stop for lunch on their way to Kruger, I suspect they did this here. Charging overnight on the slowest plug point in Kruger also delivered up to 210km range, double what they needed per day in Kruger.

It's not really an issue they were just extremely cautious, something you'd expect in South Africa if you know anything about their power grid. I believe the author was actually trying to show it is actually feasible and range is not a concern.

Comment · 2 years ago

Yes to both, the "neuron" would basically be a weighted parameter. A parameter is an expression, it's a mathematical representation of a token and it's probabilistic weighting (theyre translated from input or to output token lists entering and exiting the model). Usually tokens are pre-set small groups of character combinations like "if " or "cha" that make up a word/sentence. The recorded path your value takes down the chain of probabilities would be the "neural pathway" within the wider "neural network".

Someone please correct me if I'm wrong or my terminology is wrong.