Comment by armcat

Kind of like random projections. Pre ChatGPT there would every now and then come a paper that states something along the lines of: instantiate a large randomly populated matrix, multiply by the input, and win! It kinda makes sense because you "stretch out" the space and separation boundaries become easier, but I have never seen it fully utilised in production in any meaningful way. Anyone remember the DANs (Deep Averaging Networks)?