> Parallel inference reaches hundreds of tokens/sec.
Marketing claims, meh. It gives normal people the wrong impression.
You can’t parallelize your query because it’s sequential. I think people will be willing to wait the ~200 sec necessary to get 200 words, but it’s best to be up front about this limitation.
Also abuse is a problem. Once 4chan realizes they can poison the distributed model, they’ll have a field day. But maybe it’s too much effort for too little reward that trolls won’t bother.
Marketing claims, meh. It gives normal people the wrong impression.
You can’t parallelize your query because it’s sequential. I think people will be willing to wait the ~200 sec necessary to get 200 words, but it’s best to be up front about this limitation.
Also abuse is a problem. Once 4chan realizes they can poison the distributed model, they’ll have a field day. But maybe it’s too much effort for too little reward that trolls won’t bother.