Just poking around HuggingFace, I see a whole lot of models and finetunes claiming to be SOTA in this and that... But not a whole lot of validation. In fact, the biggest LLM validation effort seems to be with llamav1 and v2, and testers had enormous difficulty reproducing Meta's metrics.
Many of these finetunes or base models are dumped with little info and private training datasets. How much of those datasets are questionably legal or ethical? How many are accidentally or intentionally contaminated with test data? No 3rd party ever seems to test for contamination outside the rare social media post.
I see a whole lot of frameworks, papers and APIs claiming SOTA transformers/diffusion performance in this or that, but they seem barebones and janky when one actually goes and investigate them, if you can get them to work at all. Or they are grand promises asking for money now and usability later.
...I am no data scientist, but I am a straight up AI fan. I was finetuning ESRGAN 1x models for img2img/vid2vid years ago. GenAI is the best thing since sliced bread as far as I am concerned. Yet I am getting a serious crypto fraud vibe from everything that is happening, albeit one more targeted at businesses and investors than the crypto wave.
Many of these finetunes or base models are dumped with little info and private training datasets. How much of those datasets are questionably legal or ethical? How many are accidentally or intentionally contaminated with test data? No 3rd party ever seems to test for contamination outside the rare social media post.
I see a whole lot of frameworks, papers and APIs claiming SOTA transformers/diffusion performance in this or that, but they seem barebones and janky when one actually goes and investigate them, if you can get them to work at all. Or they are grand promises asking for money now and usability later.
...I am no data scientist, but I am a straight up AI fan. I was finetuning ESRGAN 1x models for img2img/vid2vid years ago. GenAI is the best thing since sliced bread as far as I am concerned. Yet I am getting a serious crypto fraud vibe from everything that is happening, albeit one more targeted at businesses and investors than the crypto wave.