Joined 10 years ago
One hour with an expert GRE tutor: $75. One equivalent hour with an AI tutor: 7 cents. 918x cheaper per point of learning, with equivalent learning (p=.044).
In 5 of 7 GRE topics, the top AI tutor beat the expert human tutors on average.
Motivation is that RSI and most AI research is focused on machines improving machines. We created this suite of benchmarks and evaluations to shift investment to focus on machines improving humans. We were also just genuinely curious how much AI could improve a human's learning gain on real actual standardized exams.
All data from students and teachers is open-sourced to advance research in augmenting human capabilities with AI.
Data: https://huggingface.co/datasets/handshake-ai-research/studen... Code: https://github.com/Handshake-AI-Research/studentbench Try it: https://studentbench.org Video: https://x.com/cgnorthcutt/status/2103171858057277532/video/1
Happy to answer questions.
Benchmarking post: https://l7.curtisnorthcutt.com/benchmarking-gpus-for-deep-le...
Positioning (speed-up) post: https://l7.curtisnorthcutt.com/gpu-positioning
GitHub with Benchmarks: https://github.com/cgnorthcutt/cnn-gpu-benchmarks
Post: https://l7.curtisnorthcutt.com/confident-learning
Paper: https://arxiv.org/abs/1911.00068
Code: https://github.com/cgnorthcutt/cleanlab/
Abstract: Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) has emerged as an approach for characterizing, identifying, and learning with noisy labels in datasets, based on the principles of pruning noisy data, counting to estimate noise, and ranking examples to train with confidence. Here, we generalize CL, building on the assumption of a classification noise process, to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This generalized CL, open-sourced as cleanlab, is provably consistent under reasonable conditions, and experimentally performant on ImageNet and CIFAR, outperforming recent approaches, e.g. MentorNet, by 30% or more, when label noise is non-uniform. cleanlab also quantifies ontological class overlap, and can increase model accuracy (e.g. ResNet) by providing clean data for training.
This is a follow-up to the previous post "I built Lambda's $12,500 deep learning rig for $6200" which had around 480 upvotes on Reddit. That previous build had only 3-GPUs and took some shortcuts. In response to the hundreds of comments on that post, including comments by the CEO of Lambda Labs, I built and tested multiple 4-GPU rigs. I'm back to share a near-perfect 4-GPU deep learning rig with the highest performance and reliability, no thermal throttling, and lowest cost. This build is nearly identical to Lambda's 4-GPU workstation, but costs around $4000 cheaper. Happy building!
