nihit-desai
Joined 6 years ago
Karma
374
Submissions
23
Member since
Feb 24
https://twitter.com/nihit_desai
https://github.com/nihit
View on Hacker NewsRecent submissions
Comment · 3 years ago
function calling, as I understand it, makes LLM outputs easier to consume by downstream APIs/functions (https://openai.com/blog/function-calling-and-other-api-updat...).
Autolabel is quite orthogonal to this - it's a library that makes interacting with LLMs very easy for labeling text datasets for NLP tasks.
We are actively looking at integrating function calling into Autolabel though, for improving label quality, and support downstream processing.
Comment · 3 years ago
Yep! I totally understand the concerns around not being able to share data externally - the library currently supports open source, self-hosted LLMs through huggingface pipelines (https://github.com/refuel-ai/autolabel/blob/main/src/autolab...), and we plan to add more support here for models like llama cpp that can be run without many constrains on hardware
Comment · 3 years ago
Hi!
The earlier post was a report summarizing LLM labeling benchmarking results. This post shares the open source library.
Neither is intended to be an ad. Our hope with sharing these is to demonstrate how LLMs can be used for data labeling, and get feedback from the community
Comment · 3 years ago
>> don't trust that there was no funny business going on in generating the results for this blog
All the datasets and labeling configs used for these experiments are available in our Github repo (https://github.com/refuel-ai/autolabel) as mentioned in the report. Hope these are useful!
Comment · 3 years ago
I mean, sure. For ground truth, we are using the labels that are part of the original dataset:
* https://huggingface.co/datasets/banking77
* https://huggingface.co/datasets/lex_glue/viewer/ledgar/train
* https://huggingface.co/datasets/squad_v2
... (exhaustive set of links at the end of the report).
Is there some noise in these labels? Sure! But the relative performance with respect to these is still a valid evaluation
Comment · 3 years ago
Hmm, I'm not suggesting training a smaller model from scratch - in most cases you'd want to finetune a pretrained model (aka, transfer learning) for your specific usecase/problem domain.
The need for labeled data for any kind of training is a constant though :)
Comment · 3 years ago
Partially agree, but it's a continuous value rather than a boolean. We've seen LLM performance largely follow this story:
https://twitter.com/karpathy/status/1655994367033884672/phot...
From benchmarking, we've been positively surprised by how effective few-shot learning and PEFT are, at closing the domain gap.
"When it encounters novel data (value) it will likely perform poorly" -- is that not true of human annotators too? :)
Comment · 3 years ago
Good question - one followup question there is value for who?
If it is to train the LLM that is labeling, then I agree.
If it is to train a smaller downstream model (e.g. finetune a pretrained BERT model) then the value is as good as coming from any human annotator and only a function of label quality
Comment · 3 years ago
Hi, one of the authors here. Good question! For this benchmarking, we evaluated performance on popular open source text datasets across a few different NLP tasks (details in the report).
For each of these datasets, we specify task guidelines/prompts for the LLM and human annotators, and compare each of their performance against ground truth labels.