joewferrara

Hey Oliver, thanks for the question! This is the PrivacyBench dataset maintainer here. The LLM-as-judge for synthesis coherence has been reviewed extensively by me. The privacy_bench_metrics evaluation code generates a report that has for each person in the data the mappings of that person's PII to the synthetic replacements as well as the LLM-as-judge's judgement for each replacement. That makes review easy and LLM-as-judge decisions look. Each decision the LLM-as-judge makes is binary (is the replacement coherent or not), so it is a strait-forward judgement for the LLM and it is easy for a human to review. All that being said, we're in the process of doing a more rigorous evaluation where we record humans judgements and compare them to the LLM-as-judge on a subset of the PrivacyBench data.

I think you mean x*x = -1, for which I agree with your point.

right now we support txt, csv, tsv, docx, xlsx, pdf, png, tif, tiff, jpg, and jpeg filetypes. we support either local files or aws s3 as the document store where the files are read from. so we can work with your messy files in s3 as they are without any standardizing!

that's a good question. for this use case, we have a custom models feature where the user can define an entity type (types) they're interested in and then a custom ner model is trained to identify the specified type. for this, a combination of llms and user input are used to generate training data for the ner model.

The built in static, I'm not sure if it's purposeful or not, but I like it. If it is purposeful, it's a nice touch.

kevin buzzard is a very good(usually math) expositor with a bent towards programming. love it

Is the staff (which includes you) is all volunteers, there’s so reason to expect or want people to have a “get shit done” attitude. You should all appreciate that you’re putting in any time together towards a shared goal.

They show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.