rnosov

select count(instruction), instruction, group_concat(context, ' ============= ') as c, group_concat(response, ' ============= ') as r, group_concat(category, ' ============= ') as cat from [databricks-dolly-15k] group by instruction having count(instruction)>1 order by count(instruction)desc limit 100

[databricks-dolly-15k] should be the name of dataset, first column is the number of instruction duplicates

Creepypastas are responses to instruction:

Imagine you are the last person on Earth. Write a diary entry describing your thoughts and feelings.

Comment · 3 years ago

They are leveraging Apple’s Metal Performance Shaders[1] not the neural engine. From the chart, it looks like you might get ~20x max boost on inference over plain CPU. Obviously, it's not like having RTX 4090 but better than nothing.

[1] https://pytorch.org/blog/introducing-accelerated-pytorch-tra...

Comment · 3 years ago

It looks like you're on python 3.11 which has some issues with Pytorch. Downgrade to python 3.10 and try running it again.

Comment · 3 years ago

A question to the author. Can you perform an ablation study with respect to the chunks? In other words, if you put in the context irrelevant/random chunks from the document would the quality of answers decrease/stay similar?

Potential issue might be that chunks just serve to activate massive knowledge of GPT4 and not actually used as a basis for an answer. For example, GPT4 has surely seen Dune in its training corpus and could be answering from memory.

Comment · 3 years ago

1) I've looked at both codebases and this one is definitely a derivative of the nanoGPT. You can compare all three implementations yourself as they are actually surprisingly compact and readable.

2) The issue whether weights are copyrightable at all has not been settled yet. If they are, there is a fair use doctrine that allows transformative works of a copyrighted work. The line is a bit blurry but consider Cariou v. Prince case[1] where addition of colour to some black and white photos was considered enough to be transformative. Similarly, full fine tuning on current news or adding visual modality could potentially create a brand new model in the eyes of a law.

[1] https://cyber.harvard.edu/people/tfisher/cx/2013_Cariou.pdf

Comment · 3 years ago

I might be missing something but it looks to me that actually running this "open" model requires special hardware only accessible with a cloud subscription with 60 000 USD / week minimum spend[1]. Can anyone confirm if you can run it on your own hardware? If software is open but hardware is locked I don't see the point.

[1] https://www.hpcwire.com/2021/09/16/cerebras-wafer-scale-engi....

EDIT: Ok, looks like I've missed the hugging face repo. The language they use is a bit confusing.

Comment · 3 years ago

The FTC link has an example of the only newspaper in town refusing to deal with customers who are also running ads on a radio station. Do you think if the newspaper dressed such refusal as a ToS violation it would fly with FTC?

Google might be banning people for enforceable violations of their ToS but imagine the uproar if they banned a Bing engineer for using Google search to find solutions for some Bing problem (which is similar to the problem here). The upside for Google or OpenAI would be somewhat limited but the downside is almost boundless.

Comment · 3 years ago

That would be anticompetitive practice that is actually against the law in many countries[1]. In the unlikely event of OpenAI ever engaging in such things they will be sued into oblivion.

[1] https://en.wikipedia.org/wiki/Refusal_to_deal

Comment · 3 years ago

ToS are not the law. It would be similar to your power company claiming copyright over the code written using "their" electricity. Not going to happen. I wouldn't be too concerned.

Comment · 3 years ago

They are instruction tuning it using the dataset released by stanford-alpaca team. The dataset itself is synthetic (created using GPT-3) and somewhat noisy and in my view can be easily recreated if OpenAI ever tries to go after it (which is very unlikely). Anyway, facebook has nothing to do with anything used by this project.

Comment · 3 years ago

This has nothing to do with facebook. The foundational model here is GPT-J which is opensource and safe to use. Sadly, it is inferior to state-of-the-art models such as LLaMA.