serendipityrecs
Joined 6 years ago
Karma
26
Submissions
12
Member since
Jul 14
james at [my username] . com
View on Hacker NewsRecent submissions
Comment · 6 years ago
How do you deal with trigger happy mods banning you even when your post is relevant to the subreddit?
Comment · 6 years ago
Hi, I don't have a comprehensive guide or anything like that, but I can quickly answer your questions:
1. Delimiting your text with <|endoftext|> lets the code know where the boundaries of your text chunks are. It makes sure that during training you're not inadvertently continuing past the end of a chunk of text into a totally unrelated one. However, it seems like there is a bug with this re how the nsheppard fork works: https://github.com/openai/gpt-2/issues/222
Even if you're not doing this, you shouldn't be getting gibberish, so you likely have another issue in your set up.
2. Finetuning just means training on top of a somewhat already trained model. GPT-2 is a language model, which means it's trying to learn (and generates from) a probability distribution over words (actually tokens). That probability distribution is going to be different depending on what corpus you're working with (wikipidia articles vs news vs reddit). GPT-2 (the models you download) is trained on a particular corpus but you as you fine-tune on your own corpus it's going to start pulling the learned probability distribution towards that of your dataset's.
I think the best way to play with this stuff is to run the code, like you're doing. When you get something that doesn't make sense, dig into the code/paper. There's also a lot of information in the readmes/issues for these repos.
Comment · 6 years ago
Hey, cool, you built the same thing as me but with less code: http://serendipityrecs.com/
One thing that's cool about GPT-3 is how versatile it can be, so I'm exploring a natural language search interface, which is something that traditional recommendation algorithms can't do.
Are you planning to incorporate GPT-3 into your startup? I'd love to hear about what use cases you envision it solving. If you're interested in connecting, my email is in my profile and my website.
Comment · 6 years ago
Any proof is also a remix of old text (and often times old ideas too)
Comment · 6 years ago
I ran some experiments with GPT3+proofs. You can read about it here: http://serendipityrecs.com/blog/posts/experiments-with-gpt-3... I wasn't able to get it to return anything interesting but I also didn't spend a lot of time refining the approach (this was just a series of quick and dirty experiments, ultimately I identified recommendations as a good use case for GPT-3 and started building a product around it.)
I think it will be interesting to use GPT-3 in conjunction with existing automated/machine proof techniques. Since GPT-3 doesn't "know" anything, automated proof systems can help the system stay logically consistent.
Comment · 6 years ago
4-6% is a very impressive number. If you're a small-mid sized business with 1-100mm ARR, getting 4-6% for no work (it sounds like these guys will do the heavy-lifting in terms of AB testing for you) is a no-brainer.
Comment · 6 years ago
Cool idea. I've been working on an adjacent product (serendipityrecs.com), but mine is more targeted towards consumers as opposed to B2B. I think I gravitated towards B2C by default because as an engineer, I don't want to deal with sales, but your product makes sense. I'll be interested in following your progress over the next few years to see how your thesis plays out.
Couple questions
- How well do your recommendations hold up against Amazon's? Since you're scraping the metadata, you should be able to generate recs for Amazon items from their own catalogue. This might be an interesting product / demo for potential customers.
- Once you hook up your system to your customer's back end, how do you learn from the behavioral data you get from them? That's straightforward for cf/mf, but can be tricky to integrate into what you already have. - You talk about Stripe pooling the data from their customers. I think the analogue for you would be pooling the behavioral data from your customers as opposed to the metadata. Have you thought about this?
- It sounds like you're doing nearest neighbors on the vector representation. You may already know this, but LSH is a fast way to do this when you have many items.
- Do you embed all items from your different customers into the same vector space? That would be ideal from the POV of creating a pooled dataset that would be helpful for all future customers, but sounds tricky given that everyone likely has their own idiosyncratic system.
Best of luck! Lmk if you'd like to talk shop sometime, I also have several years of experience with recommender systems (my email is in my profile).
Comment · 6 years ago
Solo founder here.
- I started a newsletter with progress updates to keep myself accountable but found that as time goes on I haven't needed it as much
- Motivation can be variable over time and you need to keep making progress even on your down days. I think this is (one of the ways) where a co-founder is super valuable, as you can motivate each other when one of you is down.
On the theory that this thread will be a honeypot for other solo founders: if you're interested in potentially working together or just someone to bounce ideas off of, shoot me an email (address in my profile)
Comment · 6 years ago
Counter proposal from someone who is starting a business around GPT-3 (http://serendipityrecs.com/):
- If your whole app is a wrapper around the GPT-3 API then yes, you don't have much of a moat. But GPT-3 isn't some kind of perfect oracle, so you have lots of engineering challenges in the not-GPT-3 parts of your system where you can build your moat.
- GPT-4 does not render your current efforts moot. It takes time to build business/technical expertise in a domain. That's like saying it was pointless to build a web company in the 90s because web technologies/computing power was increasing so fast. You work with the technology you have, and then when it's time you upgrade.
- I disagree with the idea that you'll have hundreds of competitors. We're just starting to figure out what we can do with this technology. The space is wide open for someone to figure out a novel application.
AI dungeon was mentioned in another comment and serves as a good counter example. Where are all the AI dungeon competitors? How easy would it be to build a competing product to AI dungeon if you wanted to? And AI dungeon started with GPT-2, but incorporated GPT-3 into their product when it came out.
To summarize, starting a business around GPT-3 is a great idea. Stay realistic, be cognizant of GPT-3's limitations, but do realize that there is a genuine opportunity here.
Comment · 6 years ago
It's mostly just GPT-3. You do need to parse it, as far as GPT-3 is concerned it's just doing text in/text out. I'm doing some ranking logic behind the scenes to make the results more consistently reliable, but you could directly ask the model for recommendations. GPT-3 has a lot of knowledge of the real world from having read so much of the internet.
Comment · 6 years ago
I built a recommendation engine on top of GPT-3: http://serendipityrecs.com/
I think other applications exist, you just have to figure out how to extract what you want from the prompt/response interface.