senecaso

Why don't sites just start publishing a dump of their site that crawlers could pull instead? I realize that won't work for dynamic content, but surely a lot of these "small" sites that are out there which are currently getting hammered, are not purely dynamic content?

Maybe we could just publish a dump, in a standard format (WARC?), at a well-known address, and have the crawlers check there? The content could be regularly updated, and use an etag/etc so that crawlers know when its been updated.

I suspect that even some dynamic sites could essentially snapshot themselves periodically, maybe once every few hours, and put it up for download to satiate these crawlers while keeping the bulk of the serving capacity for actual humans.

You site looks good, and your results are fantastic! Job well done. I did hit a server error though, so obviously still some issues to work out, but overall, really well done. Moving to semantic search was one of my top priorities before we went under, but I struggled to justify the costs of it as we were operating on a shoestring budget.

Best of luck to you and your team on user acquisition!

I didnt know about the link to checkout. That's a slightly nicer user experience for sure. Still, its confusing for users who want to do more shopping at the same time. I had users who clicked on a number of items, clicked "add to cart" in each one (all different shops), and then couldn't figure out how to checkout on the main site afterwards! Obviously people were looking for a more complete one-stop-shopping experience than I was providing at the time.

we did have channels on various sites, yes. However, its difficult to maintain a steady stream of content there for people to consume. Not only that, but you have the same discoverability problems as you do for the main site. Also, a blog outlining how you built the site may be of limited value. At least my experience on that front was it would generate short-lived bursts of traffic, but wouldnt generate returning users. So I think those articles were mostly appealing to technical users, and not necessarily users who were looking to do some shopping. Of course technical users do also shop, but after reading a technical article, they probably arent looking to immediately shop, and without some other mechanism putting the site in front of them again when they needed to shop, we would miss the opportunity.

We were intentionally limiting the number of products and shops we were indexing due to opex. We needed to keep it low enough to provide ourselves with enough runway to keep things floating for longer.

pricerunner is another site which operates in a similar space. We had plans to build out the price tracking and a number of other features, so that we would appeal more to users who had your use cases. Sadly, we weren't getting enough traction. We did have regular users from the EU, but we simply couldn't seem to get in front of enough eyeballs for it to matter. At least at first, I expect that a large amount of your traffic to a new site like this has to be driven by Google, and we failed on that front as well. I'm not an SEO expert, so there were likely many things we did wrong or didn't even do which lead to this situation.

re: a sales platform, that's a pretty big challenge to take on, which would require massive investment up front. Not sure thats a viable route for most. We did have plans to address the "without hoping to various websites" problem, as we identified that as problematic for users very early on. The solution was relatively simple, but required more money to build out. We simply ran out of funds before we could get there.

I doubt it would be that easy, but, ya, using some form of automation is necessary. We devised a few rudimentary way to filter out the chaff, and it did quite well to remove the garbage. Still some would slip through, so it still required vigilance to remove them when you happen to see them.

Many shops do double list, this is true. However, I don't think its a solved problem. There are many people who do not want to shop on Amazon for their own reasons. There are also people who want to shop locally, and Amazon provides no mechanism to do so (that I'm aware of). There are also many smaller shops who simply cannot afford to list on Amazon, as there are considerable fees associated with running a successful business there. It was these smaller shops who we were initially building to serve, to provide a funnel for them.

Still, there were problems with our solution that if addressed may have provided a better market fit. If we had had more runway, we would have worked to address them, but that simply wasn't in the cards.

Our business model revolved around referrals, so lack of users directly translated to lack of revenue. While its true that even if we had millions of users but none of them were buying sponsored items we would have had a revenue problem, that wasn't the problem we were facing, as the few users we did have were in fact purchasing sponsored items.

we didn't, no, but we spent a lot of time sifting through our catalog, and there was a _tremendous_ amount of crap in there. We manually curated and purged shops that were obviously just dropshipping or looked like out-right scams.

Ya, curation is sadly required in the Shopify ecosystem. There are millions of shops, there is a tonne of garbage. Its also difficult (but not impossible) to properly classify items so that you can better target results for a given query. One of the first problems that anyone attempting this will run into is the amount of mature content available on Shopify shops. Innocent queries turn up many NSFW images that may offend some users, so you have to be able to get on top of that one pretty quick.

I remember in once case, I found what appeared to be an escort service listing "models" on Shopify. It was super creepy. I needed to get in front of that one pretty quick as well, as it was turning up in results.

We had no capital, so advertising or solutions that basically involved "throwing money at the problem" were off the table for us.

We spent time posting in forums helping people find items they were looking for, and we had a few posts here on HN that generated short-lived, explosive traffic bursts. I remember those days we had posts get picked up on HN, it was always an exciting night!

We were looking at influencers and getting our name getting bloggers to talk about us, but, again, without capital, our options were very limited here. I'm sure someone with more of a marketing background would have found a bunch of ways we could have generated organic user growth, but neither me or my business partner had that skill set.

If I were to do it again, I think I would try to get someone with a marketing background involved to help gain traction. Without that, even the best product in the world will die of starvation if no one finds it.

Yes, this has been available for a few years now. Initially, they only indexed a very small number of shops, so it was less useful. Based on a few queries, it seems like the are still using some form of text-based search with rank boosting. Seems like they still aren't searching their entire base of shops, but they have increased the number of shops for sure, and they seem to be continuing to invest in the product, which is nice. It seems more useful now than it did the last time I checked!

I hope you have better luck than I did!

A few years ago, my partner and I built vendazzo.com (now defunct). It was an e-commerce search engine on products listed on Shopify shops (sound familiar? :)). At the time, we had > 100m products listed, and I don't remember how many shops we were indexing.. over 100k I think, but we had access to over a million. Overall, I think your approach is very similar to ours, but we managed to keep our costs lower. At the time, we were spending ~$550/mo, and our search times were under 300ms. We had established partnerships with a number of shops, and we had a few users, but not nearly enough. That's where the wheels came off. The site operated for over a year, but the monthly costs wore us down until we finally decided to pull the plug.

I still maintain that this is a good idea, and constantly have to fight off the urge to "try again", however, to do it properly, I think funding would be necessary, or finding some way to organically gain a lot of users.

Looking back, there are things I could have done to reduce my opex further, but in the end, it still wouldn't have mattered if I couldn't figure out how to acquire users.

I was thinking more along the lines of a public dialog, where a user could post a comment like "This is a great idea! However, if you could make it do X and Y, it would be even better!"

Getting access to people's contact information would be useful at a later point in the process, but early on I think immediate feedback would be the most beneficial. Even something along the lines of a HN thread like this one would be enough engagement at first. In fact, perhaps that is even sufficient :)