solso
Joined 14 years ago
Karma
285
Submissions
57
Member since
Jun 7
CS at Brave (https://search.brave.com/)
Twitter: @solso
Homepage: http://josepmpujol.net/
View on Hacker NewsRecent submissions
Comment · 4 years ago
Hi Vladimir! Good to see we are in sync. Glad to see Kagi doing well as well.
For the record, when I said "long before Kagi was even announced to the public." I meant exactly what I wrote, not that Kagi did not exist, it did.
Comment · 4 years ago
Disclaimer: Works at Brave, before at Tailcat.
Goggles white-paper was released more than a year ago, long before Kagi was even announced to the public.
Additionally, before Brave acquired Tailcat (Jan 2021) I had the pleasure to share the draft of the paper with Kagi's founder.
So no, there is no prior art.
Let me add that I do not claim that Goggles is prior art of Lenses either.
One of the key features of Goggles design is that the instructions, rules and filters are open and URL accessible.
A Goggle is not so much a personal preference configuration, but a way to collaborative come up with shareable and expandable search re-rankers.
Very different goals if you ask me. Of course, Goggles can be used for personal preferences exclusively, but that's not the use case we had in mind.
Comment · 4 years ago
An active choice is better than a passive one, if only because it requires an effort, in that respect the explicitness is an advantage over the typical personalization.
The article also mentions that Goggles will not stop polarization, it suffices to not exacerbate it.
No technology/system on any period of time has been able to suppress it, censorship included.
Disclaimer: I work at Brave search
Comment · 4 years ago
Mojeek is one of the very few players that are building their own index, my respects.
Comment · 4 years ago
There is plenty of comments discussing on the provenance of DDG results, including from Gabriel himself, which is the one we both have participated in,
"it is misleading to say our results just come from Bing."
Discussing how many sources can one bring together it's a distraction to not discuss the degree of dependency between DDG and Bing. More-so when claiming that others suffer from the same, which is factually incorrect for Brave search.
Comment · 4 years ago
This is Josep M. Pujol from Brave Search.
I'd like to correct some factually incorrect information regarding Brave Search.
Brave search crawls the web through the Web Discovery Project and has its own crawler, which fetches a bit more than 100M pages daily.
Brave search uses Bing API and Google fallback for about 8% of the results shown to the users, the remaining 92% are served from our own index, when we launched almost 1 year ago the number of results from 3rd parties was 13%.
There is no need to mention "multiple source" when a number can be given. The underlying theme here is not if DDG provides no value on top of Bing, it does, no one is questioning that. The question is whether DDG would be able to operate if Bing were to shut DDG down tomorrow.
If Bing and Google were to disappear tomorrow, for whatever reason, Brave search would continue to operate, that's the independence Brave search is building.
Comment · 5 years ago
You bring a very good point on the diversity of information sources, which is something we plan to attack in the near future with open ranking [0]
In my opinion having similar results to Google will facilitate adoption. After all, Google is pretty good for many types of queries (not all), and people in general have strong habits.
The fact that we are similar with our own index is great. It means that we have the power of deviating from it when needed, as we mature/evolve.
Allow me to repurposed your statement on why not use startpage if you want Google-like results: if tomorrow Google disappears (or for some reason becomes unusable), brave search will continue to operate as normal (similar to old Google). What will happen to searx or startpage? What till happen to ddg or swisscows if the provider turning bad is Microsoft. IMHO, no matter how much reranking or nice features they you put on top, unless you do not control the search results themselves, diversity can only be superficial.
Sorry for the "rant". Thanks a lot for the inputs and for updating the doc, appreciate it.
[0] https://brave.com/wp-content/uploads/2021/03/goggles.pdf
Comment · 5 years ago
Mixing with Google results only can happen after opt-in and only in Brave browser. You can see if a single query has been mixed clicking on the `Info`, or check the independence metrics on the `Settings` tab.
The fact that you see results similar to Google for popular queries is a by-product of the fact that our ranking is trained using anonymous query-log. There is plenty of references to the methodology (https://0x65.dev/).
The fact that we are similar to Google on certain types of queries, is good (at from the perspective of human assessment). It's easy to find other types of queries for which we are not similar to Google. It would be rather stupid if we were to "use google" on easy to solve queries but not on the complicated ones, don’t you think? In any case, very nice article besides a couple of miss-conceptions (like this one), will bookmark.
Disclaimer: work at Brave search, used to work at Cliqz
Comment · 5 years ago
It is trivial to de-anonymize if records are linkable, which is the case you mention on Dark Data DEFCON25. Another famous case was the de-anonymization of the Netflix data set.
However, you are assuming that HumanWeb data collection is record-linkable, which is not the case, precisely to avoid this attack.
If what is being collected is linkable: e.g. (user_id, url_1), ... (urser_id, url_n). No matter how you anonymize user_id, it will eventually leak. A single url containing personal identifiable information, e.g. a username, will compromise the whole session. No matter how sophisticated the user_id generation is. The real problem, privacy-wise, is the fact that record can be linked to the same origin. An attacker (or the collector) has the ability to know if two records have the same origin.
The anonymization of HumanWeb, however, ensures that linkability across data points is not present. Hence, an attacker cannot know if two records come from the same origin. As a consequence, the fact that one url might give away user data, for instance a username, it would not compromise all the urls sent by that person.
If you are interested in more details I recommend this article: https://0x65.dev/blog/2019-12-03/human-web-collecting-data-i...
[Disclaimer I'm one of the authors]
Comment · 5 years ago
The chosen excerpt omits the fact that it is predicated on the HumanWeb. In the technical papers above there is a more precise description on what and how was collected. There was no user tracking, session or history being sent as all data points are anonymous and record-unlinkable by the receiver. The vague language, required for a general audience journal, certainly does not help.
Comment · 5 years ago
Mozilla never did such a thing. The browsing history was never sent in any shape or form. As the journalistic article you quote states, Mozilla put in place the HumanWeb[1,2,3], which was a privacy preserving data collection which ensured record-unlinkability, hence no session or history. Anonymity was guaranteed and the framework was extensively tested by privacy researchers from both Cliqz and Mozilla.
Disclaimer: I worked at Cliqz.
[1]https://0x65.dev/blog/2019-12-02/is-data-collection-evil.htm... [2]https://0x65.dev/blog/2019-12-03/human-web-collecting-data-i... [3]https://0x65.dev/blog/2019-12-04/human-web-proxy-network-hpn...
Comment · 5 years ago
There was no tracking on Cliqz, nor it will be any in Brave. To know more about the underlying tech of Cliqz there are interesting posts at https://0x65.dev, some of them covering how signals are collected, data, but no tracking.
I did work at Cliqz and now I work at Brave. I can tell for a fact, that all data was, is and will be, record-unlinkable. That means that no-one, not me, not the government, not the ad department can reconstruct a session with your activity. Again, there is no tracking, full anonymity, Brave would not do it any other way.
Comment · 6 years ago
You are missing the entire point, "search" is sadly about advertising, not about the search itself :-)
Bing is interested serving DDG, Qwant, Ecosia and a lot of other unknown search engines because of the aggregated reach they provide for their ad network. Ad-networks only work if the aggregated audiences are massive, otherwise advertisers do not bother putting their ads there, only the top-3/5 ad networks get to see any action. So Bing wants/needs a bigger audience just to be on the game. They can grow in 2 paths: 1) increase Bing search reach (difficult), or 2) use partners with different value propositions.
Bing charges little for 1K query, 1USD officially but it gets cheaper, to zero :-) The real thing though, is that if you display Bing ads, you get a 70%-90% rev-share of the ad-revenue, which varies from country to country, something between $5 to $20 per 1K queries.
So, DDG basically gets around 5$ to 10$ net for each 1K, and can spend all that money on distribution and marketing so that they get even more users. Bing gets the rest, money, and what's more important, their ad-network continues to be competitive.
Everybody wins, right? :-/
Search is so cheap 1$/1K queries and people makes 2/3 queries per day, so $1/year/user (average). It makes no economic sense to build an alternative. Unless of course, you are building out of "ideals".
Comment · 6 years ago
>> You can collect data from users and still do not compromise their privacy
> This is definitionally false. The very collection of data compromises one's privacy, by nature of it having been collected.
That's not definitionally false, if it sounds false to you is because you have an implicit assumption that does not apply.
Data from users does not imply user sessions on the collector side (session as a set of multiple data points belonging to the same user).
If sessions are collected, then, privacy is impossible to guarantee. We are well aware of that, having worked on this problems for almost 20 years. But that's precisely what Cliqz never did. All messages from our users are record-unlinkable for us, meaning that we have no way to reconstruct any session.
If you are interested, check the HumanWeb posts on https://0x65.dev/ or the papers https://0x65.dev/pages/dissemination-cliqz.html
Comment · 6 years ago
There were no problems in 2017 or before, we were doing the same exactly the same during Firefox times (we went through security and privacy audits). Data collection is and always was safe wrt to privacy.
Why the ruckus then? Because some assume that is data is sent, privacy is compromised, period. They do not know how to do it, and they assume it's impossible. Instead of checking the claims for themselves (code is public, data can be inspected, documentation, etc.) they prefer to stick to their belief system, which is more comfortable and does not imply hard work. The press release that FF -- written by one of these people with a lot of biases and published without review -- did not help as it was misleading.
We did a big mistake back then. Instead of rebutting it, we chose to ignore the FUD assuming that facts would prevail. They did not.
Sadly the community is "scared", we have been congratulated and lauded by anyone who checked our systems. But never endorsed in public, there is little to gain and a lot to lose (you are getting a sneak preview right now).
Sad story, extremely frustrating too, but there is nothing we can do now.