donaldstufft

There is zero chance we require biometrics on PyPI.

You can have multiple accounts associated with a single project, so each person can have their own account and you just add them all as owners to the projects.

That can be annoying to keep in sync if you have a lot of projects, but we're rolling out organization support to make that easier for people.

TOTP is a good choice, and helps a lot on keeping everyone's account safe! Thanks a lot for taking the time to investigate your options

I do feel want to mention though (largely because I think it's pretty cool), that those security devices are using a private/public key system under the covers, and they're actually designed to be privacy friendly and phishing resistant. One of the problems with TOTP based 2FA is that since it's asking users to type the TOTP code into the website, they can be phished and tricked into typing their password and TOTP code into an attacker's website, who then quickly go and use it to sign onto their account.

Those hardware tokens prevent that phishing from happening. They basically create, on the fly, a public/private key pair that is bound to the domain name of the site in question, and then give the public key of that to the site. When you come back to log in again, the site tells the hardware token what public key it has, the token looks a the site's domain and determines if it has that key for that domain, and if it does it uses a signature to prove ownership of the private key.

It all ends up working really well, since the domain name (actually the protocol, domain name, and port) is part of the identify of the key pair, it is impossible for it to get entered on the wrong site, so it completely eliminates phishing. Then since every single site gets it's own brand new keypair generated for it, there's no way to determine that the hardware token used on Site A is the same as the hardware token used on Site B. So it's entirely privacy preserving as well!

The protocol is obviously a bit more complicated then that, but that's the general idea of it.

One important thing to remember here is that PyPI was originally started in 2002 as a weekend hack project that grew overtime to become the piece of critical infrastructure it is today. There's a lot of stuff in PyPI that exists as historical baggage and cruft and reviewing them just never bubbled up to be a priority. Likewise a lot of the policies it has have been added and grown overtime as something happened that caused us to need one.

On top of all of that, it's volunteer run and has been understaffed for basically it's entire life, so sitting down and figuring out a proper data retention policy that takes a holistic view of everything we have just never bubbled up.

In general I think we already do a pretty good job of collecting a minimal amount of data, and hopefully with proper policies we can do an even better job.

So there's a few things here:

Firstly, Debian's mirror network URLs allow a mirror operator to attack the base Debian.org site if they rely on cookies on debian.org (they may not, I'm not sure). Specifically the `ftp.<country>.debian.org` aliases cause this. On PyPI we did use cookies at the base url, so this was a non starter for us to keep.

The second thing here is that Debian and PyPI from a technical level about how mirrors are configured and hosted are generally similar. Meaning other than the above aliases, mirrors are expected to have their own domain and users are expected to configure apt or pip to point to a specific domain. Debian does have a command that will attempt to do that configuration for you to, to make it easier.

The third thing is that Debian's mirrors are as secure as the main repository is against attacks from a compromised mirror operator. This isn't the case in PyPI where you're forced to trust the mirror operator to serve you the correct packages. There is vestigal support for a scheme to support this in the mirroring PEP, but nothing ever really implemented it except the very old version of PyPI (none of the clients, etc). That scheme is also very insecure, so it doesn't really provide the security levels it was intended to.

The fourth thing is that a Debian mirror is easier to operate.

Packages on Debian don't live forever, as new versions are released old versions get removed, and as OS releases move into end of life, entire chunks of packages get rotated out. However on PyPI we don't have the concept of an OS release, or any sort of phasing out of old packages. All packages are valid for as long as the author makes them available. This means that the storage space to run a PyPI mirror (currently ~30TB) is a lot more than the storage space for a Debian mirror (~4TB).

On top of that the way apt and pip function are inherently different. Apt has users occasionally download the entire package set so that apt has a local copy of the metadata while pip asks the server for each package for the metadata (it does some light caching, but not a lot). This means that to discover what packages are available, apt might make one request a day while pip might make 100 requests for every invocation of pip. Packages on apt release a lot slower and less often than on pip. so many times people may not be needing to download more than a handful of packages, but people generally need to download a lot of packages from PyPI at a time.

I believe? the Debian mirroring protocol is rsync based, which is generally pretty reliable, while the PyPI mirroring protocol is a custom one which works, but it sometimes has a tendency to get "stuck" every few months and require operators to notice and fix themselves.

I suspect the differences between the strength of the mirror network is some combination of the two, but I suspect the the third and fourth things are the biggest differences, particularly when PyPI's CDN solved the problem in most users minds that would cause them to want to host or use a mirror.

The root of trust for uploads is the listed of signatures maintained by the archive administrators, flat out.

The requirement for having individual keys signed by Debian Developers just makes it easier for the archive administrators to decipher which keys they want to add to their root of trust. The upload system does not check those signatures at all, they do not need to exist in the slightest as far as the upload system is concerned.

Well, AFAIK it's not clear that in the US the courts have the right to compel someone to modify their software in that way. The FBI holds that it does, but so far it's been fought and they've given up when they've tried it. I think if such a thing were to happen, the fundamental ability to secure any software goes out the window. Even package signing, etc go out the window because they can just compel you to produce new software, signed with your existing key.

But let's step back a moment and presume that they do have that ability to compel. The first step here is that none of the PyPI Administrators are the legal owners of PyPI, so such an order would not be sent to any of us, but rather to the PSF itself. The PSF would then be on the hook to either comply or fight said hypothetical order, but individual members of the administration team would not be, and would be free to quit. They may not be able to say why they've quit, but quitting AFAIK would be entirely possible.

The PSF, while not having Apple's war chest, does retain counsel for dealing with things like this, and I can say personally I'd spend myself broke before I'd be willing to do so.

We are going to be implementing signing, and I'm hoping we'll be able to make strong progress on that soon.

> Before you can submit packages to Debian you have to get an existing Debian developer to sign your PGP key. In Debian the trust flows downward from older developers to newer developers.

This is not how signing works in Debian at a technical level. At at technical level uploading to Debian requires them to add your key to a list of keys maintained by the archive administrators. As a matter of policy those administrators ask you to get your key signed by an existing Debian Developer, but at no point does their upload infrastructure check that or use the Web of Trust.

This post is pretty misleading.

PyPI still fully supports mirrors (though it is becoming increasingly hard to run a full mirror of PyPI, last I looked a full copy of PyPI is about 30TB).

The only thing we ever removed was designating any particular mirror as official and an auto discovery protocol that was quite frankly extremely insecure and slow. That worked by giving every single mirror that wanted to be an "official" mirror for auto discovery a subdomain of `pypi.python.org`, labeled {a-z}.pypi.python.org. A client would determine what mirrors were available by querying last.pypi.python.org, which was a CNAME pointing to the last letter that we had assigned, that would tell it how many mirrors there were, then they could work backwards from that letter. So if the CNAME pointed to c.pypi.python.org, the client would know that a, b, and c existed.

Immediately you should be able to see a few problems with this:

- It is grossly insecure. Subdomains of a domain can set cookies on the parent domain, depending on ~things~ they can also read cookies.

- It does not scale past having 26 mirrors.

- It does not support removing a mirror, there can be no gaps in the letters.

So we needed to remove that auto discovery mechanism, which raised the question of what, if anything, we should replace it with?

Well at the time we had only ever made it up to g.pypi.python.org. So there was only 7 total mirrors that ever asked to become an official mirror. To my knowledge we never reused a letter, if a mirror went away we would just point the mirror back at the main PyPI instance. I don't remember exactly, but my email references there being only 4 mirrors left.

From my memory at the time, most of those 4 mirrors were regularly hours or days behind PyPI, would regularly go offline, etc.

But again, we never stopped anyone from running a mirror, we just removed the auto discovery mechanism and required them get their own domain name. We even linked to a third party site that would index all of the servers and keep track of how "fresh" they were, and other stats (at least until that site went away).

Running a mirror of PyPI is a non trivial undertaking, and most people simply don't want to do that. We never had many mirrors of PyPI running, and as it turns out once we improved PyPI most people decided they simply didn't care to use a mirror and preferred to just use PyPI, but still to this day we support anyone to mirror us.

I would resign from PyPI before I ever allowed a backdoor to be installed.

I haven't explicitly asked, but I would be very surprised if any of the other PyPI admins felt differently.

I agree that long lived GPG keys are a bad idea, which is yet another reason why the feature was a bad one, because it could only ever work with non expiring GPG keys.

Well PGP signatures has been part of PyPI for 18 years now, if someone was going to build a secure system on top of that, they would have by now.

PyPI should implement it though, because fundamentally the question of who is authorized to release for "requests" on PyPI is a question of who PyPI authorizes to release for that.

Sometimes? There's no global policy of doing it in Debian, it's up to individual package maintainers inside of Debian to enable it (it defaults to off AFAIK) and to hardcode the key that they expect the package to be signed by.

In the cases that it is used, AFAIK it is only used by Debian's uscan program, which is sort of like the Debian version of Dependabot, it tells them when there is a new version of something to package. As far as I know, the process of packaging that new version is still manual, and relies on the maintainer downloading the package and packaging it, so they may or may not use the signature in that case.

How useful this is, is up for debate. Many years ago when I first started taking over releasing pip, that caused the pip GPG key to change, and the reaction of the Debian maintainer at the time was to just comment out the signature bit and fall back to no signature.

I don't believe that Maven Central's use of GPG is providing a meaningful security control here, so I would dispute the idea that they're doing it "right".