brettcannon

There's a couple of reasons for the current situation in terms of the plethora of tools.

One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).

Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.

Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.

And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).

And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.

Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.

It was on purpose because TOML has not reached 1.0 yet and breaking people using a module in the stdlib for parsing TOML because TOML itself changed would be an absolute mess. Plus not a single packaging project said it was an issue to either depend on or vendor a TOML parser.

Once TOML hits 1.0 I plan to work on getting a TOML parser into Python's stdlib.

(disclaimer: I am one of the co-authors of PEP 518 that apparently handled things in way that was "a little absurd" .)

I'm not sure how old you are, but if you don't know the name "Borland" you might not realize how novel it is to have a compiler on every major OS. While Python's age means it has been around to see a lot of changes occur, that also means it had pre-existing practices and code to also consider whenever a shift in the industry occurred. Plus in hindsight things look obvious, but during the transition it isn't obvious what will stick and what won't.

It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:

- Compilers becoming available on all OSs - Linux becoming a thing - Laptops becoming common - Smartphones - Web-based email clients

As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).

The specific restriction from my understanding is that a US non-profit that accepts money for a specific thing must spend every penny of that money only on that thing. So if the PSF was given $1,000,000 to solve a specific problem and while trying to do it they go bankrupt they still couldn't touch that money to stay afloat. So non-profits avoid that situation since they typically have enough to survive a year or so, but not enough to be able to restrict what they might need to do with their cash.

It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).

Raymond came up with the idea, PyPy implemented it, and then INADA Naoki implemented it for CPython.

Poetry tries to help you manage your whole project while flit is just for building your library (and optionally uploading; you can always use twine to do the upload). It's all-in-one versus one-job-one-tool.

If you use a grepping tool like ripgrep it will automatically ignore everything in your `.gitignore`, so at least for that specific case you're covered.