pwang

What if you want to ensure that the same packages are available, built in a similar way, between your Mac and some linux servers? What if you need to share or ensure that your projects work between Linux and Windows? What if you are supporting a lot of less-sophisticated users across a number of different OSes, or even different versions of the same OS?

There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.

The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.

You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.

Fun joke but Anaconda has a track record of creating OSS and then turning it over to community governance. This includes the conda tool itself, libraries like bokeh, dask, numba, jupyterlab, and many more. And while PyScript project governance isn't in NumFOCUS, all of the code is permissively-licensed BSD/MIT.

The commercial licenses for the products and commercial repository is what supports all of this OSS development work.

Over 50% of Python users report using it for data-related things, e.g. data science, ML, etc. (Jetbrains poll)

That's numpy, pandas, sklearn, and friends.

I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.

I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.

Thanks for the feedback... I had no idea that pyenv was hosing conda envs. Do you have more details on this brokenness?

This is one of the reasons people use Anaconda/miniconda for non-data science work: conda environments are self-contained Python installs, so if you conda/pip install packages into those environments, they will not break each other. This design requirement arose from the specific needs of numerical computing (which always drags in a ton of system-level C/C++/FORTRAN dependencies), but is a generically useful design construct.

Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time.

Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)

Thanks!

FWIW, we are soon going to be releasing a much faster depedency resolver. We are also thinking hard about how best to address the "growing ecosystem" problem, in a future-proof way.

You only need a commercial license under certain circumstances, and it only pertains to accessing the default package repository at repo.anaconda.com. You can still use conda-forge etc. Conda itself (the packaging tool) remains open source.

Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.

There is a factual inaccuracy in the article: Beaker Browser is not "actually under the control of governments and big corporations". It is an independent open-source project.

Pirsig was more well known as I was coming of age in the 1990s. Seems to be less talked about among the younger crowds now?

The sequel to ZAMM is the far more illuminating book, IMO. His metaphysics of quality continue to be a useful metaphysical frame.