Jacob Kaplan-Moss kicked off this fascinating conversation about [uv](https://docs.astral.sh/uv/) on Mastodon recently. It's worth reading the whole thing, which includes input from a whole range of influential Python community members …
I don’t think it’s a dream of “everything in python”, but “python tools for python development”. It means users of the language can contribute to the tooling.
Fair, but at some point the “dream” breaks down. Python itself is written in C and plenty of packages, some vital, rely on C or Cython (or fortran) and rust now more and more. So why not the tooling that’s used all the time and doing some hard work and often in build/testing cycles?
If Guido had packaging and project management included in the standard library from ages ago, with parts written in C, no one would bat an eye lid whether users could contribute to that part of the system. Instead, they’d celebrate the “batteries included”, “ease of use” and “zen”-like achievements of the language.
Somewhere in Simon’s blog post he links to a blog post by Armin on this point, which is that the aim is to “win”, to make a singular tool that is better than all the others and which becomes the standard that everyone uses so that the language can move on from this era of chaos. With that motive, the ability for everyday users to contribute is no longer a priority.
C is the implementation language of Python and is always going to be there.
Cython is a very similar language to Python and designed to be very familiar to Python writers.
Fortran is the language that BLAS and similar libraries were historically implemented in since the 70s. Nobody in the python community has to write Fortran today. Those libraries are wrapped.
Rust is none of the above. Bringing it into the mix adds a new barrier.
Or new possibilities… See: UV, pixi, hatch, ruff, polar, pyarrow, pydantic, data fusion, deltalake, fastuuid, granian, Robyn…
I’m not a c expert and I’m not comfortable in writing python extensions in C…
But with rust you have the compiler that, if you constraint yourself to the safe part of the rust language, is checking for you for several stupid issues. In rust, I can focus on fixing logical and other implementation errors. Coming from python I feel much more at home with rust (async, yield, iterator, generator, closure, match, walrus, etc) than with C.
Whether to use a non-Python language where appropriate
Whether to use rust over C, which is already heavily used and fundamental in the ecosystem (I think we can put cython and Fortran to the side)
I think these questions are mostly independent.
If the chief criterion is accessibility to the Python user base, issue 2 isn’t a problem IMO. One could argue, as does @eraclito@feddit.it in this thread, that in fact rust provides benefits along these lines that C doesn’t. Rust being influenced by Python adds weight to that. Either way though, people like and want to program in rust and have provided marked success so far in the Python ecosystem (as eraclito cites). It’s still a new-ish language, but if the core issue is C v Rust, it’s probably best to address it on those terms.
I think for python tooling the choice is Python Vs Rust. C isn’t in the mix either.
people like and want to program in rust
I think there’s a survivor bias going on here. Those that have tried rust and stuck with it, they also like it. Far more people in the python community haven’t tried it, or have and not stuck with it. I like and want to program Haskell. I’m not going to write python tools in it because the community won’t appreciate it.
Tools should be maintained by those that use them. Python doesn’t want to rely on the portion of the venn diagram that are rust and python users because that pool of people is much smaller.
I think for python tooling the choice is Python Vs Rust. C isn’t in the mix either.
That seems fair. Though I recall Mumba making headway (at least in the anaconda / conda space) and it is a C++ project. AFAIU, their underlying internals have now been folded into conda, which would mean a fairly popular, and arguably successful portion of the tooling ecosystem (I tended to reach for conda and recommend the same to many) is reliant on a C++ foundation.
On the whole, I imagine this is a good thing as the biggest issue Conda had was performance when trying to resolve packaging environments and versions.
So, including C++ as part of C (which is probably fair for the purposes of this discussion), I don’t think C is out of the mix either. Should there ever be a push to fold something into core python, using C would probably come back into the picture too.
I think there’s a survivor bias going on here.
Your survivorship bias point on rust makes a lot of sense … there’s certainly some push back against its evangelists and that’s fair (as someone who’s learnt the language a bit). Though I think it’s fair to point out the success stories are “survivorship” stories worth noting.
But it seems we probably come back to whether fundamental tooling should be done in python or a more performant stack. And I think we just disagree here. I want the tooling to “just work” and work well and personally don’t hold nearly as much interest in being able to contribute to it as I do any other python project. If that can be done in python, all the better, but I’m personally not convinced (my experience with conda, while it was a pure python project, is informative for me here)
Personally I think python should have paid more attention to both built-in tooling (again, I think it’s important to point out how much of this is simply Guido’s “I don’t want to do that” that probably wouldn’t be tolerated these days) and built-in options for more performance (by maybe taking pypy and JIT-ing more seriously).
Maybe the GIL-less work and more performant python tricks coming down the line will make your argument more compelling to people like me.
(Thanks very much for the chat BTW, I personally appreciate your perspective as much as I’m arguing with you)
Part of it might be that I’m often having similar arguments with the team I run about introducing dependencies.
Engineers have a tendency to want to use the perfect tool for a job at the expense of other concerns. It could be ease of maintenance, availability of the skill-set, user experience, or whatever. If there’s pushback it’s normally that they are putting their own priorities above other people’s equally valid concerns.
Often I’m telling people to step-back. Stop pushing, listen to the resistance and learn from it. Maybe I’m on a bit of a crusade when I see similar situations in open-source.
I don’t think y’all quite grok what uv makes so special due to your seniority. The speed is really cool, but the reason Rust is elemental is that it’s one compiled blob that can be used to bootstrap and maintain a Python development. A blob that will never break because someone upgraded Homebrew, ran pip install or any other creative way people found to fuck up their installations. Python has shown to be a terrible tech to maintain Python
Yep! And likely the lesson to take from it for Python in general. The general utility of a singular foundation that the rest of the ecosystem can be built out from.
Even that it’s compiled is kinda beside the point. There could have been a single Python tool written in Python and bundled with its own Python runtime. But Guido never wanted to do projects and package management and so it’s been left as the one battery definitely not included.
I don’t think it’s a dream of “everything in python”, but “python tools for python development”. It means users of the language can contribute to the tooling.
Fair, but at some point the “dream” breaks down. Python itself is written in C and plenty of packages, some vital, rely on C or Cython (or fortran) and rust now more and more. So why not the tooling that’s used all the time and doing some hard work and often in build/testing cycles?
If Guido had packaging and project management included in the standard library from ages ago, with parts written in C, no one would bat an eye lid whether users could contribute to that part of the system. Instead, they’d celebrate the “batteries included”, “ease of use” and “zen”-like achievements of the language.
Somewhere in Simon’s blog post he links to a blog post by Armin on this point, which is that the aim is to “win”, to make a singular tool that is better than all the others and which becomes the standard that everyone uses so that the language can move on from this era of chaos. With that motive, the ability for everyday users to contribute is no longer a priority.
Those languages bring different things though:
Python is the language the tool is for
C is the implementation language of Python and is always going to be there.
Cython is a very similar language to Python and designed to be very familiar to Python writers.
Fortran is the language that BLAS and similar libraries were historically implemented in since the 70s. Nobody in the python community has to write Fortran today. Those libraries are wrapped.
Rust is none of the above. Bringing it into the mix adds a new barrier.
Or new possibilities… See: UV, pixi, hatch, ruff, polar, pyarrow, pydantic, data fusion, deltalake, fastuuid, granian, Robyn…
I’m not a c expert and I’m not comfortable in writing python extensions in C…
But with rust you have the compiler that, if you constraint yourself to the safe part of the rust language, is checking for you for several stupid issues. In rust, I can focus on fixing logical and other implementation errors. Coming from python I feel much more at home with rust (async, yield, iterator, generator, closure, match, walrus, etc) than with C.
I feel like this is conflating two questions now.
I think these questions are mostly independent.
If the chief criterion is accessibility to the Python user base, issue 2 isn’t a problem IMO. One could argue, as does @eraclito@feddit.it in this thread, that in fact rust provides benefits along these lines that C doesn’t. Rust being influenced by Python adds weight to that. Either way though, people like and want to program in rust and have provided marked success so far in the Python ecosystem (as eraclito cites). It’s still a new-ish language, but if the core issue is C v Rust, it’s probably best to address it on those terms.
I think for python tooling the choice is Python Vs Rust. C isn’t in the mix either.
I think there’s a survivor bias going on here. Those that have tried rust and stuck with it, they also like it. Far more people in the python community haven’t tried it, or have and not stuck with it. I like and want to program Haskell. I’m not going to write python tools in it because the community won’t appreciate it.
Tools should be maintained by those that use them. Python doesn’t want to rely on the portion of the venn diagram that are rust and python users because that pool of people is much smaller.
That seems fair. Though I recall Mumba making headway (at least in the anaconda / conda space) and it is a C++ project. AFAIU, their underlying internals have now been folded into conda, which would mean a fairly popular, and arguably successful portion of the tooling ecosystem (I tended to reach for conda and recommend the same to many) is reliant on a C++ foundation.
On the whole, I imagine this is a good thing as the biggest issue Conda had was performance when trying to resolve packaging environments and versions.
So, including C++ as part of C (which is probably fair for the purposes of this discussion), I don’t think C is out of the mix either. Should there ever be a push to fold something into core python, using C would probably come back into the picture too.
Your survivorship bias point on rust makes a lot of sense … there’s certainly some push back against its evangelists and that’s fair (as someone who’s learnt the language a bit). Though I think it’s fair to point out the success stories are “survivorship” stories worth noting.
But it seems we probably come back to whether fundamental tooling should be done in python or a more performant stack. And I think we just disagree here. I want the tooling to “just work” and work well and personally don’t hold nearly as much interest in being able to contribute to it as I do any other python project. If that can be done in python, all the better, but I’m personally not convinced (my experience with conda, while it was a pure python project, is informative for me here)
Personally I think python should have paid more attention to both built-in tooling (again, I think it’s important to point out how much of this is simply Guido’s “I don’t want to do that” that probably wouldn’t be tolerated these days) and built-in options for more performance (by maybe taking pypy and JIT-ing more seriously).
Maybe the GIL-less work and more performant python tricks coming down the line will make your argument more compelling to people like me.
(Thanks very much for the chat BTW, I personally appreciate your perspective as much as I’m arguing with you)
Part of it might be that I’m often having similar arguments with the team I run about introducing dependencies.
Engineers have a tendency to want to use the perfect tool for a job at the expense of other concerns. It could be ease of maintenance, availability of the skill-set, user experience, or whatever. If there’s pushback it’s normally that they are putting their own priorities above other people’s equally valid concerns.
Often I’m telling people to step-back. Stop pushing, listen to the resistance and learn from it. Maybe I’m on a bit of a crusade when I see similar situations in open-source.
Oh I’m with you there! And otherwise totally understandable.
I think this pretty much sums it up: https://mastodon.social/@hynek/113094547139925962
Yep! And likely the lesson to take from it for Python in general. The general utility of a singular foundation that the rest of the ecosystem can be built out from.
Even that it’s compiled is kinda beside the point. There could have been a single Python tool written in Python and bundled with its own Python runtime. But Guido never wanted to do projects and package management and so it’s been left as the one battery definitely not included.