티스토리 수익 글 보기
To read more about the performance work, and see lots of plots, I wrote a post on it: https://iscinumpy.dev/post/packaging-faster
For the release notes: Release 26.0rc1 · pypa/packaging · GitHub
Please try it out before the final release, which should be in about a week assuming no blockers.
2 posts – 2 participants
]]>This presents an interesting compatibility challenge for Python packaging standards, as I don’t think any of the PEPs that introduce toml based files (pyproject.toml and pylock.toml) specify which version of TOML should be used.
I assume that at some point TOML 1.1 will be added to tomli and then those changes will be upstreamed to the standard library, but probably not back-ported.
My question here is what would be a good strategy for adoption? My preference is to adopt reading it as quickly as possible, but don’t specially call it out, e.g. not having something in the changelog like”now supporting TOML 1.1″, and if possible emit a user warning when 1.1 specific features are used, at least for a few years.
But I’m sure others have opinions or perhaps even experiencing in adopting new format versions.
23 posts – 8 participants
]]>I’m opening this to raise awareness of a PPO change I proposed a few weeks ago, here:
TL;DR: .pypirc is the file format that twine (and others read) to access index configuration and credentials. It was never standardized by a PEP, so it’s one of the standards on PPO that sort of exists in a grandfathered state.
I’ve proposed a small change to the format, to stipulate that .pypirc SHOULD always be UTF-8 encoded, although tools MAY handle other encodings. This merely codifies the status quo, but it was implicit rather than explicit before.
I can’t imagine this would be a very controversial change, but I wanted to raise awareness here rather than merging it directly, since it exists in a weird gray area around pre-PyPA packaging standards. I welcome any thoughts/feedback!
(For background context, see the linked Twine PR in that PPO PR. The TL;DR is that some users end up with differently encoded .pypirc files through no fault of their own, and there’s no clear decision procedure for how Twine or other tools should handle that case.)
1 post – 1 participant
]]>Abstract
This PEP proposes introducing JSON encoded core metadata and wheel file format metadata files in Python packages. Python package metadata (“core metadata”) was first defined in PEP 241 to use RFC 822 email headers to encode information about packages. This was reasonable in 2001; email messages were the only widely used, standardized text format that had a parser in the standard library. However, issues with handling different encodings, differing handling of line breaks, and other differences between implementations have caused numerous packaging bugs. Using the JSON format for encoding metadata files would eliminate a wide range of these potential issues.
The full PEP text is published: PEP 819 – JSON Package Metadata | peps.python.org
Interested to see what folks think!
19 posts – 7 participants
]]>I would like to start a discussion on what is still missing, and what would be required, to make loongarch64 a well-supported and standardized platform in the Python packaging ecosystem (PyPI / manylinux / musllinux). The long-term goal would be to reach what could be considered first-class support, meaning a platform with officially defined tags, upstream tool support, and reproducible build environments.
From the loongarch64 community side, much of the low-level infrastructure is already in place. We would like guidance on the expected and recommended path toward upstream and standardized support within PyPA.
Current status
From the perspective of tooling and operating system support, loongarch64 has reached a reasonably mature state.
Build and tooling infrastructure
CI and emulation
loongarch64 can already be emulated in CI environments via docker/setup-qemu-action.
Core packaging tools
Key tools in the Python packaging workflow, including patchelf, swig, uv, and auditwheel, support loongarch64.
Cross-build environments
Community-maintained manylinux-cross and musllinux-cross environments are available for loongarch64.
Operating system support
glibc-based environments
loongarch64 is supported by several glibc-based Linux environments that provide the ABI and runtime components typically required by the manylinux specification.
musl-based environments
Alpine Linux officially supports loongarch64 and maintains container images, although official Docker Hub images are not yet published.
CPython status
- loongarch64 support has been upstreamed into CPython.
- Python ≥ 3.12 recognizes the GNU triplet without requiring downstream patches.
Gaps towards standardized integration
Despite the above progress, loongarch64 is not yet a first-class target in the official Python packaging workflow.
Upstream build tooling
Building wheels for loongarch64 currently relies on manylinux/musllinux images maintained by the loongarch64 community, as well as patched or forked versions of cibuildwheel. Upstreaming this support would significantly improve accessibility for package maintainers.
Availability of manylinux build roots
While there are glibc-based environments on loongarch64 that appear to meet the technical requirements of PEP 600, there is currently a lack of publicly available, long-term-maintained container images that can serve as reproducible manylinux build roots for this architecture. This gap makes it difficult to evaluate, standardize, and upstream loongarch64 support, even when the underlying ABI and runtime requirements are satisfied.
manylinux / musllinux platform definitions
loongarch64 is not yet included in the official manylinux or musllinux platform tags defined by existing PEPs, which prevents PyPI from recognizing loongarch64 wheels as official artifacts.
Topics for discussion
We would greatly appreciate guidance from the community on the following questions:
- What is the recommended path to introduce loongarch64 into the official manylinux and/or musllinux specifications (PEPs)?
- Would it be acceptable for the loongarch64 community to provide and maintain a reference manylinux build root for evaluation purposes, based on a glibc environment that meets the requirements of PEP 600?
- Are there specific technical or policy prerequisites from the PyPA side for accepting a new architecture?
- What level of CI maturity or hardware availability is typically expected before upstream adoption?
We are keen to collaborate and continue maintaining the necessary tooling, with the goal of moving loongarch64 from “community-supported” to “officially standardized” status.
Any feedback or pointers would be very welcome.
3 posts – 3 participants
]]>This proposal recommends that Python package registries adopt RFC 9457 (Problem Details for HTTP APIs) as a standard format for error responses in PEP 503 (Simple Repository API) and PEP 691 (JSON-based Simple API). Currently, when pip encounters HTTP errors from package registries, it displays the status code and reason phrase from the HTTP response. While HTTP/1.1 includes both fields, HTTP/2 removed the reason phrase entirely, leaving developers with only numeric status codes (e.g., “403”) and no context about why a request failed. This creates a significant user experience problem as HTTP/2 adoption accelerates. This proposal enables registries to provide structured, human-readable error information through RFC 9457’s standardized JSON format, which works consistently across both HTTP/1.1 and HTTP/2. The implementation would be optional and backwards-compatible, with clients falling back to existing behavior for non-RFC 9457 responses.
Motivation
When developers retrieve Python packages from external clients (such as UV), some registries provide error context by embedding messages in the HTTP/1.1 reason phrase field. However, HTTP/2 removed this field entirely, leaving developers with only numeric status codes under modern infrastructure. Without understanding why a package was blocked, developers must contact administrators to diagnose issues, creating unnecessary friction.
Real-World Scenario (Nexus Repository Manager):
The following is how the developer sees a custom error message when trying to download component urllib3 via pip (and using HTTP/1.1 reason phrase to display a custom message):
pip download urllib3
Looking in indexes: http://<hostName>/repository/pypi-proxy/simple
Collecting urllib3
ERROR: HTTP error 403 while getting http://<hostName>/repository/pypi-proxy/packages/urllib3/...
ERROR: Could not install requirement urllib3 because of HTTP error 403 Client Error: Requested item is quarantined ...
Technical Problems with This Approach:
- HTTP/2 Incompatibility: HTTP/2 removed the reason phrase field entirely (RFC 9113). Only the numeric status code is transmitted, meaning custom messages are lost when registries use HTTP/2.
- Not Standardized: The reason phrase format is not structured, making it impossible for clients to parse and display intelligently.
- Limited Length: Reason phrases have practical length limitations and cannot convey complex structured information (e.g. multiple errors, links, metadata).
With RFC 9457, this becomes:
{
“status”: 403,
“title”: “Package quarantined by security policy”,
“detail”: “urllib3 version 2.6.2 has been quarantined due to security policy violations.”
}
Ecosystem Context
RFC 9457 is already mandated in:
- PEP 807 Index support for Trusted Publishing
- PEP 694 Upload 2.0 API for Python Package Indexes
Rationale
Why RFC 9457?
- RFC 9457 is a well-established IETF standard (published July 2023) used by major APIs.
- Provides machine-readable + human-readable error information.
- Works consistently across HTTP/1.1 and HTTP/2
- Allows registries to add custom fields for domain-specific context
Why Optional/Recommended?
Making this recommended rather than mandatory allows:
- Gradual adoption across diverse registry implementations.
- No breaking changes for existing indices.
- Progressive enhancement: clients can improve UX when RFC 9457 is detected.
Another option to support custom error messages from registries is to add a new custom header in the pip response. Similar work was done in the HuggingFace ecosystem where the header X-Error-Message is used. However, custom headers are non-standard and don’t provide structured, machine-readable error details like RFC 9457.
Specification (High-Level)
Scope: PEP 503 (Simple Repository API) and PEP 691 (JSON-based Simple API)
When Applied: HTTP error responses (4xx and 5xx status codes) from registry endpoints when pip interacts with the registry.
Minimum Required Fields:
- status (integer): HTTP status code.
- title (string): Short, human-readable summary.
- detail (string): Detailed explanation of the error.
HTTP Content-Type: application/problem+json
Example Response:
{
“status”: 403,
“title”: “Package blocked by security policy”,
“detail”: “packageABC version 1.19.0 contains CVE-2021-12345 (Critical severity). Please upgrade to version 1.21.0 or later, for more details visit:``http://host.example.com``”
}
Client Behavior:
- Clients like pip may parse and display RFC 9457 responses.
- If
Content-Type:application/problem+jsonis detected, clients can extract structured error details. - If not present, clients continue using existing error handling (fully backwards-compatible)
Related work:
- Draft PR implementing RFC 9457 for PIP: Add RFC9457 support problem details handling by luisgcoding · Pull Request #13620 · pypa/pip · GitHub
- UV project already implements RFC 9457 for error responses: Implement RFC9457 compliant messaging by doddi · Pull Request #16199 · astral-sh/uv · GitHub
What does the community think about this?. We are maintainers of a private registry and have experienced these error reporting limitations firsthand. We recognize we may not have full visibility into ecosystem-wide impacts, so we’re seeking community feedback before proposing a formal PEP.
I’m happy to collaborate with @woodruffw and others interested in moving this forward. Looking for feedback and buy-in before drafting a formal PEP.
9 posts – 5 participants
]]>Please consider the following screenshot:
The banner is taking roughly 30% of vertical space, and making reading the site really inconvenient. The “donate today” button doesn’t hide it. Where you’d expect the closing button, there’s a button to make it bigger, and then smaller again. The bigger version has no “dismiss” button, as far as I can see. In the smaller version, the “no really, dismiss” link is barely visible, with a contrast ratio of 1.75:1 and uses a font that is smaller than anything else on the website, constituting a major accessibility problem.
Please reconsider. While raising money for Python is important, I don’t think it really justifies such an approach, and to be honest I doubt that it is actually going to convince more people into donating.
12 posts – 9 participants
]]>First, some informal definitions:
- A release is a specific version of a project, e.g.
pandas 2.3.3 - A distribution is a concrete file that containers the package, e.g.
pandas-2.3.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
A release may have one to many distributions, but a distribution belongs to exactly one release. In any given environment there may be many valid distributions available for a release. There is no guarantee, outside what is specified in the filename, that the metadata of different distributions is consistent.
When a tool has to resolve dependencies it is faced with a problem: If a given distribution is incompatible with existing requirements it does not mean other distributions of the same release are incompatible. For example, a user has the requirements foo>=1 bar!=2, the latest release of foo is 2.0.0, and the first distribution checked specifies bar==2, one could check all other distributions of foo 2.0.0 or one could mark the foo 2.0.0 release as not compatible with the users requirements.
In practice, as far as I am aware, all Python package dependency revolvers mark the release as not compatible, rather than the individual distribution, there are a number of practical reasons for not iterating through all compatible distributions for each release:
- It saves significant amount of IO
- It saves having to build the source distribution for every incompatible release
- It simplifies dependency resolution/satisfiability algorithm implementations
- It reduces dependency resolution/satisfiability algorithm computational costs, as a 2x increase in the number of possible choices for each project is an exponential increase in the possible solution space
As a consequence it is good practice to not specify different dependency metadata for different distributions of the same release. Where some dependencies are platform or API specific, it is ideal if these can be represented using markers, so tools that statically resolve multiple platforms or APIs can make one IO call per release.
It is also important that future standards not require resolver tools to make multiple IO calls for dependency metadata for each release. Further future standards should avoid requirements that invalidate basic assumptions in dependency resolution/satisfiability algorithms, such as dependencies that are affected by the current internal state of the resolver, which for example would invalidate nogood learning.
15 posts – 6 participants
]]>My goal was to be able to install in one call “locked” environments described in locked files saved in repositories on the web (hosted on GitHub or GitLab instances).
For example, one can install a good controlled and tested environment for Fluidsim supporting parallel computing with MPI with just
uvx install-locked-env https://github.com/fluiddyn/fluidsim/tree/branch/default/pixi-envs/env-fluidsim-mpi
From my point of view, this tool is useful for my students and colleagues and this is why I wrote it.
With it, it becomes very simple (basically copy/paste one command) to use and reuse prepared locked environments. It seems to me that it can in particular be very attractive for education. The teacher prepares her/his environment with tools like Pixi or UV and just commits few files in a repo on the web. The students create exactly the same environment locally on their computers, with only one very simple command.
For applications for which “in browser” or “on server“ execution is not adapted, it seems to me that it’s interesting. Moreover, it is also extremely simple for the teacher compared to alternative solutions like JupyterHub, pixi pack or even JupyterLite.
Currently, install-locked-env is just a prototype and only Pixi environments are supported.
I post here to get some feedback about this idea. Since it seems to me that nothing similar currently exist, there might be a good reason for this and it might actually be a bad idea.
I thought a bit about security and I realize that a tool like install-locked-env can be dangerous. However, it seems to me that it is not so much worse than other methods dealing with local installation. But for sure I can miss something.
Finally, a long time ago I wrote a small Python app called conda-app that was the equivalent of pipx for conda-forge. I was very happy when Pixi and UV came out so that I don’t need any longer to use and maintain conda-app. It would be perfect for me if something similar would happen with install-locked-env.
9 posts – 2 participants
]]>psutil to watch it and possibly kill it (maybe not the perfect design, but that’s not the point here). And even more stuff to build for release, as there are deps that apply only to doc building. In the past we’ve had requirements.txt files to express those, so for a developer who has done some work and wants to run the tests, pip -r requirements-dev.txt takes care of it. We do the same provisioning a CI instance. Expressing the dependencies in pyproject.toml doesn’t seem to allow the same thing – “just install these dependencies, don’t try to install the package”. Am I missing something? Or do we still need to keep parallel lists in two places? How to you best provision an environment where you’re going to test against a git checkout and aren’t at all interested in installing the package in this context.
8 posts – 4 participants
]]>RECORD.jws and RECORD.p7s. These in-archive signatures have never seen any widespread support or usage, notably they aren’t supported in pip nor uv. Users looking to verify wheel provenance should instead use index hosted attestations, which are more modern, out-of-archive and supported by tooling.
The PEP is written under the assumption that these files aren’t used. If you are a user of them or know a tool that uses them, please share!
Usage data
To add some data to the unused claim, we can use the GitHub search:
RECORD.p7s: Several hits in abandoned repos copying code that was removed frompypa/wheelin 2018 (Renamed the wheel.tool package to wheel.cli and migrated CLI stuff there · pypa/wheel@8949250 · GitHub)RECORD.jws: I only see one real usage in a fork (shuup/shuup/addons/verify.py at a34e5f51ea7cc3a8237acc01653e744142e1e6a4 · andersinno/shuup · GitHub), and that was removed in 2018 (Remove wheel validation · shuup/shuup@41d2e46 · GitHub)
Thanks to users (accidentally) committing venvs or unpacked wheels, we get additional data from GitHub:
path:**.dist-info/RECORD: 635k resultspath:**.dist-info/RECORD.jws: 8 distinct resultspath:**.dist-info/RECORD.p7s: 0 results.
We also checked Tom Forbes’ PyPI Data with the following query:
select
project_name,
max(toYear (uploaded_on)) as year,
countIf(endsWith(path, '.dist-info/RECORD.jws') OR endsWith(path, '.dist-info/RECORD.p7s')) as total_files
from pypi
group by project_name with rollup
having total_files > 0 or project_name is null
order by year desc, total_files desc;
This yields the following table, where year means “most recent year there has been a release with one of the files”: signature_files.csv · GitHub
Closing words
We usually take great care around deprecations, but unless there is usage of these signature files we have missed, I’d like to keep this as simple as possible for a deprecation with no affected tools or users.
7 posts – 6 participants
]]>pyproject.toml and uv.lock and checks each package’s PyPI metadata for either the free-threading trove classifiers or free-threading compatible wheels. You can specify overrides in [tool.free-threading-check.overrides] in your pyproject.toml.
Just run uv run check-freethreading-support.py in your project directory.
1 post – 1 participant
]]>PEP 508 has some very strange rules related to packaging markers, and packaging has never been compliant with these rules (and uv is not either). In short, the spec states any version comparison operator (like >, >=, and ==) must treat the specifier as a version, even if the table lists it as a string, trying to parse it as a version, and if it can’t parse, then using Python string comparison rules as fallback. This means implementation_name > "cpython" is valid and is supposed to evaluate False for cpython, but True for pypy. And the implementation is supposed to try to convert these to versions first, which in packaging is actually fairly expensive when done many times.
Here’s the table (minus a column for clarity here):
| Marker | Type | Sample values |
|---|---|---|
os_name |
String | posix, java |
sys_platform |
String | linux, linux2, darwin, java1.8.0_51 (note that “linux” is from Python 3 and “linux2” from Python 2) |
platform_machine |
String | x86_64 |
platform_python_implementation |
String | CPython, Jython |
platform_release |
String | 3.14.1-x86_64-linode39, 14.5.0, 1.8.0_51 |
platform_system |
String | Linux, Windows, Java |
platform_version |
String | #1 SMP Fri Apr 25 13:07:35 EDT 2014Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle CorporationDarwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015; root:xnu-2782.40.9~2/RELEASE_X86_64 |
python_version |
Version | 3.4, 2.7 |
python_full_version |
Version | 3.4.0, 3.5.0b1 |
implementation_name |
String | cpython |
implementation_version |
Version | 3.4.0, 3.5.0b1 |
extra |
String | toml |
extras |
Set of strings | {"toml"} |
dependency_groups |
Set of strings | {"test"} |
It should be noted, of these string fields, only one of them actually might look like a version: platform_release (platform.release()). On some systems, like macOS, this is a valid version. The only other one that anyone on GitHub has ever tried to use with a comparison that’s not equality is sys_platform >= "win32", which is obviously hoping that it would cover an imaginary "win64" (but "win128" would compare less!).
There is one valid use for platform_release; when you are on a system that you know has a valid PEP 440 style version, you can in theory gate it:
# scipy is not supported on Mac M1 with Mac OS < 12.0
scipy; platform_system != "Darwin" or platform_machine != "arm64" or platform_version >= "12"
In packaging<22, LegacyVersion was used, so these comparisons always returned False, and did not fall back to Python string comparison, since that’s how LegacyVersion worked. Starting in packaging 22, with the removal of LegacyVersion, these started throwing InvalidVersion errors instead (also not spec compliant). The spec does not specify if short circuit evaluation is required (since it basically has fallbacks for everything, there’s not really a point), so this means the above expression, in packaging 22-25.0 fails on any system that doesn’t have a valid PEP 440 version here, rendering it useless unless you only support the subset of systems where this does convert to a version. This was an issue adopting newer packaging versions in pip, and has basically resulted in every project that is actively maintained having to stop relying on this mechanism – less than 50 examples remain on GitHub of this. A significant number of pull requests and issues are open on packaging with various ways to fix these issues.
From some discussions with the uv team and looking at the code, uv follows the table, and does not try to convert everything to Version, though it does implement string comparisons; sys_platform == "win32" and platform_release > "9" would fail on Windows 11, for example, while it would pass on packaging – but packaging would currently crash with an error if this line was evaluated on most Linux systems.
Now we are faced with an issue: packaging 25.1 is about ready for release, and the way it handles invalid version comparisons has changed. As the code stands now, this will return False again, like it used to. Also, we’ve been really focused on performance; reading every Version on PyPI is 2x faster, and constructing/using SpecifierSet is ~3x faster (partially because we construct fewer Versions, which are costly due to needing a regex, even at 2x faster). This should improve the performance of the pip resolver, which constructs thousands of versions and specifier sets. One of the remaining areas where we are running Version on is on every dependency marker.
So there are three problems:
- The behavior is changing from an error to a non-standards complaint behavior (even though it’s the same behavior from years ago)
- We still have to try to construct versions on every single item in the above table. We are having to try to run the version regex on every marker,
"cpython","x86_64", etc. - There are weird bugs open, like
v0can’t be used as the name of an extra, because it parses as a version. We have to take an expression likething; extra == "v0"and parse the v0 as a version according to the spec, since==is a version comparison, the type of the field is meaningless according to the spec.
So I’d like to propose the following spec changes to align the spec with the way these have been handled since the beginning in packaging, and reduce our work required as well. It’s a minimal change; larger changes could be worked on later if someone wanted to work on a PEP for cleaning this up. But here’s my proposal:
- Change the spec to state only
Versionvalues must have the “convert to version if possible” behavior. This will allow implementations to fix errors like using"v0"as an extra, and provide a performance boost.uvis doing this anyway. - Make
platform_releaseaVersion(could be indicated asstring | Versionin the table to help users realize it will often fail the conversion, but it keeps the legacyVersionbehavior above.) - Define
>and<as alwaysFalse, and<=,>=as equivalent to==, for strings and failed Version conversions. This is the legacy (<22) and current (in main) behavior ofpackaging. Python string ordering is never reliable; even if it happens to work going from8to9, it will break on the next release because9is more than10. And this requires that other languages, like Rust, follow Python’s rules for string ordering.
This should only affect <50 (legacy) packages on GitHub, and it will do the right thing for them as well (making the packaging <22 behavior official). (Most of these have other typos, like 21 instead of 12 for the macOS version, so pretty sure they are dead projects, but it won’t break them).
I’d like to do this now, since we are replacing an Error with our old behavior, which was not spec compliant, so packages may start appearing expecting this behavior (again).
(I’d also be fine if we kept the string comparison (drop bullet 3 above), and changed packaging to support that; there has been pushback since this isn’t reliable even for version-like values, and it’s viewed as easier to change a False to a True than the other way around, due to the asymmetry in markers not supporting not, and historically it has returned False here. I’ll quite Konsti from the uv discussion here, hopefully he doesn’t mind: “this version-to-string-comparion fallback behavior is a big footgun and i wholeheartedly endorse removing it”.)
16 posts – 8 participants
]]>I’m deploying a Python application using PyInstaller, but instead of bundling everything normally, I’m embedding a custom Python interpreter and loading several internal packages from a separate folder. The structure looks something like this:
my_app/
main.exe (PyInstaller onefile)
embedded/
pkg1/
pkg2/
pkg3/
Each of these packages contains Python source files that my embedded interpreter imports at runtime.
My goal
I want to prevent users from accessing the source code inside these packages. Ideally, I’d like to distribute only compiled modules, not the original .py files
How do you do it?
5 posts – 3 participants
]]>In the initially version_cmp is specified:
version_cmp = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
And then in the Complete Grammar section it is specified as:
version_cmp = wsp* <'<=' | '<' | '!=' | '==' | '>=' | '>' | '~=' | '==='>
Notice that <= and < are swaped around, this is problematic for the initial version because < will match the first character of <= and therefore <= can never be matched as a version_cmp token.
Secondly both of them list === after ==, this is problematic for the same reason, == will be matched as the first two characters of === and therefore === can never be matched as a version_cmp token.
Here is my PR to fix: Fix grammar for arbitrary equality comparisons in dependency specifiers by notatallshaw · Pull Request #1969 · pypa/packaging.python.org · GitHub
3 posts – 2 participants
]]>A login attempt was made from an unrecognized device.
To complete your login, please visit the following link from the same device from which you attempted to log in:
Log in · PyPI********
Get the email like above. and then it gives me
Error:
Session invalidated. I tried three times with 2-factor auth and even changed my password.
4 posts – 4 participants
]]>As discussed, this should be a new topic. My comment from the original thread:
In particular, the backends @rgommers mentioned are:
- CMake
- Meson
- scikit-build-core
- meson-python
- maturin
To be honest, I thought scikit-build-core was the build backend corresponding to CMake, and the same for meson-python and Meson. And maturin is a slightly different case, as it’s for Rust rather than C extensions. So there’s only really two build backends for C code here, I think?
Nevertheless, I think having examples in the packaging guide that take an extremely simple C extension[1], and describe how to package it up using setuptools, scikit-build-code, and meson-python, would be really useful. By having a common description, with “tabbed” sections where you can pick your backend and see the differences, you get a great idea of all the common tasks, with a clear view of just how much (or ideally, little) difference the backends make.
With pure Python, I think this approach has been very effective, making it very clear that 99% of building a python package is backend-agnostic, and even the remaining 1% is very similar. This guides people to be less afraid of just choosing a backend based on what UI they prefer, and other “superficial” details 
To be clear, I’d strongly recommend not covering things like building for multiple platforms, bundling dependent libraries, cibuildwheel and auditwheel, in the basic page. By all means add them in a later section, but we should be showing that “write some C code and make it into an extension” isn’t fundamentally a hard problem[2].
Maybe even having an example of using Cython to wrap a C library would be useful. Again, that’s a common task that should be seen as relatively simple. But that might be a different section. Let’s keep the requirements for “Hello, world” minimal.
In case it’s not obvious, I’m very frustrated by the mystique that seems to have grown up around building C extensions, making it seem arcane and difficult. Being able to easily write your main code in Python, and the performance-critical bits in C, was what made Python so popular, and we’ve lost a lot of that these days.
Having the same for Rust would also be great, but with only one backend, and PyO3 being the core interface library, I imagine anyone writing a Rust extension already knows to look at the maturin and PyO3 docs. ↩︎
As opposed to publishing that code for a wide audience, which is…
↩︎
37 posts – 13 participants
]]>
hatch.pypa.io
Hatch v1.16.0 – Hatch
Hatch v1.16.0 brings workspace support, dependency-groups, and sbom support.
This release wouldn’t have been possible without Cary, our new co-maintainer. He picked up my unfinished workspaces branch and made it production-ready, added SBOM support to Hatchling, and landed a bunch of PRs from contributors!
My motivation took a big hit last year, in large part due to improper use of social media: I simply didn’t realize that continued mass evangelism is required nowadays. This led to some of our novel features being attributed to other tools when in fact Hatch was months ahead. I’m sorry to say that this greatly discouraged me and I let it affect maintenance. I tried to come back on several occasions but could only make incremental progress on the workspaces branch because I had to relearn the code each time. I’ve been having to make all recent releases from a branch based on an old commit because there were many prerequisite changes that were merged and couldn’t be released as is.
No more of that! Development will be much more rapid now, even better than the way it used to be. We are very excited for upcoming features 
4 posts – 4 participants
]]>
No, there are too many external dependencies needed (specifically, you have to have installed all the dependencies you’d need to build it yourself, so we suggest just building yourself – then you’ll also have sources accessible for debugging CPython, which is the only reason you’d want debug binaries).
Branching off from Python Installation Manager conversation – I thought debug binaries were required to build debug version of Python extension, since they are used for linking automatically if _DEBUG is defined and it’s defined by default in Debug configurations (at least in cmake).
pyconfig.h code:
But turned out if I just temporarily undefine _DEBUG before including Python header, then it will work fine – debug build will compile and link without issues and I will be able to use debug build with the normal Python.
#if defined(_DEBUG)
# undef _DEBUG
# include <Python.h>
# define _DEBUG 1
#else
# include <Python.h>
#endif
Is it indended way to do so? It seems there could be some kind of flag USE_NON_DEBUG_PYTHON that can avoid using python312_d.lib for linking even if _DEBUG is set and would proceed to python312.lib.
Then it would be possible to streamline this flag to the build systems used to build Python extensions, as indeed it seems debug binaries are needed only if you want to debug Python itself and not if you need debug version of your extension.
E.g. cmake’s FindPython module would be able to set this flag for Debug configs automatically when users are adding their libraries with Python_add_library, as in 99% cases they probably just want Debug version of their extension, not to debug Python. And then they also may never meet _d suffix appearing in their extensions, since it’s unlikely that they actually need it.
I’ve also found that swig already defined such option called SWIG_PYTHON_INTERPRETER_NO_DEBUG (code) to allow to compile debug version of extensions that would work with Release Python.
Maybe some flag like this can be added to the CPython itself?
—
Searching more on the topic, I’ve found a similar issue with someone using undef / define hack (github) – it was back in Python 3.4 and at the time, it seems, just undefining _DEBUG could have led to some other issue later on and python_d indeed was required and this could be the reason why debug binaries started to be included since 3.5.
But later in 3.8 (release notes) debug and release builds ABI became compatible and, maybe this is what allowed building undef _DEBUG more safely and use otherwise debug extension with release Python.
9 posts – 3 participants
]]>I’d like to know your thoughts on PEP 803 – a proposal for stable ABI for Free-Threaded Builds.
The main packaging-related points:
- Wheels tagged with the new ABI tag
abi3twould be compatible with free-threaded builds (subject to Python tags like now, e.g.cp316-abi3twouldn’t be compatible with 3.15) - C API extensions built with
Py_LIMITED_API=0x030f0000(3.15) and above would be compatible with free-threaded builds, as well as non-free-threaded ones. Wheels containing such extensions should get the compressed tagcp315-abi3.abi3tto reflect this. (Build tools for native extensions generally want to provide a knob to definePy_LIMITED_API.) - Extension filenames should be unchanged:
*.abi3.so,*.pyd, etc. - We’ll clarify explicitly that installers[1] are responsible for not putting incompatible extensions on Python’s import paths. (Packaging metadata is rich; CPython doesn’t have it and so it can’t do compatibility checks well. We will do better than in 3.14, but only to detect common mistakes (like outdated or misconfigured tools).
Please check Rejected ideas if you have a “why not?” question :)
Note that there’s a competing proposal in PEP 809, which adds a abi2026 wheel tag, to be generated when compiling with Py_LIMITED_API=0x03ff_2026.
It is possible to start preliminary testing of this in 3.15.0a2 with:
- define
_Py_OPAQUE_PYOBJECT(internal, testing only, will removed in final releases of 3.15) - don’t use API that’s getting removed (contact me for porting advice or example/test modules)
- due to a bug, free-threading builds of 3.15.0a2 don’t enable the GIL for updated modules properly
from CPython core point of view, users ↩︎
68 posts – 13 participants
]]>Trying the spec example of ===foobar, all of pip and uv and poetry fail, with an error message indicating that this is an invalid version number.
What are we even trying to be backwards-compatible with?
If no-one has implemented this for all these years then I guess that is no objection to changing the text.
However I am tempted to counter-propose that arbitrary equality just be removed altogether…
14 posts – 8 participants
]]>I am trying to fully implement arbitrary equality in packaging and I got stuck on whether the versions should be case insensitive or not.
This part of the spec is to support legacy versions that do no conform to the “modern” specification, as evidenced by this line:
This operator is special and acts as an escape hatch to allow someone using a tool which implements this specification to still install a legacy version which is otherwise incompatible with this specification.
So investigating how prior implementations worked, both LegacyVersion (implemented in 2014 in packaging) and parse_version (implemented in setuptools in 2005) are case insensitive. And prior to that StrictVersion and LooseVersion (implemented in CPython in 1998) only accepted lower case.
To clarify the situation I would like to chance change the following sentence:
Arbitrary equality comparisons are simple string equality operations which do not take into account any of the semantic information such as zero padding or local versions.
To:
Arbitrary equality comparisons are simple case-insensitive string equality operations which do not take into account any of the semantic information such as zero padding or local versions.
49 posts – 9 participants
]]>On behalf of my co-author, @Lecris and sponsor @FFY00, I’d like to announce PEP 808: Partially dynamic project metadata. You can see a rendered version here.
This PEP proposes allowing list and table dependencies to be listed statically and also to be extended by the build system if they are present in the dynamic table. So you can now do this:
[project]
name = "partially-dynamic"
version = "0.1.2
dependencies = ["numpy"]
dynamic = ["dependencies"]
Your build backend would then be allowed to add dependencies for you. Removing or changing an existing static entry is not allowed. You can see a lot of possible uses, and interactions with some other PEPs, in the PEP. There will also be a dynamic-metadata package that build backends can use to provide dynamic-metadata, including partially dynamic metadata, that is easy for users to configure, based on the mechanism inside scikit-build-core.
34 posts – 12 participants
]]>pip-tools.
Primarily this is a bugfix release. Highlights:
- Support
pipversion25.3 - Fixed a
pip-compilebug when-rwas used to include a file not in the CWD - Fixed an incorrect
returninfinallywhich could make build errors cause a crash failure
For the full changes: Release v7.5.2 · jazzband/pip-tools · GitHub
I’m very grateful for a number of community contributions which made this release possible, including both experienced and new contributors! 
1 post – 1 participant
]]>TL;DR: the current simple index standards (PEP 503 / PEP 691 and their living forms) adequately describe the shape of successful index responses, but not the shape of failure responses. As a result, installer clients (pip, uv, poetry, etc.) frequently have to assume a lowest common denominator of HTTP error code semantics, which in turn are frequently sparse or inaccurate (e.g. an index returning 404 because the client lacks access to a package, rather than because that package doesn’t exist).
For example, with PyPI, error responses are always plain text regardless of the negotiated success response:
$ curl 'https://pypi.org/simple/rlsgkljsgfslgjb/' -H 'Accept: application/vnd.pypi.simple.v1+json'
404 Not Found
whereas other indices may return HTML, JSON, or another differently structured but still plaintext response.
I’m curious if people have thoughts on this – if there’s ample interest, I’ll turn this into a pre-PEP discussion with some rough ideas on how we could accomplish this in a backwards compatible way
3 posts – 2 participants
]]>