diff --git a/peps/pep-0803.rst b/peps/pep-0803.rst index c44cf0b78f6..7b331e3f415 100644 --- a/peps/pep-0803.rst +++ b/peps/pep-0803.rst @@ -16,21 +16,22 @@ Abstract ======== Add a new variant of the Stable ABI, called “Stable ABI for Free-Threaded -Python” (or ``abi3t`` for short), and a corresponding API limitations. +Python” (or ``abi3t`` for short). ``abi3t`` will be based on the existing Stable ABI (``abi3``), but make the :c:type:`PyObject` structure opaque. This will require users to migrate to new API for common tasks like defining modules and most classes. -At least initially, extensions built for ``abi3t`` will be compatible with -the existing Stable ABI (``abi3``). +In practice, ``abi3t`` 3.15 will be compatible with ``abi3`` 3.15. +Extension authors are encouraged to explicitly compile for both ABIs at once, +and signal compatibility using the wheel tag ``abi3.abi3t``. Terminology =========== -This PEP uses “GIL-enabled build” as an antonym to “free-threaded build”, +This PEP uses "GIL-enabled build" as an antonym to "free-threaded build", that is, an interpreter or extension built without ``Py_GIL_DISABLED``. @@ -38,7 +39,7 @@ Motivation ========== The Stable ABI is currently not available for free-threaded builds. -Extensions will fail to build for a both Limited API and +Extensions will fail to build for the Stable ABI on free-threaded Python (that is, when both :c:macro:`Py_LIMITED_API` and :c:macro:`Py_GIL_DISABLED` preprocessor macros are defined). Extensions built for GIL-enabled builds of CPython will fail to load @@ -51,8 +52,8 @@ for free-threading should be prepared and defined for Python 3.15”. This PEP proposes the Stable ABI for free-threading. -Background ----------- +Background & Summary +-------------------- Python's Stable ABI (``abi3`` for short), as defined in :pep:`384` and :pep:`652`, provides a way to compile extension modules that can be loaded @@ -66,19 +67,23 @@ With free-threading builds (:pep:`703`) being on track to eventually become the default (:pep:`779`), we need a way to make a Stable ABI available to those builds. -To build against a Stable ABI, the extension must use a *Limited API*, -that is, only a subset of the functions, structures, etc. that CPython -exposes. -Both the Limited API and the current Stable ABI are versioned, and building -against Stable ABI 3.X requires using only Limited API 3.X, and yields an -extension that is ABI-compatible with CPython 3.X and *any* later version +The current Stable ABI is versioned, and an extension built for Stable ABI +3.X is ABI-compatible with CPython 3.X and *any* later version (though bugs in CPython sometimes cause incompatibilities in practice). -The Limited API is not “stable”: newer versions may remove API that -were a part of older versions. +However, this forward compatibility is only guaranteed for +a subset of the API that CPython exposes (functions, structures, etc.). +Extensions that target the Stable ABI must limit themselves to this subset, +called the :ref:`Limited API `. +When the "opt-in" preprocessor macro ``Py_LIMITED_API`` is defined, +CPython headers will expose the Limited API only -This PEP proposes additional API limitations, as required to be compatible -with both GIL-enabled and free-threaded builds of CPython. +This PEP proposes a *stable ABI for free-threading builds* +(``abi3t`` for short), which includes additional API limitations needed to +compile extensions compatible with both GIL-enabled and free-threaded builds +of CPython 3.15+, +a corresponding macro to opt in to these limitations, and naming/tagging +schemes that extensions should use to signal such compatibility. Ecosystem maintainers want decreased maintenance burden @@ -99,7 +104,7 @@ The cryptography project shipped 48 wheel files with their `most recent release somewhat unusual in that they ship 14 wheels each for both the ``cp38`` and ``cp311`` stable ABIs to enable optimizations available in newer limited API versions. They also ship 14 additional ``cp314t`` wheels and 6 wheels for -pypy. If there is no free-threaded stable ABI, then with Python 3.15, +PyPy. If there is no free-threaded stable ABI, then with Python 3.15, cryptography will be using roughly the same amount of space on PyPI to support two versions of the free-threaded build as *all* non-EOL versions of the GIL-enabled build. @@ -246,39 +251,89 @@ maintainer `David Woods said disaster since we're not getting rid of the regular compilation mode so people are free to pick their own personal trade-offs. +It should be noted that this PEP leaves some questions/work open. +Wenzel Jakob -- maintainer of ``nanobind``, a C++ binding generator -- +`noted `_ +a need for additional API that is left out of scope of *this* PEP: + + It's not clear to me how I can get a pointer to the *N*-th data entry + of [a ``PyVarObject``-derived object]. + + If that can be resolved then nanobind should be able to adopt this new + ``abi3t`` compilation target. + Rationale ========= -The design in this PEP uses several constraints: +The design of ``abi3t`` involves several choices/assumptions/constraints: Separate ABI - The new ABI (``abi3t``) will conceptually be treated as separate from the - existing Stable ABI (``abi3``). +------------ + +The new ABI (``abi3t``) will conceptually be treated as separate from the +existing Stable ABI (``abi3``) – even though all extensions compatible with +``abi3t`` will, in practice, *also* be compatible with ``abi3``. + +(In more precise wording: ``abi3t``'s set of allowed APIs will be a +*subset* of ``abi3``'s; ``abi3t``'s set of compatible interpreters will +be a *superset* of ``abi3``'s. That makes one's head spin, which is part +of the reason to keep them separate.) + +Extensions should be explicitly compiled for *both* ``abi3t`` and ``abi3``, +and should explicitly signal that they support both at once +-- via packaging wheel tags (``abi3.abi3t``) and a runtime ABI check +(:external+py3.15:c:macro:`Py_mod_abi`). + +This explicitness has several advantages over having ``abi3t`` support +*imply* ``abi3`` support: + +- The tags clearly show whether an extension is compatible with GIL-enabled + builds, and whether the existing backwards compatibility guarantees of + ``abi3`` apply. +- Implementation-wise, the set of tags a given free-threaded interpreter + supports (as returned from the :pypi:`packaging` function + ``packaging.tags.sys_tags``) It will be the same size as for a corresponding + GIL-enabled build. +- It allows the ABIs to diverge slightly in the future -- keeping + *both at once* as the preferred compilation target, but allowing + ``abi3t``-only extensions for special cases. + +One practical exception to keeping the ABIs conceptually separate is discussed +in the :ref:`803-filename-tag` section. + +See these Rejected Ideas sections for more on the alternatives: - However, it will be possible to compile a single extension module that - supports both free-threaded and GIL-enabled builds. - This should involve no additional limitations, at least in the initial - implementation, and so it will be preferred over building - ``abi3t``-only extensions. +- :ref:`803-subset` +- :ref:`803-single` No backwards compatibility now - The new stable ABI variant will not support CPython 3.14 and below. - Projects that need this support can build separate extensions specifically - for the 3.14 free-threaded interpreter, and for older stable ABI versions. +------------------------------ - However, we won't block the possibility of extending compatibility to - CPython 3.14 and below, and we recommend that package installation tools - prepare for such extensions. - See a :ref:`rejected idea ` for how this could work. +CPython headers will not allow compiling for ``abi3t`` for CPython 3.14 +and earlier. +Projects that need this can build separate extensions specifically +for the 3.14 free-threaded interpreter, and for older ``abi3``. -API changes are OK - The new Limited API variant may require extension authors to make - significant changes to their code. - Projects that cannot do this (yet) can continue using the existing Limited - API, and compile separately for GIL-enabled builds - and for specific versions of free-threaded builds. +However, it *is* technically possible to build an extension compatible +with both free-threaded and GIL-enabled builds of CPython 3.14+. +To enable experiments in this area, we recommend that package installation +tools are prepared for such extensions. +See a :ref:`rejected idea ` for more details. +Source changes are necessary in extensions +------------------------------------------ + +``abi3t`` will require extension authors to make +significant changes to their code. + +Projects that cannot do this (yet) can continue using ``abi3``, +and compile the same source for specific versions of free-threaded builds. +(Note that the APIs removed in ``api3t`` still are usable when compiling for +a specific version, including 3.15t.) + +See a Rejected Ideas sections for an alternative: +:ref:`803-freeze-pyobject` Tag name -------- @@ -287,6 +342,66 @@ The tag ``abi3t`` is chosen to reflect the fact that this ABI is similar to ``abi3``, with minimal changes necessary to support free-threading (which uses the letter ``t`` in existing, version-specific ABI tags like ``cp314t``). +See a Rejected Ideas sections for an alternative: +:ref:`803-abi4` + +.. _803-filename-tag: + +Filename tag +------------ + +On systems that use the ``abi3`` tag in filenames, a new filename tag +(``abi3t``) is added so that older stable ABI extensions +(:samp:`{name}.abi3.so`) can be installed in the same directory as ones that +support Stable ABI for free-threaded Python (:samp:`{name}.abi3t.so`). + +There can only be one ABI tag in a filename (there is no concept of "compressed +tag sets like in wheel tags), so extensions that are compatible with both ABIs +at once need to use *one* of the tags -- the new one (``abi3t``), as the +existing one has existing meaning. + +See Rejected Ideas sections for alternatives: + +- :ref:`803-combined-filename-tag` +- :ref:`803-bare-so` + +Knob name +--------- + +This PEP specifies that the C preprocessor macro ``Py_TARGET_ABI3T`` +will enable compiling for ``abi3t`` (that is, practically: it will make +``Python.h`` only expose forward-compatible definitions). + +The corresponding "knob" for ``abi3`` is named ``Py_LIMITED_API``. +This name is problematic: + +- It describes what the macro's historical internal effect (limiting which + definitions are exposed), but not the intended benefit (forward + compatibility). +- It is increasingly a misnomer: for API like ``Py_TYPE``, it selects + a forward-compatible implementation (DLL function call rather than inline + pointer deference) rather than limiting the API. +- The pair of terms *Stable ABI* and *Limited API* is technically accurate, + but quite confusing. + Avoiding the term *Limited API*, and talking about "constraints necessary + for targeting a given ABI", tends to be clearer. + +The proposed macro name (``Py_TARGET_ABI3T``) emphasizes ``abi3t`` as a +*compilation target*, with API limitations as its implicit price -- and forward +compatibility as the implicit benefit. + +As for ``Py_LIMITED_API``, this PEP proposes no change, which means keeping +it for ``abi3``. +``abi3`` is expected to eventually become irrelevant *if* free-threaded builds +replace the GIL-enabled ones (see the +`PEP 703 acceptance notice `__ +for the tentative plan). +At that point, ``Py_LIMITED_API`` will likely remain user-visible, but as an +implementation detail. + +See a Rejected Ideas sections for an alternative -- reusing an existing "knob": +:ref:`803-knob-gildisabled` + Specification ============= @@ -300,20 +415,25 @@ builds*, or ``abi3t`` for short. As with the current Stable ABI (``abi3``), ``abi3t`` will be versioned using major (3) and minor versions of Python interpreter. Extensions built for ``abi3t`` :samp:`3.{x}` will be compatible with -CPython :samp:`3.{x}` and above. - -To build a C/C++ extension for ``abi3t``, the extension will need to only -use *limited API for free-threaded builds*. -Like the existing Limited API, this will be a subset of the general CPython -C API. -Initially, it will also be a subset of the Limited API. -We will strive to keep it as a subset, but will not guarantee this. - -Since ``abi3t`` and ``abi3`` will overlap, it will be possible for a single -compiled extension to support both at once, and thus be compatible with -CPython 3.15+ (both free-threaded and GIL-enabled builds). -Initially, any extension compiled for ``abi3t`` will be compatible with +free-threading builds of CPython :samp:`3.{x}` and above. + +This mirrors the compatibility promise for the existing Stable ABI, ``abi3``, +which was defined :pep:`384#abstract` and modified in +:pep:`703#backwards-compatibility`: +Extensions built for ``abi3`` :samp:`3.{x}` will be compatible +with *GIL-enabled* builds of CPython :samp:`3.{x}` and above. + +To build a C/C++ extension for ``abi3t``, the extension will need to limit +itself to only use API for which we can promise long-term support. +This *limited API for free-threaded builds* will be a subset of the +3.15 Limited API. + +Any extension compiled for ``abi3t`` will, in practice, be compatible with ``abi3`` as well. +However, we recommend that users and tools explicitly compile for +*both at the same time*, and signal this explicitly. +(In the PyPA packaging ecosystem, this signaling means using the wheel tag +``abi3.abi3t`` as detailed below). Choosing the target ABI @@ -327,11 +447,6 @@ tool-specific UI -- will select the target ABI using the following macros: * ``Py_TARGET_ABI3T=`` (proposed here): Compile for ``abi3t`` of the given version. -These two macros are functionally very similar. -In hindsight, ``Py_TARGET_ABI3`` (without the ``T``) would be a more fitting -name for :c:macro:`Py_LIMITED_API`. -We keep the existing name for backwards compatibility. - For ease of use and implementation simplicity, respectively, ``Python.h`` will set the configuration macros automatically in the following situations: @@ -418,6 +533,8 @@ They will, however, not be removed. - :c:func:`PyModule_FromDefAndSpec`, :c:func:`PyModule_FromDefAndSpec2` +.. _803-runtime-check: + Runtime ABI checks ------------------ @@ -434,12 +551,12 @@ models are possible. .. _platform compatibility tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/ However, CPython will add a line of defense against outdated or misconfigured -tools, or human mistakes, in the form of a new *module slot* containing -basic ABI information. +tools, or human mistakes, in the form of a new *module slot*, ``Py_mod_abi``, +containing basic ABI information. This information will be checked when a module is loaded, and incompatible extensions will be rejected. The specifics are left to the C API working group -(see `issue 72 `__). +(see `capi-workgroup issue 72 `__). This slot will become *mandatory* with the new export hook added in :pep:`793`. @@ -461,63 +578,95 @@ and may be removed in future CPython versions, if the internal object layout needs to change. -The ``abi3t`` wheel tag ------------------------ +The ``abi3t`` wheel and filename tags +------------------------------------- -Wheels that use a stable ABI compatible with free-threading CPython builds +Wheels with extension modules compiled for Stable ABI for Free-Threaded Python should use a new `ABI tag`_: ``abi3t``. .. _ABI tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#abi-tag +On systems where filenames of Stable ABI extensions end with ``.abi3.so``, +extensions that support free-threading should instead use ``abi3t.so``. +This includes extensions compatible with *both* ``abi3`` and ``abi3t``. + +On these systems, all builds of CPython -- GIL-enabled and free-threaded -- +will load extensions with the ``abi3t`` tag. +Free-threaded builds will -- unlike in 3.14 -- *not* load extensions +with the ``abi3`` tag. +If files are present with both tags, GIL-enabled builds will prefer +"their" ``*.abi3.so`` over ``*.abi3t.so``. + +Put another way, ``importlib.machinery.EXTENSION_SUFFIXES`` will be +(for ``x86_64-linux-gnu`` builds of CPython): + +* ``python3.15``: + ``['.cpython-315-x86_64-linux-gnu.so', '.abi3.so', '.abi3t.so', '.so']`` +* ``python3.15t``: + ``['.cpython-315-x86_64-linux-gnu.so', '.abi3t.so', '.so']`` + +Making GIL-enabled builds load ``.abi3t.so`` files is purely a practical +choice: it this one case we break the conceptual purity of ``abi3`` and +``abi3t`` being separate ABIs. + + Recommendations for installers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Package installers should treat this tag as completely separate from ``abi3``. -They should allow ``abi3t``-tagged wheels for free-threaded builds wherever -they currently allow ``abi3``-tagged ones for (otherwise equal) +Package installers should allow ``abi3t``-tagged wheels for free-threaded +builds wherever they currently allow ``abi3``-tagged ones for (otherwise equal) non-free-threaded builds. +Note that this PEP does not provide a way to target Stable ABI for +Free-threaded Python 3.14 (``cp314-abi3t``) and below. +This may change an the future, or with experimental build tools, so +installers should be prepared for such extensions. + + Recommendations for build tools ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Build tools should give users one or two additional options, in addition to +Build tools should give users a new option: in addition to compiling for the existing CPython version-specific ABI (:samp:`cp3{nn}`) and -Stable ABI (``abi3``): - -- Compile extensions compatible with *both* ``abi3`` and ``abi3t``, by either: - - - defining both :samp:`Py_LIMITED_API={v}` and :samp:`Py_TARGET_ABI3T={v}`, or - - defining :samp:`Py_LIMITED_API={v}` and: +Stable ABI (``abi3``), they should allow +compiling extensions for *both* ``abi3`` and ``abi3t`` at once, by either: - - defining ``Py_GIL_DISABLED`` (on Windows) - - building with free-threaded CPython headers (elsewhere) +- defining both :samp:`Py_LIMITED_API={v}` and :samp:`Py_TARGET_ABI3T={v}`, or +- defining :samp:`Py_LIMITED_API={v}` and: - Such extensions should be tagged with the `compressed tag set`_ - ``abi3.abi3t``. + - defining ``Py_GIL_DISABLED`` (on Windows) + - building with free-threaded CPython headers (elsewhere) -- Compile extensions compatible with *only* ``abi3t``, by defining only - :samp:`Py_TARGET_ABI3T={v}` and tagging the result with ``abi3t``. - This will initially offer no advantages over the ``abi3.abi3t`` option - above, but there is a possibility that it will become useful in the future. - -In the above, :samp:`{v}` stands for a the lowest Python version with which +In the above, :samp:`{v}` stands for the lowest Python version with which the extension should be compatible, in :c:func:`Py_PACK_VERSION` format. -In the cases above, this version must be set to 3.15 or higher. +This version must be 3.15 or higher. + +On systems that use ABI version tagged ``.so`` files as introduced in +:pep:`3149` (Linux, macOS, and similar), the extension should be named +:samp:`{modulename}.abi3t.so`. +Otherwise, there should be no change: on Windows, use :samp:`{name}.pyd`. -.. _compressed tag set: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#compressed-tag-sets +Wheels containing such extensions should be tagged with the +`compressed ABI tag set `_ +``abi3.abi3t``, and +the `Python tag`_ :samp:`cp{3yy}` corresponding to :samp:`{v}` above. -The version of the Stable ABI, both ``abi3`` and ``abi3t``, is indicated by -the `Python wheel tag`_. For example, a wheel tagged ``cp315-abi3.abi3t`` will be compatible with 3.15, 3.16, and later versions; ``cp317-abi3.abi3t`` will be compatible with 3.17+. -.. _Python wheel tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag +.. _Python tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag -Note that this PEP does not provide a way to target Stable ABI for -Free-threaded Python 3.14 and below (``cp314-abi3t``). -This may change an the future, or with experimental build tools, so -*installers* should be prepared for such extensions. +It is discouraged, but possible, to compile extensions compatible with *only* +``abi3t`` (by defining only :samp:`Py_TARGET_ABI3T={v}`, building with +GIL-enabled CPython, and tagging the resulting wheel with ``abi3t`` rather +than ``abi3.abi3t``). +This will limit the result to free-threaded interpreters only. + +Its is also possible to build ``abi3t`` extensions compatible with CPython 3.14 +(or even lower versions), but this is unsupported and would require +detailed understanding of the limitations and guarantees, +along with thorough testing. New API @@ -527,29 +676,29 @@ Implementing this PEP will make it possible to build extensions that can be successfully loaded on free-threaded Python, but not necessarily ones that are thread-safe without a GIL. -Limited API to allow thread-safety without a GIL -- presumably ``PyMutex``, -``PyCriticalSection``, and similar -- will be added via the C API working group, +Limited API to allow thread-safety without a GIL -- presumably +``PyCriticalSection`` and similar -- will be added via the C API working group, or in a follow-up PEP. Backwards and Forwards Compatibility ==================================== -Extensions targetting ``abi3t`` will not be backwards-compatible with older -CPython releases, due to the need to use new ``PyModExport`` API added +Extensions targeting ``abi3t`` will not be backwards-compatible with older +CPython releases, neither at the source level (API) nor in compiled form (ABI), +due to the need to avoid ``PyModuleDef`` and use new ``PyModExport`` hook added in :pep:`793`. Extension authors who cannot switch may continue to use the existing ``abi3``, that is, build on GIL-enabled Python without defining ``Py_GIL_DISABLED``. For compatibility with free-threaded builds, they can compile using version-specific ABI -- that is, compile ``abi3``-compatible source -on free-threaded CPython builds without defining ``Py_LIMITED_API``. +on free-threaded CPython builds without defining ``Py_TARGET_ABI3T``. -Limited API 3.15 for free-threading is a subset of the existing Limited API, -and as such, it will be forward-compatible with future versions of CPython 3.x. -Older versions of the Limited API (that is, 3.14 and below) will continue -to be forward-compatible with GIL-enabled builds of CPython 3.x, starting with -the version that introduced the given Limited API. +As for the existing Stable ABI: ``abi3`` :samp:`3.{x}` continues to be +compatible with *GIL-enabled* CPython :samp:`3.{x}` or above, +as promised in :pep:`384#abstract` and amended in +:pep:`703#backwards-compatibility`. Compatibility Overview @@ -644,7 +793,7 @@ free-threaded one. (*): Wheels with these tags cannot be built; see table below The following table summarizes which wheel tag should be used for an extension -built with a given interpreter and ``Py_LIMITED_API`` macro: +built with the given interpreter and defined macros: +-----------------------+-------------+--------------------+---------------------+-----------+ | To get the wheel tag… | Compile on… | ``Py_LIMITED_API`` | ``Py_TARGET_ABI3T`` | Note | @@ -677,6 +826,7 @@ In the “Compile on” column, *FT* means that the :c:macro:`Py_GIL_DISABLED` macro must be defined -- either explicitly or, on non-Windows platforms, by including CPython headers configured with :option:`--disable-gil`. *GIL* means that :c:macro:`Py_GIL_DISABLED` must *not* be defined. +Rows without this note apply to both cases. In the ``Py_LIMITED_API`` and ``Py_TARGET_ABI3T``, a dash means the macro must not be defined; a version means the macro must be set to the corresponding @@ -691,6 +841,7 @@ Values in the *Note* column: this PEP, but may be added in the future. Installers should be prepared to handle the tag. + Security Implications ===================== @@ -720,30 +871,40 @@ This PEP combines several pieces, implemented individually: `python/cpython#137212 `__. - A check for older ``abi3`` was implemented in GitHub pull request `python/cpython#137957 `__. -- For wheel tags, there is no implementation yet. +- For wheel tag handling in installers, a draft pull request is at + `pypa/packaging/pull#1099 `__. +- For build tools, several individual draft pull requests are open; + contact Nathan for details. - A porting guide is not yet written. +.. _pep803-rejected-ideas: + Rejected Ideas ============== +.. _803-single: -Make ``PyObject`` opaque in Limited API 3.15 --------------------------------------------- +Single new ABI: make ``PyObject`` opaque in Stable ABI 3.15 +----------------------------------------------------------- -It would be possible to make ``PyObject`` struct opaque in Limited API 3.15 -(that is, the new version of the existing Limited API), -rather than introduce a new variant of the Stable ABI and Limited API. +It would be possible to make ``PyObject`` struct opaque in Stable ABI +rather than introduce a new variant of the Stable ABI. This would mean that extension authors would need to adapt their code to the -new limitations, or abandon Limited API altogether, in order to use any C API +new limitations, or abandon Stable ABI altogether, in order to use any C API introduced in Python 3.15. -It would also not remove the need for a new wheel tag (``abi3t``), +It would also not fully remove the case for a new wheel tag (``abi3t``), which would be required to express that an extension is compatible with both GIL-enabled and free-threaded builds of CPython 3.14 or lower. +In the `PEP discussion `__, +the ability to build for the GIL-only Stable ABI with no source changes +was deemed to be worth an extra configuration macro (now called +``Py_TARGET_ABI3T``). + .. _pep803-no-shim: @@ -777,21 +938,46 @@ extension to query the running interpreter, and for 3.14, use a ``struct`` definition corresponding to the detected build's ``PyModuleDef``. +.. _803-subset: + +Making ``abi3t`` compatible with ``abi3`` +----------------------------------------- + +It would be possible to teach packaging tools that ``abi3t`` is a "subset" +of ``abi3``, that is, all GIL-enabled interpreters are guaranteed to be +compatible with ``abi3t``-tagged builds. +This would make the ``abi3.abi3t`` compressed tag set equivalent to +``abi3t``, and thus redundant. +However, tools would still need to output the compressed tag set to support +"older installers", which do *not* consider ``abi3t`` compatible with +GIL-enabled builds. + +Here, "older installers" include ones that use or vendor an version of the +:pypi:`packaging` library that wasn't updated for ``abi3t``. +(The ``packaging`` library is what Python-based installers typically use +to implement wheel tag matching.) + +Beyond installers, the ``abi3.abi3t`` tag allow mechanisms like the ABI filter +in PyPI file list (e.g. on ``__) +to match ``abi3`` without special-casing (assuming compressed tags are handled +according to standard). +Humans wondering about compatibility with ``abi3`` also get a more +explicit signal. + +Less importantly, merging the ABIs would also remove an "escape hatch" of +possibly making ``abi3`` and ``abi3t`` diverge in the future. + + +.. _803-abi4: + Naming this ``abi4`` -------------------- Instead of ``abi3t``, we could “bump the version” and use ``abi4`` instead. The difference is largely cosmetic. -However, one thing this PEP does not propose is changing the *filename* -tag: extensions will be named with the extensions like ``.abi3.so``. -Changing this while keeping compatibility with GIL-enabled builds would be -an unnecessary technical change. - -Using ``abi3.abi4`` in wheel tags but only ``.abi3`` in filenames would -look more inconsistent than ``abi3.abi3t`` and ``.abi3``. - -If we added an ``abi4`` tag, the ``Py_LIMITED_API`` value would either need to: +If we added an ``abi4`` tag, the value of the opt-in macro (``Py_TARGET_ABI4`` +or ``Py_LIMITED_API`` or some such) would either need to: * change to start with ``4`` to match ``abi4``, but no longer correspond to ``PY_VERSION_HEX`` (making it harder to generate and check), or @@ -802,6 +988,58 @@ better as a transitional state before larger changes like :pep:`809`'s ``abi2026``. +.. _803-combined-filename-tag: + +``abi3+abi3t`` filename tag +--------------------------- + +Filename ABI tags (as introduced in :pep:`3149`) allow extensions for several +ABIs to co-exist in a directory. + +Per this PEP, extensions that are compatible with both ``abi3`` and ``abi3t`` +will use a compressed tag set (``abi3.abi3t``) in wheel metadata, +but not in filenames (``.abi3.so``/``.abi3t.so``). +We *could* add a dedicated tag for the combination -- for example, +``.abi3+abi3t.so``. + +But, there would be no need for ``.abi3+abi3t.so`` extensions to co-exist with +``.abi3t.so`` ones: free-threaded interpreters would always pick ``.abi3t.so``, +so the extension for GIL-enabled interpreters could just as well use +``.abi3.so``. +The only benefit would be clearer naming when an ``abi3.abi3t`` extension +is *not* installed together with its ``abi3``-only equivalent. + +Here, clearer naming is not worth the complexity. +We make the practical choice to make ``.abi3t.so`` mean +"abi3+abi3t", that is, "loadable by all builds". +This works for (discouraged) ``abi3t``-only extensions: on a GIL-enabled +interpreter, these will fail the mandatory :ref:`runtime ABI check <803-runtime-check>` +or, in the unlikely future where ``abi3t`` & ``abi3`` diverge, possibly fail +to load due to a missing linker symbol. + +Conceptually, filename tags do not "describe" or "name" an extension's ABI. +The current ``.abi3`` tag is already too weak for this, as it lacks a version. + + +.. _803-bare-so: + +No filename tag (bare ``.so``) +------------------------------ + +It would be possible to drop the filename ABI tag altogether, +and use ``.so`` instead of ``.abi3t.so``. +The practical meaning of these two tags is very close: +``.so`` is loadable by *any* build of CPython; +``.abi3t.so`` will be loadable by any CPython *3.15 or above* -- but the +Stable ABI filename tag already lacks version information. + +They are different semantically, though. +Bare ``.so`` means "don't care"; an ``.abi3t.so`` extension is *intentionally* +compatible with the new ABI. + + +.. _803-knob-gildisabled: + Reusing ``Py_GIL_DISABLED`` to enable the new ABI ------------------------------------------------- @@ -809,20 +1047,28 @@ It would be possible to select ``abi3t`` (rather than ``abi3``) when the ``Py_GIL_DISABLED`` macro is defined together with ``Py_LIMITED_API``. This would require annoying fiddling with build flags, and make it -especially cumbersome to target *both* ``abi3`` and ``abi3t`` at the same time. - +impossible to explicitly target both ``abi3`` and ``abi3t`` at the same time. -Making the new version of ``abi3`` compatible with free-threading ------------------------------------------------------------------ +.. _803-freeze-pyobject: -It would be possible to make ``PyObject`` opaque in Limited API 3.15, -rather than add a new stable ABI. -This would make all extensions built for the Stable ABI 3.15 and above -compatible with both free-threading and GIL-enabled Python. +Fully separate ABIs, keeping source compatibility +------------------------------------------------- -In the `PEP discussion `__, -the ability to build for the GIL-only Stable ABI with no source changes -was deemed to be worth an extra configuration macro (``Py_TARGET_ABI3T``). +It would be possible to make ``abi3`` and ``abi3t`` fully separate and +incompatible. +This would allow all current extensions to stay *source*-compatible with +``abi3t``: the ``PyObject`` struct could stay exposed, with each ABI +defining a different set of private fields, as in the version-specific +CPython ABI. + +However, exposed ``PyObject`` struct has been `noted `__ +as one of the main shortcomings of the existing Stable ABI. +It hindered or prevented optimizations and features such as immortalization +and free-threading itself. + +Exposing ``PyObject`` would mean repeating this mistake, "freezing" +its current free-threaded definition, and requiring +*yet another* variant of stable ABI if/when changes are needed. Copyright