Single-threaded re-entrancy via a recursive generator causes an access violation (segfault) in `pairwise_next`

[eendebakpt]

Crash report

What happened?

The existing partial fix in main (the it = po->it; if (it == NULL) re-read guard after the first tp_iternext call) does not cover the second tp_iternext call, where it is still a borrowed reference. In the single-threaded case a re-entrant call can trigger Py_CLEAR(po->it) which drops the generator's refcount to zero and frees it, leaving the outer call's local it as a dangling pointer.

Minimal reproducer:

from itertools import pairwise

def g():
    yield next(it)

for _ in range(10000):
    try:
        it = pairwise(g())
        next(it)
    except ValueError:
        pass

Exception ignored in: <generator object g at 0x...>
ValueError: generator already executing
Windows fatal exception: access violation

This is also the root cause of spyder-ide/qtconsole#635.

Some options to address:

• Py_INCREF(it) before the second tp_iternext(it) call + Py_DECREF after, or
• a re-entrancy guard flag (like teedataobject.running) as suggested by @rhettinger.

Note: Py_BEGIN_CRITICAL_SECTION (added in PR #144489) does not protect against same-thread re-entrancy.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Output from running 'python -VV' on the command line:

No response

[eendebakpt]

A reproducer for batched:

from itertools import batched, islice


def batched_gen():
    try:
        next(it)   # re-enters batched_next; inner call clears bo->it (frees it)
    except Exception:
        print("caught in batched_gen", flush=True)
        pass
    yield 1        # outer iternext(it) at i=0 returns 1 — loop continues
    yield 2        # outer iternext(it) at i=1: 'it' is freed → crash

print("Testing batched ...", flush=True)
for ii in range(10_000):
    try:
        it = batched(batched_gen(), 2)
        print('--')
        next(it)
        print('--')
    except (ValueError, StopIteration):
        print(ii)
        pass
print("done")

The islice is vulnerable as well.

Ping @rhettinger is code owner for itertools.