Python 3.15+ requires C-contiguous buffers for int.from_bytes()

[frenzymadness]

As you can see here: python/cpython@510ab7d

The conversion is no longer done on CPython side. Without this change, test_apply_mask_non_contiguous_memoryview fails with Python 3.15.

[frenzymadness]

I've realized I can add a check for sys.version to the if, so it makes the copy only for Python 3.15. Let me know what you prefer.

[aaugustin]

If int.from_bytes made a copy on Python < 3.15, and I don't see how it could do without a copy, then "making a copy then calling int.from_bytes" should have the same performance as "directly calling int.from_bytes (which makes a copy)", and we don't need the version check. This is easy to benchmark on a large bytestring.

IMO the comment is not needed: the code is self-explanatory and a test will fail if it is removed.

[aaugustin]

If you want to provide the benchmark, that will help confirm that there's no point making it version-conditional. Else, I can take it from there. Thank for you spotting the problem and proposing the solution.

[frenzymadness]

I'm not sure what you mean. The potential performance issue is that on Python 3.14 and older the copy is now done twice - first here in this new code and second in Python on C level. The point of version check would be in doing the copy only for Python 3.15+ when it's necessary.

[aaugustin]

I benchmarked with a 1MB non-contiguous memoryview.

Python 3.14 and earlier

Unexpectedly, int.from_bytes(bytes()) is 25-30% FASTER than justint.from_bytes() (40ms vs. 55ms) 🤯

So it's actually a performance improvement to cast to bytes on Python < 3.15 too.

Details

>>> import timeit
...
... # 1MB of non-contiguous random data
... non_contiguous = memoryview(open('/dev/urandom', 'rb').read(2 * 1024 * 1024))[::2]
...
... t = timeit.timeit(lambda: int.from_bytes(non_contiguous), number=10)
... print(f"int.from_bytes(): {t:.3f}s")
...
... t = timeit.timeit(lambda: bytes(non_contiguous), number=10)
... print(f"bytes(): {t:.3f}s")
...
... t = timeit.timeit(lambda: int.from_bytes(bytes(non_contiguous)), number=10)
... print(f"int.from_bytes(bytes()): {t:.3f}s")
...
int.from_bytes(): 54ms
bytes(): 34ms
int.from_bytes(bytes()): 40ms

I'm getting the same results on Python 3.13, 3.12, and 3.11.

Python 3.15

Even more unexpectedly, casting to int_from_bytes(bytes()) is faster than bytes() 🤔

Details

>>> import timeit
...
... # 1MB of non-contiguous random data
... non_contiguous = memoryview(open('/dev/urandom', 'rb').read(2 * 1024 * 1024))[::2]
...
... # t = timeit.timeit(lambda: int.from_bytes(non_contiguous), number=10)
... # print(f"int.from_bytes(): {t * 1000:.1f}ms")
...
... t = timeit.timeit(lambda: bytes(non_contiguous), number=10)
... print(f"bytes(): {t * 1000:.1f}ms")
...
... t = timeit.timeit(lambda: int.from_bytes(bytes(non_contiguous)), number=10)
... print(f"int.from_bytes(bytes()): {t * 1000:.1f}ms")
...
bytes(): 49.2ms
int.from_bytes(bytes()): 40.3ms

For completeness, I tested contiguous memoryviews and there's no significant perf change — expectedly 😅

[aaugustin]

I just realized that performance-sensitive users must be using the C implementation anyway so no big deal.

[aaugustin]

Thank you for your contribution!