fesco

#2825 [URGENT] Fedora 37 Python 3.11 contingency mechanism
Closed: Invalid by churchyard. Opened by churchyard.

Closed: Invalid
Reopen Issue

Hello FESCo,

The Fedora 37 Python 3.11 change is at risk. Upstream has announced that the final release of Python 3.11 might be postponed from October to December.

https://mail.python.org/archives/list/python-dev@python.org/message/3JWVCSBPBFWY5ZWSJ7RYB6FS5NIMCEOY/

Note that we don't consider the identified upstream blockers as seriously impacting Fedora. It's just troublesome timing.

Python's ABI is frozen after the first RC. This was supposed to be released in early August, allowing us to reach Fedora 37 Beta Freeze with ABI-stable Python. However, with the possible delay of 2 months, we might end up with no RC happening even during Fedora 37 Final Freeze.

We will know more about the upstream plans by the end of this week, but even if they decide not to postpone the release, nothing's stopping them from changing that decision later.

Assuming the delay will happen, we need to decide what to do. Since some of the options would require almost immediate action, this is pretty urgent to decide. We've identified the following possible courses of action:

1) Wait it out

Do nothing special, keep calm, and carry on. That's the easiest option for now but it poses significant risks if Python 3.11.0rc1 is not released sooner than Fedora 37 Final Freeze. We might need to delay the release of Fedora 37 if this happens, or we will ship a Python beta version at GA which is not cool for our users and due to ABI instability might prevent us from upgrading Python during the entire time frame of Fedora 37. This is dangerous.

If the RC release actually happens close to the Final Freeze and it changes the ABI we might need to mass rebuild all arched Python packages close to the release, possibly introducing Python-unrelated regressions at the last moment. This is also dangerous.

2) Revert before the mass rebuild

We wait for the upstream decision and if they decide to delay or if there is no decision by the end of next week, we revert to Python 3.10. We carry on a side tag rebuild with 3.10 (and we can temporarily tag in the old builds that happened before the Python 3.11 rebuilds to speed things up). We anticipate such a rebuild would take less than 1 week, but considering the overall FTBFS problems in rawhide, we cannot really make any promises. If needed, the scheduled Fedora 37 mass rebuild (2022-07-20) would need to be delayed a bit. After branching, we would update (back) to Python 3.11 in Fedora 38 and do another side tag rebuild there. This is tedious, quite noisy, but probably safe.

Additionally, we would need to introduce an epoch for the python3 package (ugly, but bearable).
What's concerning us more is that this would probably mean we push the Python updates to even Fedora releases forever and this means Fedora would no longer be the driver of the fixes in the Python ecosystem. This is sad.

https://fedoraproject.org/wiki/Changes/PythonSafePath would be deferred to Fedora 38.

3) (Be prepared to) revert after branching

A compromise is to carry on with the mass rebuild as-is and wait to see what's happening upstream. Once branching is upon us (assuming 2022-08-09), we make a decision. If Python 3.11.0rc1 has already happened or if it looks promising to happen soon, we carry on. Else, we revert to 3.10 but in f37 branches only. This still requires a targeted mass rebuild with 3.10 and still requires an epoch bump, but it leaves Fedora 38 with Python 3.11. This is what we would prefer to do.

This means we would not do https://fedoraproject.org/wiki/Changes/PythonSafePath before the mass rebuild.
If we revert to 3.10, Changes/PythonSafePath would be deferred to Fedora 38.
If we don't revert to 3.10,Changes/PythonSafePath would either happen with a targeted mini mass rebuild (245 components), or it would be deferred to Fedora 38 anyway.

Please, ask any questions, and vote for your favorite option(s). I'm tagging this with meeting - if there is no consensus on whether (2) is desired, we will need to decide on the meeting.

cc @thrnciar @adamwill @bcotton

I personally don't believe the mass rebuild matters that much here -- (close to) all packages that need a rebuild with specific Python versions were already built anyway and building them that way again does not change much. Also, a branched-only revert as described in (3) seems less disturbing to other packagers than (2), and it is definitively easier for us. Moreover, it gives us more time to decide if we want (1).

Hence, I vote for option 3.

I would rather us revert before we need an Epoch bump (that is, before branching).

We already need an epoch bump for rawhide users. If FESCo doesn't consider that a problem for rawhide, we can always coordinate with releng and branch + revert at the same time, only composing the first branched f37 compose with Python 3.10. I'd rather not do 3.10->3.11->3.10->3.11 in Rawhide just to avoid an epoch.

Devel list discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/2QE4OX3M5YQ4IMUX5XJE5LTSXZWG4XEC/

I’m curious to see if the first upstream decision point is triggered this week. If they commit to the two-month delay now, that would seem to make our decision for us.

Otherwise, I think I initially lean toward (3), but I’m open to the possibility that further listening and reflecting might convince me to prefer (2).

Note that if the upstream decision by the end of the week is to stick with the original schedule, we can probably just close this as moot (possibly keeping (3) as a backup plan if they slip).

If they commit to the two-month delay now, that would seem to make our decision for us.

And if that happens, we still need to decide between (2) and (3).

If they commit to the two-month delay now, that would seem to make our decision for us.

And if that happens, we still need to decide between (2) and (3).

So, to write out the math: if they do commit to the delay, then 3.11b4 will be released later than the current target of Monday (2022-07-11), and they will add two more betas at one month per beta. That means 3.11b5 no earlier than 2022-08-11, 3.11b6 no earlier than 2022-09-11, and therefore 3.11rc1 no earlier than 2022-10-11, which would miss the current scheduled Final Freeze date of 2022-10-04. If I haven’t miscalculated something, that would guarantee we can expect to enter the Final Freeze without an RC, and so there would be no reason to delay the contingency plan by choosing (3) over (2). Am I missing anything?

so there would be no reason to delay the contingency plan by choosing (3) over (2).

The reason is to avoid doing 3.10->3.11->3.10->3.11 in Rawhide. Waiting for branching allows us to keep Fedora 38 on Python 3.11 and revert only in Fedora 37.

The reason is to avoid doing 3.10->3.11->3.10->3.11 in Rawhide. Waiting for branching allows us to keep Fedora 38 on Python 3.11 and revert only in Fedora 37.

Thanks; now I understand your previous comment. That is a reasonable goal.

+1 on option 3

I couldn't understand option 3 at first, but @churchyard's comment here really helped me. I like the idea of keeping 3.11.0bX going in rawhide through branching and then dealing with the issues in Fedora 37 post branching if required.

It also allows us to be optimistic that upstream Python delivers without delay. I love being an optimist. 😎

We already need an epoch bump for rawhide users. If FESCo doesn't consider that a problem for rawhide, we can always coordinate with releng and branch + revert at the same time, only composing the first branched f37 compose with Python 3.10. I'd rather not do 3.10->3.11->3.10->3.11 in Rawhide just to avoid an epoch.

Epoch bumps are not required, as long as the Koji NVR remains unique. dnf distro-sync will handle it properly.

Actually, at some point, we should probably make it so DNF always does distro-sync for upgrades in Rawhide, like openSUSE does with PK-DNF for Tumbleweed.
Edited by ngompa

Ok, let's not focus on epoch bump here. If the revert comes, we can discuss that part.

I'm in favor of option 2 myself.

Option 3 seems least bad I guess. ;(

I'm certainly against option 2. It creates a lot of work before we know for sure that it's needed.

+1 for option 3.

Looking at the list of release-blocker issues that was linked, at least two of the three don't seem very serious. One is a reference miscount that will probably be fixed today, the other is a ~50% performance degradation in a very niche operation, so something that should be OK for -rc1 or even -final. The issue with reference counting might be potentially more involved, but with enough people looking into, I'm sure it too can be fixed.

If python upstream publishes an updated schedule where -rc1 is a week or two late for our schedule, I'd fine with adjusting our schedule to match. I think a small delay is better then the huge effort that'd be required to epoch-bump and rebuild python packages, and the delay of python3.11 for another six months.

Note that there were ~5 issues when the upstream email was sent. It got much better but in the meantime, at least one new was reported.

I've been thinking about this since the mail hit devel@, but I don't really like any of the choices here.

A question: how likely is an ABI change? I'm wondering this also in the context of, we already merged Python 3.11 into Rawhide, which involved a huge rebuild, right? Did we do that in the expectation that we might have to rebuild all those things again for a potential ABI change? Has that happened in the past? Does it really seem likely it'll happen now?

Can we push upstream to commit to the ABI freeze on the original schedule, even if the dates for RC and final release have to be delayed?

A question: how likely is an ABI change?

I've been told by @vstinner and @pviktori that it is likely to happen between 3.11.0b3 and 3.11.0b4. It might happen even later in case more betas are added.

I'm wondering this also in the context of, we already merged Python 3.11 into Rawhide, which involved a huge rebuild, right? Did we do that in the expectation that we might have to rebuild all those things again for a potential ABI change?

Yes, that is always the expectation. Note that we rebuilt cca 4k packages but if the ABI changes, we only need to rebuild 720 (repoquery --repo=rawhide $(repoquery --repo=rawhide --whatrequires 'libpython3.11.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.11' | grep -v 'noarch$') --source | pkgname | sort | uniq | wc -l).

Has that happened in the past?

No, but we needed to rebuild everything due to pyc magic number change in 3.8.

Does it really seem likely it'll happen now?

Now? Yes. Near October? Not so much, but we cannot count on that.

Okay. On that basis, I'm for #3 as well.

The last listed release blocker was fixed on the weekend. I assume we should get a new beta release today. However, this was not yet officially confirmed and the release hasn't happened yet.
Edited by churchyard

https://mail.python.org/archives/list/python-dev@python.org/message/VD7YNBVMEPKZLRA6VOTYAQYTEZPDH6Q7/

The fourth beta was released.

As the requirements to continue with the regular schedule are met, we are going to still target the final release of Monday, 2022-10-03.

Hence, this ticket was moot after all.

If the situation gets messy again, I will either reopen this or start a new one.

Metadata Update from @churchyard:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)

@churchyard Thanks for the ticket, anyway! I always like being prepared for the worst only to have a better outcome happen. 😉
Metadata
None

meeting system wide change

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.