It looks like the process for dealing with retired packages is not restored yet after the f41 final freeze ended and / or is just not working. I have fedpkg retired packages up to 5 days ago that are still not blocked in koji or removed from repos:
fedpkg retire
Can the new toddler code from staging be deployed to prod? Or can the check-for-dead.package-and-block-manually script be run again some time this week, please?
Soon, but it's not urgent - though before F42 branch point (2025-02-04) or F42 final freeze (2025-04-01) would be preferable :wink: Without this, many processes we have in Fedora (i.e. retirement of 6-week-orphaned packages) don't work.
N/A
Packages that should get removed from repositories are not.
Metadata Update from @phsmoura: - Issue tagged with: medium-gain, medium-trouble, ops
Can you please also retire the following packages from rawhide?
The epel9 blockings are processed by the toddler automatically, I just confirmed.
The rawhide blockings are waiting for this PR to go to prod: https://pagure.io/fedora-infra/toddlers/pull-request/295 (There was a problem with translating between f and "rawhide", dunno if something changed or I overlooked it before, but I didn't count with the "rawhide" in release => branch field in bodhi relase description.)
Can you please also retire the following packages from rawhide? RediSearch golang-github-emersion-milter golang-github-emersion-msgauth pcc rebloom rejson
These are all blocked.
The script shows these retired packages are not blocked yet:
rawhide dovecot-fts-xapian erlang-riak_core mingw-physfs python-ouimeaux python-pysignals rust-bytes0.5 rust-calloop0.12 rust-freetype-rs0.26 rust-input-linux-sys0.8 rust-libbpf-rs0.21 rust-servo-fontconfig
rawhide
note that several of these were actually unretired earlier today. :)
We need to make sure and not run things until after the retired packages files update.
Running the script this morning a see there is a new retired package on f40: java-1.8.0-openjdk. I thought the retirements don't happen on stable branches?
f40
java-1.8.0-openjdk
Metadata Update from @lenkaseg: - Issue untagged with: medium-gain, medium-trouble, ops
Ok, the current list of retired and not blocked in koji for rawhide is:
mingw-physfs python-ouimeaux python-pysignals rust-bytes0.5 rust-calloop0.12 rust-freetype-rs0.26 rust-input-linux-sys0.8 rust-libbpf-rs0.21 rust-servo-fontconfig
They shouldn't... but what happened is that they merged rawhide back to f40 and rawhide was retired (and f41, but they have another ticket to reverse that).
we should just have the script not act on stable releases normally...
php-psr-http-client python-nose-timer
also. Thanks!
Please disregard my last comment. Those are already blocked.
It looks like new retirements are processed correctly now, and epel9 stuff got processed too. So from the original list I posted, only these are remaining:
@decathorpe Sorry for the delay, I got sick this week. I just landed the PR and deployed it to production, it seems to work and all those packages you mentioned are blocked now plus some more that needed blocking:
blocked in f42: javapackages-tools-epel mingw-physfs python-ouimeaux python-pysignals rust-bytes0.5 rust-calloop0.12 rust-freetype-rs0.26 rust-input-linux-sys0.8 rust-libbpf-rs0.21 rust-libhandy rust-libhandy-sys
blocked in f42
I think the toddler starts to look good. I have only two minor changes in mind, one has the PR open already (removing the epel unblocking section), and second would be not even considering trying to block on stable releases. And another thing, to make sure everything retired is blocked before the freeze starts, which means probably blocking new retirements at some point, then run the koji blocking script and then going to freeze? rust-servo-fontconfig
Thanks, and no problem - hope you're feeling better now!
rust-libhandy and rust-libhandy-sys were false positives though, and they should get un-blocked from the f42 tag again ... I just un-retired them today: https://pagure.io/releng/issue/12458
I'm not sure why these two packages didn't get unblocked in koji as part of the unretirement process, but I unblocked them now.
Metadata Update from @lenkaseg: - Issue tagged with: medium-gain, medium-trouble, ops
@decathorpe Is there something more to be solved for this issue, or it can be closed now?
I think this is the race condition I was thinking of. Consider:
So I am not sure the best way around this. I guess we could try and make sure the daily toddler always runs after the script that updates the retired packages lists?
Generating the retired packages json files happens nightly at 4:30 and the koji blocking toddler is scheduled to run at0 4:15. I will move the toddler to 5:30 then, allowing enough time for the json files to generate (the script usually takes ~ 20 mins.
I'm not sure I understand why the unretirement process didn't unblock the packages in koji. So the package gets unblocked in the process of unretirement, but the toddler blocks it again, because it's still in the retired json file. Is that correct?
Opened a PR here to fix the schedule: https://pagure.io/fedora-infra/ansible/pull-request/2377
I'm not sure I understand why the unretirement process didn't unblock the packages in koji.
It did. I checked that rust-libhandy and rust-libhandy-sys were unblocked before I built and submitted this update: https://bodhi.fedoraproject.org/updates/FEDORA-2024-2954d32134 (hence race condition)
Maybe it would work to check the packages from the "candidates for blocking" list individually just before actually blocking them? The list of candidates should be small every time this runs, so I don't think it would make things too slow. It wouldn't fix the race condition entirely, but it would make the window as short as a few seconds.
Perhaps this is more quantifiable:
➜ ~ koji list-history --package rust-libhandy --after 2024-11-10 Thu Nov 14 04:54:11 2024 package owner changed for rust-libhandy in f42 by patrikp Thu Nov 14 04:54:11 2024 package list entry for rust-libhandy in f42 updated by patrikp blocked: True -> False Thu Nov 14 04:54:17 2024 package owner changed for rust-libhandy in f41 by patrikp Thu Nov 14 04:54:17 2024 package list entry for rust-libhandy in f41 updated by patrikp [still a ctive] blocked: True -> False Thu Nov 14 06:42:01 2024 package owner changed for rust-libhandy in f42 by bodhi owner.name: orphan -> decathorpe ... Thu Nov 14 15:33:34 2024 package owner changed for rust-libhandy in f42 by toddlers/os-control01 .iad2.fedoraproject.org Thu Nov 14 15:33:34 2024 package list entry for rust-libhandy in f42 updated by toddlers/os-cont rol01.iad2.fedoraproject.org blocked: False -> True
Here is PR that should make the check if the package is really retired (call to distgit for presence of dead.package on particular branch) for candidates for blocking (last check just before blocking) https://pagure.io/fedora-infra/toddlers/pull-request/307#
And here PR that should not allow the toddler to block on stable releases if by some accident a package would be retired anyways or a package would retire just in the window of couple of seconds when the candidates for blocking are being checked on distgit if really retired. https://pagure.io/fedora-infra/toddlers/pull-request/309
Is this still an issue or can this be closed? CC: @lenkaseg @decathorpe
This is no longer an issue, as far as I can tell, since the toddlers / poddlers were fixed ... so I'll just go ahead and close this as Fixed. :)
Metadata Update from @decathorpe: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)