#12439 package retirements are not getting processed
Closed: Fixed by decathorpe. Opened by decathorpe.

  • Describe the issue

It looks like the process for dealing with retired packages is not restored yet after the f41 final freeze ended and / or is just not working. I have fedpkg retired packages up to 5 days ago that are still not blocked in koji or removed from repos:

  • rust-calloop0.12 (rawhide)
  • rust-freetype-rs0.26 (rawhide)
  • rust-servo-fontconfig (rawhide)
  • rust-password-hash0.3 (epel9)
  • rust-unicode-xid0.1 (epel9)
  • rust-bytes0.5 (rawhide)
  • rust-input-linux-sys0.8 (rawhide)
  • rust-libbpf-rs0.21 (rawhide)

Can the new toddler code from staging be deployed to prod? Or can the check-for-dead.package-and-block-manually script be run again some time this week, please?

  • When do you need this? (YYYY/MM/DD)

Soon, but it's not urgent - though before F42 branch point (2025-02-04) or F42 final freeze (2025-04-01) would be preferable :wink: Without this, many processes we have in Fedora (i.e. retirement of 6-week-orphaned packages) don't work.

  • When is this no longer needed or useful? (YYYY/MM/DD)

N/A

  • If we cannot complete your request, what is the impact?

Packages that should get removed from repositories are not.


Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble, ops

Can you please also retire the following packages from rawhide?

  • RediSearch
  • golang-github-emersion-milter
  • golang-github-emersion-msgauth
  • pcc
  • rebloom
  • rejson

The epel9 blockings are processed by the toddler automatically, I just confirmed.

The rawhide blockings are waiting for this PR to go to prod: https://pagure.io/fedora-infra/toddlers/pull-request/295
(There was a problem with translating between f and "rawhide", dunno if something changed or I overlooked it before, but I didn't count with the "rawhide" in release => branch field in bodhi relase description.)

Can you please also retire the following packages from rawhide?

  • RediSearch
  • golang-github-emersion-milter
  • golang-github-emersion-msgauth
  • pcc
  • rebloom
  • rejson

These are all blocked.

The script shows these retired packages are not blocked yet:

rawhide
dovecot-fts-xapian
erlang-riak_core
mingw-physfs
python-ouimeaux
python-pysignals
rust-bytes0.5
rust-calloop0.12
rust-freetype-rs0.26
rust-input-linux-sys0.8
rust-libbpf-rs0.21
rust-servo-fontconfig

note that several of these were actually unretired earlier today. :)

We need to make sure and not run things until after the retired packages files update.

Running the script this morning a see there is a new retired package on f40: java-1.8.0-openjdk. I thought the retirements don't happen on stable branches?

Metadata Update from @lenkaseg:
- Issue untagged with: medium-gain, medium-trouble, ops

Ok, the current list of retired and not blocked in koji for rawhide is:

mingw-physfs
python-ouimeaux
python-pysignals
rust-bytes0.5
rust-calloop0.12
rust-freetype-rs0.26
rust-input-linux-sys0.8
rust-libbpf-rs0.21
rust-servo-fontconfig

Running the script this morning a see there is a new retired package on f40: java-1.8.0-openjdk. I thought the retirements don't happen on stable branches?

They shouldn't... but what happened is that they merged rawhide back to f40 and rawhide was retired (and f41, but they have another ticket to reverse that).

we should just have the script not act on stable releases normally...

Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble, ops

php-psr-http-client
python-nose-timer

also. Thanks!

Please disregard my last comment. Those are already blocked.

Please disregard my last comment. Those are already blocked.

It looks like new retirements are processed correctly now, and epel9 stuff got processed too. So from the original list I posted, only these are remaining:

  • rust-calloop0.12 (f42 tag)
  • rust-freetype-rs0.26 (f42 tag)
  • rust-servo-fontconfig (f42 tag)
  • rust-bytes0.5 (f42 tag)
  • rust-input-linux-sys0.8 (f42 tag)
  • rust-libbpf-rs0.21 (f42 tag)

@decathorpe Sorry for the delay, I got sick this week. I just landed the PR and deployed it to production, it seems to work and all those packages you mentioned are blocked now plus some more that needed blocking:

blocked in f42:
javapackages-tools-epel
mingw-physfs
python-ouimeaux
python-pysignals
rust-bytes0.5
rust-calloop0.12
rust-freetype-rs0.26
rust-input-linux-sys0.8
rust-libbpf-rs0.21
rust-libhandy
rust-libhandy-sys

I think the toddler starts to look good. I have only two minor changes in mind, one has the PR open already (removing the epel unblocking section), and second would be not even considering trying to block on stable releases.
And another thing, to make sure everything retired is blocked before the freeze starts, which means probably blocking new retirements at some point, then run the koji blocking script and then going to freeze?
rust-servo-fontconfig

Thanks, and no problem - hope you're feeling better now!

rust-libhandy and rust-libhandy-sys were false positives though, and they should get un-blocked from the f42 tag again ... I just un-retired them today: https://pagure.io/releng/issue/12458

I'm not sure why these two packages didn't get unblocked in koji as part of the unretirement process, but I unblocked them now.

Metadata Update from @lenkaseg:
- Issue untagged with: medium-gain, medium-trouble, ops

Metadata Update from @lenkaseg:
- Issue tagged with: medium-gain, medium-trouble, ops

@decathorpe Is there something more to be solved for this issue, or it can be closed now?

I'm not sure why these two packages didn't get unblocked in koji as part of the unretirement process, but I unblocked them now.

I think this is the race condition I was thinking of. Consider:

  • package foo is unretired, releng unblocks it.
  • the daily retirements processor runs, but package foo is still in the blocked packages json files
  • the daily generaton for retired packages runs.

So I am not sure the best way around this. I guess we could try and make sure the daily toddler always runs after the script that updates the retired packages lists?

Generating the retired packages json files happens nightly at 4:30 and the koji blocking toddler is scheduled to run at0 4:15. I will move the toddler to 5:30 then, allowing enough time for the json files to generate (the script usually takes ~ 20 mins.

I'm not sure I understand why the unretirement process didn't unblock the packages in koji.
So the package gets unblocked in the process of unretirement, but the toddler blocks it again, because it's still in the retired json file. Is that correct?

Metadata Update from @lenkaseg:
- Issue untagged with: medium-gain, medium-trouble, ops

Metadata Update from @lenkaseg:
- Issue tagged with: medium-gain, medium-trouble, ops

Opened a PR here to fix the schedule: https://pagure.io/fedora-infra/ansible/pull-request/2377

I'm not sure I understand why the unretirement process didn't unblock the packages in koji.

It did. I checked that rust-libhandy and rust-libhandy-sys were unblocked before I built and submitted this update: https://bodhi.fedoraproject.org/updates/FEDORA-2024-2954d32134 (hence race condition)

Maybe it would work to check the packages from the "candidates for blocking" list individually just before actually blocking them? The list of candidates should be small every time this runs, so I don't think it would make things too slow. It wouldn't fix the race condition entirely, but it would make the window as short as a few seconds.

Perhaps this is more quantifiable:

  ~ koji list-history --package rust-libhandy --after 2024-11-10
Thu Nov 14 04:54:11 2024 package owner changed for rust-libhandy in f42 by patrikp              
Thu Nov 14 04:54:11 2024 package list entry for rust-libhandy in f42 updated by patrikp
    blocked: True -> False                                                                      
Thu Nov 14 04:54:17 2024 package owner changed for rust-libhandy in f41 by patrikp
Thu Nov 14 04:54:17 2024 package list entry for rust-libhandy in f41 updated by patrikp [still a
ctive]      
    blocked: True -> False                                                                      
Thu Nov 14 06:42:01 2024 package owner changed for rust-libhandy in f42 by bodhi
    owner.name: orphan -> decathorpe 
...
Thu Nov 14 15:33:34 2024 package owner changed for rust-libhandy in f42 by toddlers/os-control01
.iad2.fedoraproject.org
Thu Nov 14 15:33:34 2024 package list entry for rust-libhandy in f42 updated by toddlers/os-cont
rol01.iad2.fedoraproject.org
    blocked: False -> True

Here is PR that should make the check if the package is really retired (call to distgit for presence of dead.package on particular branch) for candidates for blocking (last check just before blocking)
https://pagure.io/fedora-infra/toddlers/pull-request/307#

And here PR that should not allow the toddler to block on stable releases if by some accident a package would be retired anyways or a package would retire just in the window of couple of seconds when the candidates for blocking are being checked on distgit if really retired.
https://pagure.io/fedora-infra/toddlers/pull-request/309

Is this still an issue or can this be closed?
CC: @lenkaseg @decathorpe

This is no longer an issue, as far as I can tell, since the toddlers / poddlers were fixed ... so I'll just go ahead and close this as Fixed. :)

Metadata Update from @decathorpe:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata