I took a look at the package in question [1]. It was indeed retired because it failed to install [2].

When a retirement happens the stable tags get blocked in Koji altogether. And when an unretirement happens we manually unblock the requested tags.

For example, for the F43 tag:
$ koji list-history --package python-wxnatpy

This is when it got blocked (when it was retired):
Tue Aug 19 12:23:26 2025 package list entry for python-wxnatpy in f43 updated by toddlers/os-control01.rdu3.fedoraproject.org
blocked: False -> True

And this is when it got unblocked (when Samyak did it manually):
Sun Aug 24 12:43:14 2025 package list entry for python-wxnatpy in f43 updated by jnsamyak [still active]
blocked: True -> False

The SOP [3] we have states:
Verify the package was not retired for any reason, such as legal or license issues, that would prevent it from being re-instated.

In this case the reason for retirement was that it failed to install as stated in the commit. This should have been checked before reverting that commit.

Apologies for the inconvenience but to me personally this seems like a simple oversight (which is very human and happens to all of us) rather than an issue with the process itself. The build never got "tagged by releng". It was already tagged, the tag itself blocked and then unblocked.

If you think there is room for improvement with regards to how we handle unretirements I'd be happy to hear your thoughts.

[1] https://src.fedoraproject.org/rpms/python-wxnatpy
[2] https://src.fedoraproject.org/rpms/python-wxnatpy/c/af05f9ed6f22be1b4d274e71cffa9662d6a3063e?branch=rawhide
[3] https://docs.fedoraproject.org/en-US/infra/releng_misc_guide/sop_unretire/#_validate_package_ready_for_unretirement

Thanks for looking into this!

The build never got "tagged by releng". It was already tagged, the tag itself blocked and then unblocked.

That seems like exactly the problem. Perhaps the toddler should both block the package and untag all builds when packages are retired so that a new build is required after the package is unretired. In many cases, packages are retired because they are unmaintained or broken (whether they fail to install, fail to build, or something else) as part of the orphaned packages, FTI, or long-term FTBFS processes. Allowing the package to be unretired and the same build to be added back to repos, even if the package still fails to build or install weakens these processes.

That seems like exactly the problem. Perhaps the toddler should both block the package and untag all builds when packages are retired so that a new build is required after the package is unretired. In many cases, packages are retired because they are unmaintained or broken (whether they fail to install, fail to build, or something else) as part of the orphaned packages, FTI, or long-term FTBFS processes. Allowing the package to be unretired and the same build to be added back to repos, even if the package still fails to build or install weakens these processes.

These seem like very valid points. Allow me to ponder on it. Thank you kindly for the added context.

There were a bunch of historical reasons we didn't really want to do that... but I'm not sure any of them exist anymore... so we could do this.

For some more context, this [1] should be the toddler that handles blocking the tags.

I think the way it works right now is roughly this, happy to be corrected if I'm wrong...
The package gets retired, at some point the toddler notices it (through looking for the dead.package file) and blocks the tags that contain it.

So the idea here would be to first untag all builds (for all tags? or only for branches that contain dead.package?) and after that block the tags themselves? Or where would be the best place to handle the untagging?

https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/koji_block_retired.py

for all tags? or only for branches that contain dead.package?

The latter. We don't want the package untagged from f43 if it's only retired in rawhide.

For the record, there is no need to postpone any current unretirements because of this ticket.

I've talked some more about this with a few people. Here is what I have so far:

These FTBFS/FTI retirements/unretirements happen only occasionally. There is no need to mess with the tags via the toddler. We can add another step to our SOP (and reword/add some sections to it) and do it manually at the time of unretirement.

So...
1) The release engineer will check why the package was retired. If it was any other reason than FTBFS/FTI we proceed as usual, i.e. the way we've been doing it so far. If it IS one of those two reasons we will also...
2) koji untag-build --all PACKAGENAME TAG, and the TAG will be ONLY for those dist-git branches that contain dead.package.

Let's take a recent example.
https://src.fedoraproject.org/rpms/xmount/commits/f43

xmount was retired, for F43 and F44 (rawhide) because it FTI. In other words, these two dist-git branches contained the dead.package file. The retirement commit is now reverted in both branches.

In theory, I personally cannot see any harm in running:
$ koji untag-build --all xmount f43 and $ koji untag-build --all xmount f44.

However, I am aware that there are many things I may have missed. Can anybody think of any reason, historic or otherwise, that could potentially cause trouble if we do it as described above?

Please note that I have processed some unretirement requests in the meanwhile but haven't done any untagging of builds yet.

Thank you.

That sounds like a viable plan to me.

Yes, that seems perfectly reasonable to me. In most cases it won't be too much more work and it will insure that unretired packages are not re-added in a broken state.

One slight recommendation: when processing the unretirement request, we should add a 'Note that all old builds of the package have been untagged, please make sure you do new builds to re-introduce the package' or something like that? (or else some people won't realize they need to do a build to bring it back).

Untagging builds at unretirement time sounds feasible, but I would suggest always untagging builds — not just for certain packages. Requiring releng to make a distinction and determine whether packages FTI or FTBFS at the time the package was retired and only then untagging builds adds extra complication and inconsistencies to the process. Sometimes packages can be retired for multiple reasons, so even if the retirement commit says "Orphaned for 6+ weeks," it's very possible that the package FTBFS and/or FTI at the same time, so you'd have to also dig through the Bugzilla history to figure that out. It's also possible that a package was retired before the last mass rebuild so would also need to be rebuilt again or that it didn't FTI/FTBFS at the time of its retirement but that it would now due to other changes in rawhide.

I just think untagging builds at unretirement and requiring maintainers to always take specific action to make sure the package still builds is a good general policy. If someone has a really compelling reason to need to reuse old builds — which I don't expect — I guess they can still request that in the unretirement ticket.

Sounds reasonable to me.

Proposed approach: untag at unretirement, not at retirement. This is more feasible operationally and avoids blanket untagging of all historical builds at the moment of retirement, which can be counterproductive when a package was retired for workflow reasons (e.g. orphaning) rather than because the builds were actually bad. Instead, keep a reasons-driven path to untagging at unretirement time: if the retirement/unretirement context indicates legal/licensing/security/breakage, untag the affected builds then; if the package was merely orphaned but otherwise sound, leave historical builds tagged. This keeps history intact, reduces unnecessary churn, and focuses destructive actions only when there’s a documented reason to do so.

In short, my opinion to go with it defer untagging decisions to unretirement with clear reason codes, and stop doing broad untagging at retirement. And, doing this manually is best then implementing it in toddlers, anways the toddlers for unretirement is a dream now, not sure what's it status is right now?

For context I believe this [1] is the toddler in question that might do the UNretirements automatically in the future, perhaps @amedvede can comment on that. But as of right now we process them manually.

As for our process, @gotmax23 raises a really good point:

Sometimes packages can be retired for multiple reasons, so even if the retirement commit says "Orphaned for 6+ weeks," it's very possible that the package FTBFS and/or FTI at the same time, so you'd have to also dig through the Bugzilla history to figure that out.

I'm updating the SOP to make it more explicit what to look for. Would that be sufficient? Apart from this one point I'm inclined to agree with Samyak's approach, blanket untagging of ALL builds for EACH retired package seems a bit excessive.

Could we perhaps make it so that all the builds are untagged automatically but only when the package is retired for FTBFS/FTI reasons? I would like to hear your thoughts on the feasibility of this approach @churchyard as you seem to be the one [2] that usually handles the mass retirements.

Thank you.

[1] https://pagure.io/fedora-infra/toddlers/blob/main/f/toddlers/plugins/unretire_packages.py
[2] https://pagure.io/releng/issue/12871

I think blanket untagging of ALL builds for EACH (un)retired package is the way to go.

When packages are unretired, it is expected that the new maintainer can at least do a build. For special situations (complex bootstrap, accidental retirement, etc.), they can request manual retagging if needed.

When packages are unretired, it is expected that the new maintainer can at least do a build. For special situations (complex bootstrap, accidental retirement, etc.), they can request manual retagging if needed.

That makes sense. I'm okay with this, just in case we need to tell them on the ticket, the untagging is done. For the builds, you are requested to build one post-unretirement. In case you want us to retag a build, please let us know.

Apologies folks, it was explained to me during the releng weekly meeting that I may have completely misunderstood some of the points that were made and we may all be in agreement after all.

Just to double check, the proposed way to handle the UNretirement of EACH package REGARDLESS of the reason it was retired:
1) Do it as usual.
2) Untag ALL of the builds ONLY for the branches/tags that were requested to be unretired in the ticket.

There is no reason to mess with the toddler itself that does the retirements, all that is needed is to update our (manual) process and thus the SOP.

Please give THUMBS UP if you agree with this approach and THUMBS DOWN if you disagree.
In case of disagreement please write down your reasoning in the comment.

Thank you.

@gotmax23 @kevin @jnsamyak @churchyard

For the last few unretirements I have processed recently I already untagged the builds per the new process and nobody complained so I think we're good.

Here is the PR that updates the SOP:
https://pagure.io/infra-docs-fpo/pull-request/407

Reviews are welcome and if it looks good to you we can merge it and close this ticket. :thumbsup:

The PR that updates the SOP is now merged and this shouldn't be an issue in the future.

Thank you everyone for your patience and feedback.