@mattia can you see any easy way to test this in staging?

I don't know if the Koji prod db to the stg db can be synchronized, so that we could already have a lot of "new" builds that bodhi stg is not aware of...
Those builds could then be tagged in a side-tag and a new side-tag update submitted in bodhi stg. At least the initial tagging of all builds in pending-signing should work, after that I don't know if bodhi stg correctly lists to the message queue to process later steps.

We could do a sync of course, but not sure there would be sufficent builds that it doesn't know about? well, I guess it depends on when the last time we synced the bodhi database?

Bodhi stg db was synced last time two months ago, there F43 is still Rawhide. There are currently ~8.900 updates in prod which shouldn't be known by bodhi stg. The count surely has multiple updates for the same package, but I think we can reach a couple of thousand of unique packages.

ok. I can start a sync. It usually takes a day or so.

The sync is all done.

Can you look at making the update? or who would like to work on that? :)

Looks like I have access to koji and bodhi stg, but where can I find the builds that were just synced?

I think I don't quite understand the question, anyway we'll need to:
- create a side-tag in koji.stg
- for each package, get a list of the latest build for F43 on koji.stg (F43 because it is still "Rawhide" in bodhi.stg)
- if the build is not present in bodhi.stg, we'll need to make sure to remove all tags and add it to the side-tag
- create the side-tag update in bodhi-stg

Does anyone see any simpler way?

f43-build-side-117736 created on koji.stg

I've not been really much around lately... I will try to write a script to extract/compare/tag builds in the side-tag as soon as I get some time (a week? perhaps two...)

Ok, I've tagged ~23k builds in f43-build-side-117736 (koji is still running tag tasks ATM).
For reference, here it's what I ran (in staging bodhi-web pod terminal to overcome koji authentication and authorization):

import rpm
koji_client = request.koji
# get all F43 builds
koji_client.multicall = True
koji_client.listTagged('f43-updates-candidate', latest=True)
koji_client.listTagged('f43-updates-testing', latest=True)
koji_client.listTagged('f43-signing-pending', latest=True)
koji_client.listTagged('f43-updates-testing-pending', latest=True)
koji_client.listTagged('f43-updates-pending', latest=True)
koji_client.listTagged('f43', latest=True)
koji_client.listTagged('f43-updates', latest=True)
response = koji_client.multiCall()
# get only the latest builds and that aren't already in bodhi stg
result = dict()
pkgset = set()
for taglist in response:
    if isinstance(taglist, dict):
        continue
    else:
        for build in taglist[0]:
            item = {'nvr': build['nvr'],'id': build['id'],'package_name': build['package_\
name'],'owner_name': build['owner_name'],}
            if m.Build.get(item['nvr']) is not None:
                continue
            if not item['package_name'] in pkgset:
                pkgset.add(item['package_name'])
                result[item['package_name']] = item['nvr']
            else:
                if rpm.labelCompare(item['nvr'], result[item['package_name']]):
                    result[item['package_name']] = item['nvr']
# filter out builds from older releases (maybe pkg was retired?)
nvrs = []
for pkg, nvr in result.items():
    if nvr.endswith('fc43'):
        nvrs.append(nvr)
# tag in side-tag
koji_client.multicall = True
for build in nvrs:
    koji_client.tagBuild('f43-build-side-117736', build, force=True)
koji_client.multiCall()

... I think I might have clogged koji stg... it still running tagging tasks (ca ~1.8k left)!
This might happen in prod as well when bodhi will start to move the builds of such large updates. Perhaps there are limited resources in stg in comparison to prod, so the impact on prod will be lesser.

Yeah, stg koji has vastly smaller amount of builders... and in fact I need to rebuild some since the dc move, so it even has less than normal. :)

It seems like it's done?

https://bodhi.stg.fedoraproject.org/updates/FEDORA-2025-3d83cfd7e4

So, BTW, it wasn't done I don't think... the staging builders got in a weird state and were not checking in.

I fixed that (and moved them all to f43) eariler this week.

It had a backlog of about 13,000 tag tasks.

it's got about 2300 left. Should finish later tonight.

The question I suppose is if they got in a weird state because of this update?