releng

#12833 Fedora 43 Mass Rebuild Tracker
Closed: Fixed by jnsamyak. Opened by jnsamyak.

Closed: Fixed
Reopen Issue

The Fedora 43 schedule[1] has a mass rebuild scheduled for Wed 2025-07-23, We need to plan and coordinate all tasks in preparation for it. For the driving changes please refer[2].

Please note that if you need to exclude any packages from the rebuild,
you can use the PKG_SKIP_LIST or add a noautobuild file to the root of
your distgit repository. This will instruct the mass rebuild script to
skip these packages. Previously, some OCaml packages were rebuilt, but
they can now be opted out using this method if required.

[1] https://fedorapeople.org/groups/schedule/f-43/f-43-key-tasks.html
[2] https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild#Driving_Features

Metadata Update from @jnsamyak:
- Issue tagged with: high-gain, high-trouble

Metadata Update from @jnsamyak:
- Issue assigned to jnsamyak

That should be "2025-07-23" I guess?

@dreua, correct!

The tags & targets have been added, and the PR has been opened for the autosigning those!
I'll hold on to those details tomorrow to check before starting the process, it should be scheduled for 14 UTC.

PRs to review:
- https://pagure.io/fedora-infra/ansible/pull-request/2761
- https://pagure.io/releng/pull-request/12834

cc: @kevin @patrikp @humaton @amedvede

Edited by jnsamyak

While investigating the mass retirement script today I found out these are included in the need-rebuild but not in the pkg_skip_list, is there any specific reasons we just are missing out?

❌ Repo not found: Fedora-AtomicHost
❌ Repo not found: Fedora-AtomicHost-Vagrant
❌ Repo not found: Fedora-COSMIC-Disk
❌ Repo not found: Fedora-COSMIC-Live
❌ Repo not found: Fedora-Cloud-Atomic
❌ Repo not found: Fedora-Cloud-Atomic-Vagrant
❌ Repo not found: Fedora-Cloud-Base
❌ Repo not found: Fedora-Cloud-Base-Vagrant
❌ Repo not found: Fedora-Docker-Base
❌ Repo not found: Fedora-FEX-RootFS
❌ Repo not found: Fedora-KDE-Desktop-Disk
❌ Repo not found: Fedora-KDE-Desktop-Live
❌ Repo not found: Fedora-KDE-Mobile-Disk
❌ Repo not found: Fedora-KDE-Mobile-Live
❌ Repo not found: Fedora-LXQt-Disk
❌ Repo not found: Fedora-MiracleWM-Disk
❌ Repo not found: Fedora-MiracleWM-Live
❌ Repo not found: Fedora-Python-Classroom-Disk
❌ Repo not found: Fedora-Python-Classroom-Vagrant-VirtualBox
❌ Repo not found: Fedora-Python-Classroom-Vagrant-libvirt
❌ Repo not found: Fedora-Scientific-Vagrant
❌ Repo not found: Fedora-Scientific-Vagrant-VirtualBox
❌ Repo not found: Fedora-Scientific-Vagrant-libvirt
❌ Repo not found: Fedora-Scientific_KDE-Disk
❌ Repo not found: Fedora-Server-Guest-Generic
❌ Repo not found: Fedora-Server-Host-Generic
❌ Repo not found: Fedora-SoaS-Disk
❌ Repo not found: Fedora-WSL-Base
❌ Repo not found: Fedora-Workstation-Disk
❌ Repo not found: Fedora-Xfce-Disk
❌ Repo not found: Fedora-i3-Live
❌ Repo not found: andy-super-great-park
❌ Repo not found: fedora-container-image
❌ Repo not found: module-build-macros
❌ Repo not found: python3-doctutils

A recent update to gettext-0.25.1, has been pushed to rawhide :
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-c4d76ee10c
- https://koji.fedoraproject.org/koji/taskinfo?taskID=135147460
Edited by matiwari

@matiwari, I see it's pending for now, we have exactly an hour to run the rebuild!

A recent update to gettext-0.25.1, has been pushed to rawhide :
- https://bodhi.fedoraproject.org/updates/FEDORA-2025-c4d76ee10c

~~Unfortunately it is still gated mostly due to some CI infra issues, though we would like to use it for the mass rebuild if possible.
Dunno if you want to tag into the buildroot anyway?: it has been tested in a couple of copr mass rebuilds anyway.~~

Actually I think it is probably stuck in the signing queue?

Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43
Edited by petersen

Any chance you could kick the robosignatory before the rebuild starts? The rawhide updates have not progressed for a while

https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=rabbitmq_slash_pubsub;plugin=queues;plugin_instance=robosignatory;type=messages;begin=-43200

Also I believe @fberat is about to drop in a late autoconf patch: autoconf.2.72-6.fc43

The build is running:
https://koji.fedoraproject.org/koji/taskinfo?taskID=135152050

Edit: https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9c3ce5e46
Edited by fberat

@kevin just confirmed the vault was crashed, he restarted it, so it should be back soon! I'll see if everything looks fine will try to fire in 20 mins?

@jnsamyak Any chance both gettext and autoconf updates are taken in before the mass-rebuild kicks in ?

@fberat, I'll wait for these to sign and see if it gets into stable timely! :D

gettext -> stable
autoconf -> stable

@fberat all is good and done, we should be okay now!
Edited by jnsamyak

Mass Rebuild has been fired!

It looks like there is a problem with debuginfo in golang packages:

e.g.
https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702
error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list

https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495
error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list

It looks like there is a problem with debuginfo in golang packages:

e.g.
https://koji.fedoraproject.org/koji/taskinfo?taskID=135189702
error: Empty %files file /builddir/build/BUILD/golang-github-cloudflare-redoctober-0-build/redoctober-6a74ccc611f655e1ee9a648062a6ececfb6f2bb8/debugsourcefiles.list

https://koji.fedoraproject.org/koji/taskinfo?taskID=135180495
error: Empty %files file /builddir/build/BUILD/geoipupdate-7.1.1-build/geoipupdate-7.1.1/debugsourcefiles.list

That is upstream debugedit bug https://sourceware.org/bugzilla/show_bug.cgi?id=33204
golang 1.25 generates dwarf5 by default that is incompatible with debugedit

Does the mass rebuild .spec file update script not understand %autochangelog (yet)?

For example gnome-software had this in the .spec file:

%changelog
* Wed Jul 23 2025 Fedora Release Engineering <releng@fedoraproject.org> - 49~alpha-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_43_Mass_Rebuild
%autochangelog

I expect the mass rebuild will not add the lines when the %autochangelog is used.

@mcrha it does understand %autochangelog and in combination with %autorelease it even creates an empty commit (example). Not sure why it didn't work in this case, maybe would be best to create a separate issue to look into it. (Are there more examples where you have seen this?)
Edited by dreua

It looks like packages which use only %autochangelog are affected. The gnome-shell is fine, but evolution is not. Where do I fill the bug, just here or the script has its own home, please?
Edited by mcrha

You're right, it certainly looks like autochangelog without autorelease is triggering this. From a quick grep through "rpm-specs-latest" it looks like about 88 (false positives are possible) packages are affected by this. Many already have some added changelog entries above the %autochangelog entry from other rebuilds.

[... rpm-specs]$ grep -L "autorelease" $(grep -l "autochangelog" *) | wc -l
88

I'm not an expert here but from a look at the scripts I'd say the mass rebuild is using rpmdev-bumpspec for this task which has its own source repo here:
https://pagure.io/rpmdevtools/issues

Thanks for the pointer. I filled https://pagure.io/rpmdevtools/issue/127 , though the project does not look like the most active, thus we'll see whether it'll get fixed any time soon.

Mass rebuild is finished, everything is signed and tagged to f43; There were some blockers which shouldn't have been tagged (so I have untagged those); Finally, a rawhide is fired now, :v: hopefully nothing breaks!

We have our first compose on Monday!

Approximately 23010 builds have been tagged into f43;
There are currently 1236 failed builds that need to be addressed
by the package maintainers.  FTBFS bugs will be filed shortly.

Metadata Update from @jnsamyak:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)
Metadata

high-gain high-trouble

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.