Kevin asked me to add a link to this anecdotal build which took 37 hours. This is a little extreme, but also not entirely unusual. Kevin says: "the vm was OOM killed a few times and I had to restart it."

Um, notably I think all architectures other than s390x are actually operating more or less fine. I haven't even noticed any differences.

s390x is currently much slower than everything else.

ok. I have made the following changes today:

aarch64: removed all the buildvm-a64's from the heavybuilder channel, and brought online 5 more aarch64 buildhw's. That has a total of 7 in. 01/02 are 'bigger' but 03-07 are still not bad
aarch64: removed the buildvm-a64's from secure-boot channel so kernel builds will be faster hopefully.

x86_64: removed all the buildvm-x86's from the heavy-builder channel and brought online 5 more x86_64 buildhw's. That is a total of 8 in. They are various sizes, but all of them are bigger than the buildvm's. We also have a few more to bring on line soon.
x86_64: removed all the buildvm-x86's from secure-boot channel so kernel builds will be faster hopefully.

ppc64le: nothing right now. We are working on reconfiguring our power10s, but need some new drives and time to reconfigure.

s390x: I have disabled a compose machine there that I think we can get by without. I am waiting to see if it affects composes tomorrow. If it doesn't, I will cull it and add it's resources to the heavybuilder there.

CC: @than (chromium), @catanzaro (webkitgtk), @jforbes (kernel), @stransky (firefox)

I would appreciate feedback from all of you in coming days as you do builds, happy to do what I can to make things faster based on the constraints we have.

Hi Kevin,

I've tested your recent changes with the new Chromium builds for Fedora 41/42/43 and EPEL 9/10. The changes performed perfectly for all Chromium builds.

Architecture CPUs Used Build Time
x86_64 32 3 hours, 18 minutes
80 2 hours, 12 minutes
192 (Maximum) 1 hour, 9 minutes
aarch64 80 (Fixed) 1 hour, 36 minutes
ppc64le N/A Unchanged (No modifications were applied to this architecture.)

The significant speed improvements are very impressive. Thank you for these valuable changes; they're greatly appreciated by the team.

I'd love to join this with MariaDB and MySQL projects, @kevin

What is the process for start using the heavy buidlers?
Is it configured per-package?
MariaDB and MySQL packages are now versioned (mariadb10.11, mariadb11.8, mysql8.0, mysql8.4) and we expect to continue this trend with new upstream LTS releases.

Well, I really really really want to avoid adding more things to that channel if we can manage with out it. :)

The problem is that the more things that are added there the more that limited set of hosts will be busy, causing other packages in that channel to wait to even start.
For s390x there's only one heavybuilder instance, but there's only firefox and webkitgtk there, chromium doesn't build for s390x
For ppc64le, it's just basically all the normal builders.
For x86_64 there are 8 of them now
For aarch64 there are 7 of them now.

With mariadb / mysql... is it just long build times? Or does it have some kind of issue/failure on smaller builders?
Also, do you tend to do a bunch of builds at a time? Or are they spread out over time?
And are those packages building on all arches?

Another disadvantage of using heavybuilder channel: the architecture used to build the SRPM seems to be selected randomly, and if it's s390x then you have to wait for (all?) other s390x builds to finish before the SRPM can be built, blocking all other architectures. It's also routine to wait more than 8, 12, or even 24 hours before an s390x build can start. So it will probably actually make your builds slower.

You want to use heavybuilder if you have reliability problems with the standard builders.

srpm builds should no longer use s390x. We kept it in there a long time with the idea that it would be nice to sometimes build noarch packages on s390x in case of weird problems, but the python mass rebuilds hit that hard and so we dropped s390x from noarch tasks now.

but yeah, if there's 5 firefox builds going you will in line behind them on s390x. ;(

I just did a WebKitGTK scratch build and the results are good: 6 hours for s390x, 3h22m for ppc64le, 3h24m for aarch64, 2h24m for i686, and 1h20m. This is a best-case scenario since I was building only for rawhide. I'll test all four branches simultaneously (a more typical scenario) on Monday.

OK, I just performed four simultaneous WebKitGTK builds. All times are total time except when task time is significantly different from total time, then I have provided them both.

• Build 1 finished in 13:31:23.
• Build 2 finished in 13:34:57.
• Build 3 finished in 14:52:21.
• Build 4 finished in 21:23:42.

i686:

• Build 1: 0:49:24
• Build 2: 0:56:43
• Build 3: 1:06:46
• Build 4: total time 1:34:54, task time 0:49:58

x86_64:

• Build 1: 1:20:47
• Build 2: 0:51:53
• Build 3: 1:42:50
• Build 4: total time 1:56:04, task time 1:47:14

aarch64:

• Build 1: 1:04:24
• Build 2: 1:03:44
• Build 3: 1:11:32
• Build 4: 3:25:08

ppc64le:

• Build 1: 3:59:42
• Build 2: 3:58:59
• Build 3: 4:55:04
• Build 4: 5:37:58

s390x:

• Build 1: total time 13:27:01, task time 12:59:40
• Build 2: total time 13:29:51, task time 13:03:35
• Build 3: total time 14:47:50, task time 14:21:24
• Build 4: total time 21:19:57, task time 6:35:43

Conclusion: ppc64le is slow. s390x is extremely slow. aarch64 is usually fast, but sometimes it is slow. i686 and x86_64 are consistently fast. Every build finished reliably, which is more important than speed.

For comparison, I also performed two downstream builds on the same architectures, both of which finished in roughly one hour.

Yeah... expected.

On ppc64le, we are still working to reorg things there. After that 1/2 of the builders will be on one host and 1/2 on another (instead of right now all of them are on the same hypervisor.

On s390x I ran out of time before freeze, but will see what I can do about reorging those also. Possibly scrapping some of the normal ones and adding another heavybuilder or just more resources for the one.

Well, I really really really want to avoid adding more things to that channel if we can manage with out it. :)

Ah, I got an echo saying that you actually look for more packages to be added to the heavy builder channel, which was the reason I came here to ask for that.
Information noise, it seems. 🙂

--

However, for the record:

And are those packages building on all arches?

MySQL 8.4 and later dropped i686 support.
Other than that, both builds on all arches.

With mariadb / mysql... is it just long build times? Or does it have some kind of issue/failure on smaller builders?

While not a great solution, we do run the package testsuite during the build (in the %check phase), which has either 1000 (main suite) or 4000 (big suite) tests. (Those are the only two modes we run)

The big suite is run on rebases, and on subsequent package release bumps only the main suite is triggered.

We occasionally hit "out of resources" errors on some architectures.
The ppc64le is the most problematic in this matter, up to a point that we have to sometime disable the testing completely as it will fail in 100% cases, for few weeks / months. Which is not great.

Also, do you tend to do a bunch of builds at a time? Or are they spread out over time?

We do repeated scratch builds to check for instabilities in the suites.
Which means huge load from time to time.

--

The build time of the DBs themselves, without running any tests, is usually 30 - 60 - 120 minutes.
Varies greatly based on architecture. An hour is most common.

Running the testsuite usually makes the total package build time 2x - 4x longer.