releng

#11186 Use s390x-build channel for qclib builds
Closed: Invalid by yselkowitz. Opened by yselkowitz.

Closed: Invalid
Reopen Issue
  • Describe the issue

The qclib (an s390x-specific package) testsuite succeeds on z/VM hosts but regularly fails on KVM hosts with Warning: Capacity data inconsistent, try again later (rc=2):

https://koji.fedoraproject.org/koji/packageinfo?packageID=23510

To avoid the need for continual retries, we believe that forcing all qclib builds to the s390x-build channel (comprised of hosts 01 through 14, which I'm told are all z/VM) should avoid these failures.

  • When do you need this? (YYYY/MM/DD)

As soon as reasonably possible, to unblock ELN builds.

  • When is this no longer needed or useful? (YYYY/MM/DD)

For now indefinitely, but we are going to work with IBM on the root cause and attempt to find a fix.

  • If we cannot complete your request, what is the impact?

qclib builds will continue to be spotty, necessitating multiple retries in order to attempt to get a z/VM builder.

/cc @sharkcz

I think there is something wrong with KVM builders, because I wasn't able to reproduce the failure outside koji (IIRC).

Metadata Update from @phsmoura:
- Issue tagged with: medium-gain, medium-trouble, ops

@sharkcz I thought the ones listed as being 01-14 are not KVM but are 'direct' ish

Metadata Update from @smooge:
- Issue untagged with: medium-gain, medium-trouble, ops

Metadata Update from @smooge:
- Issue tagged with: medium-gain, medium-trouble

Exactly, the tests pass on the z/VMs (01-14) but currently fail on the KVMs (15+). Dan believes the tests should work on both and therefore suspects it may be an issue with the KVM builders themselves but is continuing to investigate.

01-14 are z/VM based (builds/tests run fine there), but the failed builds from the link above were run on the KVM ones (15-30), both kinds are VMs, but based on different technologies

I wanted to say that the KVM builders in koji maybe need to be restarted, or perhaps the vmhost needs to be rebooted. But until then or until we know more it might be useful to send the qclib builds explicitly to the z/VM based builders.

ok I will reboot the builders tomorrow (2022-12-14) and we can see if that helps any.

Builders 18->30 have been rebooted. Dan said 15,16,17 have an NFS share and should only be used fro composes so I did not reboot them. Please test.

New scratch build on 27 failed with the same error: https://koji.fedoraproject.org/koji/taskinfo?taskID=95357539

I have tried 2 different KVM guests (on 2 different hypervisors) and can't reproduce the issue. My upstream report is https://github.com/ibm-s390-linux/qclib/issues/2

Are either of the hypervisors running Fedora?

(ours is f37)

Are either of the hypervisors running Fedora?

(ours is f37)

one is F-36 with the latest virt stack (from virt-preview), the other is RHEL I support (in beaker)

Just a reminder that this is still an issue, and is affecting the ELN mass rebuild, and the F39 mass rebuild is still to come.

I wonder... does the completed rpm built on z/vm instance with tests passing, run correctly on a kvm instance?

If not, us building it so the tests pass isn't the answer, we need to fix the problem with it not working right on kvm instances?

Also, we are likely moving mainframe resources at some point later this year, and may not have any zvm at that point, it may be all kvm. ;(

ping @sharkcz

As discussed in https://github.com/ibm-s390-linux/qclib/issues/2 we made a progress with the root cause of the issue. A workaround has been applied to the qclib package, a proper fix will come later. I believe this ticket can be closed now.

thanks @sharkcz

Metadata Update from @yselkowitz:
- Issue close_status updated to: Invalid
- Issue status updated to: Closed (was: Open)
Metadata
None

medium-gain medium-trouble

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.