#8857 Fedora 31 container images are pulling in F-32 content
Closed: Fixed by mohanboddu. Opened by pbrobinson.

The Fedora 31 container images appear to be rawhide based for some reason

# podman run --rm -it fedora:31 /bin/bash
[root@3c3b229b2631 /]# dnf install -y nodejs npm nodejs-bcrypt
Fedora - Modular Rawhide - Developmental packages for the next Fedora release                                                                                                      2.0 MB/s | 2.6 MB     00:01    
Fedora - Rawhide - Developmental packages for the next Fedora release                                                                                                               22 MB/s |  54 MB     00:02    

warning: /var/cache/dnf/rawhide-135a69fc59e3201d/packages/http-parser-2.9.2-2.fc31.aarch64.rpm: Header V3 RSA/SHA256 Signature, key ID 12c944d0: NOKEY
Fedora - Rawhide - Developmental packages for the next Fedora release                                                                                                              1.6 MB/s | 1.6 kB     00:00    
GPG key at file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64 (0x3C3359C4) is already installed
The GPG keys listed for the "Fedora - Rawhide - Developmental packages for the next Fedora release" repository are already installed but they are not correct for this package.
Check that the correct key URLs are configured for this repository.. Failing package is: http-parser-2.9.2-2.fc31.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for libuv-1.31.0-0.fc32.aarch64.rpm is not installed. Failing package is: libuv-1:1.31.0-0.fc32.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for nodejs-12.10.0-1.fc32.aarch64.rpm is not installed. Failing package is: nodejs-1:12.10.0-1.fc32.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for nodejs-bcrypt-3.0.6-2.fc31.aarch64.rpm is not installed. Failing package is: nodejs-bcrypt-3.0.6-2.fc31.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for nodejs-bindings-1.3.0-5.fc31.noarch.rpm is not installed. Failing package is: nodejs-bindings-1.3.0-5.fc31.noarch
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for nodejs-libs-12.10.0-1.fc32.aarch64.rpm is not installed. Failing package is: nodejs-libs-1:12.10.0-1.fc32.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for npm-6.10.3-1.12.10.0.1.fc32.aarch64.rpm is not installed. Failing package is: npm-1:6.10.3-1.12.10.0.1.fc32.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
Public key for libnghttp2-1.39.2-1.fc32.aarch64.rpm is not installed. Failing package is: libnghttp2-1.39.2-1.fc32.aarch64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-aarch64
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED

This is in AWS on an a.1 (aarch64) instance running a fully updated Fedora 30 if that matters

I am not seeing this problem locally:

$ podman run --rm -it fedora:31 /bin/bash
Trying to pull docker.io/library/fedora:31...
...
snip
...
Storing signatures
# dnf install npm
Fedora Modular 31 - x86_64                                           599 kB/s | 5.2 MB     00:08    
Fedora Modular 31 - x86_64 - Updates                                 2.8  B/s | 257  B     01:30    
Fedora Modular 31 - x86_64 - Test Updates                             33 kB/s | 369 kB     00:11    
Fedora 31 - x86_64 - Test Updates                                    581 kB/s |  12 MB     00:21    
Fedora 31 - x86_64 - Updates                                         360  B/s | 257  B     00:00    
Fedora 31 - x86_64                                                   1.5 MB/s |  67 MB     00:44    
Killed

Oddly the process is getting killed, but its pulling the right content.

I suspect this is related to https://pagure.io/pungi-fedora/c/95d269b4bc2dbc624057804e8935dc0ef077315f?branch=f31
(all our f31 composes were using f32/master kickstarts).

We should push new container releases soon/asap... our registry has the updated one, but I don't think others do.

@kevin I dont think it might be the problem, but sure I can push all the containers to fedora registry. @cverna Can you push to docker hub?

@pbrobinson how old is the image on your AWS box ? A new image was pushed 10 days ago and does not have this problem (on x86_64).

 [cverna@localhost]  $ podman images
REPOSITORY                                      TAG       IMAGE ID       CREATED        SIZE
docker.io/library/fedora                        rawhide   e13031c001a8   10 days ago    208 MB
docker.io/library/fedora                        31        98c519110eac   10 days ago    202 MB

I recommend making sure you have the latest image. If that's the case it means that we have different images for x86_64 and aarch64 on Docker Hub, so that would be worth trying the image on registry.fedoraproject.org/fedora:31 and see if that works better.

Problem still exists for the linux/arm/v7 version of registry.fedoraproject.org/fedora:31. The image was last updated "9 months ago". FYI, I'm trying to get FreeIPA running on Raspberry Pi 4 :)

Problem still exists for the linux/arm/v7 version of registry.fedoraproject.org/fedora:31. The image was last updated "9 months ago". FYI, I'm trying to get FreeIPA running on Raspberry Pi 4 :)

The armv7 container base image build has been failing for a while now, so there is no f30 nor f31 image available. This is tracked in this ticket https://pagure.io/releng/issue/8173

This has been fixed but armv7 container failures is still an issue and being tracked in https://pagure.io/releng/issue/8173

Metadata Update from @mohanboddu:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata