releng

#13069 Atomic Desktops: issue with ostree repo: Corrupted commit object; checksum expected='...' actual='...'
Opened by siosm. Modified

Open
Close Issue
  • Describe the issue

I've tested F42 & F43 Silverblue & Kinoite, and all ostree Atomic Desktops can not update with an error that looks like:

error: While pulling fedora/43/x86_64/kinoite: Corrupted commit object; checksum expected='10ed48669028567c29acfda34e70290581e1c8d4f4538d79bea36c03710a4cf6' actual='8792806cf428563a3b0717fd515fe8d2c0c32c96ee1aafbc7a1472ef4b844711'

Something is off with the ostree repo but I don't know what yet.

  • When do you need this? (YYYY/MM/DD)

ASAP

  • When is this no longer needed or useful? (YYYY/MM/DD)

N/A

  • If we cannot complete your request, what is the impact?

No updates for Atomic Desktops users.

User reports:
- https://github.com/fedora-silverblue/issue-tracker/issues/682
- https://gitlab.com/fedora/ostree/sig/-/issues/93

I'll check if I still have a working ssh access to the server so that I can investigate.

On a Silverblue 43 system:

silverblue@fedora:~$ ostree remote summary fedora | grep -A5 fedora/43/x86_64/silverblue
* fedora/43/x86_64/silverblue
    Latest Commit (64.2 kB):
      9d6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7
    Version (ostree.commit.version): 43.20251107.0
    Timestamp (ostree.commit.timestamp): 2025-11-07T01:30:20+01
silverblue@fedora:~$ sudo rpm-ostree update --check
error: While pulling fedora/43/x86_64/silverblue: Corrupted commit object; checksum expected='9d6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7' actual='8e0ea4ea59814051812f195a5723e92a6e785008e3f59a2afb5bbce189f50298'
silverblue@fedora:~$ sudo rpm-ostree update --check
error: While pulling fedora/43/x86_64/silverblue: Corrupted commit object; checksum expected='9d6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7' actual='7189d84d1edfd63d120a992f8740e54b40a8d4d614fe8050525a6a9822db9eb8'

This changes all the time so this is really weird.

Looking at compose-branched01.rdu3.fedoraproject.org:

[siosm@compose-branched01 ~][PROD-RDU3]$ ostree log --repo /mnt/koji/compose/ostree/repo fedora/43/x86_64/silverblue | head -n6
commit 9d6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7
Parent:  1069cf0e9a9e38c436af957e93027068e37c2143256a9adf6a1958acd8e237bb
ContentChecksum:  12133b77419815ce79ef49b167ad1faeb2bd2229aca0347f8c29537d6bdaaa59
Date:  2025-11-07 00:30:20 +0000
Version: 43.20251107.0
(no subject)
[siosm@compose-branched01 ~][PROD-RDU3]$ sha256sum /mnt/koji/compose/ostree/repo/objects/9d/6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7.commit
9d6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7  /mnt/koji/compose/ostree/repo/objects/9d/6b8d6f28a2c5d1a94270060407092017286a0d047bcb4e776be9f65e7241c7.commit

So things looks OK there.

Which points to a network / proxy issue in the middle.

Seeing the same for IoT

rpm-ostree upgrade
error: While pulling fedora/stable/aarch64/iot: Corrupted commit object; checksum expected='98720c15dbdd8c4e5507d24e862114d6e9eb2ea774e90944730c13253565c155' actual='f370249927a37df3494760bd3574f53d3f1ee1f61d2a745e73394b78421258d5'

I wonder if this is affected by the 503 errors we've been generally seeing in the infra of late.

When I curl the cloudfront URL I get an Anubis challenge so it's something related to that:

$ curl https://d2uk5hbyrobdzx.cloudfront.net/ostree/repo/refs/heads/fedora/43/x86_64/silverblue
...
<!doctype html><html lang="en"><head><title>Making sure you&#39;re not a bot!</title><link rel="stylesheet" href="/.within.website/x/xess/xess.min.css?cachebuster=1.23.0-1.fc43"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="robots" content="noindex,nofollow"><style>
...

Apparently mitigated by an Anubis downgrade.

I also reported this earlier today: https://pagure.io/fedora-infrastructure/issue/12895
usernames-exhausted commented

TIL an org as storied and respectable as Fedora doesn't even check if the CDN is b0rked after updating something in its critical path, resulting in an outage approaching 24 hours.

Not too sure how I feel about that, but it's not a great feeling, tbh.
Edited by usernames-exhausted

It was actually tested and working. The issue only happened a day later when an updates push happened.
Fully testing large complex systems is not something we can fully do with the resources we have. We do the best we can.

To recap the issue as I know it:

  • We upgraded anubis to 1.23.0. Everything seemed fine.

  • A day later we hit this, which I think was caused by an updates push.

ostree.fedoraproject.org is not behind anubis, it redirects to a cloudfront cache
The cloudfront cache has a backend of kojipkgs.fedoraproject.org which is behind anubis.

Before this anubis was passing cloudfront requests fine, but with 1.23.0 somehow for some limited requests it was challenging them.
We downgraded back to 1.22.0 for now and will be looking at how we can adjust to avoid this issue moving forward.

Thanks Kevin for the details and the investigation.

Metadata Update from @jnsamyak:
- Issue tagged with: medium-gain, medium-trouble, ops, sprint-4

So looking at it; can we close this now?

well, we want to upgrade anubis again someday, so we need to actually fix the issue...

I looked a bit more and actually I was wrong here. we have: 

# Make sure that CloudFront does not cache ostree summary or refs files.
# These should always be requested directly from Fedora, so any user directly
#  hitting a cloudfront setup should update their configuration.
RewriteCond %{HTTP:X-Amz-Cf-Id} !^$
RewriteRule ^/atomic/repo/summary - [F]
RewriteCond %{HTTP:X-Amz-Cf-Id} !^$
RewriteRule ^/atomic/repo/refs - [F]
RewriteCond %{HTTP:X-Amz-Cf-Id} !^$
RewriteRule ^/ostree/repo/summary - [F]
RewriteCond %{HTTP:X-Amz-Cf-Id} !^$
RewriteRule ^/ostree/repo/refs - [F]

So, this was only affecting the refs/summary files? so it might have been ostree direct to kojipkgs...

will have to ponder on a fix.

Metadata Update from @jnsamyak:
- Issue untagged with: sprint-4
- Issue tagged with: sprint-5
Metadata
None

medium-gain medium-trouble ops sprint-5

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.