So the problem here is that nothing changed the ref to be a symlink to the updates one (like it is for other stable releases).

I can't recall what process does that. Or do we manually do that?

@dustymabe might know?

There's a manual process described here: https://docs.pagure.org/releng/sop_composing_fedora.html#update-silverblue-refs

I don't know if that's current.

We already included a script for that, it was dropped while we moved the SOP docs. Running it in a few minutes.

We use this script is it correct? https://pagure.io/releng/blob/main/f/scripts/update_ostree_refs.sh

It basically does the same thing as in the old SOP. But we got a bunch of errors about refs already updated and Onyx nonexisting.

Ah ha. I thought it might have been dropped somehow. I couldn't find any mention of it. ;(

Yes, thats likely the script, but since we have already done the release, the updates ref already exists... bodhi has created it.
So, we just need to do the other calls there, not the one to make the updates ref, just the ones to point the main ref to updates.

but I think it just errored on that first step and did the rest and we are ok now here? (ie, we are now in the desired state)

We are now good for everyone but Onyx. Not sure why:

$ ostree remote summary fedora | grep -E "39/(x86_64|aarch64|ppc64le)/(silverblue|kinoite|sericea|onyx)" -A3 --no-group-separator | grep -v 'Latest Commit' | grep -v '^      '
* fedora/39/aarch64/kinoite
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/aarch64/sericea
    Version (ostree.commit.version): 39.20231103.n.0
* fedora/39/aarch64/silverblue
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/ppc64le/kinoite
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/ppc64le/sericea
    Version (ostree.commit.version): 39.20231103.n.0
* fedora/39/ppc64le/silverblue
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/x86_64/kinoite
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/x86_64/onyx
    Version (ostree.commit.version): 39.20231103.n.0
* fedora/39/x86_64/sericea
    Version (ostree.commit.version): 39.20231110.0
* fedora/39/x86_64/silverblue
    Version (ostree.commit.version): 39.20231110.0

Might be because it's failing to build due to a pungi issue: https://pagure.io/releng/failed-composes/issue/5614#comment-883683

For Sericea on !x86_64, I don't know as well.

It looks like the container image references e.g. quay.io/fedora/fedora-sliverblue:39 aren't being updated:

$ skopeo inspect -n docker://quay.io/fedora/fedora-silverblue:39 | jq .Created
"2023-11-03T01:13:08Z"

I...should remember where the shell script is that I modified to do this, or where its logs might be, but I have no idea now and apparently google isn't indexing pagure.io reliably?

The update script is here: https://pagure.io/releng/blob/main/f/scripts/sync-ostree-base-containers.sh

Looking at the compose scripts it seems we are not running it with container composes.

https://pagure.io/pungi-fedora/pull-request/1221 is adding it to the f39 container nightly compose. https://pagure.io/pungi-fedora/pull-request/1222 will make sure next time we branch it is running with the container nightly in the next branched release.

Should be all fixed now:

skopeo inspect -n docker://quay.io/fedora/fedora-silverblue:39 | jq .Created
"2023-11-17T00:55:33Z"