releng

#12868 Fedora 39/40 didn't properly move to archives in mirrormanager
Opened by kevin. Modified

Open
Close Issue

This step:

https://docs.fedoraproject.org/en-US/infra/release_guide/release_eol/#_move_the_eol_release_to_archive

seems to have not worked correctly or was missed.

f39/f40 are still pointing to the non archive directories in mirrormanager.

I ran the playbook on f39 and it completed. We should know in an hour or so if it's fixed.

I also ran it on f40, but it failed because it's trying to move Modular repos that no longer exist in f40+.
We will need a fix in mirrormanager for it.

CC: @abompard for mirrormanager

A couple of examples of bad urls.
When pulling: https://mirrors.fedoraproject.org/metalink?repo=fedora-39&arch=x86_64

Resulting metalink.xml refers to:
https://d2lzkl7pfhq30w.cloudfront.net/pub/fedora/linux/releases/39
http://repo.jing.rocks/fedora-buffet/fedora/linux/releases/39/

instead of:
https://d2lzkl7pfhq30w.cloudfront.net/pub/archive/fedora/linux/releases/39/
http://repo.jing.rocks/fedora-buffet/archive/linux/release/39

In the case of cloudfront the URL results in an error.
In the case of jing.rocks, the mirror populates both the original and archive location so the URL still works.

So the mirrorlist ends up behaving in some non-deterministic ways depending on which mirror you end up trying to use first.

Metadata Update from @james:
- Issue assigned to abompard
- Issue tagged with: medium-gain, medium-trouble

Ooofff, I ran those playbooks after the DC move, and it ran without any errors :(

F39 seems to be OK now.
It looks like the archive primary mirror hasn't been crawled recently, because none of the directories for F40 are in the database at the moment. Also, the fullfiletimelist file seems to not have been updated in a while:

$ ls -l /srv/pub/archive/fullfiletimelist-archive 
-rw-r--r--. 1 263 263 2330491735 Jan  9  2025 /srv/pub/archive/fullfiletimelist-archive

Shouldn't this file have been updated as part of moving the repos to the archive?
Edited by abompard

@abompard, I think mirrormanager2 is moved to openshift right? Is it under oc-control01? I'm trying to find out why it didn't happen, I did run these after the dc move

/usr/local/bin/update-fullfiletimelist -l /pub/archive/update-fullfiletimelist.lock -t /pub archive

Shouldn't this file have been updated as part of moving the repos to the archive?

yes it should be ideally, I have run it again now, I'll see if this gets updated properly or not

This should be updated now:

ls -l /srv/pub/archive/fullfiletimelist-archive 
-rw-r--r--. 1 263 263 2438409405 Aug  8 11:26 /srv/pub/archive/fullfiletimelist-archive

recently, because none of the directories for F40 are in the database at the moment

Does that means things didn't get synced (for example) with /pub/fedora/linux/releases/40 -> /pub/archive/fedora/linux/releases/40/ properly?

What should the move-to-archive script do when the destination repo is not found?
1. Ignore the repo and leave it in place? This means that there will be leftovers in the database with repos still in the wrong category, pointing to directories that are now inexistant after the files move
2. Delete the repo instead of moving it? This seems risky if the destination repo doesn't exist for other reasons or hasn't been created by the primary mirror crawler yet (like here)
3. Change nothing in the script and manually delete the repos from the database before moving? I can write a script to delete repos from the DB to make that easier.
4. Something else?

I'm happy to make the change in MirrorManager when decided.

I think the core of the problem is that the script tries to move modular repositories that no longer exist in Fedora  40, so it fails when the destination isn’t there. When that happens, we end up with database entries pointing to directories that are no longer present, which is exactly right behaviour I suppse?

My suggestion would be to make the script more forgiving (maybe too much brain used here but before attempting a move the script should check whether the target repository exists. If it doesn’t, I recommend skipping that repo, logging that it was skipped, and then following up with a separate clean‑up process op3(?) to remove those entries from the MirrorManager database once the archive has been crawled. Automatically deleting repos inside the move script feels risky tbh, don’t want to accidentally remove something just because it hasn’t been crawled yet.

I think the script should detect and log missing repos rather than failing outright; I recommend running update-fullfiletimelist and triggering a mirror crawl to refresh the archive as part of the process; and my suggestion would be to add a separate tool or sub‑command to safely delete obsolete repos from the database after the move is complete. This approach keeps the process robust and gives you visibility into what was (and wasn’t) moved. And, otherwise to make life easier the option 3 seems to solve the immediate and long term problem with minimal changes.
Edited by jnsamyak

I ran the script again, it worked fine this time:

$ /opt/app-root/bin/mm2_move-to-archive --product Fedora --version 40                                                                                                                                                                [20/1859]
pub/fedora/linux/updates/testing/40/Modular/source/tree => pub/archive/fedora/linux/updates/testing/40/Modular/source/tree  
pub/fedora/linux/updates/testing/40/Everything/source/tree => pub/archive/fedora/linux/updates/testing/40/Everything/source/tree                                                                                                                               
pub/fedora/linux/updates/40/Everything/source/tree => pub/archive/fedora/linux/updates/40/Everything/source/tree                                                                                                                                               
pub/fedora/linux/updates/40/Modular/source/tree => pub/archive/fedora/linux/updates/40/Modular/source/tree                                                                                                                                                     
Repo codecs.fedoraproject.org/openh264/40/source/os (prefix fedora-cisco-openh264-source-40) is in the 'Fedora Codecs' category, skipping.       
pub/fedora/linux/releases/40/Everything/source/tree => pub/archive/fedora/linux/releases/40/Everything/source/tree                                                                                                                                             
pub/fedora-secondary/updates/testing/40/Everything/i386/debug => pub/archive/fedora-secondary/updates/testing/40/Everything/i386/debug                                                                                                                         
pub/fedora-secondary/updates/40/Everything/i386/debug => pub/archive/fedora-secondary/updates/40/Everything/i386/debug
pub/fedora-secondary/updates/testing/40/Modular/i386/debug => pub/archive/fedora-secondary/updates/testing/40/Modular/i386/debug                                                                                                                               
pub/fedora-secondary/updates/testing/40/Modular/i386 => pub/archive/fedora-secondary/updates/testing/40/Modular/i386                                                                                                                                           
pub/fedora-secondary/updates/40/Everything/i386 => pub/archive/fedora-secondary/updates/40/Everything/i386                                                                                                                                                     
pub/fedora-secondary/updates/40/Modular/i386 => pub/archive/fedora-secondary/updates/40/Modular/i386                                                                                                                                                           
pub/fedora-secondary/updates/40/Modular/i386/debug => pub/archive/fedora-secondary/updates/40/Modular/i386/debug            
pub/fedora-secondary/updates/testing/40/Everything/i386 => pub/archive/fedora-secondary/updates/testing/40/Everything/i386
Repo codecs.fedoraproject.org/openh264/40/i386/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.                                                                                                                              
Repo codecs.fedoraproject.org/openh264/40/i386/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora/linux/updates/testing/40/Everything/x86_64 => pub/archive/fedora/linux/updates/testing/40/Everything/x86_64                                                                                                                                         
pub/fedora/linux/updates/40/Modular/x86_64 => pub/archive/fedora/linux/updates/40/Modular/x86_64                                                                                                                                                               
pub/fedora/linux/updates/40/Everything/x86_64/debug => pub/archive/fedora/linux/updates/40/Everything/x86_64/debug                                                                                                                                             
pub/fedora/linux/updates/testing/40/Modular/x86_64 => pub/archive/fedora/linux/updates/testing/40/Modular/x86_64                                                                                                                                               
pub/fedora/linux/updates/testing/40/Everything/x86_64/debug => pub/archive/fedora/linux/updates/testing/40/Everything/x86_64/debug                                                                                                                             
pub/fedora/linux/updates/testing/40/Modular/x86_64/debug => pub/archive/fedora/linux/updates/testing/40/Modular/x86_64/debug
pub/fedora/linux/updates/40/Everything/x86_64 => pub/archive/fedora/linux/updates/40/Everything/x86_64
pub/fedora/linux/updates/40/Modular/x86_64/debug => pub/archive/fedora/linux/updates/40/Modular/x86_64/debug
Repo codecs.fedoraproject.org/openh264/40/x86_64/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.
Repo codecs.fedoraproject.org/openh264/40/x86_64/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora/linux/releases/40/Everything/x86_64/os => pub/archive/fedora/linux/releases/40/Everything/x86_64/os
pub/fedora/linux/releases/40/Everything/x86_64/debug/tree => pub/archive/fedora/linux/releases/40/Everything/x86_64/debug/tree
pub/fedora-secondary/updates/40/Modular/s390x => pub/archive/fedora-secondary/updates/40/Modular/s390x
pub/fedora-secondary/updates/40/Everything/s390x => pub/archive/fedora-secondary/updates/40/Everything/s390x
pub/fedora-secondary/updates/testing/40/Everything/s390x/debug => pub/archive/fedora-secondary/updates/testing/40/Everything/s390x/debug
pub/fedora-secondary/updates/testing/40/Modular/s390x/debug => pub/archive/fedora-secondary/updates/testing/40/Modular/s390x/debug
pub/fedora-secondary/updates/testing/40/Modular/s390x => pub/archive/fedora-secondary/updates/testing/40/Modular/s390x
pub/fedora-secondary/updates/testing/40/Everything/s390x => pub/archive/fedora-secondary/updates/testing/40/Everything/s390x
pub/fedora-secondary/updates/40/Everything/s390x/debug => pub/archive/fedora-secondary/updates/40/Everything/s390x/debug
pub/fedora-secondary/updates/40/Modular/s390x/debug => pub/archive/fedora-secondary/updates/40/Modular/s390x/debug
Repo codecs.fedoraproject.org/openh264/40/s390x/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.
Repo codecs.fedoraproject.org/openh264/40/s390x/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora-secondary/releases/40/Everything/s390x/os => pub/archive/fedora-secondary/releases/40/Everything/s390x/os
pub/fedora-secondary/releases/40/Everything/s390x/debug/tree => pub/archive/fedora-secondary/releases/40/Everything/s390x/debug/tree
pub/fedora/linux/updates/40/Everything/armhfp/debug => pub/archive/fedora/linux/updates/40/Everything/armhfp/debug
pub/fedora/linux/updates/testing/40/Modular/armhfp/debug => pub/archive/fedora/linux/updates/testing/40/Modular/armhfp/debug
pub/fedora/linux/updates/40/Everything/armhfp => pub/archive/fedora/linux/updates/40/Everything/armhfp
pub/fedora/linux/updates/40/Modular/armhfp/debug => pub/archive/fedora/linux/updates/40/Modular/armhfp/debug
pub/fedora/linux/updates/40/Modular/armhfp => pub/archive/fedora/linux/updates/40/Modular/armhfp
pub/fedora/linux/updates/testing/40/Everything/armhfp => pub/archive/fedora/linux/updates/testing/40/Everything/armhfp
pub/fedora/linux/updates/testing/40/Everything/armhfp/debug => pub/archive/fedora/linux/updates/testing/40/Everything/armhfp/debug
pub/fedora/linux/updates/testing/40/Modular/armhfp => pub/archive/fedora/linux/updates/testing/40/Modular/armhfp
Repo codecs.fedoraproject.org/openh264/40/armhfp/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.
Repo codecs.fedoraproject.org/openh264/40/armhfp/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora/linux/updates/testing/40/Modular/aarch64/debug => pub/archive/fedora/linux/updates/testing/40/Modular/aarch64/debug
pub/fedora/linux/updates/40/Everything/aarch64/debug => pub/archive/fedora/linux/updates/40/Everything/aarch64/debug
pub/fedora/linux/updates/testing/40/Modular/aarch64 => pub/archive/fedora/linux/updates/testing/40/Modular/aarch64
pub/fedora/linux/updates/40/Everything/aarch64 => pub/archive/fedora/linux/updates/40/Everything/aarch64
pub/fedora/linux/updates/testing/40/Everything/aarch64 => pub/archive/fedora/linux/updates/testing/40/Everything/aarch64
pub/fedora/linux/updates/40/Modular/aarch64/debug => pub/archive/fedora/linux/updates/40/Modular/aarch64/debug
pub/fedora/linux/updates/testing/40/Everything/aarch64/debug => pub/archive/fedora/linux/updates/testing/40/Everything/aarch64/debug
pub/fedora/linux/updates/40/Modular/aarch64 => pub/archive/fedora/linux/updates/40/Modular/aarch64
Repo codecs.fedoraproject.org/openh264/40/aarch64/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.
Repo codecs.fedoraproject.org/openh264/40/aarch64/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora/linux/releases/40/Everything/aarch64/debug/tree => pub/archive/fedora/linux/releases/40/Everything/aarch64/debug/tree
pub/fedora/linux/releases/40/Everything/aarch64/os => pub/archive/fedora/linux/releases/40/Everything/aarch64/os
pub/fedora-secondary/updates/40/Everything/ppc64le => pub/archive/fedora-secondary/updates/40/Everything/ppc64le
pub/fedora-secondary/updates/testing/40/Modular/ppc64le/debug => pub/archive/fedora-secondary/updates/testing/40/Modular/ppc64le/debug
pub/fedora-secondary/updates/testing/40/Everything/ppc64le/debug => pub/archive/fedora-secondary/updates/testing/40/Everything/ppc64le/debug
pub/fedora-secondary/updates/testing/40/Everything/ppc64le => pub/archive/fedora-secondary/updates/testing/40/Everything/ppc64le
pub/fedora-secondary/updates/testing/40/Modular/ppc64le => pub/archive/fedora-secondary/updates/testing/40/Modular/ppc64le
pub/fedora-secondary/updates/40/Modular/ppc64le => pub/archive/fedora-secondary/updates/40/Modular/ppc64le
pub/fedora-secondary/updates/40/Modular/ppc64le/debug => pub/archive/fedora-secondary/updates/40/Modular/ppc64le/debug
pub/fedora-secondary/updates/40/Everything/ppc64le/debug => pub/archive/fedora-secondary/updates/40/Everything/ppc64le/debug
Repo codecs.fedoraproject.org/openh264/40/ppc64le/os (prefix fedora-cisco-openh264-40) is in the 'Fedora Codecs' category, skipping.
Repo codecs.fedoraproject.org/openh264/40/ppc64le/debug/tree (prefix fedora-cisco-openh264-debug-40) is in the 'Fedora Codecs' category, skipping.
pub/fedora-secondary/releases/40/Everything/ppc64le/os => pub/archive/fedora-secondary/releases/40/Everything/ppc64le/os
pub/fedora-secondary/releases/40/Everything/ppc64le/debug/tree => pub/archive/fedora-secondary/releases/40/Everything/ppc64le/debug/tree

So it was not the Modular repos. The move-to-archive script does not work on actual directories, it only works with what's in the database anyway.

Huh. I swear it failed for me. ;( 

[root@os-control01 ~][PROD-RDU3]# oc logs job-move-to-archive-fedora-40-fxw9l
+ deactivate nondestructive
+ '[' -n '' ']'
+ '[' -n '' ']'
+ hash -r
+ '[' -n '' ']'
+ unset VIRTUAL_ENV
+ unset VIRTUAL_ENV_PROMPT
+ '[' '!' nondestructive = nondestructive ']'
+ VIRTUAL_ENV=/opt/app-root
+ export VIRTUAL_ENV
+ _OLD_VIRTUAL_PATH=/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/opt/app-root/bin:/opt/app-root/src/.local/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export PATH
+ '[' -n '' ']'
+ '[' -z '' ']'
+ _OLD_VIRTUAL_PS1=
+ PS1='(app-root) '
+ export PS1
+ VIRTUAL_ENV_PROMPT='(app-root) '
+ export VIRTUAL_ENV_PROMPT
+ hash -r
+ unset BASH_ENV PROMPT_COMMAND ENV
+ /opt/app-root/bin/mm2_move-to-archive --product Fedora --version 40
Error: Unable to find a directory in [Fedora Archive] for pub/fedora/linux/updates/testing/40/Modular/source/tree

IMHO, we should drop Modular from the script and keep it failing when it doesn't see something right... that way we know we need to fix it. We don't really add/remove kinds of things very often...

Yeah, it failed because the primary mirror for the archive hadn't been crawled because the fullfiletimelist file hadn't been updated.
The error mentions Modular because that happened to be the first path it looked at, but they were all missing.
The script does not do anything specific about Modular repos, it only looks at existing repos in the database and reassigns them to a different directory located in the archive tree. It does not look at actual filesystem paths, only the database.
Edited by abompard

Ah, right... makes complete sense.

So, I'd suggest two things:

  1. lets add to the script a warning "Make sure you have run update on the fullfiletimelist for archive: "

and

  1. Lets explictly add that to docs in https://docs.fedoraproject.org/en-US/infra/release_guide/sop_release_eol/
  1. lets add to the script a warning "Make sure you have run update on the fullfiletimelist for archive: "

Actually I think this should be done automatically, but I don't know what updates this file. In ansible there are a couple scripts and cron jobs that would touch it, but I'm not sure in which order they run.
Maybe it's running one of those scripts that should be added to the warning/docs.

It's the ./files/scripts/update-fullfiletimelist script. However... to update archive it needs to run on a machine that has that mount and has that mount rw (it's normally ro).

I guess somehow bodhi-backend01 seems to have it and rw (which seems kinda bad, but ok)

So it would run as:

sudo -u ftpsync /usr/local/bin/update-fullfiletimelist -l /pub/fedora-secondary/update-fullfiletimelist.lock -t archive

however, if it's run and files aren't yet moved, it won't do any good of course.

Metadata

medium-gain medium-trouble

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.