releng

#12316 release-candidate.sh adjustments needed
Closed: Fixed with Explanation by kevin. Opened by kevin.

Closed: Fixed with Explanation
Reopen Issue

Some issues trying a 1.1 test compose:

[kevin@compose-x86-01 pungi-fedora][PROD-IAD2]$ sudo ./release-candidate.sh 41_Beta-1.1       
First Factor:                                                                                 
Second Factor:                                                                                
========================================                                                      
CONFIG: fedora-beta.conf                                                                      
THIS IS NOT A DRY RUN.                                                                        
========================================                                                      
ASSUME THAT THE RELENG REPOSITORY IS AVAILABLE AND REBASE. CLONE IT IF IT'S NOT.              
/home/fedora/kevin/pungi-fedora/releng /home/fedora/kevin/pungi-fedora                        
remote: Enumerating objects: 382, done.                                                       
remote: Counting objects: 100% (191/191), done.                                               
remote: Compressing objects: 100% (78/78), done.                                              
remote: Total 382 (delta 151), reused 146 (delta 112), pack-reused 191                        
Receiving objects: 100% (382/382), 142.50 KiB | 8.38 MiB/s, done.                             
Resolving deltas: 100% (232/232), completed with 12 local objects.
From https://pagure.io/releng                                                                 
   56aa9c9..545e6a8  main       -> origin/main                                                
Updating 56aa9c9..545e6a8                                                                     
Fast-forward                                                                                  
 docs/source/index.rst                                       |   4 +-                         
 mail-templates/01-mass-rebuild-start.txt                    |   5 +-                         
 mail-templates/02-mass-rebuild-finished.txt                 |  10 +-                         
 scripts/block_retired.py                                    |  14 +-                         
 scripts/branching/make-koji-release-tags                    |  61 +-                         
 scripts/critpath.py                                         |  66 +-                         
 scripts/fedora-torrent-ini.py                               |  60 +-                         
 scripts/fedretire                                           | 214 ------                     
 scripts/fesco/sig-policy/sig_policy.py                      |   2 +                          
 scripts/find-bad-builds.py                                  |   1 +                          
 scripts/find_failures.py                                    |   6 +-                         
 scripts/find_unblocked_orphans.py                           | 834 +----------------------    
 scripts/ftbfs-fti/follow-policy.py                          |   4 +-                         
 scripts/get_retired_packages.sh                             |  50 ++                         
 scripts/koji_cleanup_signed.py                              |   3 +-                         
 scripts/mass-rebuild-modules.py                             |   2 +-                         
 scripts/mass-rebuild-second-run.py                          |   2 +-                         
 scripts/mass-rebuild-special.py                             |   2 +-                         
 scripts/mass-rebuild.py                                     |   4 +-                         
 scripts/mass_rebuild_file_bugs.py                           |   2 +-                         
 scripts/massrebuildsinfo.py                                 | 100 ++-                        
 scripts/misc/failed_composes_cleanup.py                     |  75 +++                        
 scripts/need-rebuild.py                                     |   8 +-                         
 scripts/orphaned-packages-process/Dockerfile                |  16 +                          
 scripts/orphaned-packages-process/find_unblocked_orphans.py | 936 ++++++++++++++++++++++++++ 
 scripts/orphaned-packages-process/pyproject.toml            |  14 +                          
 scripts/orphaned-packages-process/requirements.txt          |   8 +                          
 scripts/orphaned-packages-process/retire.py                 | 241 +++++++                    
 scripts/orphaned-packages-process/tox.ini                   |  35 +                          
 scripts/pdc/get-retired-packages.py                         | 109 +--                        
 scripts/pdc/utilities.py                                    |  12 +                          
 scripts/stage-release.sh                                    |  10 +-                         
 scripts/sync-latest-container-base-image.sh                 | 117 ++--                       
 scripts/sync-ostree-base-containers.sh                      |   4 +-                         
 scripts/update_ostree_refs.sh                               |  20 +-                         
 35 files changed, 1756 insertions(+), 1295 deletions(-)                                      
 delete mode 100755 scripts/fedretire                                                         
 mode change 100755 => 120000 scripts/find_unblocked_orphans.py                               
 create mode 100755 scripts/get_retired_packages.sh                                           
 create mode 100644 scripts/misc/failed_composes_cleanup.py                                   
 create mode 100644 scripts/orphaned-packages-process/Dockerfile                              
 create mode 100755 scripts/orphaned-packages-process/find_unblocked_orphans.py               
 create mode 100644 scripts/orphaned-packages-process/pyproject.toml                          
 create mode 100644 scripts/orphaned-packages-process/requirements.txt                        
 create mode 100755 scripts/orphaned-packages-process/retire.py                               
 create mode 100644 scripts/orphaned-packages-process/tox.ini                                 
/home/fedora/kevin/pungi-fedora                                                               
========================================                                                                                              
========================================                                                      
ANNOUNCE THAT WE ARE STARTING (EVEN THOUGH THE COMPOSE_ID IS NOT YET KNOWN).                  
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefi
x='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': Fa
lse, 'high-water-mark': 0})                                                                   
..
Traceback (most recent call last):
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <
module>
    body = json.loads(msgstring)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end) 
json.decoder.JSONDecodeError: Extra data: line 1 column 54 (char 53)
========================================
RUN THE COMPOSE. EXIT IF THE EXIT STATUS IS NOT ZERO.
usage: pungi-koji [-h] (--target-dir PATH | --compose-dir PATH) [--label LABEL] [--no-label]
                  [--supported] [--old-composes PATH] --config CONFIG [--skip-phase PHASE]
                  [--just-phase PHASE] [--nightly] [--test] [--ci] [--production]
                  [--development] [--koji-event ID] [--version]
                  [--notification-script NOTIFICATION_SCRIPT] [--no-latest-link]
                  [--latest-link-status STATUS] 
                  [--latest-link-components LATEST_LINK_COMPONENTS]
                  [--parent-compose-id PARENT_COMPOSE_ID] [--respin-of RESPIN_OF]
                  [--print-output-dir] [--quiet]
pungi-koji: error: Label in unknown format: 41_Beta-1.1
real    0m1.726s
user    0m1.459s
sys     0m0.185s
ERROR: Command failed with a non-zero exit status.
./release-candidate.sh: line 125: : command not found
ERROR: No compose ID available.
./release-candidate.sh: line 133: : command not found
ERROR: Compose ID not found.
./release-candidate.sh: line 143: : command not found
ERROR: No compose STATUS available.
./release-candidate.sh: line 150: : command not found
cat: /mnt/koji/compose/41//STATUS: No such file or directory
========================================
COMPOSE DETAILS
Compose ID: 
Compose status: 
Proceeding to rsync.
========================================
LET INTERESTED PERSONS KNOW ABOUT THE COMPOSE.
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefi
x='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': Fa
lse, 'high-water-mark': 0})                                                                   
..                                                                                            
Traceback (most recent call last):                                                            
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <
module>                                                                                       
    body = json.loads(msgstring)                                                              
           ^^^^^^^^^^^^^^^^^^^^^                                                              
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads                           
    return _default_decoder.decode(s)                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                         
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode                           
    raise JSONDecodeError("Extra data", s, end)                                               
json.decoder.JSONDecodeError: Extra data: line 1 column 71 (char 70)                          
========================================                                                      
TELL INTERESTED PERSONS THAT THE RSYNC IS STARTING.                                           
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefi
x='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': Fa
lse, 'high-water-mark': 0})                                                                   
..                                                                                            
Traceback (most recent call last):                                                            
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <
module>                                                                                       
    body = json.loads(msgstring)                                                              
           ^^^^^^^^^^^^^^^^^^^^^                                                              
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads                           
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end) 
json.decoder.JSONDecodeError: Extra data: line 1 column 72 (char 71)
========================================
CREATE THE DIRECTORY TARGETED FOR THE COPY (IF IT DOESN'T EXIST).
./release-candidate.sh: line 191: : command not found
========================================                                                      
RUN THE SYNCHRONIZATION.                                                                      
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Everything/ /pub/alt/stage/41_Beta-1.1/Everything/ --link-dest=/pub/fedora/linux/developme
nt/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Cloud/ /pub/alt/stage/41_Beta-1.1/Cloud/ --link-dest=/pub/fedora/linux/development/41/Ever
ything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Container/ /pub/alt/stage/41_Beta-1.1/Container/ --link-dest=/pub/fedora/linux/development
/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Kinoite/ /pub/alt/stage/41_Beta-1.1/Kinoite/ --link-dest=/pub/fedora/linux/development/41/
Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Labs/ /pub/alt/stage/41_Beta-1.1/Labs/ --link-dest=/pub/fedora/linux/development/41/Everyt
hing/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Server/ /pub/alt/stage/41_Beta-1.1/Server/ --link-dest=/pub/fedora/linux/development/41/Ev
erything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Silverblue/ /pub/alt/stage/41_Beta-1.1/Silverblue/ --link-dest=/pub/fedora/linux/developme
nt/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Spins/ /pub/alt/stage/41_Beta-1.1/Spins/ --link-dest=/pub/fedora/linux/development/41/Ever
ything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Workstation/ /pub/alt/stage/41_Beta-1.1/Workstation/ --link-dest=/pub/fedora/linux/develop
ment/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directo
ry
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Onyx/ /pub/alt/stage/41_Beta-1.1/Onyx/ --link-dest=/pub/fedora/linux/development/41/Everyt
hing/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/Sericea/ /pub/alt/stage/41_Beta-1.1/Sericea/ --link-dest=/pub/fedora/linux/development/41/
Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 204: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//comp
ose/metadata/ /pub/alt/stage/41_Beta-1.1/metadata/ --link-dest=/pub/fedora/linux/development/4
1/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
========================================
SET THE PERMISSIONS OF THE SYNCED COMPOSE.
cp: cannot stat '/compose/metadata/*.json': No such file or directory
========================================
LET INTERESTED PERSONS KNOW THAT THE RSYNC IS COMPLETE.
<I hit control c here>

CC: @jnsamyak @patrikp

So I think the problem here is that the label format the script expects - 39_RC-1.1 - is not a correct label format for productmd (which is what backs the check pungi-koji runs here). A canonical label does not include the release version. The label passed in to pungi must be just RC-1.1.

If we want to allow invoking this script with just a single arg, but the script needs to be told what the release version is, we shouldn't call it the "label" - we should call it something else - and we should split it to produce separate release version and label variables. Otherwise, it could be made to accept two arguments (release number and label), or it could be made to not need the release number to be passed in somehow (check what git branch it's on? I dunno).

https://pagure.io/pungi-fedora/pull-request/1369 and https://pagure.io/pungi-fedora/pull-request/1370 should fix this (on main and f41, respectively).

That fixed one issue, but there's more. ;) 

========================================
CONFIG: fedora-beta.conf
THIS IS NOT A DRY RUN.
========================================
ASSUME THAT THE RELENG REPOSITORY IS AVAILABLE AND REBASE. CLONE IT IF IT'S NOT.
/home/fedora/kevin/pungi-fedora/releng /home/fedora/kevin/pungi-fedora
Already up to date.
/home/fedora/kevin/pungi-fedora
========================================
ANNOUNCE THAT WE ARE STARTING (EVEN THOUGH THE COMPOSE_ID IS NOT YET KNOWN).
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefix='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': False, 'high-water-mark': 0})
..
Traceback (most recent call last):
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <module>
    body = json.loads(msgstring)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 54 (char 53)
========================================
RUN THE COMPOSE. EXIT IF THE EXIT STATUS IS NOT ZERO.
WARNING: productimg phase has been removed, please remove it from --skip-phase or --just-phase option
WARNING: Unrecognized config option: kiwibuild_repo_releasever. Did you mean kiwibuild_release?
Failed validation in kiwibuild.^Spins$.1: Additional properties are not allowed ('manifest_type' was unexpected)
Failed validation in kiwibuild.^Spins$.3: Additional properties are not allowed ('manifest_type' was unexpected)
real    0m1.880s
user    0m1.607s
sys 0m0.188s
ERROR: Command failed with a non-zero exit status.
./release-candidate.sh: line 124: : command not found
ERROR: No compose ID available.
./release-candidate.sh: line 132: : command not found
ERROR: Compose ID not found.
./release-candidate.sh: line 142: : command not found
ERROR: No compose STATUS available.
./release-candidate.sh: line 149: : command not found
cat: /mnt/koji/compose/41//STATUS: No such file or directory
========================================
COMPOSE DETAILS
Compose ID: 
Compose status: 
Proceeding to rsync.
========================================
LET INTERESTED PERSONS KNOW ABOUT THE COMPOSE.
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefix='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': False, 'high-water-mark': 0})
..
Traceback (most recent call last):
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <module>
    body = json.loads(msgstring)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 71 (char 70)
========================================
TELL INTERESTED PERSONS THAT THE RSYNC IS STARTING.
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefix='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': False, 'high-water-mark': 0})
..
Traceback (most recent call last):
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <module>
    body = json.loads(msgstring)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 72 (char 71)
========================================
CREATE THE DIRECTORY TARGETED FOR THE COPY (IF IT DOESN'T EXIST).
./release-candidate.sh: line 190: : command not found
========================================
RUN THE SYNCHRONIZATION.
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Everything/ /pub/alt/stage/41_Beta-1.1/Everything/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Cloud/ /pub/alt/stage/41_Beta-1.1/Cloud/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Container/ /pub/alt/stage/41_Beta-1.1/Container/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Kinoite/ /pub/alt/stage/41_Beta-1.1/Kinoite/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Labs/ /pub/alt/stage/41_Beta-1.1/Labs/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Server/ /pub/alt/stage/41_Beta-1.1/Server/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Silverblue/ /pub/alt/stage/41_Beta-1.1/Silverblue/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Spins/ /pub/alt/stage/41_Beta-1.1/Spins/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Workstation/ /pub/alt/stage/41_Beta-1.1/Workstation/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Onyx/ /pub/alt/stage/41_Beta-1.1/Onyx/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/Sericea/ /pub/alt/stage/41_Beta-1.1/Sericea/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
./release-candidate.sh: line 203: sudo -u ftpsync sh -c rsync -avhH /mnt/koji/compose/41//compose/metadata/ /pub/alt/stage/41_Beta-1.1/metadata/ --link-dest=/pub/fedora/linux/development/41/Everything/ --link-dest=/pub/alt/stage/41_Beta-1.1/Everything: No such file or directory
========================================
SET THE PERMISSIONS OF THE SYNCED COMPOSE.
cp: cannot stat '/compose/metadata/*.json': No such file or directory
========================================
LET INTERESTED PERSONS KNOW THAT THE RSYNC IS COMPLETE.
Namespace(topicprefix='', modname='compose', message='', topic='..', jsoninput=True, certprefix='releng', timeout=0, linger=0, **{'io-threads': 1, 'config-filename': '', 'print-config': False, 'high-water-mark': 0})
..
Traceback (most recent call last):
  File "/home/fedora/kevin/pungi-fedora/releng/scripts/fedora-messaging-logger", line 96, in <module>
    body = json.loads(msgstring)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 71 (char 70)

  1. pungi needs updated with the backported kiwi patches. I just did that. (so hopefully FIXED)

  2. The fedora-messaging calls don't seem right/are tracebacking.

  3. The non dry run exit 1 commands are failing because the prefix is "" and thats not a valid command. Probibly need a rework on debugging... set a 'DEBUG' var and check that?

I think the rest falls from that (ie, it would have exited for the pungi failure and never gotten to the rest of the script).

Metadata Update from @jnsamyak:
- Issue tagged with: dev, high-gain, medium-trouble

I think the problem with the messaging calls is that they're sending a single argument wrapped in double quotes:

send_fedora_message "${fedora_message_json_start} ${RELEASE} start"

if you look at nightly.sh, it does this:

send_fedora_message "${fedora_message_json_start}" ${RELEASE} start

note it's sending three args, not one long one. we probably need to fix that.

the CMD_PREFIX thing is odd because it clearly is working in other places - the git checkout block uses it and it's clearly behaving as intended there, and it's also used when calling send_fedora_message, and even though the message sending fails because we got the argument wrong, we clearly did do send_fedora_message. I wonder if the issue is that it's wrapped in double quotes in all the exit 1 calls? Compare:

$CMD_PREFIX pushd releng

with:

"$CMD_PREFIX" exit 1

also, when I try something similar at a console:

[adamw@xps13a tmp]$ testvar=''; $testvar echo "foo"
foo
[adamw@xps13a tmp]$ testvar=''; "$testvar" echo "foo"
[adamw@xps13a tmp]$

not exactly the same result, but clearly there's a difference there...

Metadata Update from @patrikp:
- Issue assigned to patrikp

Here's the PR:
https://pagure.io/pungi-fedora/pull-request/1371

I changed:
1) The double quotes in regard to messaging calls, now they should send three args instead of one long one.
2) The double quotes around CMD_PREFIX. Nice catch.
3) The argument parsing using a getopts while loop (for dry runs) was replaced by a simpler and more straightforward solution.

I am sorry that this script has been so problematic. Shell scripting is dirty business. When I do a dry run the output seems reasonable and I don't see an easy way to test it locally so that I can be sure it works exactly as intended.

It would be amazing to finally get this working and any help is greatly appreciated. (Thanks Adam!)

For clarity, here's another PR that Kevin created that fixes some of the quoting:
https://pagure.io/pungi-fedora/pull-request/1372

And it looks like the compose finished:
2024-09-11 01:55:57 [INFO] Compose finished: /mnt/koji/compose/41/Fedora-41-20240910.0

And we have a F41 Beta Release Candidate:
https://fedoraproject.org/wiki/Test_Results:Fedora_41_Beta_1.1_Summary

The syncing part [1] of the script unfortunately didn't work.
./release-candidate.sh: line 175: sudo -u ftpsync: command not found

[1] https://pagure.io/pungi-fedora/blob/main/f/release-candidate.sh#_175

I made a few more minor tweaks:

unquoting some of the rsync commands to get it to not teat 'command --args' as a single command thats not found:
https://pagure.io/pungi-fedora/pull-request/1374

drop the 'sh -c' here which caused the commands to fail because it tried to pass the entire thing to a new shell
https://pagure.io/pungi-fedora/pull-request/1375

Add fedora-secondary to link-dest so it hardlinks the already existing ppc64le/s390x files too:
https://pagure.io/pungi-fedora/pull-request/1376

With that I hope it's all working. A 1.2 is running now, I guess if that finishes ok we can close this. :)

This should be done, lets close it out. Thanks everyone!

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
Metadata

dev high-gain medium-trouble

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.