releng

#12049 coreos-ostree-importer is hitting permission error
Closed: Fixed with Explanation by kevin. Opened by dustymabe.

Closed: Fixed with Explanation
Reopen Issue
  • Describe the issue
2024-04-04 02:37:58,344 DEBUG builtins - Running command: ['ostree', '--repo=/mnt/koji/compose/ostree/repo', 'pull-local', '/tmp/tmpdyh272r0', '2f706425d44d309f1b8ac9c508ce4c66459c274eec9881a53664a0221f12b024']
2024-04-04 02:37:58,355 ERROR builtins - Command returned bad exitcode
2024-04-04 02:37:58,355 ERROR builtins - COMMAND: ['ostree', '--repo=/mnt/koji/compose/ostree/repo', 'pull-local', '/tmp/tmpdyh272r0', '2f706425d44d309f1b8ac9c508ce4c66459c274eec9881a53664a0221f12b024']
2024-04-04 02:37:58,355 ERROR builtins - STDOUT:
2024-04-04 02:37:58,355 ERROR builtins - STDERR: error: Opening lock file /mnt/koji/compose/ostree/repo/.lock failed: Permission denied

I don't really understand what is going on here. I'm a member of group 263 and the group of the file is 263 and it has group write permission. 

sh-5.2$ id
uid=1000740000(1000740000) gid=0(root) groups=0(root),263,1000740000
sh-5.2$ ostree pull-local /mnt/koji/ostree/repo/ aef34900599b2bc5b906d2621231a2fa8ae9c98b92aa74c8dcd176689fc60e9f
error: Opening lock file /mnt/koji/compose/ostree/repo/.lock failed: Permission denied
sh-5.2$ ls -ld .
drwxrwsr-x. 9 263 263 4096 Apr  2 03:14 .
sh-5.2$ ls -lh .lock
-rw-rw----. 1 263 263 0 Feb 18  2020 .lock
  • When do you need this? (YYYY/MM/DD)

ASAP so coreos builds can work again.

  • When is this no longer needed or useful? (YYYY/MM/DD)

  • If we cannot complete your request, what is the impact?

the coreos pipeline can't operate.

I'm pretty sure I used to be able to create files in those directories.. i.e. touch /mnt/koji/compose/ostree/repo/foo shouldn't give me a permission denied I don't think.

link to ticket with historical info on how we setup all the permissions here: https://pagure.io/releng/issue/8811

We did have a problem last year with a lock file, but looking I don't see it locked on the server end. ;(

Could it have been because updates pushes were going?

I don't think so. It seems to fail immediately, where if it was because it couldn't take the lock I would expect it to wait before failing because we have lock-timeout-secs=1800 in the config.

@kevin can we meet up later today to do a screenshare session on this to try to debug. @jlebon and I just met and poked around but couldn't figure it out. We're thinking it has to be some change with either the NFS mount or maybe OpenShift??

ok, dusty and I met up and looked at this more.

The problem was on the netapp side. :(

For some reason, whoever added the openshift compute nodes to the koji export policy (spoiler, it was me) didn't somehow have worker04 in there. All the rest were in there, but not 04.
So, before it worked because your pods were on a compute node that was allowed a rw mount by netapp.
When the pods landed on 04, it could only do a ro mount, so nothing would work. :(

I've fixed the policy and confirmed that all the other nodes are correct and in there.

Sorry for the mess...

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

Thanks @kevin !
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.