Some spec files have conditionally defined Sources. The Fedora packaging guidelines now forbid that in https://docs.fedoraproject.org/en-US/packaging-guidelines/SourceURL/#_do_not_conditionalize_sources but in reality, many packages in RHEL and Fedora do it.
Since the inclusion of the "not downloading unused sources" feature by @oturpe, conditionally defined sources are skipped depending on the host macros.
E.g. this:
Source0: pip-22.3.1.tar.gz %if 0%{?rhel} == 8 Source1: pip-22.3.1-man.tar.gz %endif
SHA512 (pip-22.3.1-man.tar.gz) = 6dc14a0565747e30911d23d6c618a396e30d957e6bd8d1c8248bc9e51ebebc01751b051b761e846cf0223d26a57c1bae732a004d171ba9ad18a6cc9c55a60be3 SHA512 (pip-22.3.1.tar.gz) = c7865c4ce79b0fea7ba469fe593af3be205b3bdb574a6ae019842e0f82a0af406ec905d8c6ff02fbbafe01630c8899a398d071ce74501ca830249160f2f6df98
Leads to the following problem with centpkg on Fedora:
[python3.11-pip (c8s)]$ centpkg sources Not downloading unused pip-22.3.1-man.tar.gz Downloading pip-22.3.1.tar.gz ######################################################################## 100.0%
[python3.11-pip (c8s)]$ centpkg srpm ... error: Bad file: .../centpkg/python3.11-pip/pip-22.3.1-man.tar.gz: No such file or directory
See https://git.centos.org/centos/centpkg/issue/71
Fundamentally, this problem is caused by the fact that (rh|cent|fed)pkg sources does not set the %rhel and %fedora macro at all -- the sources command works the same for any value of branch/--release.
(rh|cent|fed)pkg sources
%rhel
%fedora
sources
--release
I think that fixing this properly would require a valid branch/--release for sources which would be an UX regression for others. Hence, maybe this could work like this:
E.g.:
[python3.11-pip (c8s)]$ centpkg sources Downloading pip-22.3.1-man.tar.gz Downloading pip-22.3.1.tar.gz [python3.11-pip (c9s)]$ centpkg sources Not downloading unused pip-22.3.1-man.tar.gz Downloading pip-22.3.1.tar.gz [python3.11-pip (arbitrary-branch)]$ centpkg sources Could not set distribution macros: Could not find the release/dist from branch name arbitrary-branch. Please specify with --release. Falling back to downloading all sources. Downloading pip-22.3.1-man.tar.gz Downloading pip-22.3.1.tar.gz
WDYT?
Thank you for investigating the issue in detail. I checked, and your analysis is correct: The specfile parser that checks for used sources does not consider those macros at all. Adding them in is easy. Your proposed default action of downloading everything is also sensible.
I wrote pull request #678. I would still like to test it more. Or, if you have the time to test it and report the results to the pull request, I would be grateful.
Testing with a package that has a rhel==8 conditionalized source.
[python3.11-pip (c8s)]$ centpkg sources Downloading pip-22.3.1.tar.gz ######################################################################## 100.0% Downloading pip-22.3.1-man.tar.gz ######################################################################## 100.0%
And:
[python3.11-pip (c8s)]$ centpkg --release c9s sources Downloading pip-22.3.1.tar.gz ######################################################################## 100.0% Not downloading unused pip-22.3.1-man.tar.gz
Parsing specfile for used sources failed: Branchname: batman is not valid Falling back to downloading all sources. Downloading pip-22.3.1.tar.gz ######################################################################## 100.0% Downloading pip-22.3.1-man.tar.gz ######################################################################## 100.0%
Same behavior with actual branches.
[python3.11-pip (c10s)]$ centpkg sources Downloading pip-22.3.1.tar.gz ######################################################################## 100.0% Not downloading unused pip-22.3.1-man.tar.gz
Or
[python3.11-pip (batman)]$ centpkg sources Parsing specfile for used sources failed: Branchname: batman is not valid Falling back to downloading all sources. Downloading pip-22.3.1.tar.gz ######################################################################## 100.0% Downloading pip-22.3.1-man.tar.gz ######################################################################## 100.0%
The message is quite hard to understand here, but it does what it should do.
Can the message be amended? E.g.
Parsing specfile for used sources failed: Cannot determine release information from branch/--release "batman". Falling back to downloading all sources.
Thank you for testing.
The important part of the error message comes from centpkg. How this works is that there is shared code in rpkg, and fedpkg, centpkg etc. overloading things as needed. One of the main overload points is load_rpmdefines, where the mapping from project's branch names to rpm defines happens.
centpkg
rpkg
fedpkg
load_rpmdefines
For instance, with fedpkg I get a better error message:
$ LANG=C.UTF-8 fedpkg sources Parsing specfile for used sources failed: Could not find the release/dist from branch name batman Please specify with --release Falling back to downloading all sources. Not downloading already downloaded mercenary-0.4.0.gem [otto@ottovain rubygem-mercenary]$ LANG=C.UTF-8 centpkg sources Not downloading already downloaded mercenary-0.4.0.gem
So it seems that to improve the error message, a centpkg issue/pull request should be filed. I will leave that to people who are involved with that project. The string that needs to change is written here.
But that ain't much better in fedpkg, is it?
$ fedpkg --release batnam sources Parsing specfile for used sources failed: Could not find the release/dist from branch name batnam Please specify with --release Falling back...
It tells the user to use --release even went hey did.
IMHO we should override such messages here, only for this use case, and provide a fine-grained warning that will not confuse the users.
$ fedpkg --release batman sources Parsing specfile for used sources failed: Could not find the release/dist from branch name batman Please specify with --release Falling back...
It tells the user to use --release even when they did.
You are right, even the fedpkg variant has a weakness. I created fedpkg#518 to improve on that.
I disagree with handling this as a special case for two reasons:
In the rpkg context, there is no way to know what went wrong with the overloaded load_rpmdefines. It is a good guess that either --release was set incorrectly or the Git branch is custom, but in reality, the overriding function may be doing anything, thus be failing for any reason. So the only way to get reliable printout for the user is to rely on the resulting exception, and fix the source of that exception if it contains a bad error message.
Many commands needs to know the release, thus load_rpmdefines is very commonly called. If the bad errors are fixed here by special casing, all the other commands will remain broken. it is better to fix the problem at the source, and get the benefits for all commands.
Commit 8667d537 fixes this issue
Updated and merged rpkg part; fedpkg is pending review.
The pull request has been merged, but it is somewhat different than @churchyard suggested, so I would like to double check. Even if you did not get all your wishes in, is the fix still an acceptable solution for the issue you are/were facing?
I think so, thanks.
Commit 6a381fd9 fixes this issue
Metadata Update from @onosek: - Issue set to the milestone: 1.67