#34 `dnf` import makes the process unportable
Closed by mymindstorm. Opened by abitrolly.

https://pagure.io/fedora-packages-static/c/9ffeb1a6c87da24a7d1f08bd408c2889ec68fe27 adds dependency on dnf and hawkey, which are absent from PyPI https://pypi.org/project/dnf/ or lead to different version https://pypi.org/project/hawkey/

Resulting in failed build on non-RPM systems.

 Traceback (most recent call last):
  File "bin/fetch-repository-dbs.py", line 14, in <module>
    from dnf.subject import Subject
ModuleNotFoundError: No module named 'dnf'
make: *** [Makefile:29: sync-repositories] Error 1
Error: Process completed with exit code 2.

I am not sure what problem this commit is trying to solve, and why information in downloaded DB is not enough.


Fedora's infrastructure deals with packages using the "source package" name (i.e. the spec file name from src.fedoraproject.org). It makes sense from that standpoint to group packages using the "source package" name rather than the "built package" name. Notice the use of source package name in links on package pages such as https://packages.fedoraproject.org/pkgs/libdnf/python3-hawkey/ (related to libdnf, not python3-hawkey in Koji, sources, etc.).

In this case, libdnf is being used to parse the source package name from the database. You should still be able to build on a non-RPM system using a docker container. Non-rpm distros such as Ubuntu also provide python3-libdnf as a package via apt. Dependencies have always been listed as Fedora package names, not PyPI packages in the README.

I will add python3-libdnf as a dependency in the README and close this issue.

Commit 74be99fc fixes this issue

Commit 9ffeb1a says about packages who are sharing the same name? How that possible? And what does it mean they are always updated?

As for the name, having URLs like https://packages.fedoraproject.org/pkgs/libdnf/python3-hawkey/ instead of https://packages.fedoraproject.org/p/python3-hawkey/ makes it impossible to do package lookup.

Metadata Update from @abitrolly:
- Issue close_status updated to: None (was: Fixed)

Commit 9ffeb1a says about packages who are sharing the same name? How that possible?

It's an edge case that I ran into with real data when making that commit (I don't remember the exact package name). For example, package A could be renamed to package B. Package A produces rpms A and C, but after renaming rpm A to B, rpm C had no name change. This can be done in ways to avoid a package conflict in the same release, but packages-static collates data from multiple releases which makes things more complicated.

Packaging guidelines also acknowledge name conflicts: https://docs.fedoraproject.org/en-US/packaging-guidelines/Conflicts/#_conflicting_package_names

And what does it mean they are always updated?

Being included in a diff update unnecessarily. Assuming a package name conflict, all conflicting packages will get updated instead of only what is needed.

As for the name, having URLs like https://packages.fedoraproject.org/pkgs/libdnf/python3-hawkey/ instead of https://packages.fedoraproject.org/p/python3-hawkey/ makes it impossible to do package lookup.

What do you mean by "package lookup"?

Ah, here's a real example. Notice how source packages python3.10 and python3.9 both produce a python3 rpm with different metadata:
https://packages.stg.fedoraproject.org/pkgs/python3.9/
https://packages.stg.fedoraproject.org/pkgs/python3.10/

What do you mean by "package lookup"?

Check with a GET request that python3-hawkey package exists among Fedora packages. With link like https://packages.fedoraproject.org/p/python3-hawkey/ I can just use curl or type the URL in the browser. With https://packages.fedoraproject.org/pkgs/libdnf/python3-hawkey/ I need to know the source package name or use browser to search and then filter the results, which is hard to automate.

Ah, here's a real example. Notice how source packages python3.10 and python3.9 both produce a python3 rpm with different metadata:
https://packages.stg.fedoraproject.org/pkgs/python3.9/
https://packages.stg.fedoraproject.org/pkgs/python3.10/

Well, they generate RPMs for different versions of Fedora:

https://packages.stg.fedoraproject.org/pkgs/python3.9/python-unversioned-command/ is F34.
https://packages.stg.fedoraproject.org/pkgs/python3.10/python-unversioned-command/ is F35+.

So I don't see why these versions could not be on the same page. For me it is the way to reference to "package source" or (sourcepack?) that needs to be redone.

So URL scheme would be /p for packages and /s for sourcepacks. And URLs like /p/<package>.json containing the same info in machine readable format. Most important is info about dependencies for https://discussion.fedoraproject.org/t/extra-fast-package-index/35434

Apologies again for the lateness. I expect to have more time to work on packages-static starting this week.

So URL scheme would be /p for packages and /s for sourcepacks. And URLs like /p/.json containing the same info in machine readable format. Most important is info about dependencies for https://discussion.fedoraproject.org/t/extra-fast-package-index/35434

Except your proposed URL scheme does not account for how packages-static integrates with other Fedora systems, which all use the source package/.spec/SRPM name. As stated before, just open the Koji pages from the links you provided above for an example.

So I don't see why these versions could not be on the same page. For me it is the way to reference to "package source" or (sourcepack?) that needs to be redone.

python is only one example edge case, and it is across different Fedora versions as you noted. That does not dismiss the packaging guidelines which state that naming conflicts in the same Fedora version are possible. We cannot predict all edge cases. I'm worried that it is rare but possible for different packages to have the same name but be entirely unrelated and in the same version.


  • I can see how the current system is unintuitive, but all of the above edge cases are not a problem with the current scheme. I would rather stay on the side of representing the original data most accurately.
  • The URL scheme is not up for change at this time, especially given that there are external links to packages.fedoraproject.org present. If Fedora as a whole moved towards a simpler "sourcepak" model, I would be open to changing it.
  • My goal here is to explain to you why the URL scheme is that way since I failed to document my design decisions adequately in the commit log. I will make an effort to prevent this from happening in the future.

Regarding "package lookup" and the "extra fast index" you mentioned:

  • I appreciate your ideas and improvements, but I cannot predict features you want or need. Please open issues and discuss potential features with me before deciding on implementing something.
  • GETing is only one implementation, you could still generate a json file or similar to achieve the same result as you mentioned.
  • Are you aware of https://mdapi.fedoraproject.org/ ? packages-static is not the place for developer apis, and it seems better suited for your needs since it does not collate all Fedora releases for each package.
Metadata