The biggest problem I have is the adversarial relationship with UAPI group members have with us in Fedora. The fact the /usr/libexec is listed as "compatibility" because Lennart doesn't like it. It also does the same for /usr/lib<qual> in favor of /usr/lib/<triple> which we don't use.

Intrinsically, that specification makes no sense for us to adopt because of things like this.

As for why FHS 4.0 isn't a thing yet, the semi-lively discussion indicated there's more to work through. And I don't know how to work the DocBook stuff and want to move away to Sphinx.

I don't think UAPI has an adversarial relationship with Fedora. The spec should be evaluated on technical merit on the basis of whether it's better than FHS.

The fact the /usr/libexec is listed as "compatibility" because Lennart doesn't like it.

Until recently, /usr/libexec was really only used by Fedora. It's only recently that it gained more widespread adoption. I don't have a strong opinion on this, but systemd has been trying to reduce divergence between Linux distros, and previously the way to do that would have been to get rid of /usr/libexec.

Anyway, it's OK to use the spec as a baseline and make some Fedora-specific changes. Our packaging guidelines can simply make an exception for this and promote /usr/libexec anyway if that's what we want to do.

It also does the same for /usr/lib in favor of /usr/lib/ which we don't use.

So, multilib (Fedora-style) vs. multiarch (Debian-style). Thus far it has not really mattered very much other than presenting an annoying difference between distros, but in the future, it might matter a lot, because multiarch would notably allow us to use a different architecture triplet for a different userspace ABIs for the same CPU architecture in the future. Seems silly? We might eventually need it if an ABI break allows us to get something we really want, like memory safety, for example.

Anyway, same point applies: it's OK to use the spec as a baseline and make Fedora-specific changes if required.

What @catanzaro wrote describes the situation very well.

libexec is described, even though it is marked as "discouraged". This is a compromise between keeping the spec lean and correctly describing that

It also does the same for /usr/lib in favor of /usr/lib/ which we don't use.

This is actually an omission. /lib64 symlink is described, but /usr/lib64 is not. I'll submit a path to mention this is an alternative location for libraries. This is probably not the place to discuss this, but I think the Debian-style multiarch was a grand idea, but it turned out to be complicated to pull off and never gained dominance. That part of the spec was written a few years ago and we decided to promote multiarch over multilib, but I think it's time to revisit this.

We use multiarch in GNOME Flatpak runtimes.

Also:

We might eventually need it if an ABI break allows us to get something we really want, like memory safety, for example.

I think we will really sooner or later need a userspace ABI break. Multiarch isn't the only way to do it, but it's probably the easiest and least-disruptive way. Something to keep that in mind.

We use multiarch in GNOME Flatpak runtimes.

Also:

We might eventually need it if an ABI break allows us to get something we really want, like memory safety, for example.

I think we will really sooner or later need a userspace ABI break. Multiarch isn't the only way to do it, but it's probably the easiest and least-disruptive way. Something to keep that in mind.

There's been a discussion in the FHS project about incorporating a Exherbo/Fedora sysroot FHS structure to resolve the two worlds. We currently have a sysroot FHS for cross toolchains but Exherbo many years ago extended this to cover the whole system, and that model is fully backward compatible to both standard multilib and Debian-style multiarch libdir systems.

The sysroot style was first discussed on Fedora lists in 2017: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/F64O5RZE27JWCRDIUM5RE2V6C354PC52/

The current doc from Exherbo is here: https://www.exherbo.org/docs/multiarch.html

AGREED: Revisit on Dec 9 meeting (+6,0,-0)

Looks like we should revisit this again.

We discussed this during the meeting yesterday, but a chat is not a great place for longer discussion. I have the log open in a second tab and I'll try to respond to the points raised there in here.

zbyszek: your PR (https://pagure.io/packaging-committee/pull-request/1475#request_diff) updates the filesystem layout portion of the packager guidelines to reflect the way things are done generally now because they were out of date and didn't reflect.

This is true, but a bit misleading. Our part (i.e. the Packaging Guidelines) does reflect the way we package things. OTOH, the PGs refer to FHS 3.0, from which we have significantly diverged. My PR switches the PGs to refer to LFHS, which reflects current best practices and which we mostly follow. We have some divergence, and the PR adjusts the PGs to cover those. E.g., we recently started allowing /nix, and that needs to be and is mentioned in the "Additional Rules" section. We also use /boot/efi instead of /efi. I'll update my PR to describe this.

there is an upstream committee working across distros to try to get a new or updated set of changes on FHS

As I wrote in the chat, I think the uapi group should be considered "upstream" nowadays. This is where the systemd and kernel people working on early boot are active. The whole specs repo has dozens of contributors from many different projects and companies. It would also be good for Fedora to officially adopt that spec because then we get additional consideration and influence over later changes.

The work on FHS has picked up in very recent times, but that is mostly about the build system. And it'd be a big stretch to say that there's a "committee working across distros", since there are only two contributors in git history since the 3.0 tag.

What is the impact on Fedora packagers if we keep the docs the same

The current FHS 3.0 is a graveyard of old ideas. Just looking at the index: /bin and /sbin (gone in F17), including lengthy discussions of which binary goes in which directory and how to decide, /usr/sbin (gone in F42), /dev/MAKEDEV and /dev/MAKEDEV.local (gone 20+ years ago?), /etc/X11 (on its way out), /etc/sgml and /etc/xml, /usr/share as "architecture independent data", /usr/src as a place to "store sources", /var/lock, /var/yp. OTOH, there is no mention of UEFI (or the /boot/efi or /efi mount points); /run is mentioned, but crucial directories like /run/log or /run/lock or /run/user are not. Also, in general, FHS 3.0 describes the way that Linux systems were done 20 years ago, all mutable, rather messy, ready to be shared over NFS.

Overall, I think FHS is a great historical document, but it creates a barrier of entry for new people. In particular, it clings to the idea of splitting things up between /bin, /sbin, /usr/bin, etc. This split had its reasons (PDP-11, disk sizes, something something), but those reasons are dead and gone. We put in a lot of work in Fedora to get rid of those splits so it's not helpful to have a document that claims they are necessary. Anyone who hasn't been at this since the 2000's is misled and confused. Of course we can say that this doesn't matter, but I disagree.

The new FHS 4.0 is at this point mostly the same as 3.0. The work that is being done by the new group is so far mostly about the build system. (Which BTW. is docbook, with 1856 vendored files in docbook/xsl). The group is new. The project was ~~dumped~~donated by LF, so it's not like it's the same group up people realising a new update.

the delta is lower with FHS than LFSH

As I discussed above, this isn't the case. The LFSH is rather minimalistic in the sense that it is a much shorter document so there is much less to agree/disagree on. The text was modified in response to the feedback from Fedora to emphasize that it describes the basic structure and various local extensions and additions are expected. We have two now (/usr/<target> and /nix). The spec also extended the description of /usr/lib64 and added /opt and /usr/libexec. If people notice that things are missing, we can add them either in the upstream spec or in our PGs, whatever is more appropriate.

It is true that LFSH is more opionionated and generally recommends one way to do things, and in the few cases where there is extensive existing practice, it mentions the other approaches (/opt, /usr/libexec) but clearly marks them as a secondary choice. The goal is to steer people towards the best practices. I think this is much better than the approach taken in old "standards" like LSB and FHS which necessarilly had to include everything that the dozens of contributing distributions already had in place.

We can also take choice 4: just vendor a spec into our packaging guidelines and nope out of it

Please, no ;). Firstly, FHS 4.0 is 5k lines, LFSH is 720 lines. We have trouble to keep our docs up date, we shouldn't increase our workload gratuitously. Secondly, and more importantly, using a shared spec increases cooperation between distros and avoids pointless divergence. The introduction of systemd caused a slow increase in similarity between distros and we should continue on that path. (There is plenty that we should disagree on, e.g. how to build atomic distros, but those are areas of active research. The layout of static directories is not that.)

Alternate proposal: Fedora ceases to claim compatibility with ANY of the filesystem standards (to be revisited at a time in the future when one of them lines up better)

Also please no ;) We want people who are new contributors to be able to read a document that describes what they are seeing. And the LFSH has been updated to accomodate Fedora practices and generally lines up with what we're doing. (It also doesn't spend thousands of words to describe things which are are not doing.)

but we don't follow the UAPI spec now

Please be more specific. One mentioned /boot/efi vs. /efi above. Is there anything else?

To summarize, I still think that the proposed PR does right by our new and existing contributors and increases the quality of our documentation without much work. It also helps with cooperation with other distros. It also makes our future work smoother, because we use systemd for init and the base of the system, and that spec is co-maintained by systemd upstream.

We discussed this during the meeting yesterday, but a chat is not a great place for longer discussion. I have the log open in a second tab and I'll try to respond to the points raised there in here.

zbyszek: your PR (https://pagure.io/packaging-committee/pull-request/1475#request_diff) updates the filesystem layout portion of the packager guidelines to reflect the way things are done generally now because they were out of date and didn't reflect.

This is true, but a bit misleading. Our part (i.e. the Packaging Guidelines) does reflect the way we package things. OTOH, the PGs refer to FHS 3.0, from which we have significantly diverged. My PR switches the PGs to refer to LFHS, which reflects current best practices and which we mostly follow. We have some divergence, and the PR adjusts the PGs to cover those. E.g., we recently started allowing /nix, and that needs to be and is mentioned in the "Additional Rules" section. We also use /boot/efi instead of /efi. I'll update my PR to describe this.

there is an upstream committee working across distros to try to get a new or updated set of changes on FHS

As I wrote in the chat, I think the uapi group should be considered "upstream" nowadays. This is where the systemd and kernel people working on early boot are active. The whole specs repo has dozens of contributors from many different projects and companies. It would also be good for Fedora to officially adopt that spec because then we get additional consideration and influence over later changes.

The work on FHS has picked up in very recent times, but that is mostly about the build system. And it'd be a big stretch to say that there's a "committee working across distros", since there are only two contributors in git history since the 3.0 tag.

What is the impact on Fedora packagers if we keep the docs the same

The current FHS 3.0 is a graveyard of old ideas.

Technically Unix is a graveyard of old ideas, doesn't mean people aren't using them.

Just looking at the index: /bin and /sbin (gone in F17), including lengthy discussions of which binary goes in which directory and how to decide, /usr/sbin (gone in F42),

Yes and no. They are never going to be truly gone in Fedora because those directories are API. They exist as symlinks today and we cannot remove them. Additionally, software will still continue to have hardcoded ideas about this, even if our packaging does not. Calling them "gone" is not exactly correct.

/dev/MAKEDEV and /dev/MAKEDEV.local (gone 20+ years ago?)

Yes, and it's a good candidate for removal. But it's not even a mandatory thing, it's mentioned that it "may" exist.

/etc/X11 (on its way out),

Probably not, depending on how much of that X11 applications will require for Xwayland. We don't have a way to run X11 applications without an X server yet.

/etc/sgml and /etc/xml, /usr/share as "architecture independent data"

I mean, we definitely do this today, so no?

/usr/src as a place to "store sources"

It's not common, but we do use this in Fedora. One very prominent example is nginx, where the nginx sources are installed to /usr/src with nginx-mod-devel because the actual fully patched sources are required to build nginx modules. Another example is all the infrastructure for building kernel modules. The kernel package itself puts things there, and packaged DKMS and Akmods in third party repositories put kernel module sources there.

/var/lock, /var/yp.

Things still use these, so they have to stay there.

OTOH, there is no mention of UEFI (or the /boot/efi or /efi mount points);

It is being discussed with a broader implication that all known platform directories should probably be documented. UEFI should not be special in this regard.

/run is mentioned, but crucial directories like /run/log or /run/lock or /run/user are not.

Are those actually standard subdirectories? Also, why would you have a /run/log when it's tmpfs?

Also, in general, FHS 3.0 describes the way that Linux systems were done 20 years ago, all mutable, rather messy, ready to be shared over NFS.

This is still how Linux systems are generally used today. There's still a lot of that out there and it's probably never going away.

Overall, I think FHS is a great historical document, but it creates a barrier of entry for new people. In particular, it clings to the idea of splitting things up between /bin, /sbin, /usr/bin, etc. This split had its reasons (PDP-11, disk sizes, something something), but those reasons are dead and gone. We put in a lot of work in Fedora to get rid of those splits so it's not helpful to have a document that claims they are necessary. Anyone who hasn't been at this since the 2000's is misled and confused. Of course we can say that this doesn't matter, but I disagree.

Fedora has done a lot to make those splits invisible, but getting rid of them is something we'll never do because we value compatibility with software.

The new FHS 4.0 is at this point mostly the same as 3.0. The work that is being done by the new group is so far mostly about the build system. (Which BTW. is docbook, with 1856 vendored files in docbook/xsl). The group is new. The project was ~~dumped~~donated by LF, so it's not like it's the same group up people realising a new update.

the delta is lower with FHS than LFSH

As I discussed above, this isn't the case. The LFSH is rather minimalistic in the sense that it is a much shorter document so there is much less to agree/disagree on. The text was modified in response to the feedback from Fedora to emphasize that it describes the basic structure and various local extensions and additions are expected. We have two now (/usr/<target> and /nix). The spec also extended the description of /usr/lib64 and added /opt and /usr/libexec. If people notice that things are missing, we can add them either in the upstream spec or in our PGs, whatever is more appropriate.

It is true that LFSH is more opionionated and generally recommends one way to do things, and in the few cases where there is extensive existing practice, it mentions the other approaches (/opt, /usr/libexec) but clearly marks them as a secondary choice. The goal is to steer people towards the best practices. I think this is much better than the approach taken in old "standards" like LSB and FHS which necessarilly had to include everything that the dozens of contributing distributions already had in place.

Who decides the best practices? That's the biggest reason why we shouldn't. Fedora as the actual implementor has the right to declare what its best practices are.

The FHS document is consensus driven on how things change, which means that operating system stakeholders engage and help evolve it based on what they've done already.

Many of the things you're talking about have happened only in the past few years of consensus.

For example: /usr merge (done by Fedora in 2012) was done by Ubuntu in 2020, openSUSE did it in 2021, and Debian didn't finish the transition until last year with Debian 13. It is literally only now in 2026 that all major distribution families have done this.

Consequently, there are open discussions and considerations on how to document these new defaults in the FHS document.

We can also take choice 4: just vendor a spec into our packaging guidelines and nope out of it

Please, no ;). Firstly, FHS 4.0 is 5k lines, LFSH is 720 lines. We have trouble to keep our docs up date, we shouldn't increase our workload gratuitously. Secondly, and more importantly, using a shared spec increases cooperation between distros and avoids pointless divergence. The introduction of systemd caused a slow increase in similarity between distros and we should continue on that path. (There is plenty that we should disagree on, e.g. how to build atomic distros, but those are areas of active research. The layout of static directories is not that.)

Alternate proposal: Fedora ceases to claim compatibility with ANY of the filesystem standards (to be revisited at a time in the future when one of them lines up better)

Also please no ;) We want people who are new contributors to be able to read a document that describes what they are seeing. And the LFSH has been updated to accomodate Fedora practices and generally lines up with what we're doing. (It also doesn't spend thousands of words to describe things which are are not doing.)

but we don't follow the UAPI spec now

Please be more specific. One mentioned /boot/efi vs. /efi above. Is there anything else?

To summarize, I still think that the proposed PR does right by our new and existing contributors and increases the quality of our documentation without much work. It also helps with cooperation with other distros. It also makes our future work smoother, because we use systemd for init and the base of the system, and that spec is co-maintained by systemd upstream.

Realistically, this has zero actual effect on Fedora contributors. Fedora contributors will do what our packaging macros do, and that's pretty much it. So whether it's FHS or LFSH or something we maintain in our guidelines doesn't matter at all to them.

What does matter about which spec with use is how much autonomy and agency Fedora is allowed to have. Fedora is the implementor and the people who implement things in Fedora ultimately are supposed to drive the ideas that other communities adopt and from there we see consensus build and document them in the FHS spec so people know that these are generally available and expected.

The FHS document has always been a "lagging standard", though this may change over time as people value conformance as a property of evolution (it is something a few folks have started talking about).

Technically Unix is a graveyard of old ideas, doesn't mean people aren't using them.

By "graveyard of old ideas" I meant the ones that are actually dead, not just old. We are still using plenty of UNIX and that is fine. But some ideas are gone and we shouldn't pretend that we're using UNIX as it was originally conceived.

Just looking at the index: /bin and /sbin (gone in F17), including lengthy discussions of which binary goes in which directory and how to decide, /usr/sbin (gone in F42),

Yes and no. They are never going to be truly gone in Fedora because those directories are API. They exist as symlinks today and we cannot remove them. Additionally, software will still continue to have hardcoded ideas about this, even if our packaging does not. Calling them "gone" is not exactly correct.

No. We have the symlinks and that is fine. But we don't have the directories. And the whole complicated discussion about what belongs where, when all those symlinks point to the same place, is the part that belongs in the graveyard.

/dev/MAKEDEV and /dev/MAKEDEV.local (gone 20+ years ago?)

Yes, and it's a good candidate for removal. But it's not even a mandatory thing, it's mentioned that it "may" exist.

/etc/X11 (on its way out),

Probably not, depending on how much of that X11 applications will require for Xwayland. We don't have a way to run X11 applications without an X server yet.

In general, the approach of very verbose static configuration in /etc is what is deprecated. This applies to /etc/X11, but even more so to /etc/sgml and /etc/xml and such — we certainly don't want users to modify this as configuration.

/var/lock, /var/yp.

Things still use these, so they have to stay there.

No, actually no. /var/lock is a symlink to /run/lock, but at a wider scope, the approach of locks in the filesystem is deprecated because they have the wrong lifetime, i.e. they are not automatically removed when the process holding them dies. I'm also pretty sure we retired our implementation of NIS.

OTOH, there is no mention of UEFI (or the /boot/efi or /efi mount points);

It is being discussed with a broader implication that all known platform directories should probably be documented. UEFI should not be special in this regard.

/run is mentioned, but crucial directories like /run/log or /run/lock or /run/user are not.

Are those actually standard subdirectories? Also, why would you have a /run/log when it's tmpfs?

Phew, of course they are "standard". If you take "standard" to mean everybody has them and they make sense. Let's not go into the discussion why here.

Also, in general, FHS 3.0 describes the way that Linux systems were done 20 years ago, all mutable, rather messy, ready to be shared over NFS.

This is still how Linux systems are generally used today. There's still a lot of that out there and it's probably never going away.

Overall, I think FHS is a great historical document, but it creates a barrier of entry for new people. In particular, it clings to the idea of splitting things up between /bin, /sbin, /usr/bin, etc. This split had its reasons (PDP-11, disk sizes, something something), but those reasons are dead and gone. We put in a lot of work in Fedora to get rid of those splits so it's not helpful to have a document that claims they are necessary. Anyone who hasn't been at this since the 2000's is misled and confused. Of course we can say that this doesn't matter, but I disagree.

Fedora has done a lot to make those splits invisible, but getting rid of them is something we'll never do because we value compatibility with software.

No, they are gone. The guidelines allow the old macros (%_sbindir, etc.) to be used, but they evaluate to the fixed unified location. And when packages are actually built, the paths are all new so for the user the partial compat is invisible.

I think our disagreement on how we evaluate this stems from a difference in how we see the state of the distro. In my view, we have a layout that evolves over time, as we figure out new challenges, but is also simplified to drop old ideas that are not necessary anymore to keep things manageable. But we maintain partial compatibility, putting in shims to keep older interfaces functional, and we also have long periods when things are being moved over to the new layout. OTOH, you seem to take those old shims as the real thing. I find that just the wrong approach. If is fine to say that /var/lock exists for compat but is actually just a symlink and should not be used for anything new. It is not fine to describe /var/lock as a full-blown thing.

Who decides the best practices? That's the biggest reason why we shouldn't. Fedora as the actual implementor has the right to declare what its best practices are.

The FHS document is consensus driven on how things change, which means that operating system stakeholders engage and help evolve it based on what they've done already.

Many of the things you're talking about have happened only in the past few years of consensus.

For example: /usr merge (done by Fedora in 2012) was done by Ubuntu in 2020, openSUSE did it in 2021, and Debian didn't finish the transition until last year with Debian 13. It is literally only now in 2026 that all major distribution families have done this.

Consequently, there are open discussions and considerations on how to document these new defaults in the FHS document.

"Who" is always complicated. In this case it was "Fedora" (for some loose definition of) that decided. We drove the first implementation and documented it everywhere. The fact that "FHS" (both the old body and the new body) is still thinking about it shows that it probably is not the right entity to bind our design choices to.

At this time, I think the wider systemd camp is where those design decisions are being made… The big changes in layout of filesystem in Fedora have all come from that direction in the last 15 years. And considering that changes to initrds and early boot that are likely to happen, this trend will continue. And I think we should have a spec that keep up with that. Having a "lagging standard" that applies just as well to Debian oldstable as to what we're trying to do in Rawhide is not particularly useful.

To summarize, I still think that the proposed PR does right by our new and existing contributors and increases the quality of our documentation without much work. It also helps with cooperation with other distros. It also makes our future work smoother, because we use systemd for init and the base of the system, and that spec is co-maintained by systemd upstream.

Realistically, this has zero actual effect on Fedora contributors. Fedora contributors will do what our packaging macros do, and that's pretty much it. So whether it's FHS or LFSH or something we maintain in our guidelines doesn't matter at all to them.

If what you're saying is true, you shouldn't be opposed to the change I proposed!

But I disagree. I think it is important to describe the layout in a way that can be easily understood by a newcomer. The macros and other automatism is just just an implementation detail and the packager has to understand what is it is doing and why, and be able to override it if appropriate. So a design document that doesn't overwhelm but lists the important design choices in filesystem layout is important.

I'ma go ahead and tag this as "stalled". Last time we discussed this we were as far from a consensus as can be, and the last update was over 3 months ago.