python-daemon

#72 Excessive memory usage when `resource.RLIMIT_NOFILE` is high
Closed: Fixed by bignose. Opened by antonio-antuan.

Closed: Fixed
Reopen Issue

I investigated this: https://github.com/apache/airflow/issues/29841
tldr: inside of docker containers ulimits may be large (by default, no one cares about it:) ). so creating a set here may consume huge amount of memory

this is the place where things happens: site-packages/daemon/daemon.py:868

def get_maximum_file_descriptors():
    """ Get the maximum number of open file descriptors for this process.
        :return: The number (integer) to use as the maximum number of open
            files for this process.
        The maximum is the process hard resource limit of maximum number of
        open file descriptors. If the limit is “infinity”, a default value
        of ``MAXFD`` is returned.
        """
    (__, hard_limit) = resource.getrlimit(resource.RLIMIT_NOFILE)
    result = hard_limit
    if hard_limit == resource.RLIM_INFINITY:
        result = MAXFD
    return result
_total_file_descriptor_range = (0, get_maximum_file_descriptors())
_total_file_descriptor_set = set(range(*_total_file_descriptor_range))

inside of my image (apache/airflow:2.3.4) hard limit equals 1073741816. so memory consumption of _total_file_descriptor_set is... huge.

(Note I am maintainer of Airflow looking to find a good solution to our users).

It is indeed wreaking havoc among Airflow users starting last week when a number of OSes released the new comtainerd that sets the limit to infinity.

The issue in containerd is hrere (there is attempt to revert the change as python-daemon is not the only one impacted by it) https://github.com/containerd/containerd/pull/7566#issuecomment-12854173

We are just about to release Airflow 2.5.2 (in a few days) and if there is a little chance that this one will be fixed soon, I am considering either removing python-daemon from airflow as a dependency or some kind of patching (if possible) or even vendoring it in.

I would love to heaer if there is a chance this one will be quickly fixed ?

BTW. The solution for the problem would be:

    if hard_limit > MAXFD:
        result = MAXFD
Edited by potiuk

@antonio-antuan, you're correct that past releases of python-daemon create an in-memory list of (potentially a huge number of) file descriptors.

We have a re-implementation of the close_all_open_files logic, which avoids the creation of that temporary structure in memory. The new implementation was merged to main as of commit 297e91ab.

I suspect this will address the issue; can you test by using the current main branch of python-daemon?
Edited by bignose

@bignose checked, didn't experience the problem with main branch. looks like it is fixed.

Confirmed. Tested it also in latest Fedora with built-in containerd, installing the main version of python-daemon solves the problem and Airflow starts with no problems.

Thanks @bignose for such quick response!

Do you have plans for the release? Can we help somehow with testing it maybe to speed it up ?

Hey @bignose - any comment here? We are about to release new airflow version and we would love that one to be part of it.

I do not really "demand" this to be released quickly (it might take quite a bit of time to test and such) - but just an indication of the timeline it might take to release it - if it is days, we can likely wait, if it is weeks or months, then we will likely temporary vendor-in what is currently main version and remove the vendoring when new version gets released, but i would like to know if this is at all needed - it's a bit of hassle to vendor-in and later to revert it.

BTW. I already have vendoring-in PRs in place, so it's just a matter of a decision for us based on the indication of the timeline

https://github.com/apache/airflow/pull/29845
https://github.com/apache/airflow/pull/29848

Confirmed. Tested it also in latest Fedora with built-in containerd, installing the main version of python-daemon solves the problem and Airflow starts with no problems.

Great, I will document that this bug is fixed by the changes.

Do you have plans for the release? Can we help somehow with testing it maybe to speed it up ?

I can't say when it will appear, but it's close to completion and I'm working on it this weekend.

Metadata Update from @bignose:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

I see that python-daemon 3.0.0 has been released :) cool..

Thanks @bignose for super quick release !
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.