For more context, I'm doing this from with an Apache Airflow DAG. When running as an Airflow task, the value of sys.stdin.fileno() is >=4, not 0 like I get from CLI.

import logging
import os
import time
import daemon
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from daemon.pidfile import TimeoutPIDLockFile
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = logging.FileHandler("/tmp/my-daemon.log")
logger.addHandler(handler)
def main():
    logger.info("In main")
    timeout = time.time() + 60 * 5  # 5 minutes from now
    while True:
        test = 0
        if test == 5 or time.time() > timeout:
            logger.info("Timeout. Exiting.")
            break
        test = test - 1
def python_fn():
    rootpath = os.path.dirname(os.path.abspath(__file__))
    logger.info("Starting daemon")
    with daemon.DaemonContext(
        pidfile=TimeoutPIDLockFile("/tmp/my-daemon.pid", -1),
        stdin=handler.stream,
        stdout=handler.stream,
        stderr=handler.stream,
        umask=0o002,
        files_preserve=[handler.stream],
        working_directory=rootpath,
    ) as context:
        main()
with DAG(
    dag_id=os.path.basename(__file__).replace(".py", ""),
    catchup=False,
    start_date=days_ago(1),
) as dag:
    t = PythonOperator(
        task_id="daemon", python_callable=python_fn, provide_context=True
    )
if __name__ == "__main__":
    python_fn()

On 10-Feb-2022, Mark Richman wrote:

I'm trying to use DaemonContext as follows:
=20
python with DaemonContext( working_directory=3D"/tmp", umask=3D0o002, stdout=3Dsys.stdout, stderr=3Dsys.stderr, pidfile=3Dpidfile.TimeoutPIDLockFile("/tmp/cwmetrics.pid"), ): main()

That looks okay; though, the stdout and stderr parameters need to
have values which will be useful after the daemon context starts.

The standard streams of a process (in Python, the sys.stdout etc.
streams) are typically lost by detaching the process context; the
original file streams are no longer available to the detached process.

For this reason, the DaemonContext constructor should be passed file
objects that will survive detaching the process from its standard
streams.

I get the error ValueError: I/O operation on closed file.

That's right; typically, detaching the process context has the
apparent effect of closing the streams, which are no longer
accessible.

So, you'll need to design your program so you've got somewhere for
those streams to connect to that isn't dependent on the original
process's streams (e.g., maybe a log file, or some network service
that implements the file API, etc.), and supply those file objects as
stdin, stdout, stderr arguments as you require.

--=20
\ =E2=80=9CLaugh and the world laughs with you; snore and you sl=
eep |
`\ alone.=E2=80=9D =E2=80=
=94anonymous |
_o__) |
Ben Finney ben@benfinney.id.au

On 14-Feb-2022, Mark Richman wrote:

```python
=E2=80=A6
logger =3D logging.getLogger(name)
logger.setLevel(logging.INFO)
handler =3D logging.FileHandler("/tmp/my-daemon.log")
logger.addHandler(handler)
=E2=80=A6
=20
def python_fn():
rootpath =3D os.path.dirname(os.path.abspath(file))
logger.info("Starting daemon")
with daemon.DaemonContext(
pidfile=3DTimeoutPIDLockFile("/tmp/my-daemon.pid", -1),
stdin=3Dhandler.stream,
stdout=3Dhandler.stream,
stderr=3Dhandler.stream,
umask=3D0o002,
files_preserve=3D[handler.stream],
working_directory=3Drootpath,
) as context:
main()

Yes, this example avoids using the process's own streams, and instead
provides handler.stream. This is a good example of supplying a
stream file object that will survive detaching the process.

--=20
\ =E2=80=9CI knew it was a shocking thing to say, but =E2=80=A6 no-=
one has the |
`\ right to spend their life without being offended.=E2=80=9D =E2=
=80=94Philip |
_o__) Pullman, 2010-03-28 |
Ben Finney ben@benfinney.id.au

Yes, this example avoids using the process's own streams, and instead provides handler.stream. This is a good example of supplying a stream file object that will survive detaching the process.

I understood this to be the case as well, but I still get the same I/O error with this example. Stack trace:

[2022-02-15 15:41:35,233] {{taskinstance.py:1482}} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 128, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/dag-daemon.py", line 39, in python_fn
    working_directory=rootpath,
  File "/usr/local/lib/python3.7/site-packages/daemon/daemon.py", line 272, in __init__
    detach_process = is_detach_process_context_required()
  File "/usr/local/lib/python3.7/site-packages/daemon/daemon.py", line 819, in is_detach_process_context_required
    if is_process_started_by_init() or is_process_started_by_superserver():
  File "/usr/local/lib/python3.7/site-packages/daemon/daemon.py", line 795, in is_process_started_by_superserver
    stdin_fd = sys.__stdin__.fileno()
ValueError: I/O operation on closed file

I assumed that stdin=handler.stream would do the trick, but apparently not.

Thanks,
Mark

On 15-Feb-2022, Mark Richman wrote:

[=E2=80=A6] File "/usr/local/lib/python3.7/site-packages/daemon/daemon.py", line 79= 5, in is_process_started_by_superserver stdin_fd =3D sys.__stdin__.fileno() ValueError: I/O operation on closed file

This is a bug in the detection of a socket on the stream: it
incorrectly assumes the standard input stream will always be open.

I have made a new test case for that condition, and now implemented a
different helper for detecting whether the stream has a socket
attached.

<URL: https://pagure.io/python-daemon/c/4d447866b4376bc567acd7a9b7c2930=

0b23605e3?branch=3Dmain>

It's passing the tests; I would appreciate if you can fetch the
current unreleased code and try it in your environment?

--=20
\ =E2=80=9CBeware of bugs in the above code; I have only proved=
it |
`\ correct, not tried it.=E2=80=9D =E2=80=94Donald Knuth,=
1977-03-29 |
_o__) |
Ben Finney ben@benfinney.id.au

The URL is mangled, could you repost please?

On 19-Feb-2022, Mark Richman wrote:

The URL is mangled, could you repost please?

It was just the specific commit; but you should simply be able to
fetch the current =E2=80=98main=E2=80=99 branch.

--=20
\ =E2=80=9CWhatever you do will be insignificant, but it is v=
ery |
`\ important that you do it.=E2=80=9D =E2=80=94Mohanda=
s K. Gandhi |
_o__) |
Ben Finney ben@benfinney.id.au

There is something wrong with whatever client you are using. I literally see =E2=80=98main=E2=80=99 in your post.

On 19-Feb-2022, Mark Richman wrote:

There is something wrong with whatever client you are using. I literally =
see =3DE2=3D80=3D98main=3DE2=3D80=3D99 in your post.

Yes, that's a known bug in Pagure, unfortunately.

--=20
\ =E2=80=9CAnyone who puts a small gloss on [a] fundamental technolo=
gy, |
`\ calls it proprietary, and then tries to keep others from |
_o__) building on it, is a thief.=E2=80=9D =E2=80=94Tim O'Reilly,=
2000-01-25 |
Ben Finney ben@benfinney.id.au

This bug is resolved in ‘python-daemon’ version 2.3.1, released today.