Backround

I am calling an executable from Python and need to pass a variable to the executable. The executable however expects a file and does not read from stdin.

I circumvented that problem previously when using the subprocess module by simply calling the executable to read from /dev/stdin along the lines of:

# with executable 'foo'
cmd = ['foo', '/dev/stdin']
input_variable = 'bar'

with subprocess.Popen(
    cmd,
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    ) as process:
    stdout, stderr = process.communicate(input_variable)

    print(f"{process.returncode}, {stdout}, {stderr}")

This worked fine so far. In order to add concurrency, I am now implementing asyncio and as such need to replace the subprocess module with the asyncio subprocess module.

Problem

Calling asyncio subprocess for a program using /dev/stdin fails. Using the following async function:

import asyncio

async def invoke_subprocess(cmd, args, input_variable):
    process = await asyncio.create_subprocess_exec(
        cmd,
        args,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )

    stdout, stderr = await process.communicate(input=bytes(input_variable, 'utf-8'))

    print(f"{process.returncode}, {stdout.decode()}, {stderr.decode()}")

This generally works for files, but fails for /dev/stdin:

# 'cat' can be used for 'foo' to test the behavior
asyncio.run(invoke_subprocess('foo', '/path/to/file/containing/bar', 'not used')) # works
asyncio.run(invoke_subprocess('foo', '/dev/stdin', 'bar')) # fails with "No such device or address"

How can I call asyncio.create_subprocess_exec on /dev/stdin?

Note: I have already tried and failed via asyncio.create_subprocess_shell and writing a temporary file is not an option as the file system is readonly.

Minimal example using 'cat'

Script main.py:

import subprocess
import asyncio

def invoke_subprocess(cmd, arg, input_variable):
    with subprocess.Popen(
        [cmd, arg],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        ) as process:
        stdout, stderr = process.communicate(input_variable)

        print(f"{process.returncode}, {stdout}, {stderr}")


async def invoke_async_subprocess(cmd, arg, input_variable):
    process = await asyncio.create_subprocess_exec(
        cmd,
        arg,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )

    stdout, stderr = await process.communicate(input=input_variable)

    print(f"{process.returncode}, {stdout.decode()}, {stderr.decode()}")


cmd = 'cat'
arg = '/dev/stdin'
input_variable = b'hello world'

# normal subprocess
invoke_subprocess(cmd, arg, input_variable)
asyncio.run(invoke_async_subprocess(cmd, arg, input_variable))

Returns:

> python3 main.py
0, b'hello world', b''
1, , cat: /dev/stdin: No such device or address

Tested on:

Ubuntu 21.10, Python 3.9.7
Linux Mint 20.2, Python 3.8.10
Docker image: python:3-alpine

Backround

Problem

Minimal example using 'cat'

Recommended topics

Hot tags