python subprocess output to list or file

Question

I want to run the following bash command in Python 3:

ls -l

I know that I can do the following:

from subprocess import call
call(['ls', '-l'])

How do I save this output to a file, or put it into lists or sets?

[-rw-r--r--]  [1] [name]  [staff]   [426] [14 Jan 21:52] [HelloWorld.class]
[-rw-r--r--@] [1] [name]  [staff]   [107] [14 Jan 21:51] [HelloWorld.java]
...
etc.

I want to be able to access particular information directly, and then add it to the set, but I do not know how many items will be listed.

Any hints, snippets, or examples would really help.

Observe

If you want to access file/dir information you might want to check the built-in os library: documentation. – Hildahildagard

Answer 1

One way to access to the information in ls -l output is to parse it. For example, csv.DictReader could be use to map each column to a field in a dictionary:

import subprocess
import csv

process = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)
stdout, stderr = process.communicate()

reader = csv.DictReader(stdout.decode('ascii').splitlines(),
                        delimiter=' ', skipinitialspace=True,
                        fieldnames=['permissions', 'links',
                                    'owner', 'group', 'size',
                                    'date', 'time', 'name'])

for row in reader:
    print(row)

The code above will print a dictionary for each line in ls -l output such as:

{'group': '<group_name>',
 'name': '<filename>',
 'links': '1',
 'date': '<modified_date>',
 'time': '<modified_time>',
 'owner': '<user_name>',
 'permissions': '-rw-rw-r--',
 'size': '<size>'}

Heliotropin

I get: ` File "ip4.py", line 16 print row ^ SyntaxError: invalid syntax` – Observe

@user969617 Yes, sorry, that was for python 2.x, not for python 3 as you asked. I've updated the code to work for python 3. – Heliotropin

this looks promising!!! My next question (please bare with me, it's my second day of python) is how do I then access info from each dictionary. Do I need to give them unique names? Say I want to cross reference the date with a spreadsheet. – Observe

@user969617 There's no need to give a name to the dictionaries. You can store them in a list and access the list by index and the dictionary by key: rows[0]['name'] – Heliotropin

Thanks man your code works perfectly well for parsing hadoop fs -ls as well!!!!!!!!! – Storfer

Answer 2

With >= python3.5 you can use subprocess.run:

ls_lines = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE).stdout.splitlines()

With >= python2.7 or >= python3.0 you can use subprocess.check_output:

ls_lines = subprocess.check_output(['ls', '-l']).splitlines()

Prior to python2.7, you need to use the lower level api, which is a bit more involved.

ls_proc = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)
ls_proc.wait()
# check return code
ls_lines = ls_proc.stdout.readlines()

Dithyramb

subprocess.PIPE* – Finical

Answer 3

If what you really want is to list a directory, rather use os.listdir

import os
files = os.listdir('/path/to/dir')
for file in files:
    print(file)

Dithyramb

i actually want to list the airport tables, but thought that I would use a simple example. The actual output is going to be call(['/System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport', '-s']) – Observe

Answer 4

Read about Popen. the set you asked for you get with

import subprocess
proc = subprocess.Popen(['ls','-l'],stdout=subprocess.PIPE,stderr=subprocess.PIPE)
myset=set(proc.stdout)

or do something like

for x in proc.stdout : print x

and the same for stderr

you can examine the state of the process with

proc.poll() 

or wait for it to terminate with

proc.wait()

also read

read subprocess stdout line by line

Pustule

Answer 5

from subprocess import Popen, PIPE
output = Popen(['ls', '-l'], stdout=PIPE).communicate()[0]

You can then do whatever you want with the output. See python docs for detailed documentation

Scevor

Thanks. I had tried this one as well, but was confused by the lack of formatting... Im used to doing things with bash & awk, where printing output and then using awk '{print $n;}' lets you get to columns and awk NR=$variable lets you select the lines. Which section should I look at? – Observe

@Gary points out a nice way to do it for Python2.7+. Use that if you can :) – Scevor

Answer 6

People have already responded to it. Nevertheless, just in case somebody needs it

import subprocess
output_list=subprocess.check_output(['locate','*.nse']).decode('utf-8').split('\n')[:-1]

Gospodin