Python multi-command CLI with common options

Question

I am adding CLI for my Python application. The CLI should allow to run multiple commands in a time. The commands should have common options and personal options.

Example:

$ python mycliapp.py --common-option1 value1 --common-option2 value2 cmd1 --cmd1-option cmd2 --cmd2-option somevalue cmd3

The example has two common options used by all commands and each command can have or not the option used by the command only.

I have considered Python Click. It has rich functionality, but it does not allow (at least I didn't found) to use common options without some main command.

The above example will look as follows with Click:

$ python mycliapp.py maincmd --common-option1 value1 --common-option2 value2 cmd1 --cmd1-option cmd2 --cmd2-option somevalue cmd3

Also, considered Python Argparse. It looks that it can do what I need and I have managed to write a code, which works with common options and single command, but cannot manage to use multiple commands. This page Python argparse - Add argument to multiple subparsers has good example, but seems that command2 should be a sub-command of command1. It is a bit different since I need that the commands can be executed in any order.

Ansley

Answer 1

Click absolutely supports this sort of syntax. A simple example looks something like:

import click


@click.group(chain=True)
@click.option('--common-option1')
@click.option('--common-option2')
def main(common_option1, common_option2):
    pass


@main.command()
@click.option('--cmd1-option', is_flag=True)
def cmd1(cmd1_option):
    pass


@main.command()
@click.option('--cmd2-option')
def cmd2(cmd2_option):
    pass


@main.command()
def cmd3():
    pass


if __name__ == '__main__':
    main()

Assuming the above is in mycliapp.py, we see the common help output:

$ python example.py --help
Usage: example.py [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options:
  --common-option1 TEXT
  --common-option2 TEXT
  --help                 Show this message and exit.

Commands:
  cmd1
  cmd2
  cmd3

And for cmd1:

$ python mycliapp.py cmd1 --help
Usage: mycliapp.py cmd1 [OPTIONS]

Options:
  --cmd1-option
  --help         Show this message and exit.

And for cmd2:

$ python mycliapp.py cmd2 --help
Usage: mycliapp.py cmd2 [OPTIONS]

Options:
  --cmd2-option TEXT
  --help              Show this message and exit.

Etc.

With this we can run the command line from your question:

python mycliapp.py --common-option1 value1 --common-option2 value2 \
  cmd1 --cmd1-option \
  cmd2 --cmd2-option somevalue \
  cmd3

---

Update 1

Here's an example that implements pipelines using the callback model suggested in the documentation:

import click


@click.group(chain=True)
@click.option('--common-option1')
@click.option('--common-option2')
@click.pass_context
def main(ctx, common_option1, common_option2):
    ctx.obj = {
        'common_option1': common_option1,
        'common_option2': common_option2,
    }


@main.resultcallback()
def process_pipeline(processors, common_option1, common_option2):
    print('common_option1 is', common_option1)
    for func in processors:
        res = func()
        if not res:
            raise click.ClickException('Failed processing!')


@main.command()
@click.option('--cmd1-option', is_flag=True)
def cmd1(cmd1_option):
    def process():
        print('This is cmd1')
        return cmd1_option

    return process


@main.command()
@click.option('--cmd2-option')
def cmd2(cmd2_option):
    def process():
        print('This is cmd2')
        return cmd2_option != 'fail'

    return process


@main.command()
@click.pass_context
def cmd3(ctx):
    def process():
        print('This is cmd3 (common option 1 is: {common_option1}'.format(**ctx.obj))
        return True

    return process


if __name__ == '__main__':
    main()

Each command returns a boolean indicating whether or not it was successful. A failed command will abort pipeline processing. For example, here cmd1 fails so cmd2 never executes:

$ python mycliapp.py cmd1 cmd2
This is cmd1
Error: Failed processing!

But if we make cmd1 happy, it works:

$ python mycliapp.py cmd1 --cmd1-option cmd2
This is cmd1
This is cmd2

And similarly, compare this:

$ python mycliapp.py cmd1 --cmd1-option cmd2 --cmd2-option fail cmd3
This is cmd1
This is cmd2
Error: Failed processing!

With this:

$ python mycliapp.py cmd1 --cmd1-option cmd2  cmd3
This is cmd1
This is cmd2
This is cmd3

And of course you don't need to call things in order:

$ python mycliapp.py cmd2 cmd1 --cmd1-option
This is cmd2
This is cmd1

Holography

Can this be used together with pipelines, so I knew the result of previous command (click.palletsprojects.com/en/7.x/commands/…)? I haven't managed to combine them as shown in this example github.com/pallets/click/blob/master/examples/imagepipe/…. – Ansley

You could absolutely apply this to structure to pipelines. That's pretty much what is shown in click.palletsprojects.com/en/7.x/commands/…. If you're having trouble with that, maybe open a new question showing your code and we can figure out what's going on. – Holography

The issue is that there must be Generator and Processor. Generator shares common options with Processor (as far as I understand). But in my case there is no Generator (no main command). – Ansley

But there is a main command in this example...it's the entrypoint for the program and it's where the common options are defined. If you went with the processor model, that is where you would attach the result callback. If you want to update your question to show the specific behavior you're trying to achieve, we can try to modify this example to match. – Holography

I want to avoid main command. In the example the main command has logical relation to the next commands. Commands in my application have no relations, but they should not execute if the previous one failed. I am trying to achieve the exactly same behavior you have described in this answer, with additional feature - check return value of previous command. – Ansley

I'm not sure why you're trying to avoid the main command. I've added an example here using the callback model suggested in the documentation. – Holography

Thank you very much! This is what I need! The last question - how do I get common_option1 and common_option2 values inside cmd1, cmd2 and cmd3? – Ansley

Use the pass_context decorator and store your common options on ctx.obj. See the updated code for main and cmd3 that demonstrates this. – Holography

Cool! I was even able to create an object based on common option and pass it through the ctx.obj to the commands. I am wondering how the ctx is stored, because I thought that Python runs new process for each click command. Isn't it? – Ansley

Answer 2

You can do it without main command using argparse.

# maincmd just to tie between arguments and subparsers 
parser = argparse.ArgumentParser(prog='maincmd')
parser.add_argument('--common-option1', type=str, required=False)
parser.add_argument('--common-option2', type=str, required=False)

main_subparsers = parser.add_subparsers(title='sub_main',  dest='sub_cmd')
parser_cmd1 = main_subparsers.add_parser('cmd1', help='help cmd1')
parser_cmd1.add_argument('--cmd1-option', type=str, required=False)

cmd1_subparsers = parser_cmd1.add_subparsers(title='sub_cmd1', dest='sub_cmd1')
parser_cmd2 = cmd1_subparsers.add_parser('cmd2', help='help cmd2')

options = parser.parse_args(sys.argv[1:])
print(vars(options))

Let's check:

python test.py --common-option1 value1 --common-option2 value2
#{'common_option1': 'value1', 'common_option2': 'value2', 'sub_cmd': None}

python test.py --common-option1 value1 --common-option2 value2 cmd1
# {'common_option1': 'value1', 'common_option2': 'value2', 'sub_cmd': 'cmd1', 'cmd1_option': None, 'sub_cmd1': None}

python test.py --common-option1 value1 --common-option2 value2 cmd1 --cmd1-option cmd1-val
# {'common_option1': 'value1', 'common_option2': 'value2', 'sub_cmd': 'cmd1', 'cmd1_option': 'cmd1-val', 'sub_cmd1': None}

python test.py --common-option1 value1 --common-option2 value2 cmd1 --cmd1-option cmd1-val cmd2
# {'common_option1': 'value1', 'common_option2': 'value2', 'sub_cmd': 'cmd1', 'cmd1_option': 'cmd1-val', 'sub_cmd1': 'cmd2'}

JFYI. I worked with Click and argparse. argparse seemed to me more extensible and functional.

Hope this helps.

Sudduth

Thank you! This works when cmd2 is placed next to cmd1. When the order of the commands is different (cmd2 before cmd1), it says maincmd: error: argument sub_cmd: invalid choice: 'cmd2' (choose from 'cmd1') – Ansley

@Ansley thank you. you are right. this is because cmd1 is subparser of cmd2. but you can also add cmd2 to main parser main_subparsers.add_parser('cmd2', help='help cmd2'). In this case you can call cmd2 after cmd1 and without cmd1. – Sudduth