How can I split my Click commands, each with a set of sub-commands, into multiple files?

Question

I have one large click application that I've developed, but navigating through the different commands/subcommands is getting rough. How do I organize my commands into separate files? Is it possible to organize commands and their subcommands into separate classes?

Here's an example of how I would like to separate it:

init

import click

@click.group()
@click.version_option()
def cli():
    pass #Entry Point

command_cloudflare.py

@cli.group()
@click.pass_context
def cloudflare(ctx):
    pass

@cloudflare.group('zone')
def cloudflare_zone():
    pass

@cloudflare_zone.command('add')
@click.option('--jumpstart', '-j', default=True)
@click.option('--organization', '-o', default='')
@click.argument('url')
@click.pass_obj
@__cf_error_handler
def cloudflare_zone_add(ctx, url, jumpstart, organization):
    pass

@cloudflare.group('record')
def cloudflare_record():
    pass

@cloudflare_record.command('add')
@click.option('--ttl', '-t')
@click.argument('domain')
@click.argument('name')
@click.argument('type')
@click.argument('content')
@click.pass_obj
@__cf_error_handler
def cloudflare_record_add(ctx, domain, name, type, content, ttl):
    pass

@cloudflare_record.command('edit')
@click.option('--ttl', '-t')
@click.argument('domain')
@click.argument('name')
@click.argument('type')
@click.argument('content')
@click.pass_obj
@__cf_error_handler
def cloudflare_record_edit(ctx, domain):
    pass

command_uptimerobot.py

@cli.group()
@click.pass_context
def uptimerobot(ctx):
    pass

@uptimerobot.command('add')
@click.option('--alert', '-a', default=True)
@click.argument('name')
@click.argument('url')
@click.pass_obj
def uptimerobot_add(ctx, name, url, alert):
    pass

@uptimerobot.command('delete')
@click.argument('names', nargs=-1, required=True)
@click.pass_obj
def uptimerobot_delete(ctx, names):
    pass

Leclair

Answer 1

The downside of using CommandCollection for this is that it merges your commands and works only with command groups. The imho better alternative is to use add_command to achieve the same result.

I have a project with the following tree:

cli/
├── __init__.py
├── cli.py
├── group1
│   ├── __init__.py
│   ├── commands.py
└── group2
    ├── __init__.py
    └── commands.py

Each subcommand has its own module, what makes it incredibly easy to manage even complex implementations with many more helper classes and files. In each module, the commands.py file contains the @click annotations. Example group2/commands.py:

import click


@click.command()
def version():
    """Display the current version."""
    click.echo(_read_version())

If necessary, you could easily create more classes in the module, and import and use them here, thus giving your CLI the full power of Python's classes and modules.

My cli.py is the entry point for the whole CLI:

import click

from .group1 import commands as group1
from .group2 import commands as group2

@click.group()
def entry_point():
    pass

entry_point.add_command(group1.command_group)
entry_point.add_command(group2.version)

With this setup, it is very easy to separate your commands by concerns, and also build additional functionality around them that they might need. It has served me very well so far...

Reference: http://click.pocoo.org/6/quickstart/#nesting-commands

Godman

how to pass context to subcommand if they are in separate modules? – Otte

@vishal, have a look at this section of the documentation: click.pocoo.org/6/commands/#nested-handling-and-contexts You can pass the context object to any command using the decorator @click.pass_context. Alternatively, there is also something called Global Context Access: click.pocoo.org/6/advanced/#global-context-access. – Godman

I compiled a MWE using @Godman guidelines. You can find it here – Senaidasenalda

How can I flat all group command ? I mean, all commands in first level . – Buschi

@Buschi Use a CommandCollection. Oscar's answer has an example, and there is a really nice one in click's documentation: click.palletsprojects.com/en/7.x/commands/…. – Godman

from inspect import getmembers; for (mname, mvalue) in getmembers(group1): if isinstance(mvalue, click.core.Command): entry_point.add_command(mvalue) forgive formatting – Vaughn

Answer 2

Suppose your project have the following structure:

project/
├── __init__.py
├── init.py
└── commands
    ├── __init__.py
    └── cloudflare.py

Groups are nothing more than multiple commands and groups can be nested. You can separate your groups into modules and import them on you init.py file and add them to the cli group using the add_command.

Here is a init.py example:

import click
from .commands.cloudflare import cloudflare


@click.group()
def cli():
    pass


cli.add_command(cloudflare)

You have to import the cloudflare group which lives inside the cloudflare.py file. Your commands/cloudflare.py would look like this:

import click


@click.group()
def cloudflare():
    pass


@cloudflare.command()
def zone():
    click.echo('This is the zone subcommand of the cloudflare command')

Then you can run the cloudflare command like this:

$ python init.py cloudflare zone

This information is not very explicit on the documentation but if you look at the source code, which is very well commented, you can see how groups can be nested.

Yam

Agree. So minimal that it should be part of the documentation. Exactly what I was looking for to build complex tools! Thanks 🙏! – Salbu

It sure is great but got a question: Considering your example, should I remove @cloudflare.command() from zone function if I import zone from somewhere else? – Doering

This is an excellent information I was looking for. Another good example on how to distinguish between command groups can be found here: github.com/dagster-io/dagster/tree/master/python_modules/… – Capelin

Answer 3

It took me a while to figure this out but I figured I'd put this here to remind myself when I forget how to do i again I think part of the problem is that the add_command function is mentioned on click's github page but not the main examples page

first lets create an initial python file called root.py

import click
from cli_compile import cli_compile
from cli_tools import cli_tools

@click.group()
def main():
    """Demo"""

if __name__ == '__main__':
    main.add_command(cli_tools)
    main.add_command(cli_compile)
    main()

Next lets put some tools commands in a file called cli_tools.py

import click

# Command Group
@click.group(name='tools')
def cli_tools():
    """Tool related commands"""
    pass

@cli_tools.command(name='install', help='test install')
@click.option('--test1', default='1', help='test option')
def install_cmd(test1):
    click.echo('Hello world')

@cli_tools.command(name='search', help='test search')
@click.option('--test1', default='1', help='test option')
def search_cmd(test1):
    click.echo('Hello world')

if __name__ == '__main__':
    cli_tools()

Next lets put some compile commands in a file called cli_compile.py

import click

@click.group(name='compile')
def cli_compile():
    """Commands related to compiling"""
    pass

@cli_compile.command(name='install2', help='test install')
def install2_cmd():
    click.echo('Hello world')

@cli_compile.command(name='search2', help='test search')
def search2_cmd():
    click.echo('Hello world')

if __name__ == '__main__':
    cli_compile()

running root.py should now give us

Usage: root.py [OPTIONS] COMMAND [ARGS]...

  Demo

Options:
  --help  Show this message and exit.

Commands:
  compile  Commands related to compiling
  tools    Tool related commands

running "root.py compile" should give us

Usage: root.py compile [OPTIONS] COMMAND [ARGS]...

  Commands related to compiling

Options:
  --help  Show this message and exit.

Commands:
  install2  test install
  search2   test search

You'll also notice you can run the cli_tools.py or cli_compile.py directly as well as I included a main statement in there

Chilon

does this work if your function are split into different modules? – Predestination

I've had options split over different modules, the idea being you can have a top level menu in one module, then more sub options underneath in other modules. – Chilon

Answer 4

I'm looking for something like this at the moment, in your case is simple because you have groups in each of the files, you can solve this problem as explained in the documentation:

In the init.py file:

import click

from command_cloudflare import cloudflare
from command_uptimerobot import uptimerobot

cli = click.CommandCollection(sources=[cloudflare, uptimerobot])

if __name__ == '__main__':
    cli()

The best part of this solution is that is totally compliant with pep8 and other linters because you don't need to import something you wouldn't use and you don't need to import * from anywhere.

Niece

Can you please tell, what to put into the sub-command files? I have to import main cli from init.py, but this leads to circular imports. Could you please explain how to do it? – Jacquijacquie

@Jacquijacquie Check out my answer if you haven't figured out a solution yet. It might put you on the right track. – Godman

@Jacquijacquie I hope you figured already, but in your sub command files you just create a new click.group that's the one you import in the top level CLI. – Unstrained

Answer 5

edit: just realized that my answer/comment is little more than a rehash of what Click's official docs offer in the "Custom Multi Commands" section: https://click.palletsprojects.com/en/7.x/commands/#custom-multi-commands

Just to add to the excellent, accepted answer by @jdno, I came up with a helper function that auto-imports and auto-adds subcommand modules, which vastly cut down on the boilerplate in my cli.py:

My project structure is this:

projectroot/
    __init__.py
    console/
    │
    ├── cli.py
    └── subcommands
       ├── bar.py
       ├── foo.py
       └── hello.py

Each subcommand file looks something like this:

import click

@click.command()
def foo():
    """foo this is for foos!"""
    click.secho("FOO", fg="red", bg="white")

(for now, I just have one subcommand per file)

In cli.py, I've written a add_subcommand() function that loops through every filepath globbed by "subcommands/*.py" and then does the import and add command.

Here's what the body of the cli.py script is simplified to:

import click
import importlib
from pathlib import Path
import re

@click.group()
def entry_point():
    """whats up, this is the main function"""
    pass

def main():
    add_subcommands()
    entry_point()

if __name__ == '__main__':
    main()

And this is what the add_subcommands() function looks like:

SUBCOMMAND_DIR = Path("projectroot/console/subcommands")

def add_subcommands(maincommand=entry_point):
    for modpath in SUBCOMMAND_DIR.glob('*.py'):
        modname = re.sub(f'/', '.',  str(modpath)).rpartition('.py')[0]
        mod = importlib.import_module(modname)
        # filter out any things that aren't a click Command
        for attr in dir(mod):
            foo = getattr(mod, attr)
            if callable(foo) and type(foo) is click.core.Command:
                maincommand.add_command(foo)

I don't know how robust this is if I were to design a command that had several levels of nesting and context switching. But it seems to work all right for now :)

Parkerparkhurst

Answer 6

I'm not an click expert, but it should work by just importing your files into the main one. I would move all commands in separate files and have one main file importing the other ones. That way it is easier to control the exact order, in case it is important for you. So your main file would just look like:

import commands_main
import commands_cloudflare
import commands_uptimerobot

Mcelrath

Answer 7

When you want your user to pip install "your_module", and then use commands, you can add them in setup.py entry_points as a list:

entry_points={
    'console_scripts': [
        'command_1 = src.cli:function_command_1',
        'command_2 = src.cli:function_command_2',
    ]

each command is bounded to function in a cli file.

Thermoluminescent