call_command argument is required

Question

I'm trying to use Django's call_command in a manner very similar to this question without an answer.

The way I'm calling it is:

    args = []
    kwargs = {
        'solr_url': 'http://127.0.0.1:8983/solr/collection1',
        'type': 'opinions',
        'update': True,
        'everything': True,
        'do_commit': True,
        'traceback': True,
    }
    call_command('cl_update_index', **kwargs)

In theory, that should work, according to the docs. But it doesn't work, it just doesn't.

Here's the add_arguments method for my Command class:

def add_arguments(self, parser):
    parser.add_argument(
        '--type',
        type=valid_obj_type,
        required=True,
        help='Because the Solr indexes are loosely bound to the database, '
             'commands require that the correct model is provided in this '
             'argument. Current choices are "audio" or "opinions".'
    )
    parser.add_argument(
        '--solr-url',
        required=True,
        type=str,
        help='When swapping cores, it can be valuable to use a temporary '
             'Solr URL, overriding the default value that\'s in the '
             'settings, e.g., http://127.0.0.1:8983/solr/swap_core'
    )

    actions_group = parser.add_mutually_exclusive_group()
    actions_group.add_argument(
        '--update',
        action='store_true',
        default=False,
        help='Run the command in update mode. Use this to add or update '
             'items.'
    )
    actions_group.add_argument(
        '--delete',
        action='store_true',
        default=False,
        help='Run the command in delete mode. Use this to remove  items '
             'from the index. Note that this will not delete items from '
             'the index that do not continue to exist in the database.'
    )
    parser.add_argument(
        '--optimize',
        action='store_true',
        default=False,
        help='Run the optimize command against the current index after '
             'any updates or deletions are completed.'
    )
    parser.add_argument(
        '--do-commit',
        action='store_true',
        default=False,
        help='Performs a simple commit and nothing more.'
    )

    act_upon_group = parser.add_mutually_exclusive_group()
    act_upon_group.add_argument(
        '--everything',
        action='store_true',
        default=False,
        help='Take action on everything in the database',
    )
    act_upon_group.add_argument(
        '--query',
        help='Take action on items fulfilling a query. Queries should be '
             'formatted as Python dicts such as: "{\'court_id\':\'haw\'}"'
    )
    act_upon_group.add_argument(
        '--items',
        type=int,
        nargs='*',
        help='Take action on a list of items using a single '
             'Celery task'
    )
    act_upon_group.add_argument(
        '--datetime',
        type=valid_date_time,
        help='Take action on items newer than a date (YYYY-MM-DD) or a '
             'date and time (YYYY-MM-DD HH:MM:SS)'
    )

No matter what I do here, I get:

CommandError: Error: argument --type is required

Any ideas? If you're truly curious, you can see the entire code here.

Buhr

action=store_true and default=False contradict each other. store_true is an alias for store_const with a const=True – Scrambler

Answer 1

You defined an argument with a '--type' flag, and made it required. That command line will require a string or strings that look like --type avalue.

This looks like the relevant part of call_command:

def call_command(name, *args, **options):
    ....
    parser = command.create_parser('', name)
    if command.use_argparse:
        # Use the `dest` option name from the parser option
        opt_mapping = {sorted(s_opt.option_strings)[0].lstrip('-').replace('-', '_'): s_opt.dest
                       for s_opt in parser._actions if s_opt.option_strings}
        arg_options = {opt_mapping.get(key, key): value for key, value in options.items()}
        defaults = parser.parse_args(args=args)
        defaults = dict(defaults._get_kwargs(), **arg_options)
        # Move positional args out of options to mimic legacy optparse
        args = defaults.pop('args', ())

It creates a parser, using it's own arguments plus the ones you add.

parser._actions if s_opt.option_strings are the arguments (Actions) that take an option flag (start with - or --). opt_mapping is map between the flag strings (minus the leading -s) and the 'dest' attribute.

arg_options converts your **kwargs to something that can be merged with the parser output.

defaults = parser.parse_args(args=args) does the actual parsing. That is, it's the only code that actually uses the argparse parsing mechanism. So the *args part of your call simulates generating sys.argv[1:] from an interactive call.

Based on that reading I think this should work:

args = [
    '--solr-url', 'http://127.0.0.1:8983/solr/collection1',
    '--type', 'opinions',
    '--update'
    '--everything',
    '--do_commit',
    '--traceback',
}
call_command('cl_update_index', *args)

Instead of **kwargs I am passing in values as a list of strings. Or the two required arguments could be passed in args, and the rest in **kwargs.

args = ['--solr-url', 'http://127.0.0.1:8983/solr/collection1',
    '--type', 'opinions']
kwargs = {
    'update': True,
    'everything': True,
    'do_commit': True,
    'traceback': True,
}
call_command('cl_update_index', *args, **kwargs)

If an argument is required it needs to passed in through *args. **kwargs bypass the parser, causing it to object about missing arguments.

---

I've downloaded the latest django, but haven't installed it. But here's a simulation of call_command that should test the calling options:

import argparse

def call_command(name, *args, **options):
    """
    Calls the given command, with the given options and args/kwargs.
    standalone simulation of django.core.mangement call_command
    """
    command = name
    """
    ....
    """
    # Simulate argument parsing to get the option defaults (see #10080 for details).
    parser = command.create_parser('', name)
    if command.use_argparse:
        # Use the `dest` option name from the parser option
        opt_mapping = {sorted(s_opt.option_strings)[0].lstrip('-').replace('-', '_'): s_opt.dest
                       for s_opt in parser._actions if s_opt.option_strings}
        arg_options = {opt_mapping.get(key, key): value for key, value in options.items()}
        defaults = parser.parse_args(args=args)
        defaults = dict(defaults._get_kwargs(), **arg_options)
        # Move positional args out of options to mimic legacy optparse
        args = defaults.pop('args', ())
    else:
        # Legacy optparse method
        defaults, _ = parser.parse_args(args=[])
        defaults = dict(defaults.__dict__, **options)
    if 'skip_checks' not in options:
        defaults['skip_checks'] = True

    return command.execute(*args, **defaults)

class BaseCommand():
    def __init__(self):
        self.use_argparse = True
        self.stdout= sys.stdout
        self.stderr=sys.stderr
    def execute(self, *args, **kwargs):
        self.handle(*args, **kwargs)
    def handle(self, *args, **kwargs):
        print('args: ', args)
        print('kwargs: ', kwargs)
    def create_parser(self, *args, **kwargs):
        parser = argparse.ArgumentParser()
        self.add_arguments(parser)
        return parser
    def add_arguments(self, parser):
        parser.add_argument('--type', required=True)
        parser.add_argument('--update', action='store_true')
        parser.add_argument('--optional', default='default')
        parser.add_argument('foo')
        parser.add_argument('args', nargs='*')

if __name__=='__main__':

    testcmd = BaseCommand()
    # testcmd.execute('one','tow', three='four')

    call_command(testcmd, '--type','typevalue','foovalue', 'argsvalue', update=True)

    args = ['--type=argvalue', 'foovalue', '1', '2']
    kwargs = {
        'solr_url': 'http://127.0.0.1...',
        'type': 'opinions',
        'update': True,
        'everything': True,
    }
    call_command(testcmd, *args, **kwargs)

which produces:

python3 stack32036562.py 
args:  ('argsvalue',)
kwargs:  {'optional': 'default', 'type': 'typevalue', 'update': True, 'skip_checks': True, 'foo': 'foovalue'}
args:  ('1', '2')
kwargs:  {'optional': 'default', 'update': True, 'foo': 'foovalue', 'type': 'opinions', 'skip_checks': True, 'everything': True, 'solr_url': 'http://127.0.0.1...'}

With a bunch of stubs, I can make your cl Command work with my BaseCommand, and the following call works:

clupdate = Command()
args = ['--type','opinions','--solr-url','dummy']
kwargs = {
    'solr_url': 'http://127.0.0.1:8983/solr/collection1',
    #'type': 'opinions',
    'update': True,
    'everything': True,
    'do_commit': True,
    'traceback': True,
}
call_command(clupdate, *args, **kwargs)

performing a stub everything.

Running in update mode...
everything
args:  ()
options:  {'type': 'opinions', 'query': None, 'solr_url': 'http://127.0.0.1:8983/solr/collection1', 'items': None, 'do_commit': True, 'update': True, 'delete': False, 'datetime': None, 'optimize': False, 'skip_checks': True, 'everything': True, 'traceback': True}

Chiaroscuro

Based on reading the django code, I think required arguments need to be given via *args, not **kwargs. – Chiaroscuro

Great answer, though I still wasn't able to pass anything as a dict. Even the stuff you said should work? Didn't work. Only way this works for me is to pass everything as a list. – Buhr

I haven't installed the latest django, but I tried a simulation of call_command. As expected the kwargs pass on through to the command with mininal change. – Chiaroscuro

Interesting. I'm on Django 1.8.3. Not sure where the difference lies, but I'm sufficiently happy just to use args and call it good enough! – Buhr