Can OptionParser skip unknown options, to be processed later in a Ruby program?

Question

Is there any way to kick off OptionParser several times in one Ruby program, each with different sets of options?

For example:

$ myscript.rb --subsys1opt a --subsys2opt b

Here, myscript.rb would use subsys1 and subsys2, delegating their options handling logic to them, possibly in a sequence where 'a' is processed first, followed by 'b' in separate OptionParser object; each time picking options only relevant for that context. A final phase could check that there is nothing unknown left after each part processed theirs.

The use cases are:

1. In a loosely coupled front-end program, where various components have different arguments, I don't want 'main' to know about everything, just to delegate sets of arguments/options to each part.

2. Embedding some larger system like RSpec into my application, and I'd to simply pass a command-line through their options without my wrapper knowing those.

I'd be OK with some delimiter option as well, like -- or --vmargs in some Java apps.

There are lots of real world examples for similar things in the Unix world (startx/X, git plumbing and porcelain), where one layer handles some options but propagates the rest to the lower layer.

Out of the box, this doesn't seem to work. Each OptionParse.parse! call will do exhaustive processing, failing on anything it doesn't know about. I guess I'd happy to skip unknown options.

Any hints, perhaps alternative approaches are welcome.

Barbiturate

In your example above, myscript.rb will receive all options as ARGV. If I understand you, you say that some of those options need to be passed to "sub layers". Does myscript.rb call those sub layers? If yes, your question becomes simply how to retrieve some elements from the ARGV array, passing the rest to another program. If myscript.rb doesn't call the sub layers, what does? – Kenspeckle

Yes, myscript.rb uses those sublayers (updated the description to make that clearer). So your rephrased question is almost correct "how to retrieve some elements from the ARGV array, passing the rest to another program", except it's not necessary another program (that's why I used the more generic subsystem/component term), and I specifically asked about 'optparse'. Hence "Can optparse skip unknown options, to be processed later in a ruby program?" – Barbiturate

Answer 1

Assuming the order in which the parsers will run is well defined, you can just store the extra options in a temporary global variable and run OptionParser#parse! on each set of options.

The easiest way to do this is to use a delimiter like you alluded to. Suppose the second set of arguments is separated from the first by the delimiter --. Then this will do what you want:

opts = OptionParser.new do |opts|
  # set up one OptionParser here
end

both_args = $*.join(" ").split(" -- ")
$extra_args = both_args[1].split(/\s+/)
opts.parse!(both_args[0].split(/\s+/))

Then, in the second code/context, you could do:

other_opts = OptionParser.new do |opts|
  # set up the other OptionParser here
end

other_opts.parse!($extra_args)

Alternatively, and this is probably the "more proper" way to do this, you could simply use OptionParser#parse, without the exclamation point, which won't remove the command-line switches from the $* array, and make sure that there aren't options defined the same in both sets. I would advise against modifying the $* array by hand, since it makes your code harder to understand if you are only looking at the second part, but you could do that. You would have to ignore invalid options in this case:

begin
    opts.parse
rescue OptionParser::InvalidOption
    puts "Warning: Invalid option"
end

---

The second method doesn't actually work, as was pointed out in a comment. However, if you have to modify the $* array anyway, you can do this instead:

tmp = Array.new

while($*.size > 0)
    begin
        opts.parse!
    rescue OptionParser::InvalidOption => e
        tmp.push(e.to_s.sub(/invalid option:\s+/,''))
    end
end

tmp.each { |a| $*.push(a) }

It's more than a little bit hack-y, but it should do what you want.

Sextillion

Yes, the delimiter approach seems a viable way, I just hoped it to be nicer than tweaking arrays, with join and split; perhaps directly supported by OptionParser. The problem with your alternative solution is, when that exception is raised, the whole processing is aborted, so subsequent good arguments would be skipped too. Check this: ruby -roptparse -e 'begin OptionParser.new {|o|o.on("--ok"){puts "OK"}}.parse *ARGV;rescue OptionParser::InvalidOption;warn "BAD";end' -- --bad --ok # this says BAD although should say OK as well. – Barbiturate

Also, one of my usecases above is to wrap existing frameworks/systems like RSpec with minimal effort, something like calling Spec::Runner.run_examples, which does optparsing internally. So, unfortunately this means I do have rewrite ARGV (even though it's constant and agree with you to avoid it if possible) – Barbiturate

Is seems noone came up with a nicer solution - so maybe a hack is required indeed:( Anyhow, accepting this answer for now :). Thank you – Barbiturate

Answer 2

I needed a solution that wouldn't throw OptionParser::InvalidOption ever, and couldn't find an elegant solution among the current answers. This monkey patch is based on one of the other answers but cleans it up and makes it work more like the current order! semantics. But see below for an unsolved issue inherent to multiple-pass option parsing.

class OptionParser
  # Like order!, but leave any unrecognized --switches alone
  def order_recognized!(args)
    extra_opts = []
    begin
      order!(args) { |a| extra_opts << a }
    rescue OptionParser::InvalidOption => e
      extra_opts << e.args[0]
      retry
    end
    args[0, 0] = extra_opts
  end
end

Works just like order! except instead of throwing InvalidOption, it leaves the unrecognized switch in ARGV.

RSpec tests:

describe OptionParser do
  before(:each) do
    @parser = OptionParser.new do |opts|
      opts.on('--foo=BAR', OptionParser::DecimalInteger) { |f| @found << f }
    end
    @found = []
  end

  describe 'order_recognized!' do
    it 'finds good switches using equals (--foo=3)' do
      argv = %w(one two --foo=3 three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([3])
      expect(argv).to eq(%w(one two three))
    end

    it 'leaves unknown switches alone' do
      argv = %w(one --bar=2 two three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([])
      expect(argv).to eq(%w(one --bar=2 two three))
    end

    it 'leaves unknown single-dash switches alone' do
      argv = %w(one -bar=2 two three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([])
      expect(argv).to eq(%w(one -bar=2 two three))
    end

    it 'finds good switches using space (--foo 3)' do
      argv = %w(one --bar=2 two --foo 3 three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([3])
      expect(argv).to eq(%w(one --bar=2 two three))
    end

    it 'finds repeated args' do
      argv = %w(one --foo=1 two --foo=3 three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([1, 3])
      expect(argv).to eq(%w(one two three))
    end

    it 'maintains repeated non-switches' do
      argv = %w(one --foo=1 one --foo=3 three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([1, 3])
      expect(argv).to eq(%w(one one three))
    end

    it 'maintains repeated unrecognized switches' do
      argv = %w(one --bar=1 one --bar=3 three)
      @parser.order_recognized!(argv)
      expect(@found).to eq([])
      expect(argv).to eq(%w(one --bar=1 one --bar=3 three))
    end

    it 'still raises InvalidArgument' do
      argv = %w(one --foo=bar)
      expect { @parser.order_recognized!(argv) }.to raise_error(OptionParser::InvalidArgument)
    end

    it 'still raises MissingArgument' do
      argv = %w(one --foo)
      expect { @parser.order_recognized!(argv) }.to raise_error(OptionParser::MissingArgument)
    end
  end
end

Problem: normally OptionParser allows abbreviated options, provided there are enough characters to uniquely identify the intended option. Parsing options in multiple stages breaks this, because OptionParser doesn't see all the possible arguments in the first pass. For example:

describe OptionParser do
  context 'one parser with similar prefixed options' do
    before(:each) do
      @parser1 = OptionParser.new do |opts|
        opts.on('--foobar=BAR', OptionParser::DecimalInteger) { |f| @found_foobar << f }
        opts.on('--foo=BAR', OptionParser::DecimalInteger) { |f| @found_foo << f }
      end
      @found_foobar = []
      @found_foo = []
    end

    it 'distinguishes similar prefixed switches' do
      argv = %w(--foo=3 --foobar=4)
      @parser1.order_recognized!(argv)
      expect(@found_foobar).to eq([4])
      expect(@found_foo).to eq([3])
    end
  end

  context 'two parsers in separate passes' do
    before(:each) do
      @parser1 = OptionParser.new do |opts|
        opts.on('--foobar=BAR', OptionParser::DecimalInteger) { |f| @found_foobar << f }
      end
      @parser2 = OptionParser.new do |opts|
        opts.on('--foo=BAR', OptionParser::DecimalInteger) { |f| @found_foo << f }
      end
      @found_foobar = []
      @found_foo = []
    end

    it 'confuses similar prefixed switches' do
      # This is not generally desirable behavior
      argv = %w(--foo=3 --foobar=4)
      @parser1.order_recognized!(argv)
      @parser2.order_recognized!(argv)
      expect(@found_foobar).to eq([3, 4])
      expect(@found_foo).to eq([])
    end
  end
end

Fiber

Nice work, @ScottJ! This works well for me. I started down this path myself, and found myself trying to overwrite order with a safe_order as well as order! with safe_order! etc. It got kinda out of hand, but yours works well! – Hedge

WOW, I can't believe that OptionParser can't handle this natively! Just spent almost an hour searching for how to do it before finding this. Great solution; please please consider submitting this to Ruby itself to be included in OptionParser! – Yardmaster

Answer 3

I've got the same problem, and I found the following solution:

options = ARGV.dup
remaining = []
while !options.empty?
  begin
    head = options.shift
    remaining.concat(parser.parse([head]))
  rescue OptionParser::InvalidOption
    remaining << head
    retry
  end
end

Tradesman

another nice hack, thanks :) how does this handle parameterised options? you seem to be looking at one argument at a time, but it should be OptParser's knowledge to know which ones need how many parameters? – Barbiturate

That is definitely a good point :) Won't work with options that have arguments unless the user uses the --option=value syntax. – Tradesman

Answer 4

For posterity, you can do this with the order! method:

option_parser.order!(args) do |unrecognized_option|
  args.unshift(unrecognized_option)
end

At this point, args has been modified - all known options were consumed and handled by option_parser - and can be passed to a different option parser:

some_other_option_parser.order!(args) do |unrecognized_option|
  args.unshift(unrecognized_option)
end

Obviously, this solution is order-dependent, but what you are trying to do is somewhat complex and unusual.

One thing that might be a good compromise is to just use -- on the command line to stop processing. Doing that would leave args with whatever followed --, be that more options or just regular arguments.

Danish

Answer 5

I also needed the same... it took me a while but a relatively simple way has worked fine in the end.

options = {
  :input_file => 'input.txt', # default input file
}

opts = OptionParser.new do |opt|
  opt.on('-i', '--input FILE', String,
         'Input file name',
         'Default is %s' % options[:input_file] ) do |input_file|
    options[:input_file] = input_file
  end

  opt.on_tail('-h', '--help', 'Show this message') do
    puts opt
    exit
  end
end

extra_opts = Array.new
orig_args = ARGV.dup

begin
  opts.parse!(ARGV)
rescue OptionParser::InvalidOption => e
  extra_opts << e.args
  retry
end

args = orig_args & ( ARGV | extra_opts.flatten )

"args" will contain all command line arguments without the ones already parsed into the "options" hash. I'm passing this "args" to an external program to be called from this ruby script.

Parallelize

Clever hack, thanks; nice that it works with interface rather than internals! Some observations: 1) Why do you do orig_args & ... at the end? The whole argument space is a subset of orig_args so it does not seem to be necessary. 2) if you pass some_precious_arg --foo to this program, some_precious_arg will be lost, as the parser consumes it before hitting a snag on --foo and you never get it back. We can do: begin; abkp = ARGV.dup; ... rescue ... => e; ARGV.prepend *abkp[0...-(ARGV.size+1)]"; ... retry; end. This restores the pre-parse ARGV with the offendinge.args[0] omitted. – Padgett

Answer 6

Another solution which relies on parse! having a side effect on the argument list even if an error is thrown.

Let's define a method which tries to scan some argument list using a user defined parser and calls itself recursively when an InvalidOption error is thrown, saving the invalid option for later with eventual parameters:

def parse_known_to(parser, initial_args=ARGV.dup)
    other_args = []                                         # this contains the unknown options
    rec_parse = Proc.new { |arg_list|                       # in_method defined proc 
        begin
            parser.parse! arg_list                          # try to parse the arg list
        rescue OptionParser::InvalidOption => e
            other_args += e.args                            # save the unknown arg
            while arg_list[0] && arg_list[0][0] != "-"      # certainly not perfect but
                other_args << arg_list.shift                # quick hack to save any parameters
            end
            rec_parse.call arg_list                         # call itself recursively
        end
    }
    rec_parse.call initial_args                             # start the rec call
    other_args                                              # return the invalid arguments
end

my_parser = OptionParser.new do
   ...
end

other_options = parse_known_to my_parser

Bailar

Answer 7

I ran into a similar problem when I was writing a script that wrapped a ruby gem, which needed its own options with arguments passed to it.

I came up with the following solution in which it supports options with arguments for the wrapped tool. It works by parsing it through the first optparser, and separates what it can't use into a seperate array (which can be re-parsed again with another optparse).

optparse = OptionParser.new do |opts|
    # OptionParser settings here
end

arguments = ARGV.dup
secondary_arguments = []

first_run = true
errors = false
while errors || first_run
  errors = false
  first_run = false
  begin
    optparse.order!(arguments) do |unrecognized_option|
      secondary_arguments.push(unrecognized_option)
    end
  rescue OptionParser::InvalidOption => e
    errors = true
    e.args.each { |arg| secondary_arguments.push(arg) }
    arguments.delete(e.args)
  end
end

primary_arguments = ARGV.dup
secondary_arguments.each do |cuke_arg|
  primary_arguments.delete(cuke_arg)
end

puts "Primary Args: #{primary_arguments}"
puts "Secondary Args: #{secondary_args}"

optparse.parse(primary_arguments)
# Can parse the second list here, if needed
# optparse_2.parse(secondary_args)

Probably not the greatest or most efficient way of doing it, but it worked for me.

Illsorted

Answer 8

I've just moved from Python. Python's ArgumentParser has great method parse_known_args(). But it still doesn't accept second argument, such as:

$ your-app -x 0 -x 1

First -x 0 is your app's argument. Second -x 1 can belong to the target app that you need to forward to. ArgumentParser will raise error in this case.

Now come back to Ruby, you can use #order. Fortunately it accepts unlimited duplicate arguments. For example you need -a and -b. Your target app needs another -a and a mandatory argument some (note that there is no prefix -/--). Normally #parse will ignore mandatory arguments. But with #order, you will get the rest -- great. Note that you have to pass your own app's arguments first, then the target app's arguments.

$ your-app -a 0 -b 1 -a 2 some

And the code should be:

require 'optparse'
require 'ostruct'

# Build default arguments
options = OpenStruct.new
options.a = -1
options.b = -1

# Now parse arguments
target_app_argv = OptionParser.new do |opts|
    # Handle your own arguments here
    # ...
end.order

puts ' > Options         = %s' % [options]
puts ' > Target app argv = %s' % [target_app_argv]

Tada :-)

Runofthemine

this still throws OptionParser::InvalidOption if it finds an unrecognized --foo flag – Fiber

Answer 9

My attempt:

def first_parse
  left = []
  begin
    @options.order!(ARGV) do |opt|
      left << opt
    end
  rescue OptionParser::InvalidOption => e
    e.recover(args)
    left << args.shift
    retry
  end
  left
end

In my case, I want to scan the options and pick up any predefined options that may set debugging levels, output files, etc. Then I am going to load custom processors which may add to the options. After all the custom processors have been loaded, I call @options.parse!(left) to process the left over options. Note that --help is built in to the options so if you want to not recognize help the first time, you need to do ' OptionParser::Officious.delete('help')' before you create the OptParser and then add in your own help option

Optimum

Answer 10

Parse options up until the first unknown option ... the block might be called multiple times, so make sure that is safe ...

options = {
  :input_file => 'input.txt', # default input file
}

opts = OptionParser.new do |opt|
  opt.on('-i', '--input FILE', String,
    'Input file name',
    'Default is %s' % options[:input_file] ) do |input_file|
    options[:input_file] = input_file
  end

  opt.on_tail('-h', '--help', 'Show this message') do
    puts opt
    exit
  end
end

original = ARGV.dup
leftover = []

loop do
  begin
    opts.parse(original)
  rescue OptionParser::InvalidOption
    leftover.unshift(original.pop)
  else
    break
  end
end

puts "GOT #{leftover} -- #{original}"

Benedick