bash and readline: tab completion in a user input loop?

Question

I'm making a bash script which presents a command line to the user.

The cli code is as this:

#!/bin/bash

cmd1() {
    echo $FUNCNAME: "$@"
}

cmd2() {
    echo $FUNCNAME: "$@"
}

cmdN() {
    echo $FUNCNAME: "$@"
}

__complete() {
    echo $allowed_commands
}

shopt -qs extglob

fn_hide_prefix='__'
allowed_commands="$(declare -f | sed -ne '/^'$fn_hide_prefix'.* ()/!s/ ().*//p' | tr '\n' ' ')"

complete -D -W "this should output these words when you hit TAB"

echo "waiting for commands"
while read -ep"-> "; do
    history -s $REPLY
    case "$REPLY" in
        @(${allowed_commands// /|})?(+([[:space:]])*)) $REPLY ;;
        \?) __complete ;;
        *) echo "invalid command: $REPLY" ;;
    esac
done

Clarification: made and tested in Bash 4

So, "read -e" gives readline capabilities, i can recall commands, edit the input line, etc. What i cannot do in any way is to have readline's tab completion to work!!

I tried two things:

1. How it should be supposedly done: using the bash builtins "complete" and "compgen", which is reported to work here Update: it's not reported to work in scripts.

2. This ugly workaround

Why doesn't readline behave correctly when using "complete" inside the script? it works when i try it from bash in interactive mode...

Strangury

My bash (3.2) does not allow -D to complete. What version of bash are you using? – Marinmarina

I'm using bash 4. Sorry i didn't clarify that, i'll update the post. – Strangury

Answer 1

After trying a custom completion script that I know works (I use it every day) and running into the same issue (when rigging it up similar to yours), I decided to snoop through the bash 4.1 source, and found this interesting block in bash-4.1/builtins/read.def:edit_line():

old_attempted_completion_function = rl_attempted_completion_function;
rl_attempted_completion_function = (rl_completion_func_t *)NULL;
if (itext)
  {
    old_startup_hook = rl_startup_hook;
    rl_startup_hook = set_itext;
    deftext = itext;
  }
ret = readline (p);
rl_attempted_completion_function = old_attempted_completion_function;
old_attempted_completion_function = (rl_completion_func_t *)NULL;

It appears that before readline() is called, it resets the completion function to null for some reason that only a bash-hacking long beard might know. Thus, doing this with the read builtin may simply be hard-coded to be disabled.

EDIT: Some more on this: The wrapping code to stop completion in the read builtin occurred between bash-2.05a and bash-2.05b. I found this note in that version's bash-2.05b/CWRU/changelog file:

• edit_line (called by read -e) now just does readline's filename completion by setting rl_attempted_completion_function to NULL, since e.g., doing command completion for the first word on the line wasn't really useful

I think it's a legacy oversight, and since programmable completion has come a long way, what you're doing is useful. Maybe you can ask them to add it back in, or just patch it yourself, if that'd be feasible for what you're doing.

Afraid I don't have a different solution aside from what you've come up with so far, but at least we know why it doesn't work with read.

EDIT2: Right, here's a patch I just tested that seems to "work". Passes all unit and reg tests, and shows this output from your script when run using the patched bash, as you expected:

$ ./tabcompl.sh
waiting for commands
-> **<TAB>**
TAB     hit     output  should  these   this    when    words   you
->

As you'll see, I just commented out those 4 lines and some timer code to reset the rl_attempted_completion_function when read -t is specified and a timeout occurs, which is no longer necessary. If you're going to send Chet something, you may wish to excise the entirety of the rl_attempted_completion_function junk first, but this will at least allow your script to behave properly.

Patch:

--- bash-4.1/builtins/read.def     2009-10-09 00:35:46.000000000 +0900
+++ bash-4.1-patched/builtins/read.def     2011-01-20 07:14:43.000000000 +0900
@@ -394,10 +394,12 @@
        }
       old_alrm = set_signal_handler (SIGALRM, sigalrm);
       add_unwind_protect (reset_alarm, (char *)NULL);
+/*
 #if defined (READLINE)
       if (edit)
        add_unwind_protect (reset_attempted_completion_function, (char *)NULL);
 #endif
+*/
       falarm (tmsec, tmusec);
     }

@@ -914,8 +916,10 @@
   if (bash_readline_initialized == 0)
     initialize_readline ();

+/*
   old_attempted_completion_function = rl_attempted_completion_function;
   rl_attempted_completion_function = (rl_completion_func_t *)NULL;
+*/
   if (itext)
     {
       old_startup_hook = rl_startup_hook;
@@ -923,8 +927,10 @@
       deftext = itext;
     }
   ret = readline (p);
+/*
   rl_attempted_completion_function = old_attempted_completion_function;
   old_attempted_completion_function = (rl_completion_func_t *)NULL;
+*/

   if (ret == 0)
     return ret;

Keep in mind the patched bash would have to be distributed or made available somehow wherever people would be using your script...

Knar

If I'm reading this right, command completion with bash can be enabled by commenting out the four lines containing attempted_completion_function ? – Yearling

Maybe, but there seemed to be quite a few bugfixes for leaks in read.def, some bugs related to automatic timeouts (read -t) and completion together, and some other stuff in the ChangeLogs, as well as a few more places where rl_attempted_completion_function is referred to. I didn't trace through the entire codebase to see if that will do it, but it's worth a shot. Looks like there's quite a few unit and reg tests in the source tree, too. Probably worth running those afterwards :-) – Knar

Since read is a builtin, one doesn't strictly have to provide a patched bash. Instead, one could provide a replacement read builtin. Even better in my opinion would be to call it something different like completing_read. This routine would have to be compiled against a particular version of bash and for that you'd need the bash source around. Not ideal, but still better that patching bash. – Assail

@Assail Correct me if I'm wrong, but... the patch above is exactly a replacement (enhancement) for the read builtin (which is implemented in C), and it has to be compiled against a particular version of bash... so, I don't see where what you're suggesting differs from the above. – Knar

@Skadz I guess I was not clear, so let me try again. One does not have to provide a full bash. One just has to provide the read builtin, (say ELF code on GNU/Linux) which could be done inside your package and loaded via enable -f <path-to-read-builtin> read. – Assail

@Sdaz sorry for mispelling your name before - stack overflow won't let me correct it and the last message got truncated. Let's say I want to package this in bashdb (and I may). People who package the code need access to bash source not the end programmers who installs. And the package is tagged as requiring that version of bash. In my bashdb code I add the enable. So there is no chance that other programs that use read but don't need completion are changed. That is, things are completely upward compatible. Probably that additional code is there for a reason. – Assail

@Assail Ah, I see what you mean now. Calling externally-loaded shared objects "builtins", even though it makes sense now that I see it's possible, threw me the first time :-) Good idea. – Knar

I now understand the problem: the first-level of completion in bash is hard-wired to be bash's completion function called attempt_shell_completion_function. Reinstating the functon as the suggested patch does, will give you back the custom bash completion function, but it is not what I want in bashdb. Instead I want my debugger commands to be completed first. The patch will give a list when what you have is nothing, but what happens if you start to fill in one of the debugger commands? The patch will give shell command completions. Not what I want. – Assail

The revised code that I am now using. So far, it seems to work: bashdb.git.sourceforge.net/git/gitweb.cgi?p=bashdb/… – Assail

Rocky, Sdaz: I've never took the time to express my gratitude to both for exploring all this. Work has dragged me far from this all the year, but i still want to get it to an end. I'll play with your code as soon as I can and hope that eventually Chet gets this. – Strangury

@Assail how do I build this? – Alf

@MichaelChav You are asking me something vague and open-ended that was discussed 5 years ago? I don't understand what you are asking, and right now things are very hectic for me and I can't devote the effort right now. I suggest starting a new question. In that, please describe what you are trying to do and what you've already tried. It is probably okay to cut and paste the relevant portions here. Then possibly someone who has the time might look at and help you. – Assail

Answer 2

I've been struggling with same issue for some time now and I think I have a solution that works, in my real world case I'm using compgen to generate possible completions. But here is an example that illustrates the core logic:

#!/bin/bash

set -o emacs;
tab() {
  READLINE_LINE="foobar"
  READLINE_POINT="${#READLINE_LINE}"
}
bind -x '"\t":"tab"';
read -ep "$ ";

Set the emacs option to enable key binding, bind the tab key to a function, change READLINE_LINE to update the line after the prompt, and set READLINE_POINT to reflect the line's new longer length.

In my use case I actually mimic the COMP_WORDS, COMP_CWORD and COMPREPLY variables but this should be sufficient to understand how to go about adding custom tab completion when using read -ep.

You must update READLINE_LINE to change the prompt line (completion single match), printing to stdin prints before the prompt as readline has put the terminal in raw mode and is capturing input.

Structuralism

How this example supposed to work? I should run the script, press Tab, and I should see foobar? – Wellestablished

@pihentagy: Yes, but it requires bash 4+. Assigning a value to READLINE_LINE replaces the line being edited with that value. – Broadside

Answer 3

Well, it seems i finally stumped on the answer, and it sadly is: actually there isn't full support for readline when interfacing it via "read -e".

The answer is given by the BASH maintainer, Chet Ramey. In this thread the exact same issue is addressed:

I'm writing a script with a command line interpreter and I can most things working (eg. history etc.) except for one thing. The filename completion works well for some of the commands, but I'd like to use other completion options for others. Works well from the "real" command line, but I can't get it to work properly in my "read -e, eval" loop..

You won't be able to do it. `read -e' uses only the readline default completions.

Chet

So, unless i'm missing something //rant// while bash hands to the programmer the "read -e" mechanism as the mean for full, proper CLI user interfacing, the functionality is crippled, even though the underlying mechanism (readline) works and integrates with the rest of bash flawlessly //end rant//

I have exposed the question to the kind folks at #bash in freenode and been suggested to try with a Readline wrapper like rlfe or rlwrap.

Finally, i contacted Chet himself by mail yesterday, and he confirmed that this is by design, and that he doesn't feel like changing it as the only use case for programmable completion into "read", i.e. presenting a list of commands to the script user, doesn't look like a compelling reason to spend time working on this. Nevertheless he expressed that in case someone actually does the work he would certainly look at the result.

IMHO, not considering worth of the effort the ability to bring up a full CLI with just 5 lines of code, something one wish were possible in a lot of languages, is a mistake.

In this context, i think Simon's answer is brilliant and right in place. I'll try to follow your steps and perhaps with some luck i'll get more info. Been a while since i don't hack in C however, and i assume the amount of code i'll have to grasp to implement will not be trivial. But anyway i'll try.

Strangury

Added a bash-4.1 source patch that seems to work to my answer above. – Knar

Answer 4

I'm not sure if this exactly answers the OP question - but I was searching for which command could one use, to obtain the default bash tab completion of known executable commands (as per $PATH), as shown when pressing TAB. Since I was first led to this question (which I think is related), I thought I'd post a note here.

For instance, on my system, typing lua and then TAB gives:

$ lua<TAB>
lua       lua5.1    luac      luac5.1   lualatex  luatex    luatools

It turns out, there is a bash built-in (see #949006 Linux command to list all available commands and aliases), called compgen - and I can feed it with the same string lua as in the interactive case, and obtain the same results as if I pressed TAB:

$ compgen -c lua
luac
lua5.1
lua
luac5.1
luatex
lualatex
luatools

... and that is exactly what I was looking for :)

Hope this helps someone,
Cheers!

Mullet

Answer 5

If you're going to that much effort, why not just add the cost of a fork or two and use something that is more than capable of providing everything you want. https://github.com/hanslub42/rlwrap

#!/bin/bash

which yum && yum install rlwrap
which zypper && zypper install rlwrap
which port && port install rlwrap
which apt-get && apt-get install rlwrap

REPLY=$( rlwrap -o cat )

Or as the man page puts it:

In a shell script, use rlwrap in ’one−shot’ mode as a replacement for read

order=$(rlwrap -p Yellow -S 'Your pizza? ' -H past_orders -P Margherita -o cat)

Presurmise

+1 for mentioning rlwrap; unfortunately, I'm still in Lucid, which has version 0.34, which doesn't have the -o (--one-shot) option; one apparently needs at least v. 0.35 to use that option. Cheers! – Mullet

Answer 6

This is hacky but you can leverage hostname completion:

Bash provides a readline function complete-hostname (bound to Esc@ by default), which reads "hostnames" from any file that the $HOSTFILE variable points to.

So we can bind Tab to invoke hostname completion directly and control the contents of HOSTFILE to provide custom completion options.

(Infact, HOSTFILE doesn't have to be a regular file, we can generate it dynamically with process substitution for each read invocation if want to avoid cleanup of temp files)

This wread function will just use its arguments as completion options:

wread() {
    set -o emacs
    bind 'Tab complete-hostname'
    unset HOSTFILE    # clear any already-read hosts
    HOSTFILE=<(printf '0 %s\n' "$@") \
        read -e -p '> ' &&
        printf '%s\n' "$REPLY"
}
out=$(wread --foo --bar)
declare -p out

Notes: Bash ignores any first word on a line that starts with a digit (to avoid loading IP addresses from hosts files) so we preface every line with 0 above to make sure that our completions don't get ignored if they happen to start with a digit.

The above function runs in a subshell to avoid having the bind commands and HOSTFILE changes affect the rest of the execution environment. If used in a script, this might not be necessary, depending on your use case.

Inhuman