Get UTF-8 input with X11 Display

Question

I've been trying and reading lots of resources on the internet, trying to find a way to get an UTF-8 keyboard (composed) input from a X Display. But I could not make it work.

I have tried the example code from this link (exaple 11-4), but no success.

I also have written a simple example (below) to try to make it work. My simple test case is to print an "é", which happens by typing the acute and then the e.

What is wrong?

Thanks,

Here is my example:

#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xresource.h>
#include <X11/Xlocale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char ** argv)
{
    int screen_num, width, height;
    unsigned long background, border;
    Window win;
    XEvent ev;
    Display *dpy;
    XIM im;
    XIC ic;
    char *failed_arg;
    XIMStyles *styles;
    XIMStyle xim_requested_style;

    /* First connect to the display server, as specified in the DISPLAY 
    environment variable. */
    if (setlocale(LC_ALL, "") == NULL) {
        return 9;
    }

    if (!XSupportsLocale()) {
        return 10;
    }
    if (XSetLocaleModifiers("") == NULL) {
        return 11;
    }

    dpy = XOpenDisplay(NULL);
    if (!dpy) {
        fprintf(stderr, "unable to connect to display");
        return 7;
    }
    /* these are macros that pull useful data out of the display object */
    /* we use these bits of info enough to want them in their own variables */
    screen_num = DefaultScreen(dpy);
    background = BlackPixel(dpy, screen_num);
    border = WhitePixel(dpy, screen_num);

    width = 400; /* start with a small window */
    height = 200;

    win = XCreateSimpleWindow(dpy, DefaultRootWindow(dpy), /* display, parent */
        0,0, /* x, y: the window manager will place the window elsewhere */
        width, height, /* width, height */
        2, border, /* border width & colour, unless you have a window manager */
        background); /* background colour */

    /* tell the display server what kind of events we would like to see */
    XSelectInput(dpy, win, ButtonPressMask|StructureNotifyMask|KeyPressMask|KeyReleaseMask|KeymapStateMask);

    /* okay, put the window on the screen, please */
    XMapWindow(dpy, win);

    im = XOpenIM(dpy, NULL, NULL, NULL);
    if (im == NULL) {
        fputs("Could not open input method\n", stdout);
        return 2;
    }

    failed_arg = XGetIMValues(im, XNQueryInputStyle, &styles, NULL);

    if (failed_arg != NULL) {
      fputs("XIM Can't get styles\n", stdout);
      return 3;
    }

    int i;
    for (i = 0; i < styles->count_styles; i++) {
        printf("style %d\n", styles->supported_styles[i]);
    }
    ic = XCreateIC(im, XNInputStyle, XIMPreeditNothing | XIMStatusNothing, XNClientWindow, win, NULL);
    if (ic == NULL) {
        printf("Could not open IC\n");
        return 4;
    }

    XSetICFocus(ic);

    /* as each event that we asked about occurs, we respond.  In this
     * case we note if the window's shape changed, and exit if a button
     * is pressed inside the window */
    while(1) {
        XNextEvent(dpy, &ev);
        switch(ev.type){
        case KeymapNotify:
            XRefreshKeyboardMapping(&ev.xmapping);
            break;
        case KeyPress:
            {
                int count = 0;
                KeySym keysym = 0;
                char buf[20];
                Status status = 0;
                count = Xutf8LookupString(ic, (XKeyPressedEvent*)&ev, buf, 20, &keysym, &status);

                printf("count: %d\n", count);
                if (status==XBufferOverflow)
                    printf("BufferOverflow\n");

                if (count)
                    printf("buffer: %s\n", buf);

                if (status == XLookupKeySym || status == XLookupBoth) {
                    printf("status: %d\n", status);
                }
                printf("pressed KEY: %d\n", keysym);
            }
            break;
        case KeyRelease:
            {
                int count = 0;
                KeySym keysym = 0;
                char buf[20];
                Status status = 0;
                count = XLookupString((XKeyEvent*)&ev, buf, 20, &keysym, NULL);

                if (count)
                    printf("in release buffer: %s\n", buf);

                printf("released KEY: %d\n", keysym);
            }
            break;
        case ConfigureNotify:
            if (width != ev.xconfigure.width
                    || height != ev.xconfigure.height) {
                width = ev.xconfigure.width;
                height = ev.xconfigure.height;
                printf("Size changed to: %d by %d", width, height);
            }
            break;
        case ButtonPress:
            XCloseDisplay(dpy);
            return 0;
        }
        fflush(stdout);
    }
}

Rajkot

Answer 1

You have to do this:

if (XFilterEvent(&ev, win))
    continue;

in your event loop. This runs the input method machinery, without it you will get raw X events. For example, when you press a dead accent key followed by a letter key, and do not call XFilterEvent, you will get two KeyPress events as usual. But if you do the call, you will get three events. There are two raw events, for which XFilterEvent(&ev, win) returns True. And then there is one event synthesized by the input method, for which XFilterEvent(&ev, win) returns False. It is this third event that contains the accented character.

If you want both raw events and those synthesized by the input method, you can of course do your own raw event processing instead of continue.

Note you will need buf[count] = 0; in order to print buf correctly (or explicitly use a length), Xutf8LookupString doesn't null-terminate its output.

Finally, as mentioned in the comments, with recent versions of X11 you will need to specify a modify to XSetLocaleModifiers such as XSetLocaleModifiers("@im=none"), otherwise the extra events won't be generated.

Here is a corrected version of the code:

#include <X11/Xlib.h>
#include <X11/Xutil.h>
#include <X11/Xresource.h>
#include <X11/Xlocale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
    
int main(int argc, char ** argv)
{
    int screen_num, width, height;
    unsigned long background, border;
    Window win;
    XEvent ev;
    Display *dpy;
    XIM im;
    XIC ic;
    char *failed_arg;
    XIMStyles *styles;
    XIMStyle xim_requested_style;
    
    /* First connect to the display server, as specified in the DISPLAY 
    environment variable. */
    if (setlocale(LC_ALL, "") == NULL) {
        return 9;
    }
    
    if (!XSupportsLocale()) {
        return 10;
    }
    if (XSetLocaleModifiers("@im=none") == NULL) {
        return 11;
    }
    
    dpy = XOpenDisplay(NULL);
    if (!dpy) {
        fprintf(stderr, "unable to connect to display");
        return 7;
    }
    /* these are macros that pull useful data out of the display object */
    /* we use these bits of info enough to want them in their own variables */
    screen_num = DefaultScreen(dpy);
    background = BlackPixel(dpy, screen_num);
    border = WhitePixel(dpy, screen_num);
    
    width = 400; /* start with a small window */
    height = 200;
    
    win = XCreateSimpleWindow(dpy, DefaultRootWindow(dpy), /* display, parent */
        0,0, /* x, y: the window manager will place the window elsewhere */
        width, height, /* width, height */
        2, border, /* border width & colour, unless you have a window manager */
        background); /* background colour */
        
    /* okay, put the window on the screen, please */
    XMapWindow(dpy, win);
    
    im = XOpenIM(dpy, NULL, NULL, NULL);
    if (im == NULL) {
        fputs("Could not open input method\n", stdout);
        return 2;
    }
    
    failed_arg = XGetIMValues(im, XNQueryInputStyle, &styles, NULL);
    
    if (failed_arg != NULL) {
        fputs("XIM Can't get styles\n", stdout);
        return 3;
    }
    
    int i;
    for (i = 0; i < styles->count_styles; i++) {
        printf("style %d\n", (int)styles->supported_styles[i]);
    }
    ic = XCreateIC(im, XNInputStyle, XIMPreeditNothing | XIMStatusNothing, XNClientWindow, win, NULL);
    if (ic == NULL) {
        printf("Could not open IC\n");
        return 4;
    }

    unsigned long mask = 0;                                            
    unsigned long other_masks = ButtonPressMask|StructureNotifyMask|KeyPressMask|KeyReleaseMask;
  
    if (!XGetICValues(ic, XNFilterEvents, &mask, NULL))
    {
        XSelectInput(dpy, win, other_masks | mask);
        printf ("Got IM mask of %lx\n", mask);
    }
    else
    {
        XSelectInput(dpy, win, other_masks);
        fprintf (stderr, "Could not get an IM mask\n");
    }

    XSetICFocus(ic);
    
    /* as each event that we asked about occurs, we respond.  In this
     * case we note if the window's shape changed, and exit if a button
     * is pressed inside the window */
    while(1) {
        XNextEvent(dpy, &ev);
        if (XFilterEvent(&ev, win))
            continue;
        switch(ev.type){
        case MappingNotify:
            XRefreshKeyboardMapping(&ev.xmapping);
            break;
        case KeyPress:
            {
                int count = 0;
                KeySym keysym = 0;
                char buf[20];
                Status status = 0;
                count = Xutf8LookupString(ic, (XKeyPressedEvent*)&ev, buf, 20, &keysym, &status);
    
                printf("count: %d\n", count);
                if (status==XBufferOverflow)
                    printf("BufferOverflow\n");
    
                if (count)
                    printf("buffer: %.*s\n", count, buf);
 
                if (status == XLookupKeySym || status == XLookupBoth) {
                    printf("status: %d\n", status);
                }
                printf("pressed KEY: %d\n", (int)keysym);
            }
            break;
        case KeyRelease:
            {
                int count = 0;
                KeySym keysym = 0;
                char buf[20];
                Status status = 0;
                count = XLookupString((XKeyEvent*)&ev, buf, 20, &keysym, NULL);
    
                if (count)
                    printf("in release buffer: %.*s\n", count, buf);
    
                printf("released KEY: %d\n", (int)keysym);
            }
            break;
        case ConfigureNotify:
            if (width != ev.xconfigure.width
                    || height != ev.xconfigure.height) {
                width = ev.xconfigure.width;
                height = ev.xconfigure.height;
                printf("Size changed to: %d by %d", width, height);
            }
            break;
        case ButtonPress:
            XCloseDisplay(dpy);
            return 0;
        }
        fflush(stdout);
    }
}

Vermifuge

I have figured out this, that you said, 2 days ago and you have pointed out some other things that is important for my use case. This confirms that I'm doing the right thinks :). Thanks a lot for the answer – Rajkot

Do you put this after XNextEvent() or after case KeyPress? – Finegrained

@Finegrained after XNextEvent I think would be the right place. An input method can convert any sort of events to key presses. – Vermifuge

Note that will recent version of Xlib you must specify a modifier list to XSetLocaleModifiers() such as XSetLocaleModifiers("@im=none") or no events will be emitted by XFilterEvent. Took a while for me to figure out. – Gob

I believe there is also another small bug in the sample code: the case KeymapNotify: should read case MappingNotify:. And if so, then KeymapStateMask should be removed from XSelectInput(). – Bethanie

@JosephQuinsey you are absolutely right, I'm changing this. – Vermifuge

Sorry for the Necro, but why is XLookupString used on key up and Xutf8LookupString on key down? – Ecstatic

@Celess from the manual: "To ensure proper input processing, it is essential that the client pass only KeyPress events to XmbLookupString, XwcLookupString and Xutf8LookupString. Their behavior when a client passes a KeyRelease event is undefined." – Vermifuge

Ok thanks that takes care of it. I was asking because your example has case KeyRelease: XLookupString and I thought maybe it was a mistake you didn't put XLookupKeySym on both. Then unless I misunderstand you, maybe the example is wrong for having the second one at all. – Ecstatic

As an aside I thought that some complex key sequences to generate chars for some languages also complete on key up, but I'll have to look into that separately. – Ecstatic

Shouldn't you pass the mask obtained by XGetICValues(ic, XNFilterEvents, &mask, NULL) as XSelectInput argument ? – Biological

@Biological Frankly I have not heard about XGetICValues before, but apparently yes you are supposed to do that, and perhaps not call XFilterEvent manually, not sure. The documentation is scarce and I am not an expert. – Vermifuge

@n. m. could be an AI please take a look at SDL2 implementation. XFilterEvent is called despite calling XGetICValues. I don't understand how it works. – Biological