Progressive Web-App Soft-Keyboard and Nav Bar overlaps in Fullscreen

Question

UPDATE: Ok. So, this looks like another bug. I found some tickets at chromium.org stating basically the same problem. In their case they present it as a scrolling issue, since as a web-app it won't resize the viewport on input focus and IME popup, therefore they cannot scroll down to an input field lower than the IME, and it won't automatically scroll up. Mine doesn't use a normal input field (although xterm.js does actually use one, you just never see it), but the cause and result is the same: the viewport does not resize when the IME pops up, also therefore not firing a resize event. An excerpt from the ticket:

Note I'm not sure we want to make on-screen-keyboard work particularly well or even at all with fullscreen API, since that's primarily intended for video and fullscreen games. It's a low priority to support "regular websites" that for some reason use fullscreen API.

According to the engineers in the ticket, it is considered an extremely low priority, though there are complaints in the post ranging from 2015 to present. There are apparently no plans to address said issue, especially since Firefox has the same bug.

So, my conclusion is the only realistic way to circumvent the problem is by including a built-in HTML5 OSK, if I want it fullscreen. This is fine as I already have this in the works. Since it is a terminal-emulator (somewhat similar to Termux for Android), it needs full 101 keyboard support. Currently I use Hackers Keyboard, which works great, and set 'display' to 'standalone' for the PWA (i'm not quite sure if it qualifies as an 'instant app' but it does create and install as an apk from Chrome or Android browser.. still Chrome).

If there is a hack around this that lets the user use their native IME's and resizes the viewport appropriately, I would love to hear it, and I can see others would too, by the posts. I'm leaving this ticket open so dev's with same issue don't waste their time and effort for something that's not supported, and in the hope somebody finds a workaround. For myself, I'm going forward with built-in HTML5 OSK as stated.

For those interested you can reference the tickets here and here.

---

Previously: I have a web-app that I've been using 'installed to home screen' for a while now. It's a specialized web-based terminal, using xterm.js as the basis for the client end. I've used it in 'standalone' until recently, since porting into a PWA in Fullscreen. On mobile, screen real-estate is precious for an app like this, so fullscreen is ideal-- if I can clear the following hurdles:

1. The IME (the soft-keyboard) on Android overlaps the app, rather than resizing the viewport, making it unusable in fullscreen.

2. The Navigation Bar icons overlap the top of the app when the IME pops up, which is undesirable since the whole point of fullscreen is to claim that fraction of an inch that could be one or two extra rows of text in the terminal.

I am sure there is a fix for the first, but no luck on anything I've tried yet. The second, I'm not so sure.

I plan on implementing a custom, internal IME-- that would solve the issue, but it's only half complete, and I would like users to still have the option to use IME's natively installed on their device.

Ideas, anyone?

Alternatively, if there's a way I can work out the height difference of the soft-keyboard, I could just set the height in my script on a focus event, but I'm not sure how I would do this since it's already got a handler to adjust the font-size/cols/rows listening for resize event, which is why it works fine outside this particular case.

More info:

It installs correctly into the app drawer, prompts for install, has registering service worker, etc. It resizes correctly when the IME pops up or hides if I set "display":"standalone" in the manifest, however it does not resize when set "display":"fullscreen". It has a nice wrapper with all the recommended bells and whistles for a PWA. The app is wrapped in an iframe. As long as the iframe changes size for the IME, the rest will go along with it.

I've tried both position fixed and absolute, height 100% and 100vh, and tried setting various combinations of those on the body as well. I would have expected it to just work the same.

A while back, before setting it up as a PWA, I had plugged in a function to make it go fullscreen, and it would resize for the IME just fine. The only problem then was the icons from the Nav bar overlapping the top of the app (if I recall correctly they did not go away when the IME hides, though they do now as a PWA).

The only real difference in this case between standalone and fullscreen is in standalone the top of the app stops at the nav bar. It's a small difference, but a big deal since in landscape on a phone that's a whole extra line or two, depending on what size the user has the font set to (the font is dynamically resized, along with the number of columns/rows of text in the terminal on resize event, and via +/- graphical controls).

Here's the iframe:

<iframe src=1ndex.html style="
  position: fixed;  left: 0px;  top: 0px;  width: 100%;  height: 100%;  outline: none;  border: none;
"></iframe>

I don't think the meta viewport tag should matter, but here it is. There shouldn't be any browser page scaling going on. It's a terminal client, with explicit GUI controls to change the font size and corresponding number of rows/cols to fit in the iframe. Anyway, it works as it should when viewed any other way than 'fullscreen' set in the manifest, whether in the browser chrome, 'add to home page'd (before the port to PWA), etc. It only has the overlap when installed to the app drawer as a PWA with display:'fullscreen' (but not when set fullscreen via js before the PWA port-over).

<meta id=viewport name=viewport
  content="width=device-width, initial-scale=1, minimum-scale=1, maximum-scale=1, user-scalable=no"
>

Here' a slightly modified copy of the manifest for fullscreen:

{
  "name": "asdf",
  "short_name": "asdf",
  "description": "asdf,
  "scope":"/asdf/",
  "start_url": "https://asdf.com/asdf/index.html",
  "display": "fullscreen",
  "background_color": "#000",
  "theme_color": "#000",
  "icons": [{
    "src": "/asdf/favicon.png",
    "sizes": "192x192",
    "type": "image/png"
  }]
}

That's one problem. As far as the Nav bar icons hovering over the app when the IME is present, unless the fix is the same, I don't even know where to start. All my search terms come up with is the usual tutorial pages talking about how to set up a PWA, or toting how good it is.

I must say, I'm kind of shocked the behaviour is different, as I would think it would default to the same behavior. Just because the app is fullscreen doesn't mean you'd want the IME to overlap the bottom of your app. Anyway, if there's a workaround, it's probably something really simple that's not so obvious.

Oberg

Interestingly, Google's site advert'ing PWA's specifically denoted getting support by asking about it on Stack Overflow. developers.google.com/web/progressive-web-apps – Oberg

Does this answer your question? PWA does not resize window when virtual keyboard is active on fullscreen display mode – Bul

Answer 1

As explained above, this is a recognized bug without a plan for a fix. If you really need, or want, your app to get soft-keyboard input without the OSK/IME overlaying part of your app, you can get around this with a little DIY OSK. It's really simple (well, mostly). If your use-case is like mine, it's easier: you can send characters straight to the websocket. Assuming your websocket object is called 'socket', a soft-key will look like this:

<button onclick="socket.send(this.innerHTML)">A</button>

Special keys such as Escape will need to send the 'escaped character code' like:

<button onclick='socket.send("\x1b")'>ESC</button>

--or \033, \u001b, etc, instead of \x1b. All functionally equivalent.

Note the reversal of single and double quotes. Since that's allowed in HTML5, it's easier than using backslashes to 'escape' the quotes inside the function.

Some keys, like arrow keys, will need a terminal 'escape code', if it's for a web-based terminal emulator. This code depends on the TERM type. For an xterm it's generally "\x1bO]A", or B, C, or D, depending on the direction. VT100 is the same, but omit the O. You may have to check your termcap/terminfo db or environment variable string to be sure, depending on the system.

You can emulate Shift or Capslock by copying your lowercase or uppercase keyboard, whichever you made first, and replacing the letters with their variant, essentially making a seperate keyboard. Wrap each in a div, then style with "position: absolute; left: 0px; top: 0px;" Then wrap the two keyboards in another div. Attach an onclick handler to the shift key on each-- on the topmost keyboard use the handler to hide it. In the bottom keyboard, use it to show the top keyboard again. Effectively this works a lot like the shift key on a native IME.

If your target is a conventional input text or textarea, use the pattern:

document.getElementById('myinputelementid').value += 'A';

or equivalent in your key handlers. You really should actually 'get' the input first permanently, then reference it instead of getting it each time, but either will work.

For a conventional textarea/input you will have to handle the arrows, carrot (cursor) placement, and backspace differently. My use case is for xterm.js, which will handle things like an actual backspace character, arrow keys, etc. via terminal escape sequences. If you are needing to use conventional inputs, I leave it to you to look up the established DOM methods for those actions. On the other hand, you may be able to find a pre-made open source OSK that will fit your needs. If you use node.js, there is one I've seen on the npm site, though I cannot say with surety that it does all that, I think it does. You can find it by Googling 'npm osk', or searching npm directly for 'osk'.

One more thing, you will need to prevent the input, including xterm.js from gaining focus on click/touch, otherwise the native IME will pop up and get in the way. One way you could do this is using a click and a touchstart handler on the input to 'prevent.default'. This works good for xterm.js (I don't remember the id of the hidden input, but you can find it using Dev Tools). For a conventional input, you might just overlay it with a transparent div. You may want to fake a cursor though, since you will be avoiding actual focus it won't show up. There are some good sites on how to do this. I'm just covering the basics here though, and mostly for the xterm.js use-case, which shows a cursor anyway, you just re-style it so it's solid instead of an outline.

I apollogize if this does not technically cover the entire workaround, but it does cover most of what you'll need, and give you an idea what to do to complete it, should you choose go that route. I am not saying all of it is easy. But most of it is.

The other options are: only use inputs that are small and high up enough that the native IME cannot overlap them, or nest them in something big enough that can scroll up anyway (possibly messy), or just don't use fullscreen-- use "display":"standalone" instead.

Maybe if enough dev's bug them, the vendors will fix it. So if you don't like this problem, I suggest going to the bug reports denoted in the links above, and expressing your problem in the report. It's pretty old, and there is a list of posts going from several years ago until the present.

Good luck!

Oberg

Answer 2

I solved a similar solution for my progressive web app using CSS.

The height of your screen is way lower when using the keyboard, thus you can use CSS media queries to your advantage. Example:

@media(max-height:520px) {
    #bottom_menu {
        display:none;
    }
}

Proteus

Answer 3

As the tickets are basically marked as WontFix, I'll link my generic hack workaround posted in a duplicate question here: https://mcmap.net/q/1319642/-pwa-does-not-resize-window-when-virtual-keyboard-is-active-on-fullscreen-display-mode

Bul

Why not just put the answer here, if it works...? – Oberg

Sorry jdmayfield, I was not aware that using the stackoverflow interface to flag a question as a duplicate would result in a poorly worded system generated comment against the OP's question. I don't have the reputation to reply to your comment. I'll update this answer when I can as per your reply here. – Bul

No worries. Just, your observations regarding fullscreen not adjusting with soft-keyboard, and standalone as workaround are put forth in the original question. – Oberg