Node.js ReadLine not waiting for a full line on socket connections?

Question

I am trying to use Node.js's ReadLine with a socket, like so:

var net = require('net');
var rl = require('readline');

this.streamServer = net.createServer(function (socket) {
    var i = rl.createInterface(socket, socket);
    i.on('line', function (line) {
        socket.write(line);
    });
});
this.streamServer.maxConnections = 1;
this.streamServer.listen(7001);

When I telnet into port 7001 and start typing text, it is immediately echoed back to me before I ever push enter.

Why is ReadLine not waiting for a full line?

I have also tried .question() and I get the same results... The callback is fired upon reception of any data, without waiting for an end-of-line character.

---

Edit: This gets even stranger. When I test using the Windows telnet client, I get the behavior that I stated above. However, if I test using PuTTY as the client, ReadLine works, even on Windows. I did some packet captures. Maybe someone could shed some light on this? Unindented lines are data from the client. Indented lines are the server replies.

Using Windows Telnet

00000000  61                                               a
    00000000  61                                               a
00000001  62                                               b
    00000001  62                                               b
00000002  63                                               c
    00000002  63                                               c
00000003  64                                               d
    00000003  64                                               d
00000004  65                                               e
    00000004  65                                               e
00000005  66                                               f
    00000005  66                                               f
00000006  67                                               g
    00000006  67                                               g
00000007  68                                               h
    00000007  68                                               h
00000008  69                                               i
    00000008  69                                               i
00000009  6a                                               j
    00000009  6a                                               j
0000000A  6b                                               k
    0000000A  6b                                               k
0000000B  6c                                               l
    0000000B  6c                                               l
0000000C  6d                                               m
    0000000C  6d                                               m
0000000D  6e                                               n
    0000000D  6e                                               n
0000000E  6f                                               o
    0000000E  6f                                               o
0000000F  70                                               p
    0000000F  70                                               p
00000010  0d 0a                                            ..
    00000010  0d 0a                                            ..
00000012  0d 0a                                            ..
    00000012  0d 0a                                            ..
00000014  0d 0a                                            ..
    00000014  0d 0a                                            ..
00000016  61                                               a
    00000016  61                                               a
00000017  73                                               s
    00000017  73                                               s
00000018  64                                               d
    00000018  64                                               d
00000019  66                                               f
    00000019  66                                               f
0000001A  0d 0a                                            ..
    0000001A  0d 0a                                            ..
0000001C  61                                               a
    0000001C  61                                               a
0000001D  73                                               s
    0000001D  73                                               s
0000001E  64                                               d
    0000001E  64                                               d
0000001F  66                                               f
    0000001F  66                                               f
00000020  0d 0a                                            ..
    00000020  0d 0a                                            ..

Using PuTTY

00000000  ff fb 1f ff fb 20 ff fb  18 ff fb 27 ff fd 01 ff ..... .. ...'....
00000010  fb 03 ff fd 03                                   .....
    00000000  ef bf bd ef bf bd 1f ef  bf bd ef bf bd 20 ef bf ........ ..... ..
    00000010  bd ef bf bd 18 ef bf bd  ef bf bd 27 ef bf bd ef ........ ...'....
    00000020  bf bd 01 ef bf bd ef bf  bd 03 ef bf bd ef bf bd ........ ........
    00000030  03                                               .
00000015  61 62 63 64 65 66 67                             abcdefg
0000001C  0d 0a                                            ..
    00000031  61 62 63 64 65 66 67                             abcdefg
    00000038  0d 0a                                            ..
0000001E  61 73 64 66                                      asdf
00000022  0d 0a                                            ..
    0000003A  61 73 64 66                                      asdf
    0000003E  0d 0a                                            ..
00000024  61 73 64 66                                      asdf
00000028  0d 0a                                            ..
    00000040  61 73 64 66                                      asdf
    00000044  0d 0a                                            ..
0000002A  0d 0a                                            ..
    00000046  0d 0a                                            ..

Assize

It looks like the documentation agrees with you about how it should work. Possible bug? Or perhaps it only works on certain kinds of handles. – Eudosia

@David-SkyMesh, Yeah, I'm hoping that isn't the case, but thinking that it probably is. I also just found this post on Google Groups that looks related: groups.google.com/group/nodejs/browse_thread/thread/… – Assize

Just tried your example. It works perfectly as expected for me. – Claw

@cababunga, What version of Node.js do you run? – Assize

@cababunga, What OS? I'm running the same version under Windows. – Assize

I'm running on Linux. Sorry forgot to mention. – Claw

Hmmm, I think I'll try it on one of my Ubuntu servers tonight to see if I can duplicate your results. If so, I'll submit a bug report. Thanks. – Assize

@cababunga, I discovered that the server does work under Windows, but it depends on what Telnet client you use. Now I'm really baffled. I edited my question with some packet captures. Would you mind taking a look? – Assize

Looks like the problem is in node-v0.6.11/lib/readline.js line 232. Check it out. It's called whenever any chunk of data is received (line 75). – Claw

@cababunga, Ha, I like the comment in the code... "Very simple implementation right now. Should try to break on new lines." Alright, so any idea why it works with PuTTY then? – Assize

This is because PuTTY sends line per packet and Windows telnet sends every character per packet. It's likely both can be configured either way. On Linux the option is mode character or mode line. – Claw

Answer 1

This is a bug in node.js, ReadLine's Interface calls _normalWrite() on each 'data' event and _normaWrite has a comment that it should try to break on newlines, but currently it just calls _onLine().

Something along the lines of this should fix it for you:

i._normalWrite = function(b) {
    if(b == undefined) {
        return;
    }
    if(!this._line_buffer) {
        this._line_buffer = '';
    }
    this._line_buffer += b.toString();
    if(this._line_buffer.indexOf('\n') !=-1 ) {
        var lines = this._line_buffer.split('\n');
        // either '' or the unfinished portion of the next line
        this._line_buffer = lines.pop();
        lines.forEach(function(line) {
            this._onLine(line + '\n');
        }, this);
    }
};

I haven't tested this, it may need to take \r into account also. Please let me know if it works for you, if so then one of us should send node a pull request with it.

Nipper

That worked great! Thanks. I just made a few minor changes to your code. I edited your post to reflect those changes. Would you mind submitting a pull request? I don't have a Github account set up. (Just Bitbucket at the moment.) Thanks again!! – Assize

Here's the pull request - once that gets merged and released you'll be able to remove that hack :) github.com/joyent/node/pull/3059 – Nipper

Looks good, thanks! +200 as soon as Stack Overflow lets me, tomorrow morning. – Assize

Awesome, thanks! This actually marks my first contribution to the node.js core, so I think that nearly doubling my SO score is a good way to celebrate :D – Nipper

Thanks for submitting the fix Nathan. Your pull has been landed and will be in the v0.7.8 release. Cheers! – Wharve

Excellent answer, although is there any possible method to switch between line reading and character by character reading? There's somewhat uses for both of them and if there are methods to switch between both(either through a single function, or 2 different functions), would be excellent. – Rheumatism

@Rheumatism I don't know that there is built-in support for character-by-character, but it'd be pretty easy to make your own transform stream that accepts chunks of an arbitrary number of characters and then splits it and outputs individual characters. – Nipper

Answer 2

An alternate solution to my problem... I just needed to get some sort of line event every time there was a new line from a stream. Since I didn't need everything else that comes with readline, I found this snippet by TooTallNate here: https://gist.github.com/1785026

/**
 * By TooTallNate, originally posted at https://gist.github.com/1785026
 * A quick little thingy that takes a Stream instance and makes
 * it emit 'line' events when a newline is encountered.
 *
 *   Usage:
 *   ‾‾‾‾‾
 *  emitLines(process.stdin)
 *  process.stdin.resume()
 *  process.stdin.setEncoding('utf8')
 *  process.stdin.on('line', function (line) {
 *    console.log(line event:', line)
 *  })
 *
 */

function emitLines (stream) {
  var backlog = ''
  stream.on('data', function (data) {
    backlog += data
    var n = backlog.indexOf('\n')
    // got a \n? emit one or more 'line' events
    while (~n) {
      stream.emit('line', backlog.substring(0, n))
      backlog = backlog.substring(n + 1)
      n = backlog.indexOf('\n')
    }
  })
  stream.on('end', function () {
    if (backlog) {
      stream.emit('line', backlog)
    }
  })
}

This was posted as a comment to Nathan's pull request here: https://github.com/joyent/node/pull/3059

Assize

completely unrelated, but I've never seen while (~n). That's genius. – Nitpicking

@Nitpicking while (n >= 0) is more readable, and is at least as efficient. – Broadax