How to read non-ASCII characters from CLI standard input

Question

If I type å in CMD, fgets stop waiting for more input and the loop runs until I press ctrl-c. If I type a "normal" characters like a-z0-9!?() it works as expected.

I run the code in CMD under Windows 7 with UTF-8 as charset (chcp 65001), the file is saved as UTF-8 without bom. I use PHP 5.3.5 (cli).

<?php

echo "ÅÄÖåäö work here.\n";

while(1)
{
    echo '> '. fgets(STDIN);
}

?>

If I change charset to chcp 1252 the loop doesn't break when I type å and it print "> å" but the "ÅÄÖåäö work here" become "ÅÄÖåäö work here!". And I know that I can change the file to ANSI, but then I can't use special characters like ╠╦╗.

So why does fgets stop waiting for userinput after I have typed åäö?

And how can I fix this?

EDIT:

Also found a strange bug. echo "öäåÅÄÖåäö work here! Or?".chr(10); -> ��äåÅÄÖåäö work here! Or? re! Or?. If the first char in echo is å/ä/ö it print strange chars AND the end output duplicate's with n - 1 char.. (n = number of åäö in the begining of the string).

Eg: echo "åäö 1234" -> ??äö 123434 and echo åäöåäö 1234 -> ??äöåäö 1234 1234.

EDIT2 (solved):

The problem was chcp 65001, now I use chcp 437 (chcp 437). Big thanks to Timothy Martens!

Caucasus

A few questions about this problem: 1) what happens when you try to type and å in the CMD outside of the php? 2) It is only logical that an UTF-8 Å is not the same as an windows-1252 Å thus the resulting Ã. But what happens if you try to convert the PHP file into windows-1252? – Deadwood

1) åäö -> "command not fond", echo åäö -> åäö. So it works. Both with chcp 65001 (UTF-8) and chcp 1252. 2) I use UTF-8 in cmd AND as charset for the PHP file. If I use windows-1252 in the PHP file nothing changes. I think the problem is in windows/PHP. When I use chcp 1252 it work for ÅÄÖ (even if the PHP file is UTF-8), but then I can't use ╠╦╗ etc. – Caucasus

Man, what an interesting question ^^. You really have my attention now. I'm going to experiment myself for a bit, and I'll tell you as soon as I find anything. – Deadwood

I wonder... what if you would use fgetsc(STDIN)? Probably the same, but it might produce another(unwanted? :P) result... – Deadwood

@Deadwood var_dump(fgetc(STDIN)) -> bool(false); var_dump(fgets(STDIN)) -> bool(false); when I type å (or äö). Else it work. – Caucasus

Answer 1

Possible solution:

echo '>'; 
$line = stream_get_line(STDIN, 999999, PHP_EOL);

Notes: I was unable to reproduce your error using multiple versions of PHP. Using the following PHP version 5.3.8 gave me no issues

PHP 5.3 (5.3.8) VC9 x86 Non Thread Safe (2011-Aug-23 12:26:18) Arcitechture is Win XP SP3 32 bit

You might try upgrading PHP.

I downloaded php-5.3.5-nts-Win32-VC6-x86 and was not able to reproduce your error, it works fine for me.

Edit: Additionaly I typed the characters using my spanish keyboard.

Edit2:

CMD Command:

chcp 437

PHP Code:

<?php
$fp=fopen("php://stdin","r");
while(1){
    $str =  fgets(STDIN);
    echo mb_detect_encoding($str)."\n";
    echo '>'.stream_get_line($fp,999999,"\n")."\n";
}
?>

Output:

test
ASCII
test
>test
öïü

öïü
>öïü

Obolus

1) stream_get_line didn't work. 2) I downloaded VC9 x86 Non Thread Safe (2011-Aug-23 12:26:18) now, but it didn't work. What charset did you use in CMD and in your code? Btw I run W7 64bit. – Caucasus

@Timoth Martens On windows cmd wouldn't it be stream_get_line(STDIN, 999999, PHP_EOL);? I updated your answer anyhow. +1 anyway. seems like the best solution so far. – Florenceflorencia

NOTE: I just tested this on my Mac using PHP 5.3.6 and PHP 5.2.14 and both worked. – Florenceflorencia

@mmmshuddup Sadly it don't work for me :( Can you post your code and your charset (both console and code charset)? Because I have tested with ansi/utf8 utf8/utf8 utf8/ansi with all solutions posted here I think.. no of them work for me. – Caucasus

@Caucasus I'm using in cmd "Active code page: 437" You can change yours by running "chcp 437" in cmd. – Obolus

I'm curious as to what yours is??? You can detect it by just running the command in cmd "chcp". I found this question in on chcp if you want more information: #1259584 – Obolus

@Caucasus Curiously when I type normal letters php reports it detects ANSI but when I type the special chars PHP reports nothing. I'm using mb_detect_encoding() to tell me what the encoding is. – Obolus

Answer 2

I think that happens because PHP 5.3 does not support properly multibyte characters.

These chars: ÅÄÖåäö

Are binary: c3 85 c3 84 c3 96 c3 a5 c3 a4 c3 b6 (without BOM at beggining)

Citing PHP String:

A string is series of characters, where a character is the same as a byte. This means that PHP only supports a 256-character set, and hence does not offer native Unicode support. See details of the string type.

Normally does not affect the final result, because the browser/reader understand multibyte characters, but for CMD and STDIN buffer is ÅÄÖåäö (12 chars/bytes char array).

only MB functions handle multibyte strings basic operations.

Malcah

Yes I know about the MB functions, but they doesn't have any read resource function :( – Caucasus