c++ can't get "wcout" to print unicode, and leave "cout" working

Question

can't get "wcout" to print unicode string in multiple code pages, together with leaving "cout" to work

please help me get these 3 lines to work together.

std::wcout<<"abc "<<L'\u240d'<<" defg "<<L'א'<<" hijk"<<std::endl;
std::cout<<"hello world from cout! \n";
std::wcout<<"hello world from wcout! \n";

output:

abc hello world from cout!

i tried:

#include <io.h> 
#include <fcntl.h>
_setmode(_fileno(stdout), _O_U8TEXT);

problem: "wcout" failed

tried:

std::locale mylocale("");
std::wcout.imbue(mylocale);

and:

SetConsoleOutputCP(1251);

and

setlocale(LC_ALL, "");

and

SetConsoleCP(CP_UTF8)

Nothing worked

Skyros

@Deduplicator: I don't see how this is limited to Windows. It's fact of the C++ and C standards. – Bireme

@LightnessRacesinOrbit: You are right, sry. Did shoot too fast, due to some windows only functions used. Also, they have extra trouble for unicode. – Canberra

Related at least for windows: web.archive.org/web/20111005003105/http://blogs.msdn.com/b/… – Fayum

@LightnessRacesinOrbit Not a complete solution but at least not freezing cout\wcout cursor after printing Unicode: std::locale mylocale(""); std::wcout.imbue(mylocale); – Skyros

Answer 1

Microsoft requires a bit of non-standard set-up with _setmode() before wcout or wcin can work. This example is pretty heavy on the boilerplate, so not as clear as it could possibly be, but it runs on clang++, g++ and MSVC++:

#include <iostream>
#include <locale>
#include <locale.h>
#include <stdlib.h>

#ifndef MS_STDLIB_BUGS // Allow overriding the autodetection.
/* The Microsoft C and C++ runtime libraries that ship with Visual Studio, as
 * of 2017, have a bug that neither stdio, iostreams or wide iostreams can
 * handle Unicode input or output.  Windows needs some non-standard magic to
 * work around that.  This includes programs compiled with MinGW and Clang
 * for the win32 and win64 targets.
 */
#  if ( _MSC_VER || __MINGW32__ || __MSVCRT__ )
    /* This code is being compiled either on MS Visual C++, or MinGW, or
     * clang++ in compatibility mode for either, or is being linked to the
     * msvcrt.dll runtime.
     */
#    define MS_STDLIB_BUGS 1
#  else
#    define MS_STDLIB_BUGS 0
#  endif
#endif

#if MS_STDLIB_BUGS
#  include <io.h>
#  include <fcntl.h>
#endif

#if !HAS_APP17_FILESYSTEM && !HAS_TS_FILESYSTEM && __has_include(<filesystem>)
#  include <filesystem> /* MSVC has this header, but not the standard API. */
#  if __cpp_lib_filesystem >= 201703
#    define HAS_CPP17_FILESYSTEM 1
#  endif
#endif

#if !HAS_CPP17_FILESYSTEM && __has_include(<experimental/filesystem>)
#  include <experimental/filesystem>
/* Microsoft screws this one up, too, by not defining the feature-test
 * macro specified by the standard.
 */
#  if __cpp_lib_experimental_filesystem >= 201406 || MS_STDLIB_BUGS
#    define HAS_TS_FILESYSTEM 1
/* With g++6, this requires -lstdc++fs, AFTER this source file on the
 * command line.
 */
#  endif
#endif

#if HAS_CPP17_FILESYSTEM
  using std::filesystem::absolute;
  using std::filesystem::current_path;
  using std::filesystem::directory_entry;
  using std::filesystem::directory_iterator;
  using std::filesystem::is_directory;
  using std::filesystem::exists;
  using std::filesystem::path;
#elif HAS_TS_FILESYSTEM
  using std::experimental::filesystem::absolute;
  using std::experimental::filesystem::current_path;
  using std::experimental::filesystem::directory_entry;
  using std::experimental::filesystem::directory_iterator;
  using std::experimental::filesystem::is_directory;
  using std::experimental::filesystem::exists;
  using std::experimental::filesystem::path;
#else
#  error "This library has neither <filesystem> nor <experimental/filesystem>."
#endif

void init_locale(void)
// Does magic so that wcout can work.
{
#if MS_STDLIB_BUGS
  // Windows needs a little non-standard magic.
  constexpr char cp_utf16le[] = ".1200"; // UTF-16 little-endian locale.
  setlocale( LC_ALL, cp_utf16le );
  _setmode( _fileno(stdout), _O_WTEXT );
  /* Repeat for _fileno(stdin), if needed. */
#else
  // The correct locale name may vary by OS, e.g., "en_US.utf8".
  constexpr char locale_name[] = "";
  setlocale( LC_ALL, locale_name );
  std::locale::global(std::locale(locale_name));
  std::wcin.imbue(std::locale())
  std::wcout.imbue(std::locale());
#endif
}

using std::endl;

int main( const int argc, const char * const argv[] )
{
  init_locale();

  const path cwd = (argc > 1) ? absolute(path( argv[1], std::locale() ))
                              : absolute(current_path());

  if (exists(cwd)) {
    std::wcout << cwd.wstring() << endl;
  } else {
    std::wcerr << "Path does not exist.\n";
    return EXIT_FAILURE;
  }

  if (is_directory(cwd)) {
    for ( const directory_entry &f : directory_iterator(cwd) )
      std::wcout << f.path().filename().wstring() << endl;
  }

  return EXIT_SUCCESS;
}

That’s probably a lot more complicated than it really needed to be: std::filesystem is unsupported as of 2018, but <experimental/filesystem> is never going to be removed.

Here’s a simplified version that includes only the boilerplate to get wcout to work:

#include <iostream>
#include <locale>
#include <locale.h>

#ifndef MS_STDLIB_BUGS
#  if ( _MSC_VER || __MINGW32__ || __MSVCRT__ )
#    define MS_STDLIB_BUGS 1
#  else
#    define MS_STDLIB_BUGS 0
#  endif
#endif

#if MS_STDLIB_BUGS
#  include <io.h>
#  include <fcntl.h>
#endif

void init_locale(void)
{
#if MS_STDLIB_BUGS
  constexpr char cp_utf16le[] = ".1200";
  setlocale( LC_ALL, cp_utf16le );
  _setmode( _fileno(stdout), _O_WTEXT );
#else
  // The correct locale name may vary by OS, e.g., "en_US.utf8".
  constexpr char locale_name[] = "";
  setlocale( LC_ALL, locale_name );
  std::locale::global(std::locale(locale_name));
  std::wcin.imbue(std::locale())
  std::wcout.imbue(std::locale());
#endif
}

Confabulation

Answer 2

C++ says:

[C++11: 27.4.1/3]: Mixing operations on corresponding wide- and narrow-character streams follows the same semantics as mixing such operations on FILEs, as specified in Amendment 1 of the ISO C standard.

And the referenced document says:

The definition of a stream was changed to include the concept of an orientation for both text and binary streams. After a stream is associated with a file, but before any operations are performed on the stream, the stream is without orientation. If a wide-character input or output function is applied to a stream without orientation, the stream becomes wide-oriented. Likewise, if a byte input or output operation is applied to a stream with orientation, the stream becomes byte-oriented. Thereafter, only the fwide() or freopen() functions can alter the orientation of a stream.

Byte input/output functions shall not be applied to a wide-oriented stream and wide-character input/output functions shall not be applied to a byte-oriented stream.

By my interpretation this means, in short, do not mix std::cout and std::wcout.

Bireme

Answer 3

It's because Unicode is not representable in the codepage causing wcout to fail.

std::wcout<<"abc "<<L'\u240d'<<" defg "<<L'א'<<" hijk"<<std::endl;
if(std::wcout.fail()){
    std::cout<<"\nConversion didn't succeed\n";
    std::wcout << "This statement has no effect on the console";
    std::wcout.clear();
    std::wcout<<"hello world from wcout! \n";
}
std::cout<<"hello world from cout! \n";
std::wcout<<"hello world from wcout again! \n";

Eugenaeugene