wostream doesn't output as unicode!!!

[RwDwR]

I am using a wostream for outputting data to a file, minor problem, the wide-outpt-stream doe not output as wide...

the output is ANSI, ASCII or whatever you want to call it,
I runned it in debug, and found the characters are all unicode before I send the to the stream (<<) and I add no ascii characters

but my text file clearly showes ascii encoding in a hex viewer (unicode should show a lot of zeroes (and dots in the text view of my hex editor))

I tried to save the file in unicode with notepad, just to check what it should look like, so I am not making a mistake about this!!

 

[RwDwR]

I now found that it has to do with the << operator being overwritten by ostream, but I have no solution yet!

 

[Zyrenthian]

In response to the post in another Unicode based thread

&quot;does this also help me in having a wostream stream output unicode characters as unicode?

I checked my code, and it seems as if the << operator is overwritten in ostream, causing it to output my wide chars as ascii instead of unicode

if not, do you know the solution?&quot;

I have not used wostream but my approach to this would be to write a function that output wchar_t's


we know that a wchar_t is twice the size of a char but that brings in a few problems. We no longer have 256 characters (0-255) but we now have 65535 characters. This makes for a pretty insane loop for each charcter. What we do know however is that ascii characters remain intact so we dont need to worry about 0-255. We can start our loop at 256. The loop could look something like this

// Please keep in mind I have not used wostream but
// here is what I assume you could do

for(int i = 256;i<65535;i++)
{
   if(i%100 == 0)
   {
     int ans;
     cout<<&quot;Enter zero to break out of loop or any other number to continue:&quot;;
     cin>>ans;
     if(!ans)break;
   }
   wcout<<(wchar_t)i<<&quot; &quot;;// cast i as a wchar_t
}

Hopefully this helps.

Matt

 

[RwDwR]

There was a bug in the Metroworks Codewarrior 5.3 library that caused my problem, below is the fix:

please know that version 6.0 and higher (up to 8.x as I write this) do not have this bug, and that version 4.0 and less are probably to different from 5.x to be fixed with this code (i didn't try though)Fixed procedures:

<locale> (not locale.cpp!!)(1 function fixed)
=========================================================================
template <>
inline
int
codecvt<wchar_t, char, mbstate_t>::do_encoding() const _MSL_THROW
{
return 1;
}

--------------------------------------------------->>

template <>
inline
int
codecvt<wchar_t, char, mbstate_t>::do_encoding() const _MSL_THROW
{
return sizeof(wchar_t);
}
=========================================================================
First version returns as if it uses char's, this makes the calling
procedure use it's output functions for char's
---->
Second version returns 2 instead of 1
=========================================================================


&quot;locale.cpp&quot; (2 functions fixed)
=========================================================================
template <>
codecvt_base::result
codecvt<wchar_t, char, mbstate_t>::do_out(mbstate_t&,
const wchar_t* from, const wchar_t* from_end, const wchar_t*& from_next,
char* to, char* to_limit, char*& to_next) const
{
size_t ilen = size_t(from_end - from);
size_t xlen = size_t(to_limit - to);
size_t n = ilen;
result r = ok;
if (xlen < ilen)
{
r = partial;
n = xlen;
}
from_next = from;
to_next = to;
for (size_t e = 0; e < n; ++e)
*to_next++ = (char)*from_next++;
return r;
}

template <>
codecvt_base::result
codecvt<wchar_t, char, mbstate_t>::do_in(mbstate_t&,
const char* from, const char* from_end, const char*& from_next,
wchar_t* to, wchar_t* to_limit, wchar_t*& to_next) const
{
size_t xlen = size_t(from_end - from);
size_t ilen = size_t(to_limit - to);
size_t n = xlen;
result r = ok;
if (ilen < xlen)
{
r = partial;
n = ilen;
}
from_next = from;
to_next = to;
for (size_t e = 0; e < n; ++e)
*to_next++ = (wchar_t)*from_next++;
return r;
}

--------------------------------------------------->>

template <>
codecvt_base::result
codecvt<wchar_t, char, mbstate_t>::do_out(mbstate_t&,
const wchar_t* from, const wchar_t* from_end, const wchar_t*& from_next,
char* to, char* to_limit, char*& to_next) const
{
size_t fromlen = size_t(from_end - from);
size_t tolen = size_t(to_limit - to);
size_t n = fromlen * sizeof(wchar_t);
result r = ok;
if (tolen < n)
{
r = partial;
fromlen = tolen / sizeof(wchar_t);
if (fromlen == 0)
return error;
n = fromlen * sizeof(wchar_t);
}
tolen = n;
memcpy(to, from, n);
from_next = from + fromlen;
to_next = to + tolen;
return r;
}

template <>
codecvt_base::result
codecvt<wchar_t, char, mbstate_t>::do_in(mbstate_t&,
const char* from, const char* from_end, const char*& from_next,
wchar_t* to, wchar_t* to_limit, wchar_t*& to_next) const
{
size_t fromlen = size_t(from_end - from);
if (fromlen % sizeof(wchar_t) != 0)
return error;
size_t tolen = size_t(to_limit - to);
size_t n = fromlen;
result r = ok;
if (tolen * sizeof(wchar_t) < n)
{
r = partial;
fromlen = n = tolen * sizeof(wchar_t);
if (fromlen == 0)
return error;
}
else
tolen = n / sizeof(wchar_t);
memcpy(to, from, n);
from_next = from + fromlen;
to_next = to + tolen;
return r;
}
=========================================================================
I have no clue, these alterations were suggested by Howard Dinnant, who
writes the standard lib at metroworks
=========================================================================



<fstream> (1 function fixed)
=========================================================================
template <class charT, class traits>
typename basic_filebuf<charT, traits>::int_type
basic_filebuf<charT, traits>:verflow(int_type c)
{
....

while (r == codecvt_base:artial)
{
char* ex = const_cast<char*>(extern_buf.c_str());
char* extern_end;
codecvt_base::result r = a_codecvt_->out(state_, &intern_buf, &intern_buf + 1, intern_end,
ex, ex + extern_buf.size(), extern_end);
switch (r)
{
case codecvt_base::error:
return traits::eof();
case codecvt_base::noconv:
extern_buf = char(intern_buf);
break;
case codecvt_base:k:
extern_buf.resize(string::size_type(extern_end - ex));
break;
case codecvt_base:artial:
extern_buf.resize(2*extern_buf.size() + 1);
break;
}
}

....
}

--------------------------------------------------->>

template <class charT, class traits>
typename basic_filebuf<charT, traits>::int_type
basic_filebuf<charT, traits>:verflow(int_type c)
{
....

while (r == codecvt_base:artial)
{
char* ex = const_cast<char*>(extern_buf.c_str());
char* extern_end;
r = a_codecvt_->out(state_, &intern_buf, &intern_buf + 1, intern_end,
ex, ex + extern_buf.size(), extern_end);
switch (r)
{
case codecvt_base::error:
return traits::eof();
case codecvt_base::noconv:
extern_buf = char(intern_buf);
break;
case codecvt_base:k:
extern_buf.resize(string::size_type(extern_end - ex));
break;
case codecvt_base:artial:
extern_buf.resize(2*extern_buf.size() + 1);
break;
}
}

....
}
=========================================================================
Here a mistake is corrected, the variable r set in the while loop
is not the one used to evaluate the while loop... removing the namespace
makes the right r being set
=========================================================================

 

[RwDwR]

Corection, the Help I got with fixing this Came from Howard Hinnant, rather then Dinnant, typo, sorry Mr Hinnant!!