tlug: More generic than Linux: X

["Stephen J. Turnbull" <turnbull@example.com>]

>>>>> "Scott" == Scott Perlman <perlman@example.com> writes:

    Scott> How in zee world does X negotiate codesets and fonts?

X11 itself doesn't.  There's zero magic in the X server.  The app or a
toolkit is responsible for squirting out strings that are correctly
encoded.  When you see a -*-JISX0208*-0 font requested, that means
that X will display it correctly if and only if straight 7-bit JIS
code (no escapes, no GR codes, please) of correct endianness is sent
to the server.

The standard toolkits (maybe it's as low as libXmu, I don't think it's
libX11) provide X Compound Text, as does Motif.  Motif also provides a
more complex tagged Text structure.  X Compound Text is a member of
the ISO-2022 family and allows 8 bit codes, but the shifts are different
from the EUC conventions.  None of the standard Japanese coding
systems are compatible with Compound Text (although EUC <-> Compound
text is trivial, just add a few appropriate escape sequences).  The
tagged text structure is ugly and not accessible to the user; it's
not relevant to your problem.

So you must look at the application level.  Kterms (and probably
rxvts, exterms, and cxterms) can switch encodings on the fly using a
Ctrl-MouseButton to access menus.  Unless your LANG variable is
Japanese (ja_JP* and possibly locally defined aliases), X?Emacs/Mule
assumes X compound text, and your EUC and possibly Shift-JIS text will
be mojibake.  Mule is pretty good at guessing otherwise.  Japanese
less includes appropriate guessing magic, I believe.

Try starting a kterm with `kterm -km euc &', then in that kterm `cat
$file_containing_Japanese | nkf -e'.  That should get you correct
Japanese, regardless of your LANG setting(s).  If it doesn't, you have
big problems (or one of the rare files that nkf can't guess).

I just realized that your distinction was GUI vs CLI.  Does that mean
that kterm (rxvt, whatever) works, but other programs don't?  Which
ones?  Mule, Netscape, Japanized window manager, ...?  What's not OK,
text display or GUI button labels or both?

Did it suddenly stop working, or did it never work?

    Scott> Can anyone give me a pointer as to where this is explained?
    Scott> (Free preferred, but a good book, is also very good)

OReilly Associates, Xlib Programmer's Guide (Vol 1).  Also Vol 0 (on
the X protocol).  For X.

I don't think there's much relevant content in it but FWIW (haff, kaff, 
blush) http://turnbull.sk.tsukuba.ac.jp/Tools/I18N/LJ-I18N.html has a
bunch of references.

Unfortunately, for Japanese, you're pretty much SOL because every
Japanese program does it a little bit differently.  The
lang_COUNTRY.Encoding style locale specification wasn't really
intended for the mess we've got in Japan.  There isn't much
standardization for how to handle this stuff.

If Ken Lunde's book (revised version) is on the shelves, that's the
best bet.  (Also OReilly, I think it's called Chinese, Japanese,
Korean, Vietnamese Info Processing, commonly referred to as
Son-of-UJIP.)  UJIP goes into excruciating detail about Japanese
codes, but doesn't help much with the operating environment.  I don't
know if CJKV has more of that.

Many books about Japanese environments have cookbook approaches to
setting your various variables, that may give a hint.  None of them
are much help on debugging if you've followed the recipe and it don't
work, though.

"Use the Source, Luke."

-- 
University of Tsukuba                Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Institute of Policy and Planning Sciences       Tel/fax: +81 (298) 53-5091
__________________________________________________________________________
__________________________________________________________________________
What are those two straight lines for?  "Free software rules."
-------------------------------------------------------------------
Next Technical Meeting: February 13 (Sat), 12:30 place: Temple Univ.
** presentation: XEmacs, by Steven Baur and Martin Buchholz
Next Nomikai: March 19 (Fri), 19:30   Tengu TokyoEkiMae 03-3275-3691
-------------------------------------------------------------------
more info: http://tlug.linux.or.jp                     Sponsor: PHT