Non-ASCII Characters
********************

This chapter covers the special issues relating to non-ASCII characters and how they are stored in strings and buffers.

Text Representations
====================

Emacs has two text representations---two ways to represent text in a string or buffer.  These are called unibyte and multibyte.  Each string, and each buffer, uses one of these two representations.  For most purposes, you can ignore the issue of representations, because Emacs converts text between them as appropriate.  Occasionally in Lisp programming you will need to pay attention to the difference.

In unibyte representation, each character occupies one byte and therefore the possible character codes range from 0 to 255.  Codes 0 through 127 are ASCII characters; the codes from 128 through 255 are used for one non-ASCII character set (you can choose which one by setting the variable nonascii-insert-offset).

In multibyte representation, a character may occupy more than one byte, and as a result, the full range of Emacs character codes can be stored.  The first byte of a multibyte character is always in the range 128 through 159 (octal 0200 through 0237).  These values are called leading codes.  The first byte determines which character set the character belongs to (see Character Sets); in particular, it determines how many bytes long the sequence is.  The second and subsequent bytes of a multibyte character are always in the range 160 through 255 (octal 0240 through 0377).

In a buffer, the buffer-local value of the variable enable-multibyte-characters specifies the representation used. The representation for a string is determined based on the string contents when the string is constructed.

 -- Variable: enable-multibyte-characters
     This variable specifies the current buffer's text representation. If it is non-nil, the buffer contains multibyte text; otherwise, it contains unibyte text.

     Warning: do not set this variable directly; instead, use the function set-buffer-multibyte to change a buffer's representation.

 -- Variable: default-enable-multibyte-characters
     This variable`s value is entirely equivalent to (default-value 'enable-multibyte-characters), and setting this variable changes that default value.  Although setting the local binding of enable-multibyte-characters in a specific buffer is dangerous, changing the default value is safe, and it is a reasonable thing to do.

     The --unibyte command line option does its job by setting the default value to nil early in startup.

 -- Function: multibyte-string-p string
     Return t if STRING contains multibyte characters.

Converting Text Representations
===============================

Emacs can convert unibyte text to multibyte; it can also convert multibyte text to unibyte, though this conversion loses information.  In general these conversions happen when inserting text into a buffer, or when putting text from several strings together in one string.  You can also explicitly convert a string's contents to either representation.

Emacs chooses the representation for a string based on the text that it is constructed from.  The general rule is to convert unibyte text to multibyte text when combining it with other multibyte text, because the multibyte representation is more general and can hold whatever characters the unibyte text has.

When inserting text into a buffer, Emacs converts the text to the buffer's representation, as specified by enable-multibyte-characters in that buffer.  In particular, when you insert multibyte text into a unibyte buffer, Emacs converts the text to unibyte, even though this conversion cannot in general preserve all the characters that might be in the multibyte text.  The other natural alternative, to convert the buffer contents to multibyte, is not acceptable because the buffer's representation is a choice made by the user that cannot simply be overrided.

Converting unibyte text to multibyte text leaves ASCII characters unchanged.  It converts the non-ASCII codes 128 through 255 by adding the value nonascii-insert-offset to each character code. By setting this variable, you specify which character set the unibyte characters correspond to.  For example, if nonascii-insert-offset is 2048, which is (- (make-char 'latin-iso8859-1 0) 128), then the unibyte non-ASCII characters correspond to Latin 1.  If it is 2688, which is (- (make-char 'greek-iso8859-7 0) 128), then they correspond to Greek letters.

Converting multibyte text to unibyte is simpler: it performs logical-and of each character code with 255.  If nonascii-insert-offset has a reasonable value, corresponding to the beginning of some character set, this conversion is the inverse of the other: converting unibyte text to multibyte and back to unibyte reproduces the original unibyte text.

 -- Variable: nonascii-insert-offset
     This variable specifies the amount to add to a non-ASCII character when converting unibyte text to multibyte.  It also applies when insert-char or self-insert-command inserts a character in the unibyte non-ASCII range, 128 through 255.

     The right value to use to select character set CS is (- (make-char CS 0) 128).  If the value of nonascii-insert-offset is zero, then conversion actually uses the value for the Latin 1 character set, rather than zero.

 -- Variable: nonascii-translate-table
     This variable provides a more general alternative to nonascii-insert-offset.  You can use it to specify independently how to translate each code in the range of 128 through 255 into a multibyte character.  The value should be a vector, or nil.

 -- Function: string-make-unibyte string
     This function converts the text of STRING to unibyte representation, if it isn't already, and return the result.  If conversion does not change the contents, the value may be STRING itself.

 -- Function: string-make-multibyte string
     This function converts the text of STRING to multibyte representation, if it isn't already, and return the result.  If conversion does not change the contents, the value may be STRING itself.

Selecting a Representation
==========================

Sometimes it is useful to examine an existing buffer or string as multibyte when it was unibyte, or vice versa.

 -- Function: set-buffer-multibyte multibyte
     Set the representation type of the current buffer.  If MULTIBYTE is non-nil, the buffer becomes multibyte.  If MULTIBYTE is nil, the buffer becomes unibyte.

     This function leaves the buffer contents unchanged when viewed as a sequence of bytes.  As a consequence, it can change the contents viewed as characters; a sequence of two bytes which is treated as one character in multibyte representation will count as two characters in unibyte representation.

     This function sets enable-multibyte-characters to record which representation is in use.  It also adjusts various data in the buffer (including its overlays, text properties and markers) so that they cover or fall between the same text as they did before.

 -- Function: string-as-unibyte string
     This function returns a string with the same bytes as STRING but treating each byte as a character.  This means that the value may have more characters than STRING has.

     If STRING is unibyte already, then the value may be STRING itself.

 -- Function: string-as-multibyte string
     This function returns a string with the same bytes as STRING but treating each multibyte sequence as one character.  This means that the value may have fewer characters than STRING has.

     If STRING is multibyte already, then the value may be STRING itself.

Character Codes
===============

The unibyte and multibyte text representations use different character codes.  The valid character codes for unibyte representation range from 0 to 255---the values that can fit in one byte.  The valid character codes for multibyte representation range from 0 to 524287, but not all values in that range are valid.  In particular, the values 128 through 255 are not valid in multibyte text.  Only the ASCII codes 0 through 127 are used in both representations.

 -- Function: char-valid-p charcode
     This returns t if CHARCODE is valid for either one of the two text representations.

          (char-valid-p 65)
               => t
          (char-valid-p 256)
               => nil
          (char-valid-p 2248)
               => t

Character Sets
==============

Emacs classifies characters into various character sets, each of which has a name which is a symbol.  Each character belongs to one and only one character set.

In general, there is one character set for each distinct script.  For example, latin-iso8859-1 is one character set, greek-iso8859-7 is another, and ascii is another.  An Emacs character set can hold at most 9025 characters; therefore. in some cases, a set of characters that would logically be grouped together are split into several character sets.  For example, one set of Chinese characters is divided into eight Emacs character sets, chinese-cns11643-1 through chinese-cns11643-7.

 -- Function: charsetp object
     Return t if OBJECT is a character set name symbol, nil otherwise.

 -- Function: charset-list
     This function returns a list of all defined character set names.

 -- Function: char-charset character
     This function returns the the name of the character set that CHARACTER belongs to.

Scanning for Character Sets
===========================

Sometimes it is useful to find out which character sets appear in a part of a buffer or a string.  One use for this is in determining which coding systems (see Coding Systems) are capable of representing all of the text in question.

 -- Function: find-charset-region beg end &optional unification
     This function returns a list of the character sets that appear in the current buffer between positions BEG and END.

 -- Function: find-charset-string string &optional unification
     This function returns a list of the character sets that appear in the string STRING.

Characters and Bytes
====================

In multibyte representation, each character occupies one or more bytes.  The functions in this section convert between characters and the byte values used to represent them.

 -- Function: char-bytes character
     This function returns the number of bytes used to represent the character CHARACTER.  In most cases, this is the same as (length (split-char CHARACTER)); the only exception is for ASCII characters, which use just one byte.

          (char-bytes 2248)
               => 2
          (char-bytes 65)
               => 1

     This function's values are correct for both multibyte and unibyte representations, because the non-ASCII character codes used in those two representations do not overlap.

          (char-bytes 192)
               => 1

 -- Function: split-char character
     Return a list containing the name of the character set of CHARACTER, followed by one or two byte-values which identify CHARACTER within that character set.

          (split-char 2248)
               => (latin-iso8859-1 72)
          (split-char 65)
               => (ascii 65)

     Unibyte non-ASCII characters are considered as part of the ascii character set:

          (split-char 192)
               => (ascii 192)

 -- Function: make-char charset &rest byte-values
     Thus function returns the character in character set CHARSET identified by BYTE-VALUES.  This is roughly the opposite of split-char.

          (make-char 'latin-iso8859-1 72)
               => 2248

Coding Systems
==============

When Emacs reads or writes a file, and when Emacs sends text to a subprocess or receives text from a subprocess, it normally performs character code conversion and end-of-line conversion as specified by a particular coding system.

Character code conversion involves conversion between the encoding used inside Emacs and some other encoding.  Emacs supports many different encodings, in that it can convert to and from them.  For example, it can convert text to or from encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022.  In some cases, Emacs supports several alternative encodings for the same characters; for example, there are three coding systems for the Cyrillic (Russian) alphabet: ISO, Alternativnyj, and KOI8.

End of line conversion handles three different conventions used on various systems for end of line.  The Unix convention is to use the linefeed character (also called newline).  The DOS convention is to use the two character sequence, carriage-return linefeed, at the end of a line.  The Mac convention is to use just carriage-return.

Most coding systems specify a particular character code for conversion, but some of them leave this unspecified---to be chosen heuristically based on the data.

Base coding systems such as latin-1 leave the end-of-line conversion unspecified, to be chosen based on the data.  Variant coding systems such as latin-1-unix, latin-1-dos and latin-1-mac specify the end-of-line conversion explicitly as well.  Each base coding system has three corresponding variants whose names are formed by adding -unix, -dos and -mac.

Here are Lisp facilities for working with coding systems;

 -- Function: coding-system-list &optional base-only
     This function returns a list of all coding system names (symbols).  If BASE-ONLY is non-nil, the value includes only the base coding systems.  Otherwise, it includes variant coding systems as well.

 -- Function: coding-system-p object
     This function returns t if OBJECT is a coding system name.

 -- Function: check-coding-system coding-system
     This function checks the validity of CODING-SYSTEM. If that is valid, it returns CODING-SYSTEM. Otherwise it signals an error with condition coding-system-error.

 -- Function: detect-coding-region start end highest
     This function chooses a plausible coding system for decoding the text from START to END.  This text should be ``raw bytes'' (see Specifying Coding Systems).

     Normally this function returns is a list of coding systems that could handle decoding the text that was scanned.  They are listed in order of decreasing priority, based on the priority specified by the user with prefer-coding-system.  But if HIGHEST is non-nil, then the return value is just one coding system, the one that is highest in priority.

 -- Function: detect-coding-string
     This function is like detect-coding-region except that it operates on the contents of STRING instead of bytes in the buffer.

 -- Function: find-operation-coding-system operation &rest arguments
     This function returns the coding system to use (by default) for performing OPERATION with ARGUMENTS.  The value has this form:

          (DECODING-SYSTEM ENCODING-SYSTEM)

     The first element, DECODING-SYSTEM, is the coding system to use for decoding (in case OPERATION does decoding), and ENCODING-SYSTEM is the coding system for encoding (in case OPERATION does encoding).

     The argument OPERATION should be an Emacs I/O primitive: insert-file-contents, write-region, call-process, call-process-region, start-process, or open-network-stream.

     The remaining arguments should be the same arguments that might be given to that I/O primitive.  Depending on which primitive, one of those arguments is selected as the target.  For example, if OPERATION does file I/O, whichever argument specifies the file name is the target.  For subprocess primitives, the process name is the target.  For open-network-stream, the target is the service name or port number.

     This function looks up the target in file-coding-system-alist, process-coding-system-alist, or network-coding-system-alist, depending on OPERATION. See Default Coding Systems.

Default Coding Systems
======================

These variable specify which coding system to use by default for certain files or when running certain subprograms.  The idea of these variables is that you set them once and for all to the defaults you want, and then do not change them again.  To specify a particular coding system for a particular operation, don't change these variables; instead, override them using coding-system-for-read and coding-system-for-write (see Specifying Coding Systems).

 -- Variable: file-coding-system-alist
     This variable is an alist that specifies the coding systems to use for reading and writing particular files.  Each element has the form (PATTERN . CODING), where PATTERN is a regular expression that matches certain file names.  The element applies to file names that match PATTERN.

     The CDR of the element, VAL, should be either a coding system, a cons cell containing two coding systems, or a function symbol. If VAL is a coding system, that coding system is used for both reading the file and writing it.  If VAL is a cons cell containing two coding systems, its CAR specifies the coding system for decoding, and its CDR specifies the coding system for encoding.

     If VAL is a function symbol, the function must return a coding system or a cons cell containing two coding systems.  This value is used as described above.

 -- Variable: process-coding-system-alist
     This variable is an alist specifying which coding systems to use for a subprocess, depending on which program is running in the subprocess.  It works like file-coding-system-alist, except that PATTERN is matched against the program name used to start the subprocess.  The coding system or systems specified in this alist are used to initialize the coding systems used for I/O to the subprocess, but you can specify other coding systems later using set-process-coding-system.

 -- Variable: network-coding-system-alist
     This variable is an alist that specifies the coding system to use for network streams.  It works much like file-coding-system-alist, with the difference that the PATTERN in an elemetn may be either a port number or a regular expression.  If it is a regular expression, it is matched against the network service name used to open the network stream.

 -- Variable: default-process-coding-system
     This variable specifies the coding systems to use for subprocess (and network stream) input and output, when nothing else specifies what to do.

     The value should be a cons cell of the form (OUTPUT-CODING . INPUT-CODING).  Here OUTPUT-CODING applies to output to the subprocess, and INPUT-CODING applies to input from it.

Specifying a Coding System for One Operation
============================================

You can specify the coding system for a specific operation by binding the variables coding-system-for-read and/or coding-system-for-write.

 -- Variable: coding-system-for-read
     If this variable is non-nil, it specifies the coding system to use for reading a file, or for input from a synchronous subprocess.

     It also applies to any asynchronous subprocess or network stream, but in a different way: the value of coding-system-for-read when you start the subprocess or open the network stream specifies the input decoding method for that subprocess or network stream.  It remains in use for that subprocess or network stream unless and until overridden.

     The right way to use this variable is to bind it with let for a specific I/O operation.  Its global value is normally nil, and you should not globally set it to any other value.  Here is an example of the right way to use the variable:

          ;; Read the file with no character code conversion.
          ;; Assume CRLF represents end-of-line.
          (let ((coding-system-for-write 'emacs-mule-dos))
            (insert-file-contents filename))

     When its value is non-nil, coding-system-for-read takes precedence all other methods of specifying a coding system to use for input, including file-coding-system-alist, process-coding-system-alist and network-coding-system-alist.

 -- Variable: coding-system-for-write
     This works much like coding-system-for-read, except that it applies to output rather than input.  It affects writing to files, subprocesses, and net connections.

     When a single operation does both input and output, as do call-process-region and start-process, both coding-system-for-read and coding-system-for-write affect it.

 -- Variable: last-coding-system-used
     All operations that use a coding system set this variable to the coding system name that was used.

 -- Variable: inhibit-eol-conversion
     When this variable is non-nil, no end-of-line conversion is done, no matter which coding system is specified.  This applies to all the Emacs I/O and subprocess primitives, and to the explicit encoding and decoding functions (see Explicit Encoding).

 -- Function: keyboard-coding-system
     This function returns the coding system that is in use for decoding keyboard input---or nil if no coding system is to be used.

 -- Function: set-keyboard-coding-system coding-system
     This function specifies CODING-SYSTEM as the coding system to use for decoding keyboard input.  If CODING-SYSTEM is nil, that means do not decode keyboard input.

 -- Function: terminal-coding-system
     This function returns the coding system that is in use for encoding terminal output---or nil for no encoding.

 -- Function: set-terminal-coding-system coding-system
     This function specifies CODING-SYSTEM as the coding system to use for encoding terminal output.  If CODING-SYSTEM is nil, that means do not encode terminal output.

See also the functions process-coding-system and set-process-coding-system.  See Process Information.

See also read-coding-system in High-Level Completion.

Explicit Encoding and Decoding
==============================

All the operations that transfer text in and out of Emacs have the ability to use a coding system to encode or decode the text. You can also explicitly encode and decode text using the functions in this section.

The result of encoding, and the input to decoding, are not ordinary text.  They are ``raw bytes''---bytes that represent text in the same way that an external file would.  When a buffer contains raw bytes, it is most natural to mark that buffer as using unibyte representation, using set-buffer-multibyte (see Selecting a Representation), but this is not required.

The usual way to get raw bytes in a buffer, for explicit decoding, is to read them with from a file with insert-file-contents-literally (see Reading from Files) or specify a non-nil RAWFILE arguments when visiting a file with find-file-noselect.

The usual way to use the raw bytes that result from explicitly encoding text is to copy them to a file or process---for example, to write it with write-region (see Writing to Files), and suppress encoding for that write-region call by binding coding-system-for-write to no-conversion.

 -- Function: encode-coding-region start end coding-system
     This function encodes the text from START to END according to coding system CODING-SYSTEM.  The encoded text replaces the original text in the buffer.  The result of encoding is ``raw bytes.''

 -- Function: encode-coding-string string coding-system
     This function encodes the text in STRING according to coding system CODING-SYSTEM.  It returns a new string containing the encoded text.  The result of encoding is ``raw bytes.''

 -- Function: decode-coding-region start end coding-system
     This function decodes the text from START to END according to coding system CODING-SYSTEM.  The decoded text replaces the original text in the buffer.  To make explicit decoding useful, the text before decoding ought to be ``raw bytes.''

 -- Function: decode-coding-string string coding-system
     This function decodes the text in STRING according to coding system CODING-SYSTEM.  It returns a new string containing the decoded text.  To make explicit decoding useful, the contents of STRING ought to be ``raw bytes.''