GNU Emacs Internals
*******************

This chapter describes how the runnable Emacs executable is dumped with the preloaded Lisp libraries in it, how storage is allocated, and some internal aspects of GNU Emacs that may be of interest to C programmers.

Building Emacs
==============

This section explains the steps involved in building the Emacs executable.  You don't have to know this material to build and install Emacs, since the makefiles do all these things automatically.  This information is pertinent to Emacs maintenance.

Compilation of the C source files in the src directory produces an executable file called temacs, also called a bare impure Emacs.  It contains the Emacs Lisp interpreter and I/O routines, but not the editing commands.

The command temacs -l loadup uses temacs to create the real runnable Emacs executable.  These arguments direct temacs to evaluate the Lisp files specified in the file loadup.el.  These files set up the normal Emacs editing environment, resulting in an Emacs which is still impure but no longer bare.

It takes a substantial time to load the standard Lisp files.  Luckily, you don't have to do this each time you run Emacs; temacs can dump out an executable program called emacs which has these files preloaded.  emacs starts more quickly because it does not need to load the files.  This is the Emacs executable that is normally installed.

To create emacs, use the command temacs -batch -l loadup dump.  The purpose of -batch here is to prevent temacs from trying to initialize any of its data on the terminal; this ensures that the tables of terminal information are empty in the dumped Emacs. The argument dump tells loadup.el to dump a new executable named emacs.

Some operating systems don't support dumping.  On those systems, you must start Emacs with the temacs -l loadup command each time you use it.  This takes a substantial time, but since you need to start Emacs once a day at most---or once a week if you never log out---the extra time is not too severe a problem.

You can specify additional files to preload by writing a library named site-load.el which loads them.  You may need to increase the value of PURESIZE, in src/puresize.h, to make room for the additional files.  (Try adding increments of 20000 until it is big enough.)  However, the advantage of preloading additional files decreases as machines get faster.  On modern machines, it is usually not advisable.

You can specify other Lisp expressions to execute just before dumping by putting them in a library named site-init.el.  However, if they might alter the behavior that users expect from an ordinary unmodified Emacs, it is better to put them in default.el, so that users can override them if they wish.  See Start-up Summary.

Before loadup.el dumps the new executable, it finds the documentation strings for primitive and preloaded functions (and variables) in the file where they are stored, by calling Snarf-documentation (see Accessing Documentation).  These strings were moved out of the emacs executable to make it smaller.  See Documentation Basics.

 -- Function: dump-emacs to-file from-file
     This function dumps the current state of Emacs into an executable file TO-FILE.  It takes symbols from FROM-FILE (this is normally the executable file temacs).

     If you use this function in an Emacs that was already dumped, you must set command-line-processed to nil first for good results. See Command Line Arguments.

 -- Command: emacs-version
     This function returns a string describing the version of Emacs that is running.  It is useful to include this string in bug reports.

          (emacs-version)
            => "GNU Emacs 19.22.1 of Fri Feb 27 1994 \
          on slug (berkeley-unix)"

     Called interactively, the function prints the same information in the echo area.

 -- Variable: emacs-build-time
     The value of this variable is the time at which Emacs was built at the local site.

          emacs-build-time
               => "Fri Feb 27 14:55:57 1994"

 -- Variable: emacs-version
     The value of this variable is the version of Emacs being run.  It is a string such as "19.22.1".

The following two variables did not exist before Emacs version 19.23, which reduces their usefulness at present, but we hope they will be convenient in the future.

 -- Variable: emacs-major-version
     The major version number of Emacs, as an integer.

 -- Variable: emacs-minor-version
     The minor version number of Emacs, as an integer.  For Emacs version 19.23, the value is 23.

Pure Storage
============

Emacs Lisp uses two kinds of storage for user-created Lisp objects: normal storage and pure storage.  Normal storage is where all the new data which is created during an Emacs session is kept; see the following section for information on normal storage.  Pure storage is used for certain data in the preloaded standard Lisp files---data that should never change during actual use of Emacs.

Pure storage is allocated only while temacs is loading the standard preloaded Lisp libraries.  In the file emacs, it is marked as read-only (on operating systems which permit this), so that the memory space can be shared by all the Emacs jobs running on the machine at once.  Pure storage is not expandable; a fixed amount is allocated when Emacs is compiled, and if that is not sufficient for the preloaded libraries, temacs crashes.  If that happens, you must increase the compilation parameter PURESIZE in the file src/puresize.h.  This normally won't happen unless you try to preload additional libraries or add features to the standard ones.

 -- Function: purecopy object
     This function makes a copy of OBJECT in pure storage and returns it.  It copies strings by simply making a new string with the same characters in pure storage.  It recursively copies the contents of vectors and cons cells.  It does not make copies of other objects such as symbols, but just returns them unchanged.  It signals an error if asked to copy markers.

     This function is used only while Emacs is being built and dumped; it is called only in the file emacs/lisp/loaddefs.el.

 -- Variable: pure-bytes-used
     The value of this variable is the number of bytes of pure storage allocated so far.  Typically, in a dumped Emacs, this number is very close to the total amount of pure storage available---if it were not, we would preallocate less.

 -- Variable: purify-flag
     This variable determines whether defun should make a copy of the function definition in pure storage.  If it is non-nil, then the function definition is copied into pure storage.

     This flag is t while loading all of the basic functions for building Emacs initially (allowing those functions to be sharable and non-collectible).  Dumping Emacs as an executable always writes nil in this variable, regardless of the value it actually has before and after dumping.

     You should not change this flag in a running Emacs.

Garbage Collection
==================

When a program creates a list or the user defines a new function (such as by loading a library), that data is placed in normal storage.  If normal storage runs low, then Emacs asks the operating system to allocate more memory in blocks of 1k bytes.  Each block is used for one type of Lisp object, so symbols, cons cells, markers, etc., are segregated in distinct blocks in memory.  (Vectors, long strings, buffers and certain other editing types, which are fairly large, are allocated in individual blocks, one per object, while small strings are packed into blocks of 8k bytes.)

It is quite common to use some storage for a while, then release it by (for example) killing a buffer or deleting the last pointer to an object.  Emacs provides a garbage collector to reclaim this abandoned storage.  (This name is traditional, but ``garbage recycler'' might be a more intuitive metaphor for this facility.)

The garbage collector operates by finding and marking all Lisp objects that are still accessible to Lisp programs.  To begin with, it assumes all the symbols, their values and associated function definitions, and any data presently on the stack, are accessible.  Any objects which can be reached indirectly through other accessible objects are also accessible.

When marking is finished, all objects still unmarked are garbage.  No matter what the Lisp program or the user does, it is impossible to refer to them, since there is no longer a way to reach them.  Their space might as well be reused, since no one will miss them.  The second, ``sweep'' phase of the garbage collector arranges to reuse them.

The sweep phase puts unused cons cells onto a free list for future allocation; likewise for symbols and markers.  It compacts the accessible strings so they occupy fewer 8k blocks; then it frees the other 8k blocks.  Vectors, buffers, windows and other large objects are individually allocated and freed using malloc and free.

     Common Lisp note: unlike other Lisps, GNU Emacs Lisp does not call the garbage collector when the free list is empty.  Instead, it simply requests the operating system to allocate more storage, and processing continues until gc-cons-threshold bytes have been used.

     This means that you can make sure that the garbage collector will not run during a certain portion of a Lisp program by calling the garbage collector explicitly just before it (provided that portion of the program does not use so much space as to force a second garbage collection).

 -- Command: garbage-collect
     This command runs a garbage collection, and returns information on the amount of space in use.  (Garbage collection can also occur spontaneously if you use more than gc-cons-threshold bytes of Lisp data since the previous garbage collection.)

     garbage-collect returns a list containing the following information:

          ((USED-CONSES . FREE-CONSES)
           (USED-SYMS . FREE-SYMS)

           (USED-MARKERS . FREE-MARKERS)
           USED-STRING-CHARS
           USED-VECTOR-SLOTS
           (USED-FLOATS . FREE-FLOATS))

          (garbage-collect)
               => ((3435 . 2332) (1688 . 0)
                     (57 . 417) 24510 3839 (4 . 1))

     Here is a table explaining each element:

     used-conses
          The number of cons cells in use.

     free-conses
          The number of cons cells for which space has been obtained from the operating system, but that are not currently being used.

     used-syms
          The number of symbols in use.

     free-syms
          The number of symbols for which space has been obtained from the operating system, but that are not currently being used.

     used-markers
          The number of markers in use.

     free-markers
          The number of markers for which space has been obtained from the operating system, but that are not currently being used.

     used-string-chars
          The total size of all strings, in characters.

     used-vector-slots
          The total number of elements of existing vectors.

     used-floats
          The number of floats in use.

     free-floats
          The number of floats for which space has been obtained from the operating system, but that are not currently being used.

 -- User Option: gc-cons-threshold
     The value of this variable is the number of bytes of storage that must be allocated for Lisp objects after one garbage collection in order to trigger another garbage collection.  A cons cell counts as eight bytes, a string as one byte per character plus a few bytes of overhead, and so on; space allocated to the contents of buffers does not count.  Note that the subsequent garbage collection does not happen immediately when the threshold is exhausted, but only the next time the Lisp evaluator is called.

     The initial threshold value is 100,000.  If you specify a larger value, garbage collection will happen less often.  This reduces the amount of time spent garbage collecting, but increases total memory use. You may want to do this when running a program which creates lots of Lisp data.

     You can make collections more frequent by specifying a smaller value, down to 10,000.  A value less than 10,000 will remain in effect only until the subsequent garbage collection, at which time garbage-collect will set the threshold back to 10,000.

 -- Function: memory-limit
     This function returns the address of the last byte Emacs has allocated, divided by 1024.  We divide the value by 1024 to make sure it fits in a Lisp integer.

     You can use this to get a general idea of how your actions affect the memory usage.

Writing Emacs Primitives
========================

Lisp primitives are Lisp functions implemented in C.  The details of interfacing the C function so that Lisp can call it are handled by a few C macros.  The only way to really understand how to write new C code is to read the source, but we can explain some things here.

An example of a special form is the definition of or, from eval.c.  (An ordinary function would have the same general appearance.)

     DEFUN ("or", For, Sor, 0, UNEVALLED, 0,
       "Eval args until one of them yields non-nil, then return that value.\n\
     The remaining args are not evalled at all.\n\

     If all args return nil, return nil.")
       (args)
          Lisp_Object args;
     {
       register Lisp_Object val;
       Lisp_Object args_left;
       struct gcpro gcpro1;

       if (NULL (args))
         return Qnil;

       args_left = args;
       GCPRO1 (args_left);

       do
         {
           val = Feval (Fcar (args_left));
           if (!NULL (val))
             break;
           args_left = Fcdr (args_left);
         }
       while (!NULL (args_left));

       UNGCPRO;
       return val;
     }

Let's start with a precise explanation of the arguments to the DEFUN macro.  Here is a template for them:

     DEFUN (LNAME, FNAME, SNAME, MIN, MAX, INTERACTIVE, DOC)

lname
     This is the name of the Lisp symbol to define as the function name; in the example above, it is or.

fname
     This is the C function name for this function.  This is the name that is used in C code for calling the function.  The name is, by convention, F prepended to the Lisp name, with all dashes (-) in the Lisp name changed to underscores.  Thus, to call this function from C code, call For.  Remember that the arguments must be of type Lisp_Object; various macros and functions for creating values of type Lisp_Object are declared in the file lisp.h.

sname
     This is a C variable name to use for a structure that holds the data for the subr object that represents the function in Lisp.  This structure conveys the Lisp symbol name to the initialization routine that will create the symbol and store the subr object as its definition.  By convention, this name is always FNAME with F replaced with S.

min
     This is the minimum number of arguments that the function requires.  The function or allows a minimum of zero arguments.

max
     This is the maximum number of arguments that the function accepts, if there is a fixed maximum.  Alternatively, it can be UNEVALLED, indicating a special form that receives unevaluated arguments, or MANY, indicating an unlimited number of evaluated arguments (the equivalent of &rest).  Both UNEVALLED and MANY are macros.  If MAX is a number, it may not be less than MIN and it may not be greater than seven.

interactive
     This is an interactive specification, a string such as might be used as the argument of interactive in a Lisp function.  In the case of or, it is 0 (a null pointer), indicating that or cannot be called interactively.  A value of "" indicates a function that should receive no arguments when called interactively.

doc
     This is the documentation string.  It is written just like a documentation string for a function defined in Lisp, except you must write \n\ at the end of each line.  In particular, the first line should be a single sentence.

After the call to the DEFUN macro, you must write the argument name list that every C function must have, followed by ordinary C declarations for the arguments.  For a function with a fixed maximum number of arguments, declare a C argument for each Lisp argument, and give them all type Lisp_Object.  If the function has no upper limit on the number of arguments in Lisp, then in C it receives two arguments: the first is the number of Lisp arguments, and the second is the address of a block containing their values.  They have types int and Lisp_Object *.

Within the function For itself, note the use of the macros GCPRO1 and UNGCPRO.  GCPRO1 is used to ``protect'' a variable from garbage collection---to inform the garbage collector that it must look in that variable and regard its contents as an accessible object.  This is necessary whenever you call Feval or anything that can directly or indirectly call Feval.  At such a time, any Lisp object that you intend to refer to again must be protected somehow. UNGCPRO cancels the protection of the variables that are protected in the current function.  It is necessary to do this explicitly.

For most data types, it suffices to protect at least one pointer to the object; as long as the object is not recycled, all pointers to it remain valid.  This is not so for strings, because the garbage collector can move them.  When the garbage collector moves a string, it relocates all the pointers it knows about; any other pointers become invalid. Therefore, you must protect all pointers to strings across any point where garbage collection may be possible.

The macro GCPRO1 protects just one local variable.  If you want to protect two, use GCPRO2 instead; repeating GCPRO1 will not work.  Macros GCPRO3 and GCPRO4 also exist.

These macros implicitly use local variables such as gcpro1; you must declare these explicitly, with type struct gcpro.  Thus, if you use GCPRO2, you must declare gcpro1 and gcpro2. Alas, we can't explain all the tricky details here.

Defining the C function is not enough to make a Lisp primitive available; you must also create the Lisp symbol for the primitive and store a suitable subr object in its function cell.  The code looks like this:

     defsubr (&SUBR-STRUCTURE-NAME);

Here SUBR-STRUCTURE-NAME is the name you used as the third argument to DEFUN.

If you add a new primitive to a file that already has Lisp primitives defined in it, find the function (near the end of the file) named syms_of_SOMETHING, and add the call to defsubr there.  If the file doesn't have this function, or if you create a new file, add to it a syms_of_FILENAME (e.g., syms_of_myfile).  Then find the spot in emacs.c where all of these functions are called, and add a call to syms_of_FILENAME there.

This function syms_of_FILENAME is also the place to define any C variables which are to be visible as Lisp variables. DEFVAR_LISP makes a C variable of type Lisp_Object visible in Lisp.  DEFVAR_INT makes a C variable of type int visible in Lisp with a value that is always an integer. DEFVAR_BOOL makes a C variable of type int visible in Lisp with a value that is either t or nil.

Here is another example function, with more complicated arguments. This comes from the code for the X Window System, and it demonstrates the use of macros and functions to manipulate Lisp objects.

     DEFUN ("coordinates-in-window-p", Fcoordinates_in_window_p,
       Scoordinates_in_window_p, 2, 2,
       "xSpecify coordinate pair: \nXExpression which evals to window: ",
       "Return non-nil if POSITIONS is in WINDOW.\n\
       \(POSITIONS is a list, (SCREEN-X SCREEN-Y)\)\n\

       Returned value is list of positions expressed\n\
       relative to window upper left corner.")
       (coordinate, window)
          register Lisp_Object coordinate, window;
     {
       register Lisp_Object xcoord, ycoord;

       if (!CONSP (coordinate)) wrong_type_argument (Qlistp, coordinate);
       CHECK_WINDOW (window, 2);
       xcoord = Fcar (coordinate);
       ycoord = Fcar (Fcdr (coordinate));
       CHECK_NUMBER (xcoord, 0);
       CHECK_NUMBER (ycoord, 1);

       if ((XINT (xcoord) < XINT (XWINDOW (window)->left))
           || (XINT (xcoord) >= (XINT (XWINDOW (window)->left)
                                 + XINT (XWINDOW (window)->width))))
         return Qnil;
       XFASTINT (xcoord) -= XFASTINT (XWINDOW (window)->left);

       if (XINT (ycoord) == (screen_height - 1))
         return Qnil;

       if ((XINT (ycoord) < XINT (XWINDOW (window)->top))
           || (XINT (ycoord) >= (XINT (XWINDOW (window)->top)
                                 + XINT (XWINDOW (window)->height)) - 1))
         return Qnil;

       XFASTINT (ycoord) -= XFASTINT (XWINDOW (window)->top);
       return (Fcons (xcoord, Fcons (ycoord, Qnil)));
     }

Note that C code cannot call functions by name unless they are defined in C.  The way to call a function written in Lisp is to use Ffuncall, which embodies the Lisp function funcall.  Since the Lisp function funcall accepts an unlimited number of arguments, in C it takes two: the number of Lisp-level arguments, and a one-dimensional array containing their values.  The first Lisp-level argument is the Lisp function to call, and the rest are the arguments to pass to it.  Since Ffuncall can call the evaluator, you must protect pointers from garbage collection around the call to Ffuncall.

The C functions call0, call1, call2, and so on, provide handy ways to call a Lisp function conveniently with a fixed number of arguments.  They work by calling Ffuncall.

eval.c is a very good file to look through for examples; lisp.h contains the definitions for some important macros and functions.

Object Internals
================

GNU Emacs Lisp manipulates many different types of data.  The actual data are stored in a heap and the only access that programs have to it is through pointers.  Pointers are thirty-two bits wide in most implementations.  Depending on the operating system and type of machine for which you compile Emacs, twenty-four to twenty-six bits are used to address the object, and the remaining six to eight bits are used for a tag that identifies the object's type.

Because Lisp objects are represented as tagged pointers, it is always possible to determine the Lisp data type of any object.  The C data type Lisp_Object can hold any Lisp object of any data type.  Ordinary variables have type Lisp_Object, which means they can hold any type of Lisp value; you can determine the actual data type only at run time.  The same is true for function arguments; if you want a function to accept only a certain type of argument, you must check the type explicitly using a suitable predicate (see Type Predicates).

Buffer Internals
----------------

Buffers contain fields not directly accessible by the Lisp programmer. We describe them here, naming them by the names used in the C code. Many are accessible indirectly in Lisp programs via Lisp primitives.

name
     The buffer name is a string which names the buffer.  It is guaranteed to be unique.  See Buffer Names.

save_modified
     This field contains the time when the buffer was last saved, as an integer. See Buffer Modification.

modtime
     This field contains the modification time of the visited file.  It is set when the file is written or read.  Every time the buffer is written to the file, this field is compared to the modification time of the file.  See Buffer Modification.

auto_save_modified
     This field contains the time when the buffer was last auto-saved.

last_window_start
     This field contains the window-start position in the buffer as of the last time the buffer was displayed in a window.

undo_list
     This field points to the buffer's undo list.  See Undo.

syntax_table_v
     This field contains the syntax table for the buffer.  See Syntax Tables.

downcase_table
     This field contains the conversion table for converting text to lower case. See Case Table.

upcase_table
     This field contains the conversion table for converting text to upper case. See Case Table.

case_canon_table
     This field contains the conversion table for canonicalizing text for case-folding search.  See Case Table.

case_eqv_table
     This field contains the equivalence table for case-folding search. See Case Table.

display_table
     This field contains the buffer's display table, or nil if it doesn't have one.  See Display Tables.

markers
     This field contains the chain of all markers that currently point into the buffer.  Deletion of text in the buffer, and motion of the buffer's gap, must check each of these markers and perhaps update it. See Markers.

backed_up
     This field is a flag which tells whether a backup file has been made for the visited file of this buffer.

mark
     This field contains the mark for the buffer.  The mark is a marker, hence it is also included on the list markers.  See The Mark.

mark_active
     This field is non-nil if the buffer's mark is active.

local_var_alist
     This field contains the association list describing the variables local in this buffer, and their values, with the exception of local variables that have special slots in the buffer object.  (Those slots are omitted from this table.)  See Buffer-Local Variables.

keymap
     This field holds the buffer's local keymap.  See Keymaps.

overlay_center
     This field holds the current overlay center position.  See Overlays.

overlays_before
     This field holds a list of the overlays in this buffer that end at or before the current overlay center position.  They are sorted in order of decreasing end position.

overlays_after
     This field holds a list of the overlays in this buffer that end after the current overlay center position.  They are sorted in order of increasing beginning position.

Window Internals
----------------

Windows have the following accessible fields:

frame
     The frame that this window is on.

mini_p
     Non-nil if this window is a minibuffer window.

buffer
     The buffer which the window is displaying.  This may change often during the life of the window.

dedicated
     Non-nil if this window is dedicated to its buffer.

pointm
     This is the value of point in the current buffer when this window is selected; when it is not selected, it retains its previous value.

start
     he position in the buffer which is the first character to be displayed in the window.

force_start
     If this flag is non-nil, it says that the window has been scrolled explicitly by the Lisp program.  This affects what the next redisplay does if point is off the screen: instead of scrolling the window to show the text around point, it moves point to a location that is on the screen.

last_modified
     The modified field of the window's buffer, as of the last time a redisplay completed in this window.

last_point
     The buffer's value of point, as of the last time a redisplay completed in this window.

left
     This is the left-hand edge of the window, measured in columns.  (The leftmost column on the screen is column 0.)

top
     This is the top edge of the window, measured in lines.  (The top line on the screen is line 0.)

height
     The height of the window, measured in lines.

width
     The width of the window, measured in columns.

next
     This is the window that is the next in the chain of siblings.  It is nil in a window that is the rightmost or bottommost of a group of siblings.

prev
     This is the window that is the previous in the chain of siblings.  It is nil in a window that is the leftmost or topmost of a group of siblings.

parent
     Internally, Emacs arranges windows in a tree; each group of siblings has a parent window whose area includes all the siblings.  This field points to a window's parent.

     Parent windows do not display buffers, and play little role in display except to shape their child windows.  Emacs Lisp programs usually have no access to the parent windows; they operate on the windows at the leaves of the tree, that actually display buffers.

hscroll
     This is the number of columns that the display in the window is scrolled horizontally to the left.  Normally, this is 0.

use_time
     This is the last time that the window was selected.  The function get-lru-window uses this field.

display_table
     The window's display table, or nil if none is specified for it.

update_mode_line
     Non-nil means this window's mode line needs to be updated.

base_line_number
     The line number of a certain position in the buffer, or nil. This is used for displaying the line number of point in the mode line.

base_line_pos
     The position in the buffer for which the line number is known, or nil meaning none is known.

region_showing
     If the region (or part of it) is highlighted in this window, this field holds the mark position that made one end of that region.  Otherwise, this field is nil.

Process Internals
-----------------

The fields of a process are:

name
     A string, the name of the process.

command
     A list containing the command arguments that were used to start this process.

filter
     A function used to accept output from the process instead of a buffer, or nil.

sentinel
     A function called whenever the process receives a signal, or nil.

buffer
     The associated buffer of the process.

pid
     An integer, the Unix process ID.

childp
     A flag, non-nil if this is really a child process. It is nil for a network connection.

mark
     A marker indicating the position of end of last output from this process inserted into the buffer.  This is usually the end of the buffer.

kill_without_query
     If this is non-nil, killing Emacs while this process is still running does not ask for confirmation about killing the process.

raw_status_low
raw_status_high
     These two fields record 16 bits each of the process status returned by the wait system call.

status
     The process status, as process-status should return it.

tick
update_tick
     If these two fields are not equal, a change in the status of the process needs to be reported, either by running the sentinel or by inserting a message in the process buffer.

pty_flag
     Non-nil if communication with the subprocess uses a PTY; nil if it uses a pipe.

infd
     The file descriptor for input from the process.

outfd
     The file descriptor for output to the process.

subtty
     The file descriptor for the terminal that the subprocess is using.  (On some systems, there is no need to record this, so the value is nil.)