Merge from lexical-binding branch.

* doc/lispref/eval.texi (Eval): Discourage the use of `eval'.
Document its new `lexical' argument.

* doc/lispref/variables.texi (Defining Variables): Mention the new meaning of `defvar'.
(Lexical Binding): New sub-section.

* lisp/Makefile.in (BIG_STACK_DEPTH, BIG_STACK_OPTS, BYTE_COMPILE_FLAGS):
New variables.
(compile-onefile, .el.elc, compile-calc, recompile): Use them.
(COMPILE_FIRST): Add macroexp and cconv.
* lisp/makefile.w32-in: Mirror changes in Makefile.in.

* lisp/vc/cvs-status.el:
* lisp/vc/diff-mode.el:
* lisp/vc/log-edit.el:
* lisp/vc/log-view.el:
* lisp/vc/smerge-mode.el:
* lisp/textmodes/bibtex-style.el:
* textmodes/css.el:
* lisp/startup.el:
* lisp/uniquify.el:
* lisp/minibuffer.el: 
* lisp/newcomment.el: 
* lisp/reveal.el: 
* lisp/server.el: 
* lisp/mpc.el: 
* lisp/emacs-lisp/smie.el: 
* lisp/doc-view.el: 
* lisp/dired.el: 
* lisp/abbrev.el: Use lexical binding.

* lisp/custom.el (custom-initialize-default, custom-declare-variable):
Use `defvar'.

* lisp/files.el (lexical-binding): Declare safe.

* lisp/help-fns.el (help-split-fundoc): Return nil if there's nothing else
than the arglist.
(help-add-fundoc-usage): Don't add `Not documented'.
(help-function-arglist): Handle closures, subroutines, and new
byte-code-functions.
(help-make-usage): Remove leading underscores.
(describe-function-1): Handle closures.
(describe-variable): Use special-variable-p for completion.

* lisp/simple.el (with-wrapper-hook, apply-partially): Move to subr.el.

* lisp/subr.el (apply-partially): Use new closures rather than CL.
(--dolist-tail--, --dotimes-limit--): Don't declare dynamic.
(dolist, dotimes): Use slightly different expansion for lexical code.
(functionp): Move to C.
(letrec): New macro.
(with-wrapper-hook): Use it and apply-partially instead of CL.
(eval-after-load): Preserve lexical-binding.
(save-window-excursion, with-output-to-temp-buffer): Turn them
into macros.

* lisp/emacs-lisp/advice.el (ad-arglist): Use help-function-arglist.

* lisp/emacs-lisp/autoload.el (make-autoload): Don't burp on trivial macros.

* lisp/emacs-lisp/byte-opt.el: Use lexical binding.
(byte-inline-lapcode): Remove (to bytecomp).
(byte-compile-inline-expand): Pay attention to inlining to/from
lexically bound code.
(byte-compile-unfold-lambda): Don't handle byte-code-functions
any more.
(byte-optimize-form-code-walker): Don't handle save-window-excursion
any more and don't call compiler-macros.
(byte-compile-splice-in-already-compiled-code): Remove.
(byte-code): Don't inline any more.
(disassemble-offset): Receive `bytes' as argument rather than via
dynamic scoping.
(byte-compile-tag-number): Declare before first use.
(byte-decompile-bytecode-1): Handle new byte-codes, don't change
`return' even if make-spliceable.
(byte-compile-side-effect-and-error-free-ops): Add stack-ref, remove
obsolete interactive-p.
(byte-optimize-lapcode): Optimize new lap-codes.
Don't trip up on new form of `byte-constant' lap code.

* lisp/emacs-lisp/byte-run.el (make-obsolete): Don't set the `byte-compile'
handler any more.

* lisp/emacs-lisp/bytecomp.el: Use lexical binding instead of
a "bytecomp-" prefix.  Macroexpand everything as a separate phase.
(byte-compile-initial-macro-environment):
Handle declare-function here.
(byte-compile--lexical-environment): New var.
(byte-stack-ref, byte-stack-set, byte-discardN)
(byte-discardN-preserve-tos): New lap codes.
(byte-interactive-p): Don't use any more.
(byte-compile-push-bytecodes, byte-compile-push-bytecode-const2):
New macros.
(byte-compile-lapcode): Use them and handle new lap codes.
(byte-compile-obsolete): Remove.
(byte-compile-arglist-signature): Handle new byte-code arg"lists".
(byte-compile-arglist-warn): Check late def of inlinable funs.
(byte-compile-cl-warn): Don't silence warnings for compiler-macros
since they should have been expanded by now.
(byte-compile--outbuffer): Rename from bytecomp-outbuffer.
(byte-compile-from-buffer): Remove unused second arg.
(byte-compile-preprocess): New function.
(byte-compile-toplevel-file-form): New function to distinguish
file-form calls from outside from file-form calls from hunk-handlers.
(byte-compile-file-form): Simplify.
(byte-compile-file-form-defsubst): Remove.
(byte-compile-file-form-defmumble): Simplify now that
byte-compile-lambda always returns a byte-code-function.
(byte-compile): Preprocess.
(byte-compile-byte-code-maker, byte-compile-byte-code-unmake):
Remove, not used any more.
(byte-compile-arglist-vars, byte-compile-make-lambda-lexenv)
(byte-compile-make-args-desc): New funs.
(byte-compile-lambda): Handle lexical functions.  Always return
a byte-code-function.
(byte-compile-reserved-constants): New var, to make up room for
closed-over variables.
(byte-compile-constants-vector): Obey it.
(byte-compile-top-level): New args `lexenv' and `reserved-csts'.
(byte-compile-macroexpand-declare-function): New function.
(byte-compile-form): Call byte-compile-unfold-bcf to inline immediate
byte-code-functions.
(byte-compile-form): Check obsolescence here.
(byte-compile-inline-lapcode, byte-compile-unfold-bcf): New functions.
(byte-compile-variable-ref): Remove.
(byte-compile-dynamic-variable-op): New fun.
(byte-compile-dynamic-variable-bind, byte-compile-variable-ref)
(byte-compile-variable-set): New funs.
(byte-compile-discard): Add 2 args.
(byte-compile-stack-ref, byte-compile-stack-set)
(byte-compile-make-closure, byte-compile-get-closed-var): New funs.
(byte-compile-funarg, byte-compile-funarg-2): Remove, handled in
macroexpand-all instead.
(byte-compile-quote-form): Remove.
(byte-compile-push-binding-init, byte-compile-not-lexical-var-p)
(byte-compile-bind, byte-compile-unbind): New funs.
(byte-compile-let): Handle let* and lexical binding.
(byte-compile-let*): Remove.
(byte-compile-catch, byte-compile-unwind-protect)
(byte-compile-track-mouse, byte-compile-condition-case):
Handle a new :fun-body form, used for lexical scoping.
(byte-compile-save-window-excursion)
(byte-compile-with-output-to-temp-buffer): Remove.
(byte-compile-defun): Simplify.
(byte-compile-stack-adjustment): New fun.
(byte-compile-out): Use it.
(byte-compile-refresh-preloaded): Don't reload byte-compiler files.

* lisp/emacs-lisp/cconv.el: New file.

* lisp/emacs-lisp/cl-extra.el (cl-macroexpand-all): Properly quote CL
closures.

* lisp/emacs-lisp/cl-macs.el (cl-byte-compile-block)
(cl-byte-compile-throw): Remove.
(cl-block-wrapper, cl-block-throw): Use compiler-macros instead.

* lisp/emacs-lisp/cl.el (pushnew): Silence warning.

* lisp/emacs-lisp/disass.el (disassemble-internal): Handle new
`closure' objects.
(disassemble-1): Handle new byte codes.

* lisp/emacs-lisp/edebug.el (edebug-eval-defun)
(edebug-eval-top-level-form): Use eval-sexp-add-defvars.
(edebug-toggle): Avoid `eval'.

* lisp/emacs-lisp/eieio-comp.el: Remove.

* lisp/emacs-lisp/eieio.el (byte-compile-file-form-defmethod):
Don't autoload.
(eieio-defgeneric-form-primary-only-one): Use `byte-compile' rather
than the internal `byte-compile-lambda'.
(defmethod): Don't hide code under quotes.
(eieio-defmethod): New `code' argument.

* lisp/emacs-lisp/float-sup.el (pi): Don't declare as dynamically bound.

* lisp/emacs-lisp/lisp-mode.el (eval-last-sexp-1):
Use eval-sexp-add-defvars.
(eval-sexp-add-defvars): New fun.

* lisp/emacs-lisp/macroexp.el: Use lexical binding.
(macroexpand-all-1): Check obsolete macros.  Expand compiler-macros.
Don't convert ' to #' without checking that it's indeed quoting
a lambda.

* lisp/emacs-lisp/pcase.el: Don't use destructuring-bind.
(pcase--memoize): Rename from pcase-memoize.  Change weakness.
(pcase): Add `let' pattern.
Change memoization so it actually works.
(pcase-mutually-exclusive-predicates): Add byte-code-function-p.
(pcase--u1) <guard, pred>: Fix possible shadowing problem.
<let>: New case.

* src/alloc.c (Fmake_symbol): Init new `declared_special' field.

* src/buffer.c (defvar_per_buffer): Set new `declared_special' field.

* src/bytecode.c (Bstack_ref, Bstack_set, Bstack_set2, BdiscardN):
New byte-codes.
(exec_byte_code): New function extracted from Fbyte_code to handle new
calling convention for byte-code-functions.  Add new byte-codes.

* src/callint.c (Fcall_interactively): Preserve lexical-binding mode for
interactive spec.

* src/doc.c (Fdocumentation, store_function_docstring):
* src/data.c (Finteractive_form): Handle closures.

* src/eval.c (Fsetq): Handle lexical vars.
(Fdefun, Fdefmacro, Ffunction): Make closures when needed.
(Fdefconst, Fdefvaralias, Fdefvar): Mark as dynamic.
(FletX, Flet): Obey lexical binding.
(Fcommandp): Handle closures.
(Feval): New `lexical' arg.
(eval_sub): New function extracted from Feval.  Use it almost
everywhere where Feval was used.  Look up vars in lexical env.
Handle closures.
(Ffunctionp): Move from subr.el.
(Ffuncall): Handle closures.
(apply_lambda): Remove `eval_flags'.
(funcall_lambda): Handle closures and new byte-code-functions.
(Fspecial_variable_p): New function.
(syms_of_eval): Initialize the Vinternal_interpreter_environment var,
but without exporting it to Lisp.

* src/fns.c (concat, mapcar1): Accept byte-code-functions.

* src/image.c (parse_image_spec): Use Ffunctionp.

* src/keyboard.c (eval_dyn): New fun.
(menu_item_eval_property): Use it.

* src/lisp.h (struct Lisp_Symbol): New field `declared_special'.

* src/lread.c (lisp_file_lexically_bound_p): New function.
(Fload): Bind Qlexical_binding.
(readevalloop): Remove `evalfun' arg.
Bind Qinternal_interpreter_environment.
(Feval_buffer): Bind Qlexical_binding.
(defvar_int, defvar_bool, defvar_lisp_nopro, defvar_kboard):
Mark as dynamic.
(syms_of_lread): Declare `lexical-binding'.

* src/window.c (Ftemp_output_buffer_show): New fun.
(Fsave_window_excursion):
* src/print.c (Fwith_output_to_temp_buffer): Move to subr.el.

1 files changed, 182 insertions, 33 deletions

diff --git a/lisp/subr.el b/lisp/subr.el
index 8ea4becdc11..e6e0c62e0b4 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -116,6 +116,17 @@ BODY should be a list of Lisp expressions.
   ;; depend on backquote.el.
   (list 'function (cons 'lambda cdr)))
 
+;; Partial application of functions (similar to "currying").
+;; This function is here rather than in subr.el because it uses CL.
+(defun apply-partially (fun &rest args)
+  "Return a function that is a partial application of FUN to ARGS.
+ARGS is a list of the first N arguments to pass to FUN.
+The result is a new function which does the same as FUN, except that
+the first N arguments are fixed at the values with which this function
+was called."
+  `(closure (t) (&rest args)
+            (apply ',fun ,@(mapcar (lambda (arg) `',arg) args) args)))
+
 (if (null (featurep 'cl))
     (progn
   ;; If we reload subr.el after having loaded CL, be careful not to
@@ -163,8 +174,6 @@ value of last one, or nil if there are none.
   ;; If we reload subr.el after having loaded CL, be careful not to
   ;; overwrite CL's extended definition of `dolist', `dotimes',
   ;; `declare', `push' and `pop'.
-(defvar --dolist-tail-- nil
-  "Temporary variable used in `dolist' expansion.")
 
 (defmacro dolist (spec &rest body)
   "Loop over a list.
@@ -176,18 +185,29 @@ Then evaluate RESULT to get return value, default nil.
   ;; It would be cleaner to create an uninterned symbol,
   ;; but that uses a lot more space when many functions in many files
   ;; use dolist.
+  ;; FIXME: This cost disappears in byte-compiled lexical-binding files.
   (let ((temp '--dolist-tail--))
-    `(let ((,temp ,(nth 1 spec))
-	   ,(car spec))
-       (while ,temp
-	 (setq ,(car spec) (car ,temp))
-	 ,@body
-	 (setq ,temp (cdr ,temp)))
-       ,@(if (cdr (cdr spec))
-	     `((setq ,(car spec) nil) ,@(cdr (cdr spec)))))))
-
-(defvar --dotimes-limit-- nil
-  "Temporary variable used in `dotimes' expansion.")
+    ;; This is not a reliable test, but it does not matter because both
+    ;; semantics are acceptable, tho one is slightly faster with dynamic
+    ;; scoping and the other is slightly faster (and has cleaner semantics)
+    ;; with lexical scoping.
+    (if lexical-binding
+        `(let ((,temp ,(nth 1 spec)))
+           (while ,temp
+             (let ((,(car spec) (car ,temp)))
+               ,@body
+               (setq ,temp (cdr ,temp))))
+           ,@(if (cdr (cdr spec))
+                 ;; FIXME: This let often leads to "unused var" warnings.
+                 `((let ((,(car spec) nil)) ,@(cdr (cdr spec))))))
+      `(let ((,temp ,(nth 1 spec))
+             ,(car spec))
+         (while ,temp
+           (setq ,(car spec) (car ,temp))
+           ,@body
+           (setq ,temp (cdr ,temp)))
+         ,@(if (cdr (cdr spec))
+               `((setq ,(car spec) nil) ,@(cdr (cdr spec))))))))
 
 (defmacro dotimes (spec &rest body)
   "Loop a certain number of times.
@@ -200,15 +220,30 @@ the return value (nil if RESULT is omitted).
   ;; It would be cleaner to create an uninterned symbol,
   ;; but that uses a lot more space when many functions in many files
   ;; use dotimes.
+  ;; FIXME: This cost disappears in byte-compiled lexical-binding files.
   (let ((temp '--dotimes-limit--)
 	(start 0)
 	(end (nth 1 spec)))
-    `(let ((,temp ,end)
-	   (,(car spec) ,start))
-       (while (< ,(car spec) ,temp)
-	 ,@body
-	 (setq ,(car spec) (1+ ,(car spec))))
-       ,@(cdr (cdr spec)))))
+    ;; This is not a reliable test, but it does not matter because both
+    ;; semantics are acceptable, tho one is slightly faster with dynamic
+    ;; scoping and the other has cleaner semantics.
+    (if lexical-binding
+        (let ((counter '--dotimes-counter--))
+          `(let ((,temp ,end)
+                 (,counter ,start))
+             (while (< ,counter ,temp)
+               (let ((,(car spec) ,counter))
+                 ,@body)
+               (setq ,counter (1+ ,counter)))
+             ,@(if (cddr spec)
+                   ;; FIXME: This let often leads to "unused var" warnings.
+                   `((let ((,(car spec) ,counter)) ,@(cddr spec))))))
+      `(let ((,temp ,end)
+             (,(car spec) ,start))
+         (while (< ,(car spec) ,temp)
+           ,@body
+           (setq ,(car spec) (1+ ,(car spec))))
+         ,@(cdr (cdr spec))))))
 
 (defmacro declare (&rest specs)
   "Do not evaluate any arguments and return nil.
@@ -249,20 +284,6 @@ Any list whose car is `frame-configuration' is assumed to be a frame
 configuration."
   (and (consp object)
        (eq (car object) 'frame-configuration)))
-
-(defun functionp (object)
-  "Non-nil if OBJECT is a function."
-  (or (and (symbolp object) (fboundp object)
-	   (condition-case nil
-	       (setq object (indirect-function object))
-	     (error nil))
-	   (eq (car-safe object) 'autoload)
-	   (not (car-safe (cdr-safe (cdr-safe (cdr-safe (cdr-safe object)))))))
-      (and (subrp object)
-           ;; Filter out special forms.
-           (not (eq 'unevalled (cdr (subr-arity object)))))
-      (byte-code-function-p object)
-      (eq (car-safe object) 'lambda)))
 
 ;;;; List functions.
 
@@ -1258,6 +1279,67 @@ the hook's buffer-local value rather than its default value."
 	    (kill-local-variable hook)
 	  (set hook hook-value))))))
 
+(defmacro letrec (binders &rest body)
+  "Bind variables according to BINDERS then eval BODY.
+The value of the last form in BODY is returned.
+Each element of BINDERS is a list (SYMBOL VALUEFORM) which binds
+SYMBOL to the value of VALUEFORM.
+All symbols are bound before the VALUEFORMs are evalled."
+  ;; Only useful in lexical-binding mode.
+  ;; As a special-form, we could implement it more efficiently (and cleanly,
+  ;; making the vars actually unbound during evaluation of the binders).
+  (declare (debug let) (indent 1))
+  `(let ,(mapcar #'car binders)
+     ,@(mapcar (lambda (binder) `(setq ,@binder)) binders)
+     ,@body))
+
+(defmacro with-wrapper-hook (var args &rest body)
+  "Run BODY wrapped with the VAR hook.
+VAR is a special hook: its functions are called with a first argument
+which is the \"original\" code (the BODY), so the hook function can wrap
+the original function, or call it any number of times (including not calling
+it at all).  This is similar to an `around' advice.
+VAR is normally a symbol (a variable) in which case it is treated like
+a hook, with a buffer-local and a global part.  But it can also be an
+arbitrary expression.
+ARGS is a list of variables which will be passed as additional arguments
+to each function, after the initial argument, and which the first argument
+expects to receive when called."
+  (declare (indent 2) (debug t))
+  ;; We need those two gensyms because CL's lexical scoping is not available
+  ;; for function arguments :-(
+  (let ((funs (make-symbol "funs"))
+        (global (make-symbol "global"))
+        (argssym (make-symbol "args"))
+        (runrestofhook (make-symbol "runrestofhook")))
+    ;; Since the hook is a wrapper, the loop has to be done via
+    ;; recursion: a given hook function will call its parameter in order to
+    ;; continue looping.
+    `(letrec ((,runrestofhook
+               (lambda (,funs ,global ,argssym)
+                 ;; `funs' holds the functions left on the hook and `global'
+                 ;; holds the functions left on the global part of the hook
+                 ;; (in case the hook is local).
+                 (if (consp ,funs)
+                     (if (eq t (car ,funs))
+                         (funcall ,runrestofhook
+                                  (append ,global (cdr ,funs)) nil ,argssym)
+                       (apply (car ,funs)
+                              (apply-partially
+                               (lambda (,funs ,global &rest ,argssym)
+                                 (funcall ,runrestofhook ,funs ,global ,argssym))
+                               (cdr ,funs) ,global)
+                              ,argssym))
+                   ;; Once there are no more functions on the hook, run
+                   ;; the original body.
+                   (apply (lambda ,args ,@body) ,argssym)))))
+       (funcall ,runrestofhook ,var
+                ;; The global part of the hook, if any.
+                ,(if (symbolp var)
+                     `(if (local-variable-p ',var)
+                          (default-value ',var)))
+                (list ,@args)))))
+
 (defun add-to-list (list-var element &optional append compare-fn)
   "Add ELEMENT to the value of LIST-VAR if it isn't there yet.
 The test for presence of ELEMENT is done with `equal',
@@ -1630,6 +1712,8 @@ This function makes or adds to an entry on `after-load-alist'."
     (unless elt
       (setq elt (list regexp-or-feature))
       (push elt after-load-alist))
+    ;; Make sure `form' is evalled in the current lexical/dynamic code.
+    (setq form `(funcall ',(eval `(lambda () ,form) lexical-binding)))
     (when (symbolp regexp-or-feature)
       ;; For features, the after-load-alist elements get run when `provide' is
       ;; called rather than at the end of the file.  So add an indirection to
@@ -2763,6 +2847,71 @@ nor the buffer list."
 	 (when (buffer-live-p ,old-buffer)
 	   (set-buffer ,old-buffer))))))
 
+(defmacro save-window-excursion (&rest body)
+  "Execute BODY, preserving window sizes and contents.
+Return the value of the last form in BODY.
+Restore which buffer appears in which window, where display starts,
+and the value of point and mark for each window.
+Also restore the choice of selected window.
+Also restore which buffer is current.
+Does not restore the value of point in current buffer.
+
+BEWARE: Most uses of this macro introduce bugs.
+E.g. it should not be used to try and prevent some code from opening
+a new window, since that window may sometimes appear in another frame,
+in which case `save-window-excursion' cannot help."
+  (declare (indent 0) (debug t))
+  (let ((c (make-symbol "wconfig")))
+    `(let ((,c (current-window-configuration)))
+       (unwind-protect (progn ,@body)
+         (set-window-configuration ,c)))))
+
+(defmacro with-output-to-temp-buffer (bufname &rest body)
+  "Bind `standard-output' to buffer BUFNAME, eval BODY, then show that buffer.
+
+This construct makes buffer BUFNAME empty before running BODY.
+It does not make the buffer current for BODY.
+Instead it binds `standard-output' to that buffer, so that output
+generated with `prin1' and similar functions in BODY goes into
+the buffer.
+
+At the end of BODY, this marks buffer BUFNAME unmodifed and displays
+it in a window, but does not select it.  The normal way to do this is
+by calling `display-buffer', then running `temp-buffer-show-hook'.
+However, if `temp-buffer-show-function' is non-nil, it calls that
+function instead (and does not run `temp-buffer-show-hook').  The
+function gets one argument, the buffer to display.
+
+The return value of `with-output-to-temp-buffer' is the value of the
+last form in BODY.  If BODY does not finish normally, the buffer
+BUFNAME is not displayed.
+
+This runs the hook `temp-buffer-setup-hook' before BODY,
+with the buffer BUFNAME temporarily current.  It runs the hook
+`temp-buffer-show-hook' after displaying buffer BUFNAME, with that
+buffer temporarily current, and the window that was used to display it
+temporarily selected.  But it doesn't run `temp-buffer-show-hook'
+if it uses `temp-buffer-show-function'."
+  (let ((old-dir (make-symbol "old-dir"))
+        (buf (make-symbol "buf")))
+    `(let* ((,old-dir default-directory)
+            (,buf
+             (with-current-buffer (get-buffer-create ,bufname)
+               (prog1 (current-buffer)
+                 (kill-all-local-variables)
+                 ;; FIXME: delete_all_overlays
+                 (setq default-directory ,old-dir)
+                 (setq buffer-read-only nil)
+                 (setq buffer-file-name nil)
+                 (setq buffer-undo-list t)
+                 (let ((inhibit-read-only t)
+                       (inhibit-modification-hooks t))
+                   (erase-buffer)
+                   (run-hooks 'temp-buffer-setup-hook)))))
+            (standard-output ,buf))
+       (prog1 (progn ,@body)
+         (internal-temp-output-buffer-show ,buf)))))
+
 (defmacro with-temp-file (file &rest body)
   "Create a new buffer, evaluate BODY there, and write the buffer to FILE.
 The value returned is the value of the last form in BODY.