summaryrefslogtreecommitdiff
path: root/doc/misc/ert.texi
diff options
context:
space:
mode:
authorPaul Eggert <eggert@cs.ucla.edu>2017-11-02 21:01:44 -0700
committerPaul Eggert <eggert@cs.ucla.edu>2017-11-02 21:01:44 -0700
commit1441eb76fd08572c25e0128c247cc1a628a79ff2 (patch)
treeaaa9a86485836a33618d258a55d8cdbe6082c91a /doc/misc/ert.texi
parent3ca4a3c8d40ea89ed717e0673fa85f5c1ac851ed (diff)
parent460fe4a1bc40f2ba39deda6448a7baf57e0e6b76 (diff)
downloademacs-1441eb76fd08572c25e0128c247cc1a628a79ff2.tar.gz
emacs-1441eb76fd08572c25e0128c247cc1a628a79ff2.tar.bz2
emacs-1441eb76fd08572c25e0128c247cc1a628a79ff2.zip
Merge from origin/emacs-26
460fe4a1bc ; Doc fixes 41adf3281e Avoid duplicate calls to xfree for the same pointer 3e7ebbe1bd Don't clobber docstrings of explicitly-defined mode hook v... 9c8fe0248b Avoid fullscreen ediff control frames by default (Bug#29026) 7d32176acc Fix the bug#24034 change (revno 9eb028f) causing infloop (... ee493663ba Merge branch 'emacs-26' of git.savannah.gnu.org:/srv/git/e... ca5eb0d0b7 Fix a typo in the ELisp manual d10c9479ca Fix doc and doc-strings for minibuffer window related func... 18331d00da Fix "Args out of range" error in c-determine-limit. Fixes... edde35e6f8 * lisp/progmodes/perl-mode.el: Fix electric indentation wi... 056587c45f Extend mhtml submode region when in comment 4c4ad80848 Fix mthml submode lighting at end of buffer 557e252aa2 Remember more variables in mhtml-mode 336cd0a11a ; * lisp/select.el (select-enable-primary): Add missing pe... 134099bc90 ; * etc/NEWS (EUDC): Mark as not requiring documentation. 1a340274bf * etc/NEWS (EUDC): Deprecate BBDB 2.x backward compatibility. 4189d0ef7b Fix minibuffer window related docs and strings (Bug#28978) 2ebdde6e9c Add ChkTeX flymake backend for latex-mode 5b59841791 Fix doc typos 82a16c547b Fix some duplicate word typos 266888b1d7 * doc/lispref/commands.texi (Adjusting Point): Fix wording... 00c3c6d88d Avoid segfaults in 64-bit Windows builds a8e6741066 Fix conversion of pixel coordinates to buffer position d43b486f6c Fix doc strings in desktop.el 9102fb603e Add Index to ERT manual 68182a4710 Make manuals and NEWS consistent 451823b0e5 Don't allow (minibuffer-window-active-p nil) to return t cc8f72ca22 Clarify obsolescence message for 'whitespace-tab' 50f711e7fa Fix some duplicate words typos 8bd9524a7c * lisp/button.el (button-activate): Fix doc typo. 0b0d91e60a * lisp/calendar/todo-mode.el (todo-toggle-mark-item): Fix ... e6b4e5ffdf Fix some doc typos e8636ac8cc Fix startup display on Cygwin 3926c5ad83 * src/fileio.c (Fset_default_file_modes): Fix typo in doc ... 9715317dfd * lisp/dired.el (dired-find-alternate-file): Doc fix. (Bu... 9e442a001a Improve documentation of how faces are applied to display ... 1bda71ec3b Improve pixel-scroll-mode 196106d37d Support Certification Authority Authorization in dns-mode.el ec08d70b4f Improve documentation of set-default-file-modes # Conflicts: # etc/NEWS
Diffstat (limited to 'doc/misc/ert.texi')
-rw-r--r--doc/misc/ert.texi113
1 files changed, 96 insertions, 17 deletions
diff --git a/doc/misc/ert.texi b/doc/misc/ert.texi
index 2a17a211665..de71aca8aea 100644
--- a/doc/misc/ert.texi
+++ b/doc/misc/ert.texi
@@ -3,6 +3,10 @@
@setfilename ../../info/ert.info
@settitle Emacs Lisp Regression Testing
@include docstyle.texi
+@syncodeindex fn cp
+@syncodeindex vr cp
+@syncodeindex pg cp
+@syncodeindex ky cp
@c %**end of header
@dircategory Emacs misc features
@@ -59,6 +63,7 @@ traditional software development methods.
* How to Debug Tests:: What to do if a test fails.
* Extending ERT:: ERT is extensible in several ways.
* Other Testing Concepts:: Features not in ERT.
+* Index:: Concept, Function and Variable Index
* GNU Free Documentation License:: The license for this documentation.
@detailmenu
@@ -92,6 +97,10 @@ Other Testing Concepts
* Mocks and Stubs:: Stubbing out code that is irrelevant to the test.
* Fixtures and Test Suites:: How ERT differs from tools for other languages.
+Index
+
+* Index:: Concept, Function and Variable Index
+
Appendix
* GNU Free Documentation License:: The license for this documentation.
@@ -102,6 +111,7 @@ Appendix
@node Introduction
@chapter Introduction
+@cindex introduction to ERT
ERT allows you to define @emph{tests} in addition to functions,
macros, variables, and the other usual Lisp constructs. Tests are
@@ -169,6 +179,7 @@ Environment}.
@node How to Run Tests
@chapter How to Run Tests
+@cindex how to run ert tests
You can run tests either in the Emacs you are working in, or on the
command line in a separate Emacs process in batch mode (i.e., with no
@@ -187,7 +198,10 @@ different Emacs versions.
@node Running Tests Interactively
@section Running Tests Interactively
+@cindex running tests interactively
+@cindex interactive testing
+@findex ert
You can run the tests that are currently defined in your Emacs with
the command @kbd{@kbd{M-x} ert @kbd{RET} t @kbd{RET}}. (For an
explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop
@@ -232,6 +246,7 @@ F list-test
(different-atoms c d))))
@end example
+@cindex test results buffer
At the top, there is a summary of the results: we ran all tests defined
in the current Emacs (@code{Selector: t}), 31 of them passed, and 2
failed unexpectedly. @xref{Expected Failures}, for an explanation of
@@ -245,20 +260,29 @@ unexpected result. In the example above, there are two failures, both
due to failed @code{should} forms. @xref{Understanding Explanations},
for more details.
+@kindex TAB@r{, in ert results buffer}
+@kindex S-TAB@r{, in ert results buffer}
In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between
buttons. Each name of a function or macro in this buffer is a button;
moving point to it and typing @kbd{RET} jumps to its definition.
+@kindex r@r{, in ert results buffer}
+@kindex d@r{, in ert results buffer}
+@kindex .@r{, in ert results buffer}
+@kindex b@r{, in ert results buffer}
+@cindex backtrace of a failed test
Pressing @kbd{r} re-runs the test near point on its own. Pressing
@kbd{d} re-runs it with the debugger enabled. @kbd{.} jumps to the
definition of the test near point (@kbd{RET} has the same effect if
point is on the name of the test). On a failed test, @kbd{b} shows
the backtrace of the failure.
+@kindex l@r{, in ert results buffer}
@kbd{l} shows the list of @code{should} forms executed in the test.
If any messages were generated (with the Lisp function @code{message})
in a test or any of the code that it invoked, @kbd{m} will show them.
+@kindex L@r{, in ert results buffer}
By default, long expressions in the failure details are abbreviated
using @code{print-length} and @code{print-level}. Pressing @kbd{L}
while point is on a test failure will increase the limits to show more
@@ -267,7 +291,11 @@ of the expression.
@node Running Tests in Batch Mode
@section Running Tests in Batch Mode
+@cindex running tests in batch mode
+@cindex batch-mode testing
+@findex ert-run-tests-batch
+@findex ert-run-tests-batch-and-exit
ERT supports automated invocations from the command line or from
scripts or makefiles. There are two functions for this purpose,
@code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}.
@@ -283,6 +311,7 @@ with a zero exit status if all tests passed, or nonzero if any tests
failed or if anything else went wrong. It will also print progress
messages and error diagnostics to standard output.
+@findex ert-summarize-tests-batch-and-exit
You can also redirect the above output to a log file, say
@file{output.log}, and use the
@code{ert-summarize-tests-batch-and-exit} function to produce a neat
@@ -314,6 +343,8 @@ files that it requires are on your @code{load-path}.
@node Test Selectors
@section Test Selectors
+@cindex test selector
+@cindex selecting tests
Functions like @code{ert} accept a @emph{test selector}, a Lisp
expression specifying a set of tests. Test selector syntax is similar
@@ -328,17 +359,22 @@ to Common Lisp's type specifier syntax:
@item A string is a regular expression that selects all tests with matching names.
@item A test (i.e., an object of @code{ert-test} data type) selects that test.
@item A symbol selects the test that the symbol names.
-@item @code{(member TESTS...)} selects the elements of TESTS, a list of
-tests or symbols naming tests.
-@item @code{(eql TEST)} selects TEST, a test or a symbol naming a test.
-@item @code{(and SELECTORS...)} selects the tests that match all SELECTORS.
-@item @code{(or SELECTORS...)} selects the tests that match any SELECTOR.
-@item @code{(not SELECTOR)} selects all tests that do not match SELECTOR.
-@item @code{(tag TAG)} selects all tests that have TAG on their tags list.
+@item @code{(member @var{tests}...)} selects the elements of
+@var{tests}, a list of tests or symbols naming tests.
+@item @code{(eql @var{test})} selects @var{test}, a test or a symbol
+naming a test.
+@item @code{(and @var{selectors}@dots{})} selects the tests that match
+all @var{selectors}.
+@item @code{(or @var{selectors}@dots{})} selects the tests that match
+any of the @var{selectors}.
+@item @code{(not @var{selector})} selects all tests that do not match
+@var{selector}.
+@item @code{(tag @var{tag})} selects all tests that have @var{tag} on
+their tags list.
(Tags are optional labels you can apply to tests when you define them.)
-@item @code{(satisfies PREDICATE)} selects all tests that satisfy PREDICATE,
-a function that takes a test as argument and returns non-@code{nil} if
-it is selected.
+@item @code{(satisfies @var{predicate})} selects all tests that
+satisfy @var{predicate}, a function that takes a test as argument and
+returns non-@code{nil} if it is selected.
@end itemize
Selectors that are frequently useful when selecting tests to run
@@ -354,7 +390,9 @@ result in the last run, and tag-based selectors such as @code{(not
@node How to Write Tests
@chapter How to Write Tests
+@cindex how to write tests
+@findex ert-deftest
ERT lets you define tests in the same way you define functions. You
can type @code{ert-deftest} forms in a buffer and evaluate them there
with @code{eval-defun} or @code{compile-defun}, or you can save the
@@ -375,6 +413,7 @@ to find where a test was defined if the test was loaded from a file.
@node The @code{should} Macro
@section The @code{should} Macro
+@findex should@r{, ert macro}
Test bodies can include arbitrary code; but to be useful, they need to
check whether the code being tested (or @emph{code under test})
does what it is supposed to do. The macro @code{should} is similar to
@@ -410,6 +449,8 @@ test failed, it helps to know that the function @code{+} returned 3
here. ERT records the return value for any predicate called directly
within @code{should}.
+@findex should-not@r{, ert macro}
+@findex should-error@r{, ert macro}
In addition to @code{should}, ERT provides @code{should-not}, which
checks that the predicate returns @code{nil}, and @code{should-error}, which
checks that the form called within it signals an error. An example
@@ -438,7 +479,10 @@ default.
@node Expected Failures
@section Expected Failures
+@cindex expected failures
+@cindex known bugs
+@vindex :expected-result
Some bugs are complicated to fix, or not very important, and are left as
@emph{known bugs}. If there is a test case that triggers the bug and
fails, ERT will alert you of this failure every time you run all
@@ -492,6 +536,9 @@ versions, specific architectures, etc.:
@node Tests and Their Environment
@section Tests and Their Environment
+@cindex skipping tests
+@cindex test preconditions
+@cindex preconditions of a test
Sometimes, it doesn't make sense to run a test due to missing
preconditions. A required Emacs feature might not be compiled in, the
function to be tested could call an external binary which might not be
@@ -505,6 +552,7 @@ available on the test machine, you name it. In this case, the macro
...)
@end lisp
+@cindex tests and their environment
The outcome of running a test should not depend on the current state
of the environment, and each test should leave its environment in the
same state it found it in. In particular, a test should not depend on
@@ -559,6 +607,8 @@ hook variables to @code{nil}. This avoids the above problems.
@node Useful Techniques
@section Useful Techniques when Writing Tests
+@cindex useful techniques
+@cindex tips and tricks
Testing simple functions that have no side effects and no dependencies
on their environment is easy. Such tests often look like this:
@@ -596,6 +646,8 @@ Here's a more complicated test:
" signal(ert-test-failed (\"foo\"))")))))))
@end lisp
+@findex make-ert-test
+@findex ert-equal-including-properties
This test creates a test object using @code{make-ert-test} whose body
will immediately signal failure. It then runs that test and asserts
that it fails. Then, it creates a temporary buffer and invokes
@@ -653,6 +705,8 @@ a test failed.
@node Understanding Explanations
@section Understanding Explanations
+@cindex understanding explanations
+@cindex explanations, understanding
Failed @code{should} forms are reported like this:
@@ -720,41 +774,55 @@ function registered. @xref{Defining Explanation Functions}.
@node Interactive Debugging
@section Interactive Debugging
+@cindex interactive debugging
+@cindex debugging failed tests
Debugging failed tests essentially works the same way as debugging any
other problems with Lisp code. Here are a few tricks specific to
tests:
@itemize
-@item Re-run the failed test a few times to see if it fails in the same way
+@cindex re-running a failed test
+@item
+Re-run the failed test a few times to see if it fails in the same way
each time. It's good to find out whether the behavior is
deterministic before spending any time looking for a cause. In the
ERT results buffer, @kbd{r} re-runs the selected test.
-@item Use @kbd{.} to jump to the source code of the test to find out exactly
+@cindex jump to the test source code
+@item
+Use @kbd{.} to jump to the source code of the test to find out exactly
what it does. Perhaps the test is broken rather than the code
under test.
-@item If the test contains a series of @code{should} forms and you can't
+@item
+If the test contains a series of @code{should} forms and you can't
tell which one failed, use @kbd{l}, which shows you the list of all
@code{should} forms executed during the test before it failed.
-@item Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run
+@cindex show backtrace of failed test
+@item
+Use @kbd{b} to view the backtrace. You can also use @kbd{d} to re-run
the test with debugging enabled, this will enter the debugger and show
the backtrace as well; but the top few frames shown there will not be
relevant to you since they are ERT's own debugger hook. @kbd{b}
strips them out, so it is more convenient.
-@item If the test or the code under testing prints messages using
+@item
+If the test or the code under testing prints messages using
@code{message}, use @kbd{m} to see what messages it printed before it
failed. This can be useful to figure out how far it got.
-@item You can instrument tests for debugging the same way you instrument
+@cindex instrumenting test for Edebug
+@item
+You can instrument tests for debugging the same way you instrument
@code{defun}s for debugging: go to the source code of the test and
type @kbd{@kbd{C-u} @kbd{C-M-x}}. Then, go back to the ERT buffer and
re-run the test with @kbd{r} or @kbd{d}.
-@item If you have been editing and rearranging tests, it is possible that
+@cindex discard obsolete test results
+@item
+If you have been editing and rearranging tests, it is possible that
ERT remembers an old test that you have since renamed or removed:
renamings or removals of definitions in the source code leave around a
stray definition under the old name in the running process (this is a
@@ -765,6 +833,7 @@ forget about the obsolete test.
@node Extending ERT
@chapter Extending ERT
+@cindex extending ert
There are several ways to add functionality to ERT.
@@ -776,6 +845,7 @@ There are several ways to add functionality to ERT.
@node Defining Explanation Functions
@section Defining Explanation Functions
+@cindex defining explanation functions
The explanation function for a predicate is a function that takes the
same arguments as the predicate and returns an @emph{explanation}.
@@ -786,6 +856,7 @@ comprehensible printed representation. If the return value of the
predicate needs no explanation for a given list of arguments, the
explanation function should return @code{nil}.
+@vindex ert-explainer@r{, property}
To associate an explanation function with a predicate, add the
property @code{ert-explainer} to the symbol that names the predicate.
The value of the property should be the symbol that names the
@@ -794,6 +865,7 @@ explanation function.
@node Low-Level Functions for Working with Tests
@section Low-Level Functions for Working with Tests
+@cindex low-level functions
Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch}
are implemented on top of the lower-level test handling code in the
@@ -821,6 +893,7 @@ For information on mocks, stubs, fixtures, or test suites, see below.
@node Mocks and Stubs
@section Other Tools for Emacs Lisp
+@cindex mocks and stubs
Stubbing out functions or using so-called @emph{mocks} can make it
easier to write tests. See
@@ -834,6 +907,7 @@ offers mocks for Emacs Lisp and can be used in conjunction with ERT.
@node Fixtures and Test Suites
@section Fixtures and Test Suites
+@cindex fixtures
In many ways, ERT is similar to frameworks for other languages like
SUnit or JUnit. However, two features commonly found in such
@@ -891,6 +965,11 @@ e.g., to run quick tests during interactive development and slow tests less
often. This can be achieved with the @code{:tag} argument to
@code{ert-deftest} and @code{tag} test selectors.
+@node Index
+@unnumbered Index
+
+@printindex cp
+
@node GNU Free Documentation License
@appendix GNU Free Documentation License
@include doclicense.texi