For PR1319:

Rewrite much of the DejaGnu section to bring it in line with the new
facilities in llvm.exp.


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@36015 91177308-0d34-0410-b5e6-96231b3b80d8

1 files changed, 191 insertions, 73 deletions

diff --git a/docs/TestingGuide.html b/docs/TestingGuide.html
index 4092231..584b632 100644
--- a/docs/TestingGuide.html
+++ b/docs/TestingGuide.html
@@ -275,81 +275,199 @@ location of these external programs is configured by the llvm-test
 <!--=========================================================================-->
 <div class="doc_section"><a name="dgstructure">DejaGNU Structure</a></div>
 <!--=========================================================================-->
-
 <div class="doc_text">
-<p>The LLVM test suite is partially driven by DejaGNU and partially
-driven by GNU Make. Specifically, the Features and Regression tests
-are all driven by DejaGNU. The <tt>llvm-test</tt>
-module is currently driven by a set of Makefiles.</p>
-
-<p>The DejaGNU structure is very simple, but does require some
-information to be set. This information is gathered via <tt>configure</tt> and
-is written to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The
-<tt>llvm/test</tt>
-Makefile does this work for you.</p>
-
-<p>In order for DejaGNU to work, each directory of tests must have a
-<tt>dg.exp</tt> file. This file is a program written in tcl that calls
-the <tt>llvm-runtests</tt> procedure on each test file. The
-llvm-runtests procedure is defined in
-<tt>llvm/test/lib/llvm-dg.exp</tt>. Any directory that contains only
-directories does not need the <tt>dg.exp</tt> file.</p>
-
-<p>In order for a test to be run, it must contain information within
-the test file on how to run the test. These are called <tt>RUN</tt>
-lines. Run lines are specified in the comments of the test program
-using the keyword <tt>RUN</tt> followed by a colon, and lastly the
-commands to execute. These commands will be executed in a bash script,
-so any bash syntax is acceptable. You can specify as many RUN lines as
-necessary.  Each RUN line translates to one line in the resulting bash
-script. Below is an example of legal RUN lines in a <tt>.ll</tt>
-file:</p>
-<pre>
-; RUN: llvm-as < %s | llvm-dis > %t1
-; RUN: llvm-dis < %s.bc-13 > %t2
-; RUN: diff %t1 %t2
-</pre>
-<p>There are a couple patterns within a <tt>RUN</tt> line that the
-llvm-runtest procedure looks for and replaces with the appropriate
-syntax:</p>
-
-<dl style="margin-left: 25px">
-<dt>%p</dt> 
-<dd>The path to the source directory. This is for locating
-any supporting files that are not generated by the test, but used by
-the test.</dd> 
-<dt>%s</dt> 
-<dd>The test file.</dd> 
-
-<dt>%t</dt>
-<dd>Temporary filename: testscript.test_filename.tmp, where
-test_filename is the name of the test file. All temporary files are
-placed in the Output directory within the directory the test is
-located.</dd> 
-
-<dt>%prcontext</dt> 
-<dd>Path to a script that performs grep -C. Use this since not all
-platforms support grep -C.</dd>
-
-<dt>%llvmgcc</dt> <dd>Full path to the llvm-gcc executable.</dd>
-<dt>%llvmgxx</dt> <dd>Full path to the llvm-g++ executable.</dd>
-</dl>
+  <p>The LLVM test suite is partially driven by DejaGNU and partially driven by 
+  GNU Make. Specifically, the Features and Regression tests are all driven by 
+  DejaGNU. The <tt>llvm-test</tt> module is currently driven by a set of 
+  Makefiles.</p>
+
+  <p>The DejaGNU structure is very simple, but does require some information to 
+  be set. This information is gathered via <tt>configure</tt> and is written 
+  to a file, <tt>site.exp</tt> in <tt>llvm/test</tt>. The <tt>llvm/test</tt> 
+  Makefile does this work for you.</p>
+
+  <p>In order for DejaGNU to work, each directory of tests must have a 
+  <tt>dg.exp</tt> file. DejaGNU looks for this file to determine how to run the
+  tests. This file is just a Tcl script and it can do anything you want, but 
+  we've standardized it for the LLVM regression tests. It simply loads a Tcl 
+  library (<tt>test/lib/llvm.exp</tt>) and calls the <tt>llvm_runtests</tt> 
+  function defined in that library with a list of file names to run. The names 
+  are obtained by using Tcl's glob command.  Any directory that contains only
+  directories does not need the <tt>dg.exp</tt> file.</p>
+
+  <p>The <tt>llvm-runtests</tt> function lookas at each file that is passed to
+  it and gathers any lines together that match "RUN:". This are the "RUN" lines
+  that specify how the test is to be run. So, each test script must contain
+  RUN lines if it is to do anything. If there are no RUN lines, the
+  <tt>llvm-runtests</tt> function will issue an error and the test will
+  fail.</p>
+
+  <p>RUN lines are specified in the comments of the test program using the 
+  keyword <tt>RUN</tt> followed by a colon, and lastly the command (pipeline) 
+  to execute.  Together, these lines form the "script" that 
+  <tt>llvm-runtests</tt> executes to run the test case.  The syntax of the
+  RUN lines is similar to a shell's syntax for pipelines including I/O
+  redirection and variable substitution.  However, even though these lines 
+  may <i>look</i> like a shell script, they are not. RUN lines are interpreted 
+  directly by the Tcl <tt>exec</tt> command. They are never executed by a 
+  shell. Consequently the syntax differs from normal shell script syntax in a 
+  few ways.  You can specify as many RUN lines as needed.</p>
+
+  <p>Each RUN line is executed on its own, distinct from other lines unless
+  its last character is <tt>\</tt>. This continuation character causes the RUN
+  line to be concatenated with the next one. In this way you can build up long
+  pipelines of commands without making huge line lengths. The lines ending in
+  <tt>\</tt> are concatenated until a RUN line that doesn't end in <tt>\</tt> is
+  found. This concatenated set or RUN lines then constitutes one execution. 
+  Tcl will substitute variables and arrange for the pipeline to be executed. If
+  any process in the pipeline fails, the entire line (and test case) fails too.
+  </p>
+
+  <p> Below is an example of legal RUN lines in a <tt>.ll</tt> file:</p>
+  <pre>
+  ; RUN: llvm-as &lt; %s | llvm-dis &gt; %t1
+  ; RUN: llvm-dis &lt; %s.bc-13 &gt; %t2
+  ; RUN: diff %t1 %t2
+  </pre>
+</div>
 
-<p>There are also several scripts in the llvm/test/Scripts directory
-that you might find useful when writing <tt>RUN</tt> lines.</p>
-
-<p>Lastly, you can easily mark a test that is expected to fail on a
-specific platform or with a specific version of llvmgcc by using the
- <tt>XFAIL</tt> keyword. Xfail lines are
-specified in the comments of the test program using <tt>XFAIL</tt>,
-followed by a colon, and one or more regular expressions (separated by
-a comma) that will match against the target triplet or llvmgcc version for the
-machine. You can use * to match all targets. You can specify the major or full
- version (i.e. 3.4) for llvmgcc. Here is an example of an
-<tt>XFAIL</tt> line:</p>
-<pre>
-; XFAIL: darwin,sun,llvmgcc4
-</pre>
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsection"><a name="dgvars">Vars And Substitutions</a></div>
+<div class="doc_text">
+  <p>With a RUN line there are a number of substitutions that are permitted. In
+  general, any Tcl variable that is available in the <tt>substitute</tt> 
+  function (in <tt>test/lib/llvm.exp</tt>) can be substituted into a RUN line.
+  To make a substitution just write the variable's name preceded by a $. 
+  Additionally, for compatibility reasons with previous versions of the test
+  library, certain names can be accessed with an alternate syntax: a % prefix.
+  These alternates are deprecated and may go away in a future version.
+  </p>
+  Here are the available variable names. The alternate syntax is listed in
+  parentheses.</p>
+  <dl style="margin-left: 25px">
+    <dt><b>$test</b> (%s)</dt>
+    <dd>The full path to the test case's source. This is suitable for passing
+    on the command line as the input to an llvm tool.</dd>
+    <dt><b>$srcdir</b></dt>
+    <dd>The source directory from where the "<tt>make check</tt>" was run.</dd>
+    <dt><b>objdir</b></dt>
+    <dd>The object directory that corresponds to the </tt>$srcdir</tt>.</dd>
+    <dt><b>subdir</b></dt>
+    <dd>A partial path from the <tt>test</tt> directory that contains the 
+    sub-directory that contains the test source being executed.</dd>
+    <dt><b>srcroot</b></dt>
+    <dd>The root directory of the LLVM src tree.</dd>
+    <dt><b>objroot</b></dt>
+    <dd>The root directory of the LLVM object tree. This could be the same
+    as the srcroot.</dd>
+    <dt><b>path</b><dt>
+    <dd>The path to the directory that contains the test case source.  This is 
+    for locating any supporting files that are not generated by the test, but 
+    used by the test.</dd>
+    <dt><b>tmp</b></dt>
+    <dd>The path to a temporary file name that could be used for this test case.
+    The file name won't conflict with other test cases. You can append to it if
+    you need multiple temporaries. This is useful as the destination of some
+    redirected output.</dd>
+    <dt><b>llvmlibsdir</b> (%llvmlibsdir)</dt>
+    <dd>The directory where the LLVM libraries are located.</dd>
+    <dt><b>target_triplet</b> (%target_triplet)</dt>
+    <dd>The target triplet that corresponds to the current host machine (the one
+    running the test cases). This should probably be called "host".<dd>
+    <dt><b>prcontext</b> (%prcontext)</dt>
+    <dd>Path to the prcontext tcl script that prints some context around a 
+    line that matches a pattern. This isn't strictly necessary as the test suite
+    is run with its PATH altered to include the test/Scripts directory where
+    the prcontext script is located. Note that this script is similar to 
+    <tt>grep -C</tt> but you should use the <tt>prcontext</tt> script because
+    not all platforms support <tt>grep -C</tt>.</dd>
+    <dt><b>llvmgcc</b> (%llvmgcc)</dt>
+    <dd>The full path to the <tt>llvm-gcc</tt> executable as specified in the
+    configured LLVM environment</dd>
+    <dt><b>llvmgxx</b> (%llvmgxx)</dt>
+    <dd>The full path to the <tt>llvm-gxx</tt> executable as specified in the
+    configured LLVM environment</dd>
+    <dt><b>llvmgcc_version</b> (%llvmgcc_version)</dt>
+    <dd>The full version number of the <tt>llvm-gcc</tt> executable.</dd>
+    <dt><b>llvmgccmajvers</b> (%llvmgccmajvers)</dt>
+    <dd>The major version number of the <tt>llvm-gcc</tt> executable.</dd>
+    <dt><b>gccpath</b></dt>
+    <dd>The full path to the C compiler used to <i>build </i> LLVM. Note that 
+    this might not be gcc.</dd>
+    <dt><b>gxxpath</b></dt>
+    <dd>The full path to the C++ compiler used to <i>build </i> LLVM. Note that 
+    this might not be g++.</dd>
+    <dt><b>compile_c</b> (%compile_c)</dt>
+    <dd>The full command line used to compile LLVM C source  code. This has all 
+    the configured -I, -D and optimization options.</dd>
+    <dt><b>compile_cxx</b> (%compile_cxx)</dt>
+    <dd>The full command used to compile LLVM C++ source  code. This has 
+    all the configured -I, -D and optimization options.</dd>
+    <dt><b>link</b> (%link)</dt> 
+    <dd>This full link command used to link LLVM executables. This has all the
+    configured -I, -L and -l options.</dd>
+    <dt><b>shlibext</b> (%shlibext)</dt>
+    <dd>The suffix for the host platforms share library (dll) files. This
+    includes the period as the first character.</dd>
+  </dl>
+  <p>To add more variables, two things need to be changed. First, add a line in
+  the <tt>test/Makefile</tt> that creates the <tt>site.exp</tt> file. This will
+  "set" the variable as a global in the site.exp file. Second, in the
+  <tt>test/lib/llvm.exp</tt> file, in the substitute proc, add the variable name
+  to the list of "global" declarations at the beginning of the proc. That's it,
+  the variable can then be used in test scripts.</p>
+</div>
+  
+<!-- _______________________________________________________________________ -->
+<div class="doc_subsection"><a name="dgfeatures">Other Features</a></div>
+<div class="doc_text">
+  <p>To make RUN line writing easier, there are several shell scripts located
+  in the <tt>llvm/test/Scripts</tt> directory. For example:</p>
+  <dl>
+    <dt><b>ignore</b></dt>
+    <dd>This script runs its arguments and then always returns 0. This is useful
+    in cases where the test needs to cause a tool to generate an error (e.g. to
+    check the error output). However, any program in a pipeline that returns a
+    non-zero result will cause the test to fail. This script overcomes that 
+    issue and nicely documents that the test case is purposefully ignoring the
+    result code of the tool</dd>
+    <dt><b>not</b></dt>
+    <dd>This script runs its arguments and then inverts the result code from 
+    it. Zero result codes become 1. Non-zero result codes become 0. This is
+    useful to invert the result of a grep. For example "not grep X" means
+    succeed only if you don't find X in the input.</dd>
+  </dl>
+
+  <p>Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.
+  You can easily mark a test as XFAIL just by including  <tt>XFAIL: </tt> on a
+  line near the top of the file. This signals that the test case should succeed
+  if the test fails. Such test cases are counted separately by DejaGnu. To
+  specify an expected fail, use the XFAIL keyword in the comments of the test
+  program followed by a colon and one or more regular expressions (separated by
+  a comma). The regular expressions allow you to XFAIL the test conditionally
+  by host platform. The regular expressions following the : are matched against
+  the target triplet or llvmgcc version number for the host machine. If there is
+  a match, the test is expected to fail. If not, the test is expected to
+  succeed. To XFAIL everywhere just specify <tt>XFAIL: *</tt>. When matching
+  the llvm-gcc version, you can specify the major (e.g. 3) or full version 
+  (i.e. 3.4) number. Here is an example of an <tt>XFAIL</tt> line:</p>
+  <pre>
+   ; XFAIL: darwin,sun,llvmgcc4
+  </pre>
+
+  <p>To make the output more useful, the <tt>llvm_runtest</tt> function wil
+  scan the lines of the test case for ones that contain a pattern that matches
+  PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
+  is related to the test case. The numer after "PR" specifies the LLVM bugzilla
+  number. When a PR number is specified, it will be used in the pass/fail
+  reporting. This is useful to quickly get some context when a test fails.</p>
+
+  <p>Finally, any line that contains "END." will cause the special
+  interpretation of lines to terminate. This is generally done right after the
+  last RUN: line. This has two side effects: (a) it prevents special
+  interpretation of lines that are part of the test program, not the
+  instructions to the test case, and (b) it speeds things up for really big test
+  cases by avoiding interpretation of the remainder of the file.</p>
 
 </div>