Cross Reference: /test/stcnv/usr/src/suites/net/nc/README

#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#

#
# Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident	"@(#)README	1.3	09/03/15 SMI"
#

#
# README file for nc(1) CTI-TET test suite.
#

1.  Introduction
2.  Test coverage
3.  System requirements
4.  System changes
5.  Installation
6.  Running the tests
7.  Building, configuring and running the tests from source code
8.  Notes
9.  Future work
10. Information for developers
11. Contact us


1. Introduction
---------------

The Netcat test suite residing here is set of basic functionality tests
for Netcat implementation shipped with Solaris.  It is based on unit
tests developed for pre-RTI testing of nc(1) integration into ON.

The test suite uses CTI-TET, Common Test Infrastructure (CTI) for Test
Environment Toolkit (TET).  Please see the following link for more
information about CTI-TET both in OpenSolaris context and in general:

  http://opensolaris.org/os/community/testing/testsuites/ctifortet/

The test suite is intended to be run by anyone from OpenSolaris community
who wants to contribute a bug fix or RFE to Netcat shipped with Solaris.

If you want to make a change in this test suite be sure you read "Information
for developers" section before progressing any further.

2. Test coverage
----------------

With every RFE, new test purpose or test case should be added.  The same
for serious bugs.  Most of the functionality is covered by the tests with the
exception of features which require setup of servers (e.g. SOCKS support).

All tests should PASS.  If not, please report a bug.

3. System requirements
----------------------

3.1 Access to your local machine
--------------------------------

All tests run on local machine only.  Root access is not required
unless the tests are set to use privileged port (see section 8) or test
suite package is installed/removed.

The test suite can be run both on SPARC and i386.  Those two are the only
relevant modes because we use only shell scripts in this test suite.

3.2 Network interfaces
----------------------

All tests need two loopback interface (lo0) instances, configured with
IPv4 (127.0.0.1) and IPv6 address (::1/128).

4. System changes
-----------------

This test suite does not currently modify system configuration in any way.

5. Installation
---------------

This section is for those who want to install the test suite package.  If you
want to run the tests from source code, see section 7.

The package will be installed into /opt.  It is recommended that you install
the package from scratch, rather than on top of an existing installation.
Thus, if an old version of the tests is installed, remove it and install it
again:

  root# pkgrm SUNWstc-net-nc
  root# pkgadd -d <package-location> SUNWstc-net-nc

Note: SUNWstc-tetlite which contains CTI-TET framework tools is prerequisite
      for SUNWstc-net-nc.

6. Running the tests
--------------------

Run the tests from the package like this:

  $ cd /opt/SUNWstc-net-nc && \
    export CTI_ROOT=/opt/SUNWstc-tetlite/contrib/ctitools; \
    export PATH=$PATH:$CTI_ROOT/bin; \
    run_test nc

The results files will be stored in /var/tmp/results.<PID> directory.
The default journal file name will be /var/tmp/results.<PID>/testlog
where PID is pid of run_test process.

6.1 Running single test case
----------------------------

To run single test case (which can contain several test purposes)
use additional argument for run_test, e.g. for Tflag test case:

  $ run_test nc Tflag

6.2 Running predefined scenarios
---------------------------------

nc test suite is based on CTI-TET framework.  It supports scenarios
and the test suite has predefined the following scenarios:

  - short
  - longer

Scenario definitions reside in the 'tet_scen' file.

To run e.g. the short scenario use the following command:

  $ run_test nc short

6.3  Testing custom nc program
------------------------------

To test nc program stored at different location than the default
/usr/bin/nc, set STC2_NC_PATH environment variable to
an alternative location.

7. Building, configuring and running the tests from source code
---------------------------------------------------------------

- checkout the test suite source, if desired.  For example,

  $ hg clone ssh://anon@hg.opensolaris.org/hg/test-dev/stcnv-gate

  If the previous Mercurial command was performed in directory
  /local/userfoo/testing (say) then the source code of the nc
  test suite will be placed at 'usr/src/suites/net/nc'.  For
  the rest of this discussion, we will use the string <WS_ROOT>
  to refer to /local/userfoo/testing.

- to build the tests, set CTI-TET variables and run 'make':

  $ export CTI_ROOT=/opt/SUNWstc-tetlite/contrib/ctitools; \
    export PATH=$PATH:$CTI_ROOT/bin; \
    cd <WS_ROOT>/usr/src/suites/net/nc \
    && make

- run the test suite

  $ cd <WS_ROOT>/usr/src/suites/net/nc && run_test nc

- examine results

  $ less /var/tmp/results.<number>/testlog

  Full path to the journal file is printed by run_test before start
  of the tests and at the end of the run.

- at some later time the journal can be displayed in comprehensive form
  via the following command:

  $ report_writer /var/tmp/results.<number>/testlog /tmp/report.$$

- you can also build the SUNWstc-net-nc package:

  $ cd <WS_ROOT>/usr/src/suites/net/nc && make package

  The package will be constructed in
  <WS_ROOT>/packages/`uname -p`/SUNWstc-net-nc/ directory.

  After that it can be converted to the cpio package format via:

  $ pkgtrans -s <WS_ROOT>/packages/`uname -p` \
	/tmp/SUNWstc-net-nc.pkg SUNWstc-net-nc

  The result package /tmp/SUNWstc-net-nc.pkg can be simply copied to
  remote sytem and installed:

  $ pkgadd -d SUNWstc-net-nc.pkg all

8. Notes
--------

8.1 Solution for bad latency to the tools
-----------------------------------------

If you run this test suite far from where the gate is exported from it
might be very slow.  In that case it is enough just to recursively copy
tetlite directory from proto area to your local proto/tools directory
and then to set PATH accordingly.

8.2 Internal variables
----------------------

If you need to test nc binary which is not in default location (for
example you want to test it without copying it from a build server)
or need to change port used for testing there is a list of variables
that you can set and export before running the test suite, together
with their default values:

	variable			default value
	------------------------------+----------------
	STC2_NC_PATH			/usr/bin/nc
	STC2_NC_TEST_PORT 		4444
	STC2_NC_TEST_SOURCE_PORT_FIRST	55555
	STC2_NC_TEST_SOURCE_PORT_LAST	55566
	STC2_NC_WATCHMALLOC 		1
	STC2_NC_DEBUG			0
	STC2_NC_STARTUP_TIMEOUT		10

8.3 Memory checking
-------------------

By default every nc command is run with watchmalloc(3MALLOC) checking.
Setting STC2_NC_WATCHMALLOC environment variable to 0 will cause nc to be run
without the checks.

8.4 Binding to ports and privileges
-----------------------------------

All of the tests utilizing TCP/UDP connections are using default port
number STC2_NC_TEST_PORT which is set to unprivileged port.

This could lead to setup phase (for test purposes which start nc in server
mode first and then perform the test by connecting with nc client process)
failures where bind(3SOCKET) call fails because the port is already
bound by different process.  To overcome this limitation,
the default value can be overriden by setting STC2_NC_TEST_PORT
environment variable to privileged port (smaller than 1024) and
running the test suite with net_privaddr privilege (e.g. under root).

STC2_NC_TEST_SOURCE_PORT_{FIRST,LAST} port range is used (by calling
try_connect_srcport()) in test purposes using -p flag for setting source
TCP port such as those in vflag or pflag test cases.  Same rules apply w.r.t.
privileged port number as above.

9. Future work
--------------

Aside from better test suite coverage, changes should be done which
would make the test suite more robust:

  - addition of timeouts for test purposes
  - detection of bind(3socket) failures and restart or using
    net_privaddr privilege for running all tests with privileged port

As with nc(1), patches and contributions are welcome.

10. Information for developers
------------------------------

If you want or need to debug the test suite, set STC2_NC_DEBUG variable to
positive value before running the tests.  It will put some more
information into the journal.

nc should be executed either using predefined functions such as
listen_wrapper() or start_nc().  In some cases it is needed to execute
it directly.  In such cases it should be started like this:

  $ eval $NC arguments

This is because of watchmalloc(3MALLOC) checking (see NC_PRELOADS
definition in include/vars for details).

10.1 Start and end of a test purpose
------------------------------------

Each test purpose should begin with a call to init_test() and should
end with a call to finish_test().

init_test() prints test purpose ID and description to journal file
and sets test_port/test_source_port internals variable to
STC2_NC_TEST_PORT/STC2_NC_TEST_SOURCE_PORT values, respectively.

finish_test() performs cleanup by killing any potentional left-over nc
processes via stop_nc() which uses internal variable NC_PIDFILE.

Also note that tet_startup,tet_cleanup variables are set to common_startup
and common_cleanup (defined in src/lib/ksh/common_funcs.ksh), respectively.

10.2 Internal variables
-----------------------

start_nc() sets two internal variables which accessible from test purposes:

	Name	     Meaning
	------------+----------------------------------------------
	NC_PIDFILE   file containing pid of nc process
	NC_ERRFILE   file containing stderr produced by nc process

Both files have to be removed when the program started by start_nc() exits.
Normally, finish_test() takes care of that but in case start_nc()
is executed with custom pid file and error file the caller has to remove
them.

To remove custom pid files and kill the processes associated with them
finish_test() can be passed the variables.

10.3 Adding new tests
---------------------

10.3.1 Adding new test purpose
------------------------------

The following steps are needed:

  1. create the file for the test purpose in given test case directory
     e.g. tests/uflag/tp_uflag_004_neg.ksh

  2. add new test purpose to TARGET_KSH variable in Makefile
     for given test case

  3. include the test purpose in test case script
     (e.g. tests/uflag/tc_uflag.ksh)

10.3.2 Adding new test case
---------------------------

To add new test case the following steps are needed:

  1. add new subdirectory in tests/ directory with all necessary files
     (Makefile, .ksh script for the test case, files for the test purposes)

  2. modify the following files:

     - tests/Makefile
       append new test case directory to SUBDIRS variable definition

     - tet_scen
       add a new test case name to appropriate scenario and define
       the test case with a path to the script

     - Targetdirs
       add new directory for the test case to ROOT.TEST variable

10.4 Code style
---------------

When writing new ksh code make sure that the shell style rules are
followed:

  http://www.opensolaris.org/os/project/shell/shellstyle/

All tests should run with both ksh88 and ksh93.

10.5 Using internal functions
-----------------------------

Temporary files or directories should be created using the nc_mktemp()
or nc_mktemp_dir() routines, respectively. These functions use common
prefix from the NC_TMPFILE_PREFIX variable defined in include/vars file.

Whenever nc_mktemp() is used directly, its output should be checked and
if it fails (empty string is returned) the caller should bail out.
For example (assuming we are in a function which returns 1 on failure):

    typeset -r tmp_out=$( nc_mktemp test_purpose.out )
    [[ -z $tmp_out ]] && return 1

In a test purpose, finish_test() should be called before returning.

Upon failure, nc_mktemp() will set the result of the currently running test
purpose to UNRESOLVED.

Assuming all test cases/purposes use these functions it should be easy to
check for leftover temporary files at the end of test suite run (signaling
there is a problem in test suite code).  This command will perform the
check (with current definition of the common prefix used for
temporary files/directories) using the NC_TMPFILE_PREFIX definition from
include/vars:

  $ /usr/bin/find /tmp/ -name 'nc_tet-*' 2>/dev/null

The output of the command should be empty.

11. Contact us
--------------

Please e-mail the STC_CONTACT listed in the STC.INFO file if you want to
contact us - your feedback is appreciated.  If you want to file a bug, use
stc/net/nc category.