Test Case Parameters Description
There is long list of parameters which can be applied to any test case. Here is the description of all possible parameters which can be used to build test scenarios.
Test Report Configuration
There are test report parameters which must be set in the main script file in order to generate HTML report from the test. These parameters have no effect is they are set inside the test case description.
- -version = 2.0.0 sets the test script version. This is used to easily detect incompatibility issues when the test suite loads a script created for more recent version of the suite and may not work properly for this version.
- -output-format = (html | html-content) sets the output format for the test report. There is actually only one format possible right now - HTML. The only difference between these 2 options is that the html format creates full HTML page with HTML header and body. The html-content format on the other hand creates only what is inside
<body/> element. And is used to embed test result inside other HTML content.
- -output-file = "report-file.html" sets the file name for the test report.
- -output-history = (yes | no) sets logging of the all protocol data sent between test suite and the XMPP server. Normally for functional tests it is recommended to set it to yes but for all other tests like performance or load tests it should be set to no.
- -history-format = separate-file sets protocol data logging to a separate file. Currently this is the only possible option.
-output-cols = (5 | 7) Only valid values are:
5: "Test name", "Result", "Test time", "Description" [, "History" ]
7: "Test name", "Result", "Total time", "OK", "Average", "Description" [, "History" ]
- -title = "The title of the report page" This parameter sets the test report title which is placed in the HTML page in the
<title/> element as well as in the first page header.
These parameters can be set on per-test case basis but usually they are set in the main script file to apply them to all test cases.
- -base-ns = "jabber:client" sets the XML name space used for the XML stream in the XMPP connection. Some test cases can be used to test client to server protocol as well as server to server protocol and possibly different protocols added in the future.
- -debug switches debugging mode on. All the communication between the test suite and the server is printed out to the text console and all other debugging information including java exceptions are displayed as well. It is especially useful when some test fails and you want to find out why.
- -debug-on-error switches on debugging mode on error detection. Normally debug output generates lots of message which makes the output very hard to read. Especially in the performance tests not only you can read fast scrolling lines of the protocol data but also it slows the test down. This option however turns debugging off if everything is working well and then generates debug output if any test error us detected.
- -def-auth = (auth-plain | auth-digest | auth-sasl) sets the default authentication method for the user connection.
- -def-stream = (stream-client | stream-server | stream-component | stream-bosh) sets the connection stream to be tested and the name space for the connection.
- -host = "host.name" the vhost name the tested server runs for. It may be the real DNS name or just configured for testing purposes hostname. It must match however the server configuration.
- -keys-file = "certs/keystore" sets the location of the keys store file. No need to touch it.
- -keys-file-password = keystore sets the password for the keystore file. Normally you don’t have to touch it.
- -serverip = "127.0.0.1" defines the XMPP server IP address. You may omit this parameter and then the IP address will be determined automatically based on the server DNS address. However if the DNS address can not be correctly resolved or if you run tests on the localhost you can use this parameter to enforce the IP address.
- -socket-wait = 10000 sets the network socket timeout in milliseconds that is maximum time the test suite will wait for the response from the server. You may want to increase the timeout for some specific tests which require lots of computation or database activity on the server. Normally 10 seconds is enough for most cases.
- -stop-on-fail = true causes the script to terminate all actions on the first failed test case. It helps diagnosing the server state at the failure point.
- -trust-file = "certs/client_truststore" sets the file name for the client trust store file. No need to change it.
- -trust-file-password = truststore sets the password for the trust store file. Normally you don’t have to touch it.
- -user-name = tester sets the user name used for the XMPP connections between the test suite and the XMPP server. It is usually set globally the same for all tests and for some tests like receiving the server configuration you may want to use a different account (with admin permissions). Then you can set a different user for this specific test case.
- -user-pass = tester-password sets the password for the user used for the XMPP connection between the test suite and the XMPP server.
- -user-resr = resource sets the user JID resource part for the XMPP connection between the test suite and the XMPP server.
Test parameters which are normally set on per-test case basis and apply only to the test they are set for and all inherited tests. Some of the parameters though are applied only to inherited test cases. Please look in the description below to find more details.
- -active-connection is a similar parameter to -on-one-socket option. If set the suite doesn’t close the network socket and if the test is run in loop each loop run re-uses the network connection. Unlike in the -on-one-socket mode the whole test is executed on each run including XMPP stream initialization and user authentication. This option is currently not recommended in a normal use. It is useful only to debug the server behavior in very special use cases.
- -background executes the test in a separate thread in background and immediately returns control to the test suite program without waiting for the test to complete. Default behavior is to execute all tests sequentially and run next test when previous one has been completed. This parameter however allows to run tests concurrently. This a bit similar option to the -daemon parameter. The daemon test/task however is ignored completely and results from the daemon are not collected where the background test is a normal test which is run concurrently with another one or possibly many other tests.
- -daemon creates a task running in background in a separate thread. Such a test runs infinitely as a daemon, it is not recorded in the test report and it’s result is not calculated. The purpose of such test/task is to work as a helper for other test cases. A good example of such daemon test is message responder - the test which runs under a different user name and waits for messages and responding to the sender.
- -delay = 1000 sets the waiting time in milliseconds after the test case is completed. You may use it if you want to introduce short delay between each test cases run in the loop or if you start the helper daemon thread and you have to add the delay to make sure it is ready to work before next real test starts sending requests to the daemon.
- -expect-type = error sets the type for a packet expected as a response. Some test cases like message sender expects sometimes response with the same type it has sent the packet ( chat ) but in some other cases when it sends a message to a user who has privacy lists set to block messages the response should be with an error. This way we can use the same test cases for testing different responses scenarios.
- -loop = 10 sets the number of times the test (and all inherited tests) are repeated. You can use a $(loop) pseudo-variable to obtain and use the current loop run number. This is useful if you want to run every loop run for a different user name like registering 10 different user accounts. To do this you stick the $(loop) variable to the user name string: -user-name = "nick_name_$(loop)".
- -loop-delay = 10 sets a delay in milliseconds between each individual loop run for the tests which is run multiple times. This is similar parameter to the -delay one but the -delay option introduces a delay after the whole test (or all loop runs) has been completed. The loop delay options adds waiting time between each run of the looped test.
- -loop-start = 5 sets the loop starting value. It doesn’t affect number of loop runs in a any way. It only affects the value of the $(loop) variable. Let’s say you want to run a load test for the server with 100k concurrent users and you want to run the test from 3 different machines. To make sure each machine uses distinct user accounts you have to set a different -loop-start parameter on each to prevent from overlapping.
- -messages = 10 sets the number of messages to send to the server. This is another way of looping the test. Instead of repeating the whole test with opening network connection, XMPP stream, authentication and so on it causes only to send the message this many times. This parameters is accepted by some test cases only which send messages. For the messages listeners - test cases which is supposed to respond to the messages the number set here specifies how many times the the response must be sent before the test successfully terminates it’s work.
- -multi-thread option causes to run the test case and all inherited in all levels test cases in separate threads. Normally the test case where you put the parameter doesn’t have a test ID (what you put between @ and : characters so it doesn’t run a test on it’s own. Instead it contains a series of test cases inside which are then run in a separate thread each. This is a key parameter to run tests for many concurrent users. (Not a load tests though.) For example you can see whether the server behaves correctly when 5 simultaneous modifies their roster. The execution time all inherited tests run in a separate threads is added together and also results from each individual test is calculated and added to the total main test results.
- -no-record is used for kind of configuration tests (tasks) which are used to prepare the XMPP server or database for later tests. As an example can be creation of the test user account which is later on used for the roster tests. Usually you don’t want to include such tests in the test report and using this parameter you essentially exclude the test from the report. The test and the result however shows in the command line output so you can still track what is really going on.
- -on-one-socket is a modifier for a looped test case. Normally when we switch looping on using -loop parameter the suite resets the state, closes the network socket and runs the test from the very beginning including opening network socket, XMPP stream, authentication and so on. This parameter however changes this behavior. The network socket is not closed when the test run is completed (successfully) and next run executes only the last part of the test omitting the XMPP stream initialization, authentication and all others but last. This is useful when you want to send many messages to the server (although this effect may be accomplished using -messages parameter as well) or registering many user accounts on the server, unregistering user accounts and any other which might make sense repeating many times.
- -port = 5223 this parameter is similar to the IP address setting and can be also set globally for all tests. Normally however you set it for a selected tests only to check SSL connection. For all other tests default port number is used. Therefore this parameters has been included in this section instead of "Basic test parameters".
- -presence this parameter enables sending initial presence with positive priority after connection and binding the session.
- -repeat-script = 100 and -repeat-wait = 10 are 2 parameters are specific to the common test cases. (The test cases which reads the test input/output data from the pseudo-xml text file. The first parameter is another variation of test looping. It sets how many times the test has to be repeated. It works very much like the -on-one-socket parameter. The only difference is that the common test can preserve some internal states between runs and therefore it has more control over the data. The second parameter sets the timeout in milliseconds to wait/delay between each individual test run and it is a very similar parameter to the -delay one but it sets a timeout inside the common test instead.
- -source-file = "dir/path/to/file.cot" is a parameter to set the "common test" script file. The common test is a test cases which depends on the authentication test case and can read data to send and responses to expect from the text file. The "cot" file is a pseudo-xml file with stanzas to send and stanzas to expect. The the test cases compares the received packets with those in the text file and reports the test result. This is usually a more convenient way to write a new test cases than coding them in Java.
- -time-out-ok is set for a test case when we expect socket timeout as a correct result from the test case. Normally the timeout means that the test failed and there was no response from the server at all or the response was incorrect. For some tests however (like sending a message to the user who is blocking messages through privacy lists) the timeout is the desired correct test result.
- -to-jid = "email@example.com" sets the destination address for packets sending packets somewhere. As an example is the test case sending <message /> packet. You can set the destination address for the packet. Mind, normally every test expects some response for the data sent so make sure the destination end-point will send back the data expected by the test case.