Automated Testing Support
dsTest, dsClient Terminal, and dsClient Desktop all provide various features in support of automated testing from test configuration through reporting test results. How you choose to combine those features depends on the automation tools available to you and on your test goals. In this topic we'll discuss the various automation options available as well as how you can optimize your test configurations and process.
Which Method to Use?
There are several ways you can control dsTest through automation, each with its own advantages. If you choose to work with XML, whether directly or through dsClient Terminal, you can use dsClient Desktop in headless mode (i.e., no visual display) to generate the XML for your automated tests from your desktop workspaces during the test cycle. We use this feature in our build process to generate the XML for all of the example test configurations.
This is the simplest option for a lightweight approach. You can use shell scripts or other scripting languages to control dsTest through dsClient Terminal commands and capture the responses for later inspection, or parse the textual responses into a test report. All of the options discussed below are available with this approach, and you can work with the same XML configurations that you manually execute.
Read more about automating tests with dsClient Terminal.
For a more robust approach you could use the same mechanism used by the dsClient applications and directly communicate with dsTest using XML. With this approach you'll need to use a script language or automation tool that provides TCP socket management. As with the shell script approach you would be using your XML configurations with the added advantage that you would also be dealing with structured responses rather than parsing text. This approach would simplify data collection, enabling you to produce more comprehensive test reports as well as allowing you to design a more abstract automation approach.
Read more about using XML/TCP.
dsTest also provides a RESTful API that enables full control of your tests. If your automation tool supports REST this may be the option for you. When you use this API dsTest will respond with JSON content. We provide a utility that converts dsTest XML documents to JSON.
Read more about using the RESTful API.
A primary goal of test automation is to implement constant testing. In our lab we have over one hundred platforms involved in round-the-clock feature and regression testing. In order to achieve this goal and also have platforms available for test development and troubleshooting, we optimized our configurations using the strategies listed below.
Portability - develop an alias strategy that will allow any test to be run on any compatible platform.
Scalability - use variables and formulas to scale tests up or down.
Efficiency - optimize memory and licensed resource utilization.
If your implementation will utilize a floating license and a pool of platforms, it is vital that you understand how your licensed resources are consumed and released. Contention for those resources can foul your test cycle with failures that have nothing to do with the DUT.
If your test suite includes short conformance tests or load tests with dozens or hundreds of nodes, consider replacing the subscribers to load new tests rather than disconnecting and deleting your nodes at the end of each test.
Collecting Test Results
dsTest provides various types of notifications that can alert you to adverse conditions while a test is running. Failure and status change notifications can alert you to hard failures while threshold notifications can be useful for determining when a DUT's performance began to degrade. In addition, status change notifications inform you when your nodes have started and when they have successfully connected to peers, which your process could use as a trigger to start the test's actions. Your automation tool can, in most cases, register for the desired notifications at the beginning of the test cycle and receive (or collect in the case of the RESTful API) those notifications as all of your tests are executed.
While you can explicitly deregister from notifications, dsTest will automatically deregister a client from all notifications when that client disconnects from dsTest. If you're using dsClient Terminal notifications will be written to standard out until you exit dsClient. If you're using XML/TCP dsTest will transmit notifications as long as the TCP socket is active. With the RESTful API dsTest will queue notifications until the client collects them with a GET request as long as the HTTP connection is maintained using a Connection: keep-alive header.
You can also configure your conformance tests to capture Subscriber Event Logs, as they can contain message content, or you can use dsTest's Packet Capture feature to capture traffic for any type of test.
Operational Measurements (OMs)
dsTest's comprehensive OMs are the definitive measure of your test. As your automation tool runs each test it should also collect the measurements associated with each test. At the very least it should collect the OM databases at the appropriate times, taking care not to corrupt them and waiting for them to become available. You could also use the powerful diag om command to collect and/or summarize key OMs periodically and at the end of a test to determine the outcome.
Here again, dsClient Desktop can play a part in your automated testing. With it you can define pass/fail criteria for your tests and then run the application in headless mode to produce HTML Test Results after a test is complete, composed using templates that you can customize. dsClient will analyze the measurements from the referenced database and write the SmartReport dashboard summaries indicated by your template to the file you specify. You can include any other HTML content in your template - a description of the test, for example - and you can use your own pass/fail icons or branding. Your automation tool could then email the results to interested parties.
Putting it all Together
Using the tools and features described above, you should be able to construct a process similar to the process depicted.
Register for the desired notifications. Failure and status changes notifications are recommended.
Load any aliases required for the next test or adjust the aliases previously loaded as needed.
Load the test configuration.
Start the test. Typically that would be with at least one action or SmartAction command. These actions may kick off SmartEvents state machines or execute node actions. You can execute additional actions at any time. If you're using XML configurations, these would be Stored Commands, loaded with your configuration, that you reference by name. If you're using REST, each command would be issued using a PUT request.
Monitor the test by assessing notifications received or collected and possibly by querying OMs. You may choose to stop the test immediately if error indications are present.
Stop the test by either stopping and deleting the configured nodes or by stopping any remaining actions and releasing the subscribers involved depending on your testing strategy. If you're using the latter strategy of replacing subscribers rather than replacing nodes, you must also issue a command to close the OM database if you want to isolate the measurements for each test.
Collect the final test results. These results would include at least a final OM report - from OM query commands, HTML test results, or the OM database itself. In the case of either of the latter two options, check for the presence of a database lock file before copying, moving, or accessing the database in any way.
If this is the end of the test cycle you can deregister from notifications explicitly or simply close the TCP socket (XML/TCP), the HTTP connection (RESTful API), or exit dsClient Terminal (shell script). dsTest will automatically deregister a client from all notifications when that client disconnects from dsTest.