²ÝÁñÉçÇø

Introduction: Ensuring the Validity of Tests

Next to providing one of the best test tools in the embedded sector, we also test software products on behalf of customers from the automotive industry (including driver assistance functions, drive components, control software for charging and battery systems

Over time, we have also experienced bugs in the testing process. To avoid process errors, we have developed various strategies and methods. Always with the aim ¨C to be able to provide our customers with high-quality statements on their developments quickly. 

In the following, we would like to explain one of these methods in detail. It was developed by our test engineers and is used daily in practice. 

Incorrect links can occur in different ways: 

1. the tester creates a wrong link between test case and requirements 
2. an existing link loses its significance over time due to a change in the test item

There is a simple and quickly implementable solution that solves this problem.  

In our method, each test case is reported as failed if it does not test the linked requirement correctly. A failure due to an incorrect link is detailed in the report. 

Our approach is essentially based on the possibility to define test data and expected values separately.  

In TPT, the expected results of a test item ¨C in this context we also speak of the test oracle ¨C can be described with the help of Assesslets. Assesslets can be used for the evaluation of several test cases simultaneously.

Implementation: Five-Step Process

Step 1: Import of Requirements into TPT 

Import can be done in several ways. For this method, it is only relevant that the requirements are available in TPT.

Step 2: Creation of 1 Assesslet per Requirement

The purpose of an Assesslet is to specify the expected behavior of a test object under defined conditions. This single-source-of-truth definition can then be used for multiple test cases.  

How to do this?

Create a new script Assesslet for each requirement in the Assesslet folder, name it accordingly and implement it.

The implementation of an Assesslet contains the following elements:

1.define conditions or case distinctions (usually derived from the requirements)
2. define the expected value of each condition (some are simple, some are complex) 
3. add an annotation of which requirement is covered by which expected value 

For our lights control example above, here is a reference implementation for an Assesslet that checks the requirement with the ID 2018 ¡°If light switch is ON, then headlight shall immediately be ON¡±: 

Step 5: Linking of Test Cases with Imported Requirements

The linking of requirements with tests or vice versa can be done by drag and drop. Select some test cases and drag them over the requirements. Done.

Benefits: Advantages of the Method

The advantage of this procedure is that incorrect links in the reporting are immediately and easily visible. Each incorrectly linked test case is identified as a failed test case in the report.  

The report thus gives the user a quick overview of whether relevant test cases have been created for all requirements. At the same time, this increases productivity, as analyses of the degree of completion can be omitted. 

Guidelines: Considerations for Applying the Method

Assesslets should be checked for correctness and consistency with requirements. Only if the Assesslets are correct do they have any significance. This is the actual engineering work in testing. We cannot (yet) take this off your hands. 

Further hints and recommendations: 

In some of our projects, we do not link the script Assesslets directly to requirements. However, a mapping is done by naming convention: in that each requirement-testing script Assesslet has the following structure ¡°Ass_¡± & <Requirements-ID>. The requirements for bidirectional traceability (e.g. from Automotive SPICE) are thus fulfilled in principle, as a pairing can be determined at any time. 

Summary

Our method for ensuring the significance of tests is compliant with the requirements of Automotive SPICE and ISO26262. 

In its application, it requires basic functions of the test automation used, such as the separation of test data for stimulation and a separate definition of expected behavior for the test object.  

We have been using this method successfully in safety-critical automotive projects for several years.  

Our engineers are convinced by the intuitive procedure and no longer want to do without it, among other things because time-consuming manual checks of the correctness of links can be omitted.  

The effort required to write scripts and check them for the correctness of Assesslets and requirements is manageable and significantly lower than alternative measures such as reviews & walkthroughs.