Testing Basics

A test case when testing in Kompics is a method describing the expected behavior of the component under test (CUT) for a given scenario. This behavior is specified in terms of the expected sequence of events that go in and out of the component. You can think of a behavior simply as a sequence of events while a test case describes a set of behaviors, any of which the CUT’s runtime execution must match in order for a test case to be a successful. The framework executes an instance of the CUT, observing the events that actually occur at runtime and matching them against the next expected event according to the test case. If a match is unsuccessful then the test case fails immediately. Kompics allows you to test your component either in a realistic environment, complete isolation and anywhere in between. A test case may also inject events into the environment.

The TestContext class implements the DSL for writing test cases in this manner.

In this part of the tutorial we add an id field to identify our Ping and Pong events using java.util.Comparator instances. We also create two components Pinger and Ponger.

package se.sics.test;
import se.sics.kompics.KompicsEvent;
import java.util.Comparator;

public class Ping implements KompicsEvent {
  public int id;
  public Ping(int id) {
    this.id = id;
  }
  public static Comparator<Ping> comparator = new Comparator<Ping>() {
    public int compare(Ping p1, Ping p2) {
      return p1.id - p2.id;
    }
  };
}

package se.sics.test;
import se.sics.kompics.KompicsEvent;
import java.util.Comparator;

public class Pong implements KompicsEvent {
  public int id;
  public Pong(int id) {
    this.id = id;
  }
  public static Comparator<Pong> comparator = new Comparator<Pong>() {
    public int compare(Pong p1, Pong p2) {
      return p1.id - p2.id;
    }
  };
}

package se.sics.test;
import se.sics.kompics.PortType;

public class PingPongPort extends PortType {{
  request(Ping.class);
  indication(Pong.class);
}}

package se.sics.test;
import se.sics.kompics.ComponentDefinition;
import se.sics.kompics.Handler;
import se.sics.kompics.Positive;
import se.sics.kompics.Start;

public class Pinger extends ComponentDefinition {
  public static int pongsReceived = 0;
  Positive<PingPongPort> ppp = requires(PingPongPort.class);
  Handler<Start> startHandler = new Handler<Start>(){
    public void handle(Start event) {
      trigger(new Ping(8), ppp);
    }
  };
  Handler<Pong> pongHandler = new Handler<Pong>() {
    public void handle(Pong pong) {
      pongsReceived++;
    }
  };
  {
    subscribe(startHandler, control);
    subscribe(pongHandler, ppp);
  }
}

package se.sics.test;
import se.sics.kompics.ComponentDefinition;
import se.sics.kompics.Handler;
import se.sics.kompics.Negative;

public class Ponger extends ComponentDefinition {
  public static int pingsReceived = 0;
  Negative<PingPongPort> ppp = provides(PingPongPort.class);
  Handler<Ping> pingHandler = new Handler<Ping>() {
    public void handle(Ping ping) {
      pingsReceived++;
      if (ping.id == 0) {
        trigger(new Pong(1), ppp);
        trigger(new Pong(2), ppp);
      } else {
        // echo the id
        trigger(new Pong(ping.id), ppp);
      }
    }
  };
  {
    subscribe(pingHandler, ppp);
  }
}

The behavior of Pinger is straightforward - it triggers a single Ping (with id 8) event when started. On the other hand, Ponger listens for incoming Ping s and responds with a Pong of the same id unless the id happens to be 0 in which case it responds with two Pongs of id s 1, 2 in that order.

The following example shows a test case for this behavior of Ponger. The entire test case can also be downloaded here.

package se.sics.test;

import se.sics.kompics.Component;
import se.sics.kompics.Positive;
import se.sics.kompics.testing.Direction;
import se.sics.kompics.testing.TestContext;
import static org.junit.Assert.assertTrue;

public class Main {
  public static void main(String[] args) { // basicsExample
    TestContext<Ponger> tc = TestContext.newInstance(Ponger.class);
    Component ponger = tc.getComponentUnderTest();
    Component pinger = tc.create(Pinger.class);
    Positive<PingPongPort> pongerPort = ponger.getPositive( PingPongPort.class);
    tc.connect(pongerPort, pinger.getNegative(PingPongPort.class));
    tc.setComparator(Ping.class, Ping.comparator);
    tc.setComparator(Pong.class, Pong.comparator);
    // setup done
    tc.body()
        .expect(new Ping(8), pongerPort, Direction.IN)          // a
        .expect(new Pong(8), pongerPort, Direction.OUT)         // b
        .trigger(new Ping(0), pongerPort)
        .either()
            .expect(new Pong(1), pongerPort, Direction.OUT)     // c
            .expect(new Pong(2), pongerPort, Direction.OUT)     // d
        .or()
            .expect(new Pong(3), pongerPort, Direction.OUT)     // e
            .expect(new Pong(4), pongerPort, Direction.OUT)     // f
        .end();
    assertTrue(tc.check());
  }
}

The newInstance method creates a new TestContext specifically for our Ponger component - an instance of which is retrieved by calling getComponentUnderTest. The framework begins observing and matching events only when the check method is called. As such, this method should only be called finally when the test case has been specified.

Method calls in the body of the test case form statements that instruct the framework to perform specified actions.

The expect method causes the framework to observe a single event and match it against the specified arguments - failing the test case if no event is observed or the match is not successful. The body of our test case says that first, we expect a Ping event, with an id 8, going into the Ponger’s port (as sent by our connected Pinger component). This should then be followed by a Pong response with the same id coming out of it.

The trigger method causes the framework to trigger an event on the specified port. In our example we next trigger an event on the Ponger’s port. In a sense, we mock the behavior of Pinger by providing stimuli (a Ping with id 0) to our CUT just as if it were sent by Pinger.

Next, using a conditional statement, our test case says that the framework should expect either a Pong of id 1 followed by another with id 2 or a Pong 3 followed by another with id 4. Depending on what events actually show up at runtime, the framework chooses which branch of the statement to match. So if event Pong 1 occurs then it is matched successfully and the rest of the either branch is executed (the framework tries to observe and match an equivalent event to Pong 2). Here, we know that this will be the case and that the test case should pass. Have Ponger send different id s in response to a Ping 0 or change the expected events and see that the test case fails instead.

If we assign each event a symbol, like in our example the letters a-f as shown using comments, then the set of behaviors that our test case describes is {abcd, abef} so that the test cases passes if either of these sequences are observed and no other events occur. An equivalent notation to describe this set is the regular expression ab(cd|ef), where the either-or conditional statement is used as the boolean or operator |. The DSL also explicitly supports constructs for the quantification operators of regular expressions * and {n} for matching any number of occurrences of a sequence. This might be helpful when thinking of concise ways to write test cases.

Setup

As shown in the previous example, we might need to set up our test environment before we begin matching events. Any setup activites must be carried out before calling body for the first time - signalling that we would now like to start describing the expected behaviors of our component. We refer to this part of the test case (before the body is called) as the setup header of the test case.

Within the setup header, the following methods can be called:

create

Create instances of other components that may communicate with the CUT.

connect

Connect any two ports.

setComparator

Register a java.util.Comparator instance for an event class to be used when matching events of that type.

setDefaultAction

By default unmatched events cause the test case to fail. This method overrides this behavior for desired events by registering a Function.

setTimeout

To ensure test case termination and correctness, the framework needs to know how long to wait for an event to occur before making a decision. This method overrides the default timeout.

Modes

Using method calls to write our test cases necessarily mean that the order in which the methods are called matter. But more importantly, some combinations of methods calls are simply not legal. The DSL uses the concept of modes to enforce the legality of test cases by assigning each method a set of valid modes in which they can be used. Additionally, calling certain method calls change the current mode of the test case and calling a method in an illegal mode always throws an exception. The possible modes can be found here se.sics.kompics.testing.MODE. Please refer to the javadoc se.sics.kompics.testing.TestContext for a list of all methods and their valid modes.

Conditionals

A conditional is a compound statement of the form either() - or() - end() where - represents the either and or branches of the conditional and may contain other statements that would normally appear in the body of a test case, including other conditionals. When either() is called, it changes the current mode X of the test case to CONDITIONAL and when the matching end() is called, the current mode is restored to X.

In a typical conditional statement like the if-then(-else) constructs of most programming languages, the necessary condition for executing the if or else branch is always clear - If the boolean condition is true then the if branch is executed otherwise the else branch is executed. In our either-or construct the condition for matching the branches is inferred from the actual events that are observed at runtime. If an observed event e is matched by the first statement in the either branch then the framework continues to execute statements in that branch and the same goes for the or branch. If the first statements in both branches match e the framework proceeds to execute the next statements of both branches and so on until there is a divergence (a branch’s statement fails to match an observed event).

Blocks

The DSL uses the concepts of blocks as a way to split the statements in a test case into partitions that make up a block. By doing so we may specify instructions to the framework like executing the statements of a block a particular number of times, whitelisting or blacklisting events that can occur while the framework is executing statements within a block etc.

The following code snippet shows an example of a block.

tc.repeat(5)
     // block header
  .body()
     // block body
    .expect(new Ping(1), pingerPort, IN) // a
    .expect(new Ping(2), pingerPort, IN) // b
  .end();

A block is of the form repeat() A body() B end() where A and B are the header and body respectively of the block. The statements in a block body is executed sequentially by the framework while those at the header are used to filter out events that may occur while the framework is executing the statements in the body. Once repeat is called, the test case enters HEADER mode from some previous mode X and statements allowed in the block header can be called (these statements are optional as a block header may be empty). Calling body() moves the test case into BODY mode and finally calling the matching end() restores the current mode back to X.

In our example, the repeat(5) instructs the framework to execute the statements in the block body exactly 5 times in succession. As a single execution of the block body matches the sequence ab, the entire block matches the sequence ababababab - that is 5 occurrences of ab similar to the regular expression (ab){5}.

Block Headers

Within a block, some events may be disallowed by the test case - the occurrence of such events at any point in the block’s execution is undesirable so that the test case should fail if observed. In some other cases, some events may occur but they are not required for a successful test case - that is, they are allowed if they do occur. Finally, some events may be dropped if they do occur - if incoming then they are not delivered to the CUT and if outgoing they are not forwarded to any other components in the environment. For these three cases Kompics offers the respective statements disallow, allow and drop that can be called within the header of a block.

The following code snippet shows the previous example with a single header statement.

tc.repeat(5)
     // block header
     .allow(new Ping(0), pingerPort, IN) // c
  .body()
     // block body
    .expect(new Ping(1), pingerPort, IN) // a
    .expect(new Ping(2), pingerPort, IN) // b
  .end(); // end of block

The allow in this case whitelists the Ping 0 event within this block (the event may be observed while executing a statement within the block body). In terms of set of behaviors being described, since the block body already describes the sequence ab, this means that any number of c events can occur at any positions of this expected sequence. As such, the regular expression (c*ac*bc*){5} describes the same set of sequences.

For a concrete example, we utilize the setup code from this example

public static void main(String[] args) { // blocksIntroDemo
  // ... setup code as in basicsExample ...
  // setup done
  tc.body()
      .repeat(2)
          .allow(new Ping(8), pongerPort, Direction.IN)
          .allow(new Pong(8), pongerPort, Direction.OUT)
      .body()
          // this is executed twice
          .trigger(new Ping(0), pongerPort)
          .expect(new Pong(1), pongerPort, Direction.OUT)
          .expect(new Pong(2), pongerPort, Direction.OUT)
      .end()

      // the same scenario but this time do not forward Pong 1 events
      .repeat(3)
          .drop(new Pong(1), pongerPort, Direction.OUT)
      .body()
          // this is executed thrice
          .trigger(new Ping(0), pongerPort)
          .expect(new Pong(2), pongerPort, Direction.OUT)
      .end()
  ;
  assertTrue(tc.check());
  assertEquals(6, Ponger.pingsReceived);
  assertEquals(8, Pinger.pongsReceived);
}

The test case consists of two explicit blocks. In the first block repeat(2) we allow the Ping 8 event sent by Pinger on start and it’s response while we execute the same scenario of triggering Ping 0 and expecting two Pong s in response. Within this block Pinger receives a total of 5 Pong events while Ponger receives 3 Ping events. In the second block we replay the same scenario three times while dropping outgoing Pong 1 events. Consequently, Pinger receives only the three Pong 2 events that are actually forwarded, bringing its total to 8.

Note that in this example we assumed that the Ping 8 event would be sent at some point within the first block. Our assumption is not entirely correct since we haven’t taken into consideration any possible delays. For example, execution of the start handler of Pinger could be delayed so that the event actually arrives at Ponger later on when the framework executes statements in the repeat(3) block. This would cause our test case to fail since we have not explicitly allowed the Ping event (or its response) there. For the purpose of this illustration we erred on the side of naiveté, but to be sure one would have to include the same allow statements in the second block as well.

Nesting Blocks

A block N may be nested within the body of another block B. Header statements of block B are in scope while the framework executes statements within B as well as those of it’s nested blocks such as N. In the case that some event is specified in the header of a block, the same event may be also be specified in the header of nested blocks, in which case the original specification of that event (e.g allow) is shadowed while the nested block is in scope (its statements are being executed).

The following example utilizes nested blocks and shadows headers.

public static void main(String[] args) { // nestedBlockExample
  // ... setup code as in basicsExample ...
  // setup done
  tc.body()
      // first, handle event from pinger
      .expect(new Ping(8), pongerPort, Direction.IN)
      .expect(new Pong(8), pongerPort, Direction.OUT)

      .repeat(4).body()
          .trigger(new Ping(0), pongerPort)
      .end()

      // after triggering four Ping 0s, ponger should respond with four Pong 1,2 sequences
      .repeat(2)
          // drop Pong 1 events
          .drop(new Pong(1), pongerPort, Direction.OUT)
      .body()
          .repeat(1) // nested block
              // here, handle Pong 1 events (previous drop is shadowed!)
              .allow(new Pong(1), pongerPort, Direction.OUT)
          .body()
              .expect(new Pong(2), pongerPort, Direction.OUT) // preceeding Pong 1 is handled
          .end()
          // drop(...) is back in scope
          .expect(new Pong(2), pongerPort, Direction.OUT) // preceeding Pong 1 is dropped
      .end();
  assertTrue(tc.check());
  assertEquals(7, Pinger.pongsReceived);
}

Note

In addition to shadowing headers within nested blocks, you may also specify multiple header statements within the same block that end up matching the same event at runtime. The framework’s uses a last-statement-wins policy for such cases - choosing the matching statement that was declared last in that block’s specification.

Finally, each statement must belong to some block. The TestContext instance is created with an implicit block repeat(1) within which all statements and explicitly created blocks are nested and the entire test case is run. The calls to repeat and end for this block are handled by the framework itself. However the body() must be called to denote its body. This is the reason for calling body() before any sequence can be described - the setup header is in fact the header of this implicit block and as such can be treated as any normal header.

Kleene Blocks

Omitting an integer argument when creating a block (e.g calling repeat()) instead executes the statements of the block zero or more times. The next statement of the block is executed as long as the previous one was successful (e.g a successful match using expect or a trigger that always succeeds).

As an example, the following code snippet matches any number of outgoing Pong 0 events followed by a Pong 1 event. This construct provides a variant of the Kleene star operator *. In this case, the described set is a*b.

tc.repeat().body()
      .expect(new Pong(0), pongerPort, OUT)   // a
  .end()
 .expect(new Pong(1), pongerPort, OUT)        // b

Block Entry Functions

An instance of the se.sics.kompics.testing.EntryFunction interface may be provided as an argument when creating a block. This interface declares a single void run() method that is called on entering the block and might be useful if some desired code needs to be run upon executing statements within that block.

The following code snippet shows a minimal usage of an EntryFunction instance to increment a counter. Although the body is empty, the framework calls run for each iteration.

EntryFunction increment = new EntryFunction() {
        public void run() {
            myCounter.increment();
        }
};
tc.repeat(5, increment)
  .body()
  .end();
//...
assert myCounter.count == 5;

Warning

Be careful when using entry functions. If the first statement of a block’s body can be successfully executed, its entry function is called whether or not the rest of the statements of the block is executed successfully. This might not be the intended behavior but it might be best is to avoid writing such test cases.

Ambiguous Testcases

In order to correctly verify the behavior of a component, the provided test case should be unambiguous. Unfortunately, the addition of some constructs like Kleene blocks and Conditionals, while powerful, make it possible to write test cases whose intention can not be accurately inferred by the framework.

Consider a scenario where a trigger is the first statement of a Kleene block. The following code snippet shows a minimal example.

tc.repeat().body
        .trigger(new Ping(0), pongerPort)
  .end();

It is not possible to infer how many times the trigger statement should be executed - since such a statement always succeeds, the framework would not know when to exit the block.

The second scenario uses the trigger statement as the first statement in both branches of a conditional as shown in the following example. Again, it is unclear which of the Ping events to trigger since the choice of what branch to execute depends on successfully matching events but none of the branches here expect to match an event.

tc.either()
        .trigger(new Ping(0), pongerPort)
        //...
  .or()
        .trigger(new Ping(1), pongerPort)
        //...
  .end();

Test cases containing statements of these forms are deemed ambiguous and the behavior of the framework is left unspecified. As such they should be avoided entirely.

Specifying Nondeterministic Testcases

The DSL offers two explicit constructs for expressing cases where we are not exactly sure of the ordering of events at certain points even though we know that the events will occur.

Unordered

The first is the unordered statement of the form unordered() - end() where - represents a sequence of expect statements which are executed by the framework in any order such that they are successful. That is, the events described by the statements are matched in the order that they actually occur at runtime as opposed to the normal sequence of expect statements whose events must be matched in sequential order. Calling unordered() switches the current mode to UNORDERED while calling the matching end() restores the previous mode. The following example verifies the initial ping-pong exchange initiated by Pinger in its start handler, then using unordered, we verify the Ping 0 behavior of our CUT - the expect statements are matched in the order that the events occur. Comment out the call to unordered() and its matching end() and verify that the test case now fails since we would then be expecting d to happen before c.

public static void main(String[] args) { // unorderedExample
  // ... setup code as in basicsExample ...
  // setup done
  tc.body()
      .expect(new Ping(8), pongerPort, Direction.IN)          // a
      .expect(new Pong(8), pongerPort, Direction.OUT)         // b
      .trigger(new Ping(0), pongerPort)
      .unordered()
          // the specified order of statements does not matter here
          // since 'c' will actually happen before 'd', it will be executed first
          .expect(new Pong(2), pongerPort, Direction.OUT)     // d
          .expect(new Pong(1), pongerPort, Direction.OUT)     // c
      .end();
  assertTrue(tc.check());
}

Block Expect

This range of nondeterministic scenarios that can be expressed with unordered are limited since the ordering of events are still specified relative to others within the block. The second construct is the header statement blockExpect, specifying that an event must occur at any point within the block. Since we are able to group any statements into a single block and use blockExpect statements to interleave our nondeterministic events with those matched by the block body, we can express an even wider range of scenarios. The following shows a variant of the previous example using this construct.

public static void main() { // blockExpectExample
  // ... setup code as in basicsExample ...
  // setup done
  tc.body()
      .repeat(1)
          .blockExpect(new Ping(8), pongerPort, Direction.IN)          // a
          .blockExpect(new Pong(8), pongerPort, Direction.OUT)         // b
      .body()
          // 'a' and 'b' can interleave the execution of any of these three statements
          .trigger(new Ping(0), pongerPort)
          .expect(new Pong(1), pongerPort, Direction.OUT)     // c
          .expect(new Pong(2), pongerPort, Direction.OUT)     // d
      .end();
  assertTrue(tc.check());
}

Expecting Exceptions

If you want to test that the component throws an exception while handling an event, the expectFault statements are available. Place this statement right after the event whose handling causes the exception otherwise the framework treats the thrown exception unexpectedly and fails the test case. The following code snippet matches an exception of class IllegalStateException on handling the triggered Ping event. An instance of com.google.common.base.Predicate<Throwable> may also be provided to match the actual expected event if desired.

tc.trigger(new Ping(-1), pongerPort) // handling this event throws an exception
  .expectFault(IllegalStateException.class)
//... continue execution normally

Specifying Default Actions

Events that are not explicitly matched by any (header) statements in scope cause the test case to fail. However, you can override this behavior for particular events using the setDefaultAction statement. It specifies a com.google.common.base.Function<? extends KompicsEvent, Action> instance which will be called with an unmatched event of the specified class (or subclass) and returns an se.sics.kompics.testing.Action, telling the framework whether to DROP or HANDLE the event or FAIL the test case. This is shown in the following example where we only explicitly match the outgoin Pong 8 in response to the Pinger. However, since we provide a default action to handle the Ping 8 event, the test case succeeds. Change the implementation of handlePing8 (e.g so that the event is dropped) and verify that the test case does in fact fail.

Function<Ping, Action> handlePing8 = new Function<Ping, Action> () {
        public Action apply(Ping ping) {
            if (ping.id == 8) {
                return Action.HANDLE;
            } else {
                return Action.FAIL;
            }
        }
};

public static void main(String[] args) { // defaultActionExample
  // ... setup code as in basicsExample ...
  tc.setDefaultAction(Ping.class, handlePing8);
  // setup done
  tc.body()
          .expect(new Pong(8), pongerPort, Direction.OUT);
  assertTrue(tc.check());
}

Matching Events

Statements such as expect, allow, disallow etc require the framework to determine whether or not an observed event matches what was specified by the statement. Besides the port and direction, they require an event description. This could be an equivalent instance of the desired event, the event’s class or an instance of com.google.common.base.Predicate<? extends KompicsEvent> that returns true only when called with the expected event.

As an example, in the statement expect(new Ping(8), pongerPort, IN) used in the basics example, our desired event was a Ping with id 8. If the framework observes a Ping event p when it executes this statement, it checks if a java.util.Comparator instance was registered for Ping events or any of its subclasses using the setComparator method (the closest match in the class heirarchy is selected) and if found, uses the comparator to determine equivalence. Otherwise it defaults to the equals(Object) method. In our example we did register Ping.comparator to extract and compare the id fields although the same logic could have been implemented by overriding the equals method of the Ping class.

If we were not interested in the id and simply wanted to match an event of type Ping then the statement expect(Ping.class, pongerPort, IN) suffices.

Finally, the following code snippet shows how one may match the same id using a predicate instance.

Predicate<Ping> matchesPing8 = new Predicate<Ping>() {
          public boolean apply(Ping ping) {
                  return ping.id == 8;
          }
}
tc.expect(Ping.class, matchesPing8, pongerPort, IN);