Package one.fonda

This package provides a framework for regression testing and benchmarking.

See:
Description

Interface Summary

Harness	Definition of the testing harness.
TestCollection	Definition of a collection of regression tests.

Class Summary

EmptyEvent	An empty event is an event with no data.
Exercise	Implementation of the exercise program.
Minimum	Implementation of a minimal main application component.

Package one.fonda Description

This package provides a framework for regression testing and benchmarking. It defines the testing harness and the interface for collections of related tests.

Classes that implement a collection of regression tests should be defined within the package for which they provide the regression tests. Such a class should be named "Test<Name>", where <Name> is the name of the abstraction or concept being tested.

TupleStore benchmarks

Description

There are three sets of benchmarks designed to evaluate tuple store performance. These tests are BenchmarkTupleStore, BenchmarkBDB, and BenchmarkTSpaces. BenchmarkTupleStore is a microbenchmark of TupleStore performance. BenchmarkBDB is a microbenchmark of raw Berkeley DB performance, designed to show the latency penalty imposed by one.world over raw Berkeley DB. BenchmarkTSpaces is a benchmark of IBM TSpaces performance meant as a point of comparison to Berkeley DB.

The data used in each test are PIM contact tuples modeled after the digime datatype (this exact datatype was not used to allow a closer comparison with TSpaces, which does not support nested tuples). An unlimited number of random tuples (with a fixed number of distinct first names) can be created.

Their are four basic tests (only 3 of which are supported in BDB):

• Read test: Before beginning the test, a number of tuples are written to the store. The test consists of sequentially reading a fixed number of random tuples based on their id. The average latency per read is measured by measuring the latency of the set of reads.
• Write test: A number of tuples are created prior to the test. The test consists of sequentially writing a fixed number of randomly selected tuples in random order. The latency is again measured.
• Query test: Before beginning the test, a number of tuples are written to the store. The test consists of sequentially issuing queries which return all of the entries in the tuple store. The latency is again measured.
• Query by name test: Before beginning the test, a number of tuples are written to the store. The test consists of sequentially issuing queries which select tuples based on the first name inside the tuple. The latency is again measured. This test is not supported in BDB since it lacks the concept of a record.

In addition, the TupleStore and TSpaces benchmarks allow throughput to be measured by allowing differing amounts of concurrency. In BenchmarkTupleStore, this means that a user specified number of operations can be concurrently outstanding. In BenchmarkTSpaces, this means a user specified number of threads are run concurrently.

Specific information on running each benchmark is listed with each class

Results

If the concurrency is 1, the latency per operation is calculated. This is simply the average elapsed time per test divided by the number of operations per test.

The throughput in operations/second is always returned. This is simply the number of operations per test divided by the average test time. For concurrency of 1, this is simply the inverse of the latency. The reported standard deviation is the standard deviation of operations/second between tests.

For named queries, the average number of matches per query is also returned.

Networking benchmarks

These benchmarks are provided as a comparison to one.world DatagramIO and NetworkIO performance (benchmarks for these components are implemented by BenchmarkDatagramIO and BenchmarkNetworkIO). The Java tests in this package are implemented in BenchmarkUdp and BenchmarkTcp, respectively.

There are 2 types of tests in each of these classes:

• Latency test: Send data to an echo server and time the round-trip, repeating a specified number of times.
• Throughput test: Send data as fast a possible to another host, measuring the throughput.

Tests are performed a specified number of times and the average result is printed. These tests can be performed by sending either Tuples or raw bytes, so that the overhead of using an object (Tuple) wire format may be determined.

Specific information on running each benchmark is listed with each class

REP benchmarks

Benchmarks are provided for the comparison of remote event passing (REP) to Java RMI. These benchmarks test latency and throughput. The number of tests to run and the number of remote events or remote method invocations in each test can be chosen by the user. See BenchmarkRmi and BenchmarkREP for more information on running these benchmarks.

Two machines are required to run the REP benchmarks. Designate one the client and one the server.

To run the REP latency benchmark, from the one.world shell on the server, type

mk echo one.world.rep.BenchmarkREP echo
run echo

On the client, type

mk el one.world.rep.BenchmarkREP el servername echo
run el

The reported result on the client machine will be the average over 100 tests of the time required to send and receive 1000 remote events.

To run the REP throughput benchmark, from the one.world shell on the server, type

mk etr one.world.rep.BenchmarkREP etr
run etr

On the client, type

mk ets one.world.rep.BenchmarkREP ets servername etr
run ets

The reported result on the server machine will be the average over 100 tests of the time required to receive 1000 remote events from the client.

To run the Java RMI benchmarks, the RMI registry must be running. To start the RMI registry under Linux, run

rmiregistry &

To run the RMI registry under Windows, run

start rmiregistry

To run the RMI call latency benchmark, go to the ${JAVA_DEV_ROOT}/src/one/fonda/java directory on the server and type

java -Djava.security.policy=rmi.policy one.fonda.java.BenchmarkRmi cls cls

From the client, type

java one.fonda.java.BenchmarkRmi clc servername cls

The reported result on the client machine will be the average over 100 tests of the time required to make 1000 remote method invocations.

To run the RMI call throughput benchmark, go to the ${JAVA_DEV_ROOT}/src/one/fonda/java directory on the server and type

java -Djava.security.policy=rmi.policy one.fonda.java.BenchmarkRmi cts cts

From the client, type

java one.fonda.java.BenchmarkRmi ctc servername cts numthreads

The reported result on the server machine will be the average over 100 tests of the time required to receive 1000 remote method invocations from the client. numthreads is the number of threads used by the client to perform remote method invocations. If numthreads is 1, the result reported will be nearly the latency. To determine throughput, numthreads should be increased until it no longer reduces the time reported.

To run the RMI event latency benchmark, go to the ${JAVA_DEV_ROOT}/src/one/fonda/java directory on the server and type

java -Djava.security.policy=rmi.policy one.fonda.java.BenchmarkRmi els els

From the client, type

java one.fonda.java.BenchmarkRmi elc servername els

The reported result on the client machine will be the average over 100 tests of the time required to make 1000 remote method invocations.

To run the RMI event throughput benchmark, go to the ${JAVA_DEV_ROOT}/src/one/fonda/java directory on the server and type

java -Djava.security.policy=rmi.policy one.fonda.java.BenchmarkRmi ets ets

From the client, type

java one.fonda.java.BenchmarkRmi etc servername ets numthreads

The reported result on the server machine will be the average over 100 tests of the time required to receive 1000 remote method invocations from the client. numthreads is the number of threads used by the client to perform remote method invocations. If numthreads is 1, the result reported will be nearly the latency. To determine throughput, numthreads should be increased until it no longer reduces the time reported.