Use Case One - Setting up a simple client:server test definition.

Use Case One - Setting up a simple client:server test definition.

One device needs to obtain / prepare some data and then make the data available to another device in the same group.

Requirements

  1. A mechanism to obtain the data, presumably from some third-party source
  2. A sync to ensure that the file is ready to be offered to the other device
2.1. This ensures that the attempt to receive does not start early
  1. A message to the original board that the data has been received and verified
3.1. This ensures that any cleanup of the data does not happen before the transfer is complete.

Methods

  • Install a package which can obtain the data from the third party source
  • Install a package which can provide the means to get the data to the other board

Control flow

sender starts receiver starts
sender obtains the data receiver waits for sender to be ready
sender modifies the data wait
sender notifies receiver wait
sender waits for completion receiver initiates transfer
wait receiver notifies sender of completion
sender cleans up receiver processes the modified data

It is clear from the flow that the sender and the receiver are doing different things at different times and may well need different packages installed. The simplest way to manage this is to have two YAML files.

In this example, sender is going to use wget to obtain the data and apache to offer it to the receiver. The receiver will only need wget. The example won’t actually modify the data, but for the purposes of the example, the documentation will ignore the fact that the receiver could just get the data directly.

Preparing the YAML

The name field specified in the YAML will be used later as the basis of the filter. To start each YAML file, ensure that the metadata contains two metadata fields:

  • format : Lava-Test Test Definition 1.0
  • description : your own descriptive text

It is useful to also add the maintainer field with your email address as this will be needed later if the test is to be added to one of the formal test sets.

metadata:
   format: Lava-Test Test Definition 1.0
   name: multinode-usecaseone
   description: "MultiNode network test commands"
   maintainer:
       - neil.williams@linaro.org

Installing packages for use in a test

If your test image raises a usable network interface by default on boot, the YAML can specify a list of packages which need to be installed for this test definition:

install:
   deps:
       - wget
       - apache2

If your test needs to raise the network interface itself, the package installation will need to be done in the run steps:

run:
   steps:
       - lava-test-case linux-linaro-ubuntu-route-ifconfig-up --shell ifconfig eth0 up
       - lava-test-case apt-update --shell apt-get update
       - lava-test-case install-deps --shell apt-get -y install wget apache2

Note that although KVM devices can use apt, the network interface fails the LAVA test, so use the manual install steps for non-bridged KVM devices.

Preparing the test to send data

modify-data.sh would, presumably, unpack the data, modify it in some way and pack it back up again. In this example, it would be a no-op but note that it still needs to exist in the top level directory of your VCS repo and be executable.

Any packages required by modify-data.sh need to be added to the install deps of sender.yaml. Providing useful contents of modify-data.sh is left as an exercise for the reader.

Modification happens before the lava-sync download which tells the receiver that the data is ready to be transferred.

The sender then waits for the receiver to acknowledge a correct download using lava-sync received and cleans up.

sender.yaml

install:
   deps:
       - wget
       - apache2

run:
  steps:
       - lava-test-case multinode-network --shell lava-network broadcast eth0
       - lava-test-case wget-file --shell wget -O /var/www/testfile http://releases.linaro.org/latest/android/arndale/userdata.tar.bz2
       - ./modify-data.sh
       - lava-test-case file-sync --shell lava-sync download
       - lava-test-case done-sync --shell lava-sync received
       - lava-test-case remove-tgz --shell rm /var/www/testfile

Handling the transfer to the receiver

The receiver needs to know where to find the data. The sender can ensure that the file is in a particular location, it is up to the YAML to get the rest of the information of the network address of the sender. This example assumes that the data is modified in some undisclosed manner by the ./modify-data.sh script which is part of your testdef_repo before the receiver is notified.

The LAVA MultiNode API provides ways of querying the network information of devices within the group. In order to offer the data via apache, the sender needs to raise a suitable network interface, so it calls ifconfig as a lava test case first and then uses the lava-network API call to broadcast network information about itself.

Equally, the receiver needs to raise a network interface, broadcast it’s network information and then collect the network information for the group.

Note that collect is a blocking call - each of the devices needs to broadcast before collect will return. (There is support for collecting data only for specific roles but that’s outside the scope of this example.)

receiver.yaml

install:
   deps:
       - wget

run:
  steps:
       - lava-test-case linux-linaro-ubuntu-route-ifconfig-up --shell ifconfig eth0 up
       - lava-test-case multinode-network --shell lava-network broadcast eth0
       - lava-test-case multinode-get-network --shell lava-network collect eth0
       - lava-test-case file-sync --shell lava-sync download
       - lava-test-case wget-from-group --shell ./get-data.sh
       - lava-test-case get-sync --shell lava-sync received
       - lava-test-case list-file --shell ls -l /tmp/testfile
       - lava-test-case remove-file --shell rm /tmp/testfile

The receiver then needs to obtain that network information and process it to get the full URL of the data. To do command line processing and pipes, a helper script is needed:

get-data.sh

Always use set -x in any wrapper / helper scripts which you expect to use in a test run to be able to debug test failures.

Ensure that the scripts are marked as executable in your VCS and that the appropriate interpreter is installed in your test image.

#!/bin/sh
set -e
set -x
DEVICE=`lava-group | grep -m1 receiver|cut -f2`
SOURCE=`lava-network query $DEVICE ipv4|grep -v LAVA|cut -d: -f2`
wget -O /tmp/testfile http://${SOURCE}/testfile

The $DEVICE simply matches the first device name in this group which contains the string ‘receiver’ (which comes from the role specified in the JSON) and returns the full name of that device, e.g. multinode-kvm02 or staging-beagleblack03

This device name is then passed to lava-network query to get the ipv4 details of that device within this group. The value of $SOURCE is an IPv4 address of the sender (assuming that your JSON has defined a role for the sender which would contain the ‘receiver’ string in the name.)

Finally, get-data.sh does the work of receiving the data from the sender. The verification of the data is left as an exercise for the reader - one simple method would be for the sender to checksum the (modified) data and use lava-send to make that checksum available to devices within the group. The receiver can then use lava-wait to get that checksum.

Once get-data.sh returns, the receiver notifies the sender that the transfer is complete, processes the data as it sees fit and cleans up.

Preparing the JSON

The JSON ties the YAML test definition with the hardware and software to run the test definition. The JSON is also where multiple test definitions are combined into a single MultiNode test.

General settings

Warning

Timeout values need to be reduced from single node examples

  • each synchronisation primitive uses the timeout from the general settings,
  • always check your timeout value - 900 is recommended.
{
   "health_check": false,
   "logging_level": "DEBUG",
   "timeout": 900,
   "job_name": "client-server test",
}

device_group

The device_group collates the device-types and the role of each device type in the group along with the number of boards to allocate to each role.

If count is larger than one, enough devices will be allocated to match the count and all such devices will have the same role and use the same commands and the same actions. (The job will be rejected if there are not enough devices available to satisfy the count.)

{
   "device_group": [
       {
           "role": "sender",
           "count": 1,
           "device_type": "beaglebone-black"
       },
       {
           "role": "receiver",
           "count": 1,
           "device_type": "kvm"
       }
   ],
}

actions

When mixing different device_types in one group, the images to deploy will probably vary, so use the role parameter to determine which image gets used on which board(s).

deploy_linaro_image

{
   "actions": [
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "image": "http://images.validation.linaro.org/kvm-debian-wheezy.img.gz",
               "role": "receiver"
           }
       },
       {
           "command": "deploy_linaro_image",
           "parameters": {
               "image": "http://linaro-gateway/beaglebone/beaglebone_20130625-379.img.gz",
               "role": "sender"
           }
       }
}

lava_test_shell

If specific actions should only be used for particular roles, add a role field to the parameters of the action.

If any action has no role specified, it will be actioned for all roles.

For Use Case One, we have a different YAML file for each role, so we have two lava_test_shell commands.

{
       {
           "command": "lava_test_shell",
           "parameters": {
               "testdef_repos": [
                   {
                       "git-repo": "git://git.linaro.org/people/neilwilliams/multinode-yaml.git",
                       "testdef": "forwarder.yaml"
                   }
               ],
               "role": "sender"
           }
       },
       {
           "command": "lava_test_shell",
           "parameters": {
               "testdef_repos": [
                   {
                       "git-repo": "git://git.linaro.org/people/neilwilliams/multinode-yaml.git",
                       "testdef": "receiver.yaml"
                   }
               ],
               "role": "receiver"
           }
       }
}

submit_results

The results for the entire group get aggregated into a single result bundle. Ensure that the bundle stream exists on the specified server and that you have permission to add to that stream.

{
       {
           "command": "submit_results_on_host",
           "parameters": {
               "stream": "/anonymous/use-cases/",
               "server": "http://validation.linaro.org/RPC2/"
           }
       }
}

Prepare a filter for the results

Now decide how you are going to analyse the results of tests using this definition, using the name of the test definition specified in the YAML metadata.

Unique names versus shared names

Each YAML file can have a different name or the name can be shared amongst many YAML files at which point those files form one test definition, irrespective of what each YAML file actually does. Sharing the name means that the results of the test definition always show up under the same test name. Whilst this can be useful, be aware that if you subsequently re-use one of the YAML files sharing a name in a test which does not use the other YAML files sharing the same name, there will be gaps in your data. When the filter is later used to prepare a graph, these gaps can make it look as if the test failed for a period of time when it was simply that the not all of the tests in the shared test definition were run.

A single filter can combine the results of multiple tests, so it is generally more flexible to have a unique name in each YAML file and combine the tests in the filters.

If you use a unique test definition name for every YAML file, ensure that each name is descriptive and relevant so that you can pick the right test definition from the list of all tests when preparing the filter. If you share test definition names, you will have a shorter list to search.

Filters also allow results to be split by the device type and, in Multi-Node, by the role. Each of these parameters is defined by the JSON, not the YAML, so care is required when designing your filters to cover all uses of the test definition without hiding the data in a set of unrelated results.

Create a filter

To create or modify filters (and the graphs which can be based on them) you will need appropriate permissions on the LAVA instance to which are you submitting your JSON.

On the website for the instance running the tests, click on Dashboard and Filters. If you have permissions, there will be a link entitled Add new filter....

The filter name should include most of the data about what this filter is intended to do, without whitespace. This name will be preserved through to the name of the graph based on this filter and can be changed later if necessary. Choose whether to make the filter public and select the bundle stream(s) to add into the filter.

If the filter is to aggregate all results for a test across all devices and all roles, simply leave the Attributes empty. Otherwise, Add a required attribute and start typing to see the available fields.

To filter by a particular device_type, choose target.device_type.

To filter by a particular role (Multi-Node only), choose role.

Click Add a test to get the list of test definition names for which results are available.

Within a test definition, a filter can also select only particular test cases. In this Use Case, for example, the filter could choose only the multinode-network, multinode-get-network or file-sync test cases. Continue to add tests and/or test cases - the more tests and/or test cases are added to the filter, the fewer results will match.

Click the Preview button to apply the filter to the current set of results without saving the filter.

In the preview, if there are columns with no data or rows with no data for specific columns, these will show up as missing data in the filter and in graphs based on this filter. This is an indication that you need to refine either the filter or the test definitions to get a cohesive set of results.

If you are happy with the filter, click on save.

The suggested filter for this use case would simply have a suitable name, no required attributes and a single test defined - using a shared name specified in each of the YAML files.

Bundle streams     /anonymous/instance-manager/
Test cases         multinode-network   any

Prepare a graph based on the filter

A graph needs an image and the image needs to be part of an image set to be visible in the dashboard image reports. Currently, these steps need to be done by an admin for the instance concerned.

Once the image exists and it has been added to an image set, changes in the filter will be reflected in the graph without the need for administrator changes.

Each graph is the result of a single image which itself is basde on a single filter. Multiple images are collated into image sets.

Summary

The full version of this use case are available:

http://git.linaro.org/people/neil.williams/multinode-yaml.git/blob_plain/HEAD:/json/kvm-beagleblack-group.json

Example test results are visible here:

http://multinode.validation.linaro.org/dashboard/image-reports/kvm-multinode

http://multinode.validation.linaro.org/dashboard/streams/anonymous/instance-manager/bundles/da117e83d7b137930f98d44b8989dbe0f0c827a4/

This example uses a kvm device as the receiver only because the test environment did not have a bridged configuration, so the internal networking of the kvm meant that although the KVM could connect to the beaglebone-black, the beaglebone-black could not connect to the kvm.

https://git.linaro.org/people/neil.williams/multinode-yaml.git/blob_plain/HEAD:/json/beagleblack-use-case.json

https://staging.validation.linaro.org/dashboard/image-reports/beagleblack-usecase

https://staging.validation.linaro.org/dashboard/streams/anonymous/codehelp/bundles/cf4eb9e0022232e97aaec2737b3cd436cd37ab14/

This example uses two beaglebone-black devices.