Test Discovery and Linting¶

When compiling the binary the makefile will try detect litmus tests and collect any new ones.

Discovery¶

By default the tool responsible is a small Python script located at litmus/makegroups.py.

This file will read all the .c files under litmus/litmus_tests/ and build 3 files:

litmus/test_list.txt
litmus/group_list.txt
litmus/groups.c

Group and test lists¶

The group list contains the LITMUS_TESTS argument from the run of the Makefile. This means repeated executions of the Makefile remembers the old setting.

The test listing contains a file like the one below:

litmus/litmus_tests/amo/MP+po_p_amoswap_pl-coi+addr.c 1594228641.6208193 MP_pos AArch64 MP+po_p_amoswap_pl-coi+addr all amo
litmus/litmus_tests/checks/check4.c 1590352298.4354115 check4 check4 all checks
litmus/litmus_tests/checks/check3.c 1590339837.875848 check3 check3 all checks
litmus/litmus_tests/checks/check2.c 1590339935.373299 check2 check2 all checks
litmus/litmus_tests/checks/check6.c 1590421869.2080476 check6 check6 all checks
litmus/litmus_tests/checks/check1.c 1590339787.7471209 check1 check1 all checks
litmus/litmus_tests/checks/check5.c 1592996319.996931 check5 check5 all checks
litmus/litmus_tests/pgtable/WRC.TRR+addr+dmb.c 1594290381.603 WRCtrr_addr_dmb WRC.TRR+addr+dmb all pgtable

Each line is a record, and a record is made from whitespace separated fields. The fields are, in-order:

Whether the test is currently enabled and is compiled each time (1=yes, 0=no).
the source file that contains the test
the last modified timestamp
the name of the litmus test
the groups it belongs to

There may be many groups (in general, at least 2) for a test and so the records are not a fixed number of fields.

This list is mostly useful for bookkeeping, allowing the file to check for new tests that weren’t detected last time, or for collection of results from devices (TODO).

groups.c¶

The groups.c file contains some C declarations of the test lists and group lists, so they can be read at runtime on the remote device for parsing command-line args.

An example groups.c is given below for LITMUS_TESTS=@data

/************************
*  AUTOGENERATED FILE  *
*     DO NOT EDIT      *
************************/

/* this file was generated with the following command:
* `make LITMUS_TESTS='@data'`
*
* please re-run that command to re-generate this file
*/
#include "lib.h"

extern litmus_test_t
  LB_pos,
  MP_dmbs,
  MP_pos,
  SB_dmbs,
  SB_pos,
  WRC_addrs,
  WRC_pos;


const litmus_test_group grp_data = {
  .name="@data",
  .tests = (const litmus_test_t*[]){
    &LB_pos,
    &MP_dmbs,
    &MP_pos,
    &SB_dmbs,
    &SB_pos,
    &WRC_addrs,
    &WRC_pos,
    NULL
  },
  .groups = (const litmus_test_group*[]){
    NULL
  },
};

const litmus_test_group grp_all = {
  .name="@all",
  .tests = (const litmus_test_t*[]){
    NULL
  },
  .groups = (const litmus_test_group*[]){
    &grp_data,
    NULL
  },
};

The extern litmus_test_t declarations just bring all the C identifiers for the litmus_test_t declarations into scope to squash warnings, then a group @groupName is defined by a C declaration grp_groupName.

Each litmus_test_group has 3 parts:

a @name
a NULL-terminated list of litmus_test_t s
a NULL-terminated list of litmus_test_group s

Linting¶

During compilation a small ‘linter’ script runs over the litmus tests files. This linter script is intended to just detect mistakes in the source that can easily be done when writing the litmus tests, but are still valid C.

Below is an example of an intentionally badly written litmus test:

#include <stdint.h>

#include "lib.h"

#define VARS x, y, z
#define REGS p0x2

static void sync_handler(void) {
  asm volatile (
    "mov x2, #0\n\t"

    ERET_TO_NEXT(x10)
  );
}

static void P0(litmus_test_run* data) {
  asm volatile (
    /* move from C vars into machine regs */
      "mov x0, %[ydesc]\n\t"
      "mov x1, %[xpte]\n\t"
      "mov x3, %[x]\n\t"
      "mov x6, #42\n\t"

      /* test */
      "str x0, [x1]\n\t"
      "ldr x2, [x3]\n\t"

      /* output */
      "str x2, [%[outp0r2]]\n\t"
  :
  : ASM_VARS(data, VARS),
    ASM_REGS(data, REGS)
  : "cc", "memory", "x0", "x2", "x3"
  );
}


litmus_test_t CoWTinv = {
  "CoWT+wrong-name.inv",
  MAKE_THREADS(2),
  MAKE_VARS(VARS),
  MAKE_REGS(REGS),
  INIT_STATE(
    2,
    INIT_UNMAPPED(x),
    INIT_VAR(y, 1),
    INIT_VAR(z, 0)
  ),
  .interesting_result = (uint64_t[]){
    /* p0:x2 =*/0,
  },
  .thread_sync_handlers = (uint32_t**[]){
    (uint32_t*[]){(uint32_t*)sync_handler, NULL},
    (uint32_t*[]){(uint32_t*)sync_handler, NULL},
  },
  .no_sc_results = 1,
};

Running the linter on this file reveals the following issues:

$ python3 litmus/linter.py litmus/litmus_tests/pgtable/CoWT.inv.c
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: File name and Litmus name mismatch.
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: Thread count mismatch.
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: Thread count does not match number of exception handler entries.
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: clobber missing register x1 in Thread 0
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: clobber missing register x10 in Thread 0
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: clobber missing register x6 in Thread 0
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: register x6 in Thread 0 appears to be unused
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: initial state contains 3 states but INIT_STATE arguments claim it has 2 states
! [litmus/litmus_tests/pgtable/CoWT.inv.c] Warning: missing requires_pgtable=1?

Linting can be totally disabled by running make with NO_LINT=1. Or the linter command can be customized by running make with LINTER='custom cmd'.