Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

0011 - Tests follow the specification, not a corpus

Status: accepted

Context

A real package can contain non-conforming data. Pinning a value measured from one turns its defect into a rule the suite then defends. Separately, a component that checks itself agrees with itself, and a test that cannot fail proves nothing: checks that passed on empty sets have been found in this project.

Decision

  • The shape of an assertion comes from the specification; the values come from the data. Only the first has authority. A test that states a rule cites the document, page and paragraph in a comment.
  • Reference documents are cited, never read at run time. Specifications and schemas are not opened by tests; a test restates the rule instead.
  • Triangulate. Never ask a component to verify itself. Each cross-check confronts two independent sources of the same fact, at least one of which is not code written here: the bytes of the HDF5 file, the part names of the zip, the counts the XML declares.
  • A test must be able to fail. When adding one, ask what would have to be broken for it to go red; if the answer is “nothing”, break something on purpose to find out.
  • An unexplained discrepancy is pursued, not pinned.
  • Real files remain the only real test, so they are exercised in a separate private campaign; the public suite does not depend on them (ADR 0018).

Consequences

  • Tests stay valid when a producer is fixed.
  • Writing a cross-check costs more than writing an assertion; that is the point.