How to Write W3C Reftests

Zhiqiang Zhang (zqzhang), Intel

Start from Example in Spec

Example 14.3 in CSS3 background-size

To Checkpoints

  • background-size: length auto:
    scale the image width to 'length' and height to (1) the image's intrinsic ratio and the size of width, or (2) the image's intrinsic size, or (3) 100%
  • background-repeat: round repeat:
    scale the image in width so that it fits a whole number of times in the background positioning area
  • round for width and auto for height, so height is scaled to keep the image's original aspect ratio

Copy Test Case Template

<!DOCTYPE html>
<title>[Test Area]: [Title/Scope of Test]</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
<link rel="help" href="[direct link to tested 
<link rel="match" href="[path to reference file]">
<meta name="flags" content="[requirement flags]">
<meta name="assert" content="Test checks that [explanation of what 
  you're trying to test].">
    [CSS for test]
    <p> Test passes if [description of pass condition].</p>
    [Content of test]

Reftest Test Case Template

To Test Case: Style

To Test Case: Pass Condition

Try Test Case on Browsers

Try Test Case on Browsers

Complete Test File

Reference File

    The reference file should look exactly like the test file, except that the code behind it is different.

  • The assert metadata is taken out.
  • The title is generic, as this file can and may be shared among multiple tests
  • The match (or mismatch) metadata not necessary
  • The markup that created the actual test data is different: here, the same effect is created with very mundane, dependable technology.

Copy Reference File Template

<!DOCTYPE html>
<title>[Test Area] Reference File</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
    [CSS for test]
    <p> Test passes if [description of pass condition].</p>
    [Content of test]

Reftest Reference Template

Reference File Source

Try Reference File on Browsers


Continue to Improve the Test

Commits to background-size-029.html

background-size-029.html: approved v.s. initial

Tests to avoid: long test

  • Manual test
  • Need to scroll to complete testing
  • Example of a test that is too long

Tests to avoid: "this should be red" test

  • In the Test Style Guidelines document, red indicates a bug
  • Nothing should ever be red in a test.
  • One exception: the test for the 'red' value for the color properties!

Tests to avoid: unobvious test

The last subtest on this page shows this problem.

Tips: vendor prefix

  • You may need to add a vendor prefix to the CSS property you're testing
  • If the prefix is necessary, your test will appear to fail without it
  • However, do not include vendor prefixes when pushing tests to the W3C

How to Review W3C Tests

or How to Have Tests that Pass Review

Zhiqiang Zhang (zqzhang), Intel

Who Can Review a W3C Test?

Anyone but the test author!

  • Spec editors
  • Test coordinator
  • Implementors
  • Anyone with good working knowledge of the spec


Checklist for all tests

  • The test follows the format and style guidelines
  • The test passes when it's supposed to pass
  • The test fails when it's supposed to fail
  • The test is testing what it thinks it's testing
  • The spec backs up the expected behavior in the test
  • The test is automatable as either reftest or a script test

Checklist for Reftests only

  • The test has a self-describing statement
  • The self-describing statement is accurate, precise, simple, and self-explanatory.
  • The reference file is accurate and will render pixel-perfect identically to the test on all platforms.
  • The reference file uses a different technique that won't fail in the same way as the test.
  • The title is descriptive but not too wordy.
  • The test is as cross-platform as reasonably possible


  • A reviewer can be a colleague of the contributor as long as the proceedings are public
  • A reviewer cannot review his/her own tests and changes
  • All reviews should be fully transparrent by having all review comments on GitHub

Where to Report Bugs

Zhiqiang Zhang (zqzhang), Intel

Where to Report a Bug?



Ready? Go!

  • Let's start the Test Hackathon!

  • Write test cases for the 8 specs and beyond.

  • File good bug reports.