How to Write W3C Reftests

Zhiqiang Zhang (zqzhang), Intel

Start from Example in Spec

Example 14.3 in CSS3 background-size

To Checkpoints

  • background-size: length auto:
    scale the image width to 'length' and height to (1) the image's intrinsic ratio and the size of width, or (2) the image's intrinsic size, or (3) 100%
  • background-repeat: round repeat:
    scale the image in width so that it fits a whole number of times in the background positioning area
  • round for width and auto for height, so height is scaled to keep the image's original aspect ratio

Copy Test Case Template


<!DOCTYPE html>
<title>[Test Area]: [Title/Scope of Test]</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
  http://CONTACT_PAGE">
<link rel="help" href="http://www.w3.org/TR/[direct link to tested 
  section]">
<link rel="match" href="[path to reference file]">
<meta name="flags" content="[requirement flags]">
<meta name="assert" content="Test checks that [explanation of what 
  you're trying to test].">
<style>
    [CSS for test]
</style>
<body>
    <p> Test passes if [description of pass condition].</p>
    [Content of test]
</body>

Reftest Test Case Template

To Test Case: Style

To Test Case: Pass Condition

Try Test Case on Browsers

Try Test Case on Browsers

Complete Test File

http://test.csswg.org/source/contributors/intel/submitted/css3-background/background-size-029.html

Reference File

    The reference file should look exactly like the test file, except that the code behind it is different.


  • The assert metadata is taken out.
  • The title is generic, as this file can and may be shared among multiple tests
  • The match (or mismatch) metadata not necessary
  • The markup that created the actual test data is different: here, the same effect is created with very mundane, dependable technology.

Copy Reference File Template


<!DOCTYPE html>
<title>[Test Area] Reference File</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
  http://CONTACT_PAGE">
<style>
    [CSS for test]
</style>
<body>
    <p> Test passes if [description of pass condition].</p>
    [Content of test]
</body>

Reftest Reference Template

Reference File Source

Try Reference File on Browsers

Reftest

Continue to Improve the Test

Commits to background-size-029.html

background-size-029.html: approved v.s. initial

Tests to avoid: long test

  • Manual test
  • Need to scroll to complete testing
  • Example of a test that is too long

Tests to avoid: "this should be red" test

  • In the Test Style Guidelines document, red indicates a bug
  • Nothing should ever be red in a test.
  • One exception: the test for the 'red' value for the color properties!

Tests to avoid: unobvious test

The last subtest on this page shows this problem.

Tips: vendor prefix

  • You may need to add a vendor prefix to the CSS property you're testing
  • If the prefix is necessary, your test will appear to fail without it
  • However, do not include vendor prefixes when pushing tests to the W3C

How to Review W3C Tests

or How to Have Tests that Pass Review

Zhiqiang Zhang (zqzhang), Intel

Who Can Review a W3C Test?

Anyone but the test author!

  • Spec editors
  • Test coordinator
  • Implementors
  • Anyone with good working knowledge of the spec

Merged?

Checklist for all tests

  • The test follows the format and style guidelines
  • The test passes when it's supposed to pass
  • The test fails when it's supposed to fail
  • The test is testing what it thinks it's testing
  • The spec backs up the expected behavior in the test
  • The test is automatable as either reftest or a script test

Checklist for Reftests only

  • The test has a self-describing statement
  • The self-describing statement is accurate, precise, simple, and self-explanatory.
  • The reference file is accurate and will render pixel-perfect identically to the test on all platforms.
  • The reference file uses a different technique that won't fail in the same way as the test.
  • The title is descriptive but not too wordy.
  • The test is as cross-platform as reasonably possible

Notes

  • A reviewer can be a colleague of the contributor as long as the proceedings are public
  • A reviewer cannot review his/her own tests and changes
  • All reviews should be fully transparrent by having all review comments on GitHub

Where to Report Bugs

Zhiqiang Zhang (zqzhang), Intel

Where to Report a Bug?

References

Acknowledgements

Ready? Go!

  • Let's start the Test Hackathon!

  • Write test cases for the 8 specs and beyond.

  • File good bug reports.