W3C Test Case and Bug Report

Zhiqiang Zhang (zqzhang), Intel

Agenda

How to write a manual test case?
How to write a Reftest?
How to write a Script test?
How to file a good bug report?

Start from Example in Spec

Example 14.3 in CSS3 background-size

To Checkpoints

background-size: length auto:
scale the image width to 'length' and height to (1) the image's intrinsic ratio and the size of width, or (2) the image's intrinsic size, or (3) 100%
background-repeat: round repeat:
scale the image in width so that it fits a whole number of times in the background positioning area
round for width and auto for height, so height is scaled to keep the image's original aspect ratio

Copy Test Case Template


<!DOCTYPE html>
<title>[Test Area]: [Title/Scope of Test]</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
  http://CONTACT_PAGE">
<link rel="help" href="http://www.w3.org/TR/[direct link to tested 
  section]">
<link rel="match" href="[path to reference file]">
<meta name="flags" content="[requirement flags]">
<meta name="assert" content="Test checks that [explanation of what 
  you're trying to test].">
<style>
    [CSS for test]
</style>
<body>
    <p> Test passes if [description of pass condition].</p>
    [Content of test]
</body>

test-templates.md#reftest-test-template

To Test Case: Style

To Test Case: Pass Condition

Try Test Case on Browsers

Complete Test File

http://test.csswg.org/source/contributors/intel/submitted/css3-background/background-size-029.html

Reference File

The reference file should look exactly like the test file, except that the code behind it is different.

The assert metadata is taken out.
The title is generic, as this file can and may be shared among multiple tests
The match (or mismatch) metadata not necessary
The markup that created the actual test data is different: here, the same effect is created with very mundane, dependable technology.

Copy Reference File Template


<!DOCTYPE html>
<title>[Test Area] Reference File</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
  http://CONTACT_PAGE">
<style>
    [CSS for test]
</style>
<body>
    <p> Test passes if [description of pass condition].</p>
    [Content of test]
</body>

test-templates.md#reftest-ref-template

Reference File Source

Try Reference File on Browsers

Reftest

Definition:

A reftest is a test that compares the visual output of one file (the test case) with the output of one or more other files (the references).

Components:

Test file: background-size-029.html
Reference file: background-size-029-ref.html
Comparison:


<link rel="match" href="reference/background-size-029-ref.xht">

Continue to Improve the Test

Commits to background-size-029.html

background-size-029.html: approved v.s. initial

Tests to avoid: long test

Manual test
Need to scroll to complete testing
Example of a test that is too long

Tests to avoid: "this should be red" test

In the W3C test style guidelines document, red indicates a bug
Nothing should ever be red in a test.
One exception: the test for the 'red' value for the color properties!

Tests to avoid: unobvious test

The last subtest on this page shows this problem.

Tips: vendor prefix

You may need to add a vendor prefix to the CSS property you're testing
If the prefix is necessary, your test will appear to fail without it
However, do not include vendor prefixes when pushing tests to the W3C

What is Script Test?


<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<link rel="stylesheet" href="/resources/testharness.css">

<div id="log"></div>

testharness.js

testharness-documentation.md

From IDL Fragment in Spec

Blob Interface in FileAPI

And Description in the Spec

Blob Constructor in FileAPI

To Checkpoints

blob = Blob():

blob instanceof Blob
blob.size === 0
blob.type === ""

Copy Script Test Template


<!DOCTYPE html>
<title>[Test Area]: [Title/Scope of Test]</title>
<link rel="author" title="[Name of Author]" href=mailto:EMAIL OR 
  http://CONTACT_PAGE">
<link rel="help" href="http://www.w3.org/TR/[direct link to tested 
  section]">
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<link rel="stylesheet" href="/resources/testharness.css">
<div id="log"></div>
<script>

test(function() {
  [body of test function - must include at least one assert*() 
  function]
}, [Test Name]);

/* Include as many test() functions as you need */

</script>

test-templates.md#script-test-shared-metadata