Testing Effort for Crosswalk on Android 4.0


Because Google is freezing Chrome for Ice Cream Sandwich, Chrome 42 will be the last supported release of Chrome for Android on all Ice Cream Sandwich (Android 4.0) devices. The Crosswalk Project comes to a real crosswalk: maintain Crosswalk on Android 4.0 or not?

Copy the reason of this change from Chromium blog post here:

In the last year, we’ve seen the number of Chrome users running ICS drop by thirty percent. Developing new features on older phones has become increasingly challenging, and supporting ICS takes time away from building new experiences on the devices owned by the vast majority of our users. So, with Chrome’s 42nd release, we’ll stop updating Chrome on ICS devices. After Chrome 42, users on ICS devices can continue to use Chrome but won’t get further updates.


As Crosswalk Web QA, I was asked a question as:

If Google doesn’t run upstream tests on 4.0 we should do it ourselves, and we should make sure 4.0 is part of the weekly test. Is that feasible and at what cost?

Some intuitive testing efforts might be:

Testing for Chromium for Android

From the Android Test Instructions wiki page, we can see that there are 4 types of tests running on Chromium for Android.

Per the overview dashboard in the test result server, we can get the flaky and total count of the tests.

Test Type Flaky Count Total Count Applicable to Crosswalk
Gtests (content_unittests) 3 3263 Yes
Instrumentation Tests (ContentShellTest) 1 403 Yes
Instrumentation Tests (ChromeShellTest) 31 357 No
Instrumentation Tests (AndroidWebViewTest) 3 428 No
Blink Layout Tests (layout-tests) 3774 42485 Yes
GPU Tests (context_lost) 5 6 Yes
GPU Tests (memory_test) 1 1 Yes
GPU Tests (webgl_conformance) 7 679 Yes
GPU Tests (maps) 1 1 Yes
GPU Tests (angle_unittests) 0 4832 Yes
GPU Tests (content_gl_tests) 0 19 Yes
GPU Tests (gl_tests) 0 94 Yes
GPU Tests (gles2_conform_test) 0 143 Yes