Android CTS 测试研究之二

Android CTS 测试研究之二

作者： 宋立新

Email ： [email protected]

 

前言

继续较深入了解 CTS 的细节

先研究一下具体用法

命令行下敲 cts 进入命令行 ,

先看一下帮助

Help 一下

cts_host > help

Usage: command options

Avaiable commands and options:

  Host:

    help: show this message

    exit: exit cts command line

  Plan:

    ls --plan: list available plans

    ls --plan plan_name: list contents of the plan with specified name

    add --plan plan_name: add a new plan with specified name

    add --derivedplan plan_name -s/--session session_id -r/--result result_type: derive a plan from the given session

    rm --plan plan_name/all: remove a plan or all plans from repository

    start --plan test_plan_name: run a test plan

    start --plan test_plan_name -d/--device device_ID: run a test plan using the specified device

    start --plan test_plan_name -t/--test test_name: run a specific test

    start --plan test_plan_name -p/--package java_package_name: run a specific java package

    start --plan test_plan_name -t/--test test_name -d/--device device_ID: run a specific test using the specified device

    start --plan test_plan_name -p/--package java_package_name -d/--device device_ID: run a specific java package using the specified device

  Package:

    ls -p/--package: list available packages

    ls -p/--package package_name: list contents of the package with specified name

    add -p/--package root: add packages from root to repository

    rm -p/--package package_name/all: remove a package or all packages from repository

  Result:

     ls -r/--result: list all result of sessions

    ls -r/--result -s/--session session_id: list detail case result of a specified session

    ls -r/--result [pass/fail/notExecuted/timeout] -s/--session session_id: list detail cases of a specified session by the specified result.

  History:

    history/h: list all commands in command history

    history/h count: list the latest count records in command history

    history/h -e num: run the command designated by 'num' in command history

  Device:

    ls -d/--device: list available devices

列出所有的 plan

cts_host > ls --plan  

List of plans (8 in total):

CTS

Signature

AppSecurity

Performance

RefApp

Java

VM

Android

 

查看某 plan 的内容

cts_host > ls --plan Android

Packages of plan Android (29 in total):

=================================

android.apidemos.cts

android.accessibilityservice

android.accounts

android.app

android.bluetooth

android.content

android.database

android.dpi

android.dpi2

android.example

android.gesture

android.graphics

android.hardware

android.jni

android.location

android.media

android.net

android.os

android.permission2

android.permission

android.provider

android.speech

android.telephony

android.text

android.util

android.view

android.webkit

android.widget

android.tests.appsecurity

 

添加一个新的 plan

cts_host > add --plan marvell

[Choose package] SignatureTest: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m] Y

Invalid input. Please chose 'y', 'n', or 'm' (default is 'y')

[Choose package] SignatureTest: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m] y

[Choose package] android.accessibilityservice: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m] y

[Choose package] android.accounts: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m] y

[Choose package] android.apidemos.cts: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m]

[Choose package] android.app: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m] y

。。。。

  [Choose package] android.widget: select[Y], reject[n], or choose suite in it[m]?  [Y/n/m]

 

 

cts_host > ls --plan marvell

The following package(s) contained in plan marvell have been removed:

    SignatureTest

Packages of plan marvell (55 in total):

=================================

android.accessibilityservice

android.accounts

android.apidemos.cts

android.app

。。。

android.widget

 

删除一个 plan

cts_host > rm --plan marvell

 

执行一个 plan

cts_host > start --plan Performance

start test plan Performance

==============================================================

Test package: android.performance2

android.performance2.cts.AppStartup#testStartup........................................................................................................................................................(timeout)

==============================================================

 

CTS_INFO >>> Max ADB operations reached. Restarting ADB...

 

CTS_INFO >>> Restarting device ...

 

CTS_INFO >>> Restart complete.

==============================================================

Test package: android.performance3

android.performance3.cts.AppStartup#testStartup..............(pass)

CTS_INFO >>> Max ADB operations reached. Restarting ADB...

 

CTS_INFO >>> Restarting device ...

 

CTS_INFO >>> Restart complete.

==============================================================

Test package: android.performance4

android.performance4.cts.AppStartup#testStartup.......(pass)

 

CTS_INFO >>> Max ADB operations reached. Restarting ADB...

 

CTS_INFO >>> Restarting device ...

 

CTS_INFO >>> Restart complete.

==============================================================

Test package: android.performance5

android.performance5.cts.AppStartup#testStartup........(fail)

junit.framework.AssertionFailedError: App Took too long to startup: 715 650 at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)

at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)

at java.lang.reflect.Method.invokeNative(Native Method)

at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)

at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)

at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)

at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)

at android.performance5.cts.AppStartup.testStartup(AppStartup.java:67)

at java.lang.reflect.Method.invokeNative(Native Method)

at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)

at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)

at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)

at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)

at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:1404)

at android.app.Instrumentation.execStartActivity(Instrumentation.java:1378)

at android.app.ApplicationContext.startActivity(ApplicationContext.java:555)

at android.performance.cts.MultiAppStartupTest.launchActivity(MultiAppStartupTest.java:52)

at android.performance.cts.MultiAppStartupTest.testMultipleApps(MultiAppStartupTest.java:89)

at java.lang.reflect.Method.invokeNative(Native Method)

at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:205)

at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:195)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)

at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)

at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:430)

at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)

 

Test summary:   pass=2   fail=2   timeOut=1   notExecuted=0   Total=5

Time: 635.927s

 

查看有多少个包

cts_host > ls -p

Available packages (56 in total):

android.util

android.hardware

。。。

android.core.tests.luni.net

android.jni

android.core.vm-tests

 

查看某个包

cts_host > ls -p android.net

Test suites (3 in total):

android.net.cts

android.net.wifi.cts

android.net.http.cts

 

查看测试结果

cts_host > ls -r

List of all results:

Session        Test result                Start time        End time        Test plan name

        Pass    Fail    Timeout    NotExecuted

1        2    2    1    0        2010.06.01 16:18:03    2010.06.01 16:28:39    Performance

2        1    0    0    0        2010.06.01 16:39:38    2010.06.01 16:41:33    Signature

研究 cts 生成文件

cts 脚本文件位于： ~/mydroid/out/host/linux-x86/bin ，它是一个 100 行不到的脚本，最终执行：

java -Xmx512M

-cp ./../framework/ddmlib.jar:./../framework/junit.jar:./../framework/hosttestlib.jar:./../framework/cts.jar:./cts.jar

-DHOST_CONFIG=./../cts/android-cts/repository/host_config.xml

com.android.cts.TestHost

可见， CTS 的 Java 主程序为： com.android.cts.TestHost ，同时使用了配置文件： host_config.xml. 

 

两个 cts.jar: ./out/host/linux-x86/cts/android-cts/tools/cts.jar ./out/host/linux-x86/framework/cts.jar 相同。

 

  cts 使用的目录则为： ~/mydroid/out/host/linux-x86/cts

文件列表 1

songlixin @zjujoe-desktop:~/mydroid/out/host/linux-x86/cts$ tree -L 3

.

├── all_cts_core_files_stamp

├── all_cts_files_stamp

├── android-cts

│    ├── docs

│    ├── repository

│    │    ├── host_config.xml

│    │    ├── log_2010.06.01_16.02.21_.txt

│    │    ├── log_2010.06.01_16.12.22_.txt

│    │    ├── log_2010.06.01_16.38.30_.txt

│    │    ├── log_2010.06.01_16.59.50_.txt

│    │    ├── plans

│    │    ├── results

│    │    └── testcases

│    └── tools

│        ├── cts.jar

│        ├── hosttestlib.jar

│        ├── junit.jar

│        └── startcts

├── android-cts.zip

└── temp

└── description.xml

 

我们关心的主要是 repository 目录。

 

文件列表 2

songlixin @zjujoe-desktop:~/mydroid/out/host/linux-x86/cts$ tree android-cts/repository/

android-cts/repository/

├── host_config.xml

├── plans

│    ├── Android.xml

│    ├── AppSecurity.xml

│    ├── CTS.xml

│    ├── Java.xml

│    ├── Performance.xml

│    ├── RefApp.xml

│    ├── Signature.xml

│    └── VM.xml

├── results

│    ├── 2010.06.01_16.18.03

│    │    ├── cts_result.css

│    │    ├── cts_result.xsl

│    │    ├── logo.gif

│    │    ├── newrule-green.png

│    │    └── testResult.xml

│    ├── 2010.06.01_16.18.03.zip

│    ├── 2010.06.01_16.39.38

│    │    ├── cts_result.css

│    │    ├── cts_result.xsl

│    │    ├── logo.gif

│    │    ├── newrule-green.png

│    │    └── testResult.xml

│    ├── 2010.06.01_16.39.38.zip

│    ├── cts_result.css

│    ├── cts_result.xsl

│    ├── logo.gif

│    └── newrule-green.png

└── testcases

    ├── android.core.tests.annotation.apk

    ├── android.core.tests.annotation.xml

    ├── android.core.tests.archive.apk

  。。。。。。。。。。。。。。。。。。

    ├── CtsWidgetTestCases.xml

    ├── SignatureTest.apk

    ├── SignatureTest.xml

    └── TestDeviceSetup.apk

 

5 directories, 156 files

 

可见 plans  目录下为以 xml 格式定义的测试方案

     results 目录为 测试结果。

     testcases 目录则为一个个测试用例包

host_config.xml 为配置文件

 

研究 cts 源文件

从根目录开始

Cts 的 source 包位于 android 根目录下：

songlixin @zjujoe-desktop:~/mydroid/cts$ tree -L 2

.

├── Android.mk

├── tests

│    ├── accessibilityservice

│    ├── AndroidManifest.xml

│    ├── Android.mk

│    ├── ApiDemosReferenceTest

│    ├── appsecurity-tests

│    ├── assets

│    ├── config_demo

│    ├── core

│    ├── ProcessTest

│    ├── res

│    ├── SignatureTest

│    ├── src

│    ├── tests

│    └── vm-tests

└── tools

    ├── Android.mk

    ├── annotation-helper

    ├── cts-reference-app-lib

    ├── dasm

    ├── device-setup

    ├── dex-tools

    ├── dx-tests

    ├── host

    ├── signature-tools

    ├── spec-progress

    ├── test-progress

    ├── test-progress-new

    ├── utils

    └── vm-tests

 

根目录下的 make file 非常简单：

 

songlixin @zjujoe-desktop:~/mydroid/cts$ cat Android.mk

include $(call all-subdir-makefiles)

 

看来主要内容在两个子目录下。

 

子目录 1 ： tools

根目录下的 Android.mk 与其父目录相同。

各子目录代表不同的工具，比如 dasm 编译后的位置是： ./out/host/linux-x86/bin/dasm

这里就不仔细分析这些工具了。

子目录 2 ： tests

看一下根目录下的 Android.mk:

LOCAL_PATH:= $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE_TAGS := tests

LOCAL_SRC_FILES := $(call all-java-files-under, src)/

              $(call all-java-files-under, core/runner/src)/

              src/android/app/cts/ISecondary.aidl/

              src/android/os/cts/IEmptyService.aidl

 

LOCAL_JAVA_LIBRARIES := android.test.runner

 

# Resource unit tests use a private locale and some densities

LOCAL_AAPT_FLAGS = -c xx_YY -c cs -c 32dpi -c 240dpi -c 160dpi

 

LOCAL_PACKAGE_NAME := CtsTestStubs

 

include $(BUILD_PACKAGE)

 

# Build the test APK using its own makefile, and any other CTS-related packages

include $(call all-makefiles-under,$(LOCAL_PATH))

 

所有 src 及 core/runner/src 下的 java 文件被编译成：

./out/host/linux-x86/cts/android-cts/repository/testcases/CtsTestStubs.apk

 

剩下的就是很多子目录以及子目录的子目录会被编译成 *.apk

 

Tests 目录下每个子目录包括一个 test suite.

songlixin @zjujoe-desktop:~/mydroid/cts/tests/tests$ tree -L 1

.

├── accessibilityservice

├── accounts

├── Android.mk

。。。

├── webkit

└── widget

 

Core 目录下每个子目录包含一个 test suite, 但是他们的代码在其他地方

songlixin @zjujoe-desktop:~/mydroid/cts/tests/core$ tree -L 2

.

├── Android.mk

├── annotation

│    ├── AndroidManifest.xml

│    └── Android.mk

├── archive

│    ├── AndroidManifest.xml

│    └── Android.mk

。。。

├── text

│    ├── AndroidManifest.xml

│    └── Android.mk

├── xml

│    ├── AndroidManifest.xml

│    └── Android.mk

└── xnet

    ├── AndroidManifest.xml

    └── Android.mk

 

appsecurity-tests/test-apps 目录下每个子目录包括一个 test case. 都特别简单 .

 

songlixin @zjujoe-desktop:~/mydroid/cts/tests/appsecurity-tests/test-apps$ tree -L 1

.

├── Android.mk

├── AppAccessData

├── AppWithData

├── InstrumentationAppDiffCert

├── PermissionDeclareApp

├── SharedUidInstall

├── SharedUidInstallDiffCert

├── SimpleAppInstall

├── SimpleAppInstallDiffCert

├── TargetInstrumentationApp

└── UsePermissionDiffCert

 

总结

Cts 其实就是基于 instrumentation test, Test case 有很多类型，不同的实现方式。 对我们来说，需要：

1 ）     能够执行现有的 test case

2 ）     能够模仿 api demo test case 写一些比较简单的：

可以参考： http://www.netmite.com/ /

android/mydroid/cupcake/development/pdk/docs/instrumentation_testing.html