Firebase Test Lab for Android provides cloud-based infrastructure for testing
Android apps, including full integration with the
gcloud command-line client.
This document covers the installation and configuration required
to get started using Test Lab from the gcloud
command line interface.
Create a Firebase project
If you don't have a Firebase project for your app, go to the Firebase console and click Create Project to create one now. You will need ownership or edit permissions in your project.
A Firebase "Blaze" plan is required to use Test Lab. To learn more, see Test Lab billing.
Configure your local Google Cloud SDK environment
-
If not present, install the Google Cloud SDK on your local system.
-
Make sure your Cloud SDK installation is up to date and includes the
gcloud beta
commands:gcloud components update beta
-
Set your current gcloud project to your project ID:
gcloud config set project <YOUR-PROJECT-ID>
-
Make sure your authentication credentials are current:
gcloud auth login
Choosing test configurations
In this example, you will run some tests on a simple note-taking Android app named Notepad.
-
Download the binary APK file for the Notepad app (app-debug-unaligned.apk) and its corresponding instrumentation tests (app-debug-test-unaligned.apk) provided in the NotePad/app/build/outputs/apk/ directory of notepad.zip.
-
Get the current list of Android devices available to test against:
$ gcloud beta test android devices list
The first column of the command output,
DEVICE_ID
, contains the identifier that you can use later to run tests on a device. TheOS_VERSION_ID
column lists the operating system versions supported by that device. If you don't specify which device ID(s) to test against, the default noted under theTAGS
column is used. -
Get the current list of Android OS versions available to test against:
$ gcloud beta test android versions list
You can use an identifier from either of the first two columns of command output (
OS_VERSION_ID
andVERSION
), to later run tests against an Android OS version. If you don't specify the Android OS versions to test against, the default noted under theTAGS
column is used. -
Get the current list of locales available to test against:
The first column of the command output,$ gcloud beta test android locales list
LOCALE
, contains the identifier that you can use later to run tests against a locale. If you don't specify the locales to test against, English is used as the default locale. Command output is not shown here because hundreds of locales are available.
Running the Robo test
Even if you don't have any instrumentation tests, you can still look for bugs in your app. Use the Robo test to perform automated review of your app's user interface. Robo test exercises the app by performing a static analysis of the various paths through the app's user interface, and then crawling through the app to find crashes and other potential issues.
Let's start by running an example command:
gcloud beta test android run \ --type robo \ --app app-debug-unaligned.apk \ --device-ids Nexus6,Nexus7 \ --os-version-ids 19,21 \ --locales en,fr \ --orientations portrait,landscape \ --timeout 90s
The --type robo
parameter is implicit if no --type
value is specified.
You can see the complete set of command line options for running tests by
typing: gcloud help beta test android run
.
As an alternative to specifying all these arguments on the command line,
you can optionally specify your arguments in a YAML-formatted argument file.
Run gcloud topic arg-files
to learn how to use this feature.
See Analyze your test results section to learn how to investigate the test results from the Robo test.
Running your instrumentation tests
Now let's use the gcloud
command line tool to run the Notepad app's
Espresso
tests on your specified Android device configurations, using the
instrumentation
test type to run the tests in app-debug-test-unaligned.apk
.
gcloud beta test android run \ --type instrumentation \ --app app-debug-unaligned.apk \ --test app-debug-test-unaligned.apk \ --device-ids Nexus6,Nexus7 \ --os-version-ids 19,21 \ --locales en,fr \ --orientations portrait,landscape
The --type instrumentation
parameter is implicit if a test APK has been
specified with --test
. As an alternative to specifying all these arguments on
the command line, you can optionally specify your arguments in a YAML-formatted
argument file. Run gcloud topic arg-files
to learn how to use this feature.
Note: You can also control how Test Lab runs your instrumentation tests
using additional flags that are not shown above. For example, you can use the
--test-targets
flag to test just one class used by your test APK, or to test
just one method from a class used by your test APK. To learn more, see
gcloud beta test android run.
Analyze your test results
After a few minutes, a basic summary of your test results is printed by the gcloud tool:
The output of your command line test run will also include a link to view test results. To learn more about how to interpret these results, see Analyzing Test Results.
Scripting gcloud commands with Test Lab
You can use shell scripts or batch files to automate mobile app testing commands that you would otherwise run using the gcloud command line. The following example bash script runs an instrumentation test with a two minute timeout, and reports if the test run completed successfully:
if gcloud beta test android run --app my-app.apk --test my-test.apk --timeout 2m then echo "Test matrix successfully finished" else echo "Test matrix exited abnormally with non-zero exit code: " $? fi
Script exit codes
Test Lab provides several exit codes that you can use to better understand the results of tests that you run using scripts or batch files.
Scripting exit codes for Test Lab
Exit Code | Notes |
---|---|
0 | All test executions passed. |
1 | A general failure occurred. Possible causes include: a filename that does not exist, or an HTTP/network error. |
2 | Testing exited because unknown commands or arguments were provided. |
10 | One or more test cases (tested classes or class methods) within a test execution did not pass. |
15 | Firebase Test Lab for Android could not determine if the test matrix passed or failed because of an unexpected error. |
18 | The test environment for this test execution is not supported because of incompatible test dimensions. This error might occur if the selected Android API level is not supported by the selected device type. |
19 | The test matrix was canceled by the user. |
20 | A test infrastructure error occurred. |