There are multiple ways to use Firebase Test Lab for Android to run tests on your Android app, including the command line interface, Android Studio, and the Test Lab UI in the Firebase console. However you choose to start your tests, the results are stored in the Firebase project that you specify. This page describes how to review and analyze these test results.
Key concepts
To see the results from all your previous test runs, select Test Lab in the left navigation panel of your project in the Firebase console. This page displays all the test runs from the apps that you have tested with your project using Test Lab.
To review test results, you first need to understand three concepts:
-
Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are device type, Android API level, locale, and screen orientation. Before testing your app, you select attributes from each dimension, and Test Lab creates a list of all valid combinations of these attributes. Test Lab supports testing your app on up to 200 unique combinations at a time.
-
Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. For example, if you select five device types (Asus Google Nexus 7, LG G3, Samsung Galaxy Note 2, etc.), two Android API levels (21, 22), two locales (en_US, es_ES), and two screen orientations (landscape, portrait), the test contains 5*2*2*2 = 40 test executions. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.
-
Test Matrix. The test matrix is the set of test executions defined by the combinations of the test dimension attributes that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.
The following sections explain how to navigate test results.
Interpreting test history results
When you navigate to your test results by selecting Test Lab, you see the results of tests you have run so far.
Testing history is grouped by app. Only the most recent five test matrices are shown for each app; if more are available, you can click the All Matrices link at the bottom of the app test list to see the complete list for that app.
Interpreting test matrix results
When starting a test through Test Lab UI, you are redirected to a page where you can see your test matrix and click a specific test execution to view test results. Android Studio and the gcloud command provide a URL for the test matrix results page as well.
In a typical test matrix, you might run a test across a dozen or so different devices. Each test execution can have a different outcome. The possible outcomes for any test execution in a test matrix include:
- Passed
: No failures were encountered. - Failed
: At least one failure was
encountered. - Inconclusive
: Test results were
inconclusive, possibly due to a Test Lab error. - Skipped
: The selected dimension values
for some test executions in the matrix were incompatible. This occurs when
devices that you selected are incompatible with one or more of the Android API
levels that you selected.
To review aggregated test results for all test matrices for a given app in your Firebase project, click the name of the app, as shown in the example test matrix results page shown below.
Example of a test matrix results page with only four test executions
This takes you to the test matrix list for your app, where you can click the name of any test matrix to see the test matrix results, and where you can click the name of the app (shown in the red box below) to view the test matrix list for other apps associated with your Firebase project.
Example test matrix list page
A test matrix can either pass, fail, or be inconclusive. A test matrix is shown as failed or inconclusive if any test executions in that matrix fail or are inconclusive.
Interpreting results from a single test execution
From the test matrix results page, click on one of the test executions to see the result of that specific test execution.
Example test execution results page
On this page, you can see the results for specific test cases (which correspond to methods in your test APK), test log output, screenshots, and videos for that particular test.
For Robo test, you can also see an Activity Map tab that graphically shows the UI paths that were visited by Robo test.
Need help? Visit our