Firebase Test Lab for Android Overview

Firebase Test Lab for Android runs tests against a test matrix that you create to validate your app across a range of device types and device configurations. To understand how this works, we first need to explain test dimensions and test executions.

• Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are device type, Android API level, locale, and screen orientation. Before testing your app, you select attributes from each dimension, and Test Lab creates a list of all valid combinations of these attributes. Test Lab supports testing your app on up to 200 unique combinations at a time.

• Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. For example, if you select five device types (Asus Google Nexus 7, LG G3, Samsung Galaxy Note 2, etc.), two Android API levels (21, 22), two locales (en_US, es_ES), and two screen orientations (landscape, portrait), the test contains 5*2*2*2 = 40 test executions. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.

• Test Matrix. The test matrix is the set of test executions defined by the combinations of the test dimension attributes that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.

You can choose how to initiate testing of your app:

• Android Studio integration lets you test your app from your development environment.
• The gcloud command line interface enables you run tests from the command line interactively, and is also well suited for scripting as part of your automated build and testing process.
• The Test Lab UI in the Firebase console lets you upload an app binary and initiate testing from anywhere.

You can also test your app with Test Lab when you upload and publish your app's APK files to the Play Store using either the alpha or beta channel. For more information, see Use pre-launch reports to identify issues and Robo test.

Choose your device type

Test Lab supports testing on several makes and models of Android devices installed and running in a Google data center. Tests on devices help you to detect issues that might not occur when testing your app using the Android emulator.

Currently, the following types of devices are available:

Table 1: Devices used by Test Lab

Choose your testing method

You can use Test Lab to validate the stability of your app's user experience and the compatibility of your app across a range of devices.

Two testing methods are available to help you to test your app: You can run Instrumentation tests and the Test Lab Robo test.

Instrumentation tests

Instrumentation tests are tests that you or your team have written specifically to test your app, using the Espresso, Robotium and UI Automator 2.0 Android test frameworks. Instrumentation tests run for up to 30 minutes on physical devices, and for up to 60 minutes on virtual devices. Test results are provided for all test cases that complete execution during that time.

When you write instrumentation tests, you create a second .APK module that you will later upload to Test Lab along with the .APK module for your app. To learn about creating test APKs, see Test APK Module.

Robo test

Even if you have not written any instrumentation tests for your app, you can use Robo test to exercise your app by simulating user actions. To learn more, see Robo test.

Review your test results

Regardless of how you initiate your tests, all your test results are managed by Test Lab and can be viewed online. Test results include test logs and the details of any app failure. For Robo test, videos and screenshots are also available. See Analyzing Test Results for details.

When you initiate a test from Android Studio, you can also review test results from inside your development environment.

Billing

Firebase is available to use with three pricing plans: Spark (free), Flame, and Blaze.

To use Test Lab, upgrade to the Firebase Blaze plan. To upgrade your billing plan, click UPGRADE in the Firebase console, and then click SELECT PLAN in the Blaze plan column.

With the Blaze plan, you can use Test Lab to test your app at the following hourly rates:

• $5 per hour for each physical device
• $1 per hour for each virtual device

Charges are calculated on a per-minute basis, and you are only charged for the time required to run your tests (not for time required to install your app or collect test results).

Test Lab and mobile advertising

For app developers that use or work with digital advertising providers (e.g., ad networks, demand-side platforms), and for digital advertising providers:

Test Lab provides developers with a scalable app testing infrastructure that automates app testing. Unfortunately, this capability could be misused by malicious apps designed to generate fraudulent ad revenue.

To mitigate this issue:

• App developers should notify any digital advertising providers they work with to filter out revenues and all corresponding traffic generated from devices that belong to testing providers, including Test Lab.
• Digital advertising providers can filter ad revenues and all corresponding traffic generated from Test Lab by filtering traffic originating from the following IP address blocks: 108.177.6.0/24 (for physical devices) and 104.196.0.0/16 (for virtual devices).