<!-- Copyright 2015 The Chromium Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.
-->
# Telemetry
Telemetry is the performance testing framework used by Chrome. It allows you
to perform arbitrary actions on a set of web pages (or any android application!)
and report metrics about it. The framework abstracts:
* Launching a browser with arbitrary flags on any platform.
* Opening a tab and navigating to the page under test.
* Launching an Android application with intents through ADB.
* Fetching data via the Inspector timeline and traces.
* Using [Web Page Replay](https://github.com/chromium/web-page-replay) to
cache real-world websites so they don’t change when used in benchmarks.
## Design Principles
* Write one performance test that runs on major platforms - Windows, Mac,
Linux, Chrome OS, and Android for both Chrome and ContentShell.
* Run on browser binaries, without a full Chromium checkout, and without
having to build the browser yourself.
* Use Web Page Replay to get repeatable test results.
* Clean architecture for writing benchmarks that keeps measurements and use
cases separate.
**Telemetry is designed for measuring performance rather than checking
correctness. If you want to check for correctness,
[browser tests](http://www.chromium.org/developers/testing/browser-tests) are
your friend.**
**If you are a Chromium developer looking to add a new Telemetry benchmark to
[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/),
please make sure to read our
[Benchmark Policy](https://docs.google.com/document/d/1bBKyYCW3VlUUPDpQE4xvrMFdA6tovQMZoqO9KCcmqqQ/preview)
first.**
## Code Concepts
Telemetry provides two major functionality groups: those that provide test
automation, and those that provide the capability to collect data.
### Test Automation
The test automation facilities of Telemetry provide Python wrappers for a number
of different system concepts.
* _Platforms_ use a variety of libraries & tools to abstract away the OS
specific logic.
* _Browser_ wraps Chrome's
[DevTools Remote Debugging Protocol](https://developer.chrome.com/devtools/docs/remote-debugging)
to perform actions and extract information from the browser.
* _Android App_ is a Python wrapper around
[`adb shell`](http://developer.android.com/tools/help/adb.html).
The Telemetry framework lives in
[`src/third_party/catapult/telemetry/`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/)
and performance benchmarks that use Telemetry live in
[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/).
### Data Collection
Telemetry offers a framework for collecting metrics that quantify the
performance of automated actions in terms of benchmarks, measurements, and story
sets.
* A
[_benchmark_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark.py)
combines a _measurement_ together with a _story set_, and optionally a set
of browser options.
* We strongly discourage benchmark authors from using command-line flags
to specify the behavior of benchmarks, since benchmarks should be
cross-platform.
* Benchmarks are discovered and run by the
[benchmark runner](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py),
which is wrapped by scripts like
[`run_benchmark`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/run_benchmark)
in `tools/perf`.
* A _measurement_ (called
[`StoryTest`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/story_test.py)
in the code) is responsible for setting up and tearing down the testing
platform, and for collecting _metrics_ that quantify the application
scenario under test.
* Measurements need to work with all story sets, to provide consistency
and prevent benchmark rot.
* You probably don't need to override `StoryTest` (see "Timeline Based
Measurement" below). If you think you do, please talk to us.
* A
[_story set_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story_set.py)
is a set of _stories_ together with a
[_shared state_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/shared_state.py)
that describes application-level configuration options.
* A
[_story_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story.py)
is an application scenario and a set of actions to run in that scenario. In
the typical Chromium use case, this will be a web page together with actions
like scrolling, clicking, or executing JavaScript.
* A _metric_ describes how to collect data about the story run and compute
results.
* New metrics should generally be
[timeline-based](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/timeline_based_metric.py).
* Metrics can specify many different types of results, including numbers,
histograms, traces, and failures.
* _Timeline Based Measurement_ is a built-in `StoryTest` that runs all
available timeline-based metrics, and benchmarks that use it can filter
relevant results.
## Next Steps
* [Run Telemetry benchmarks locally](/telemetry/docs/run_benchmarks_locally.md)
* [Record a story set](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set)
with Web Page Replay
* [Add a measurement](https://sites.google.com/a/chromium.org/dev/developers/telemetry/add_a_measurement)
* [Feature guidelines](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-feature-guidelines)
* [Profiling with Telemetry](https://sites.google.com/a/chromium.org/dev/developers/telemetry/profiling)
* [Profile generation](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-profile-generation)
* [Telemetry unittests](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-unittests)
## Contact Us or Follow Along
If you have questions, please email telemetry@chromium.org.
You can keep up with Telemetry related discussions by joining the
[telemetry group](https://groups.google.com/a/chromium.org/forum/#!forum/telemetry).
[For Googlers](http://go/telemetry)
## Frequently Asked Questions
### I get an error when I try to use recorded story sets.
The recordings are not included in the Chromium source tree. If you are a Google
partner, run `gsutil config` to authenticate, then try running the test again.
If you don't have `gsutil` installed on your machine, you can find it in
`build/third_party/gsutil/gsutil`.
If you are not a Google partner, you can run on live sites with
--use-live-sites` or
[record your own](http://dev.chromium.org/developers/telemetry/record_a_page_set)
story set archive.
### I get mysterious errors about device\_forwarder failing.
Your forwarder binary may be outdated. If you have built the forwarder in
src/out that one will be used. if there isn't anything there Telemetry will
default to downloading a pre-built binary. Try re-building the forwarder, or
alternatively wiping the contents of `src/out/` and running `run_benchmark`,
which should download the latest binary.
### I'm having problems with keychain prompts on Mac.
Make sure that your keychain is
[correctly configured](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-mac-keychain-setup).