Tideways

Tideways is an APM tool that can give insight into the performance of applications across multiple environments. It provides both a high-level profiler that can show the time spent performing operations in different "spans" and can take lower-level traces, instrumenting all function calls.

How it works

Tideways is made up of a few different components:

  • The Tideways app itself, which allows you to view profiles and traces.
  • Browser extensions for toggling profiling of your requests.
  • A PHP extension based on XHProf for collecting data.
  • A daemon for sampling data to send to the Tideways app for processing.
  • A proxy for sampling requests across multiple daemons.

Tideways components

Concepts

  • Applications are the what we're monitoring, either individual services or monoliths.
  • Environments provide separation between production, UAT, testing and development environments, allowing you to compare performance across environments.
  • Traces come in two varieties:
    • Timeline traces are the lighter-weight of the two, providing a very high level overview of what an application is doing in the form of timespans for high-level categories of operations (e.g. SQL queries or Redis cache operations).
    • Callgraph traces provide much deeper profiling insight into how a request performed.

Traces are taken from a configurable number of requests, then the collected data is sampled by the daemons before being sent to Tideways for analysis.

Profiling a request

Requests are sampled at random, but it's possible to alter the sampling behaviour if you need profile data in a pinch:

  • If you know the route, but don't have a reliable test case for the problem, you can boost traces for the route that interests you. This instructs Tideways to increase the volume of samples to retain for aa specified route. To start this, browse to a trace for the route and click the Boost traces button in the upper-right.
  • If you can reliably reproduce an issue and want to record your own traffic, install the browser extension, then click the extension button. Click Take profile to profile the current page and all embedded resources or click the more arrow to the right and choose a time to sample all requests made in that time window.

Declaring custom timespans

The timeline view represents the time spent in different function calls. We can declare custom timespans that we want Tideways to represent here:

use Tideways\Profiler;

Profiler::watchCallback('sqlsrv_native_moodle_database::do_query', 'handle_do_query');

function handle_do_query($context) {
    $sql = $context['args'][0];
    $params = array_key_exists(1, $context['args'])
            ? $context['args'][1] : [];

    $span = Profiler::createSpan('sql');
    $span->annotate([
        'sql' => $sql,
        'params' => $params,
    ]);

    return $span;
}

Troubleshooting

Proxy doesn't respect daemon environment values

This is a known (but strangely undocumented) limitation of the Tideways proxy; it requires that all daemons it's collecting for submit to the same environment and that the --env option is specified for each proxy instance.

Daemon "cannot validate certificate" for proxy

When configuring the Tideways proxy to collect data across multiple daemons, the following error can be seen in the daemon log:

sender: 2020/02/05 17:23:50 Could not fetch configurations: Get https://192.168.120.100:8137/api/config: x509: cannot validate certificate for 192.168.120.100 because it doesn't contain any IP SANs
2020/02/05 17:23:50 Fetching key hash algorithm failed, retrying after 64 seconds.

Likewise, in the proxy log:

2020/02/05 17:16:22 http: TLS handshake error from 192.168.120.80:39798: remote error: tls: bad certificate

This indicates that the X.509 certificate used for authentication between the daemon and the proxy doesn't contain a SAN for the server's IP address. To rectify this we'll need to replace the certificate with one that includes the appropriate names.