Cloud Trace

Cloud Trace is Google Cloud's distributed tracing service. It collects latency data from your applications and displays it in near real-time, helping you understand how requests propagate through your distributed system, identify performance bottlenecks, and diagnose latency issues. It is the GCP equivalent of AWS X-Ray or Azure Application Insights distributed tracing.

---

What is Distributed Tracing?

In a monolithic application, a single request is handled by one process, and profiling that process reveals where time is spent. In a distributed system — where a single user request may traverse an API gateway, multiple microservices, a cache, a database, and a message queue — understanding end-to-end latency requires distributed tracing.

Trace Terminology

Term	Description
Trace	The complete journey of a single request through the system
Span	A single unit of work within a trace (e.g., one HTTP call, one database query)
Root span	The first span in a trace — typically the initial user request
Child span	A span that is initiated by another span (parent-child relationship)
Trace context	Metadata propagated between services to correlate spans into a single trace
Latency	The time taken to complete a span or trace

Trace Visualisation

User Request (root span) --------- 450ms total ---------
  |
  +-- API Gateway (span 1) -------- 20ms --------
  |
  +-- Auth Service (span 2) ------- 35ms --------
  |
  +-- Order Service (span 3) ------ 380ms -------
        |
        +-- Database Query (span 4) -- 120ms --
        |
        +-- Payment API (span 5) ----- 230ms --
              |
              +-- Fraud Check (span 6) - 180ms -

This waterfall view immediately shows that the Payment API call (and specifically the Fraud Check) is the primary source of latency.

---

Automatic Tracing on GCP

Several Google Cloud services automatically generate trace data without any code changes:

Service	Automatic Tracing
Cloud Run	Incoming HTTP requests are automatically traced
App Engine	Requests are traced by the runtime
Cloud Functions	Function invocations are traced
Cloud Load Balancing	Load balancer latency is captured as a span
Cloud SQL	Database query latency (when using supported client libraries)

Trace Context Propagation

For traces to span multiple services, each service must propagate the trace context. Google Cloud uses the X-Cloud-Trace-Context header and also supports the W3C traceparent header:

X-Cloud-Trace-Context: TRACE_ID/SPAN_ID;o=TRACE_TRUE
traceparent: 00-TRACE_ID-SPAN_ID-FLAGS

---

Instrumenting Your Application

Using OpenTelemetry (Recommended)

OpenTelemetry is the industry-standard, vendor-neutral framework for distributed tracing. Google Cloud fully supports OpenTelemetry for trace export:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

# Set up the tracer provider
provider = TracerProvider()
exporter = CloudTraceSpanExporter()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Create spans in your application
tracer = trace.get_tracer(__name__)

@app.route('/api/orders/<order_id>')
def get_order(order_id):
    with tracer.start_as_current_span("get_order") as span:
        span.set_attribute("order.id", order_id)

        with tracer.start_as_current_span("fetch_from_database"):
            order = db.query(f"SELECT * FROM orders WHERE id = %s", order_id)

        with tracer.start_as_current_span("enrich_with_customer_data"):
            customer = customer_service.get(order.customer_id)

        return jsonify(order)

Using OpenTelemetry (Node.js)

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { TraceExporter } = require('@google-cloud/opentelemetry-cloud-trace-exporter');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

const provider = new NodeTracerProvider();
provider.addSpanProcessor(new BatchSpanProcessor(new TraceExporter()));
provider.register();

registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

---

Trace Explorer