In modern microservice architectures, a single user request can trigger a cascade of calls across numerous services. When something goes wrong, pinpointing the source of failure or a performance bottleneck becomes a significant challenge. This is where distributed tracing comes in, and OpenTelemetry has emerged as the industry standard for implementing it.

This post will guide you through integrating OpenTelemetry for distributed tracing into your ASP.NET Core applications. We’ll cover the core concepts, set up a basic project, and visualize traces in a backend like Jaeger.

What is Distributed Tracing?

Distributed tracing allows you to follow a single request’s journey from start to finish as it moves through different services. It stitches together individual operations into a unified view, helping you understand the flow, identify latency issues, and debug errors effectively.

Key concepts include:

  • Trace: Represents the entire end-to-end path of a request. A trace is a collection of spans.
  • Span: Represents a single unit of work or operation within a trace, like an HTTP call or a database query. Each span has a start time, duration, and associated metadata (tags).
  • Trace Context: A set of unique identifiers (like a TraceId and SpanId) that are passed between services with each request. This context is what allows the system to link all related spans into a single trace.

Think of it like tracking a package. The trace is the entire journey from the warehouse to your door. Each stop or transfer (e.g., leaving the warehouse, arriving at a sorting facility, out for delivery) is a span. The tracking number is the trace context that links all these events together.

Why OpenTelemetry?

Before OpenTelemetry (OTel), instrumentation was often tied to a specific observability vendor. If you wanted to switch vendors, you had to re-instrument your entire application.

OpenTelemetry solves this by providing a single, vendor-neutral set of APIs, libraries, and agents for collecting telemetry data (traces, metrics, and logs). By instrumenting your code with OTel, you can send your data to any OTel-compatible backend without changing your application code. This flexibility is a massive win for long-term maintainability.

Setting Up Tracing in ASP.NET Core

Let’s set up a simple two-service application and trace a request between them. We’ll have a GatewayService that calls a ProductService.

Step 1: Install NuGet Packages

First, create two new web API projects. In both projects, install the necessary OpenTelemetry packages.

# In both GatewayService and ProductService projects
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

# For demonstration, we'll use the console and Jaeger exporters
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Exporter.Jaeger
  • OpenTelemetry.Extensions.Hosting: Integrates OTel with the generic host.
  • OpenTelemetry.Instrumentation.AspNetCore: Automatically creates spans for incoming ASP.NET Core requests.
  • OpenTelemetry.Instrumentation.Http: Automatically creates spans for outgoing HttpClient requests.
  • OpenTelemetry.Exporter.Console: Prints traces to the console, great for quick debugging.
  • OpenTelemetry.Exporter.Jaeger: Sends traces to a Jaeger backend.

Step 2: Configure OpenTelemetry in Program.cs

In both GatewayService and ProductService, configure OpenTelemetry in your Program.cs file. The setup is nearly identical for both services. We’ll use the minimal API syntax available in modern .NET.

// Program.cs
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Define a service name for the resource
var serviceName = "GatewayService"; // Change to "ProductService" in the other project

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(serviceName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter()
        .AddJaegerExporter(opts =>
        {
            // The default endpoint is http://localhost:6831, using UDP.
            // You can configure it if your Jaeger agent is running elsewhere.
        }));

// Add HttpClient for the GatewayService to call the ProductService
builder.Services.AddHttpClient(); 

// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();
app.MapControllers();
app.Run();

Callout: The Importance of Service Name Setting a clear and unique serviceName using ConfigureResource is crucial. This is the identifier that will appear in your tracing backend (like Jaeger). Without it, it’s difficult to distinguish which service generated which span, defeating the purpose of distributed tracing.

Step 3: Create Endpoints to Simulate a Call Chain

In ProductService, create a simple endpoint.

// Controllers/ProductsController.cs in ProductService
[ApiController]
[Route("[controller]")]
public class ProductsController : ControllerBase
{
    private static readonly string[] Products = new[] { "Laptop", "Mouse", "Keyboard" };

    [HttpGet]
    public IEnumerable<string> Get()
    {
        // Simulate some work
        Thread.Sleep(50); 
        return Products;
    }
}

In GatewayService, create an endpoint that calls the ProductService.

// Controllers/GatewayController.cs in GatewayService
[ApiController]
[Route("[controller]")]
public class GatewayController : ControllerBase
{
    private readonly HttpClient _httpClient;

    public GatewayController(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    [HttpGet("products")]
    public async Task<IActionResult> GetProducts()
    {
        // Ensure you use the correct URL for your ProductService
        var products = await _httpClient.GetStringAsync("https://localhost:7001/products");
        return Ok(products);
    }
}

Remember to update the port number (7001) to match the actual port your ProductService is running on.

Step 4: Visualize Traces with Jaeger

The console exporter is useful, but a graphical UI is much better. Let’s run Jaeger using Docker.

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 16686:16686 \
  -p 14268:14268 \
  jaegertracing/all-in-one:latest

Now, run both your GatewayService and ProductService applications. Make a GET request to https://localhost:PORT/gateway/products (using the port for your GatewayService).

Open the Jaeger UI in your browser at http://localhost:16686. In the “Service” dropdown, you should see GatewayService and ProductService. Select GatewayService and click “Find Traces.”

You will see a trace that contains two spans:

  1. A parent span for the incoming request to /gateway/products in GatewayService.
  2. A child span for the outgoing HTTP call from GatewayService to ProductService.

This waterfall view instantly shows you the entire request flow and how much time was spent in each service.

Adding Custom Instrumentation

Automatic instrumentation is powerful, but sometimes you need to trace specific business logic within a method. You can create custom spans using System.Diagnostics.ActivitySource and Activity.

  1. Define an ActivitySource: Create a static ActivitySource instance with a unique name.

    // In a new static class or within your service class
public static class Tracing
{
    public static readonly ActivitySource MyActivitySource = new("MyCompany.MyApp.MyLibrary");
}

  2. Start an Activity (Span): Wrap the code you want to trace in a using block with StartActivity.

    // Inside a method in ProductService
[HttpGet]
public IEnumerable<string> Get()
{
    using var activity = Tracing.MyActivitySource.StartActivity("FetchingProductsFromDb");

    // Add custom metadata (tags)
    activity?.SetTag("db.statement", "SELECT * FROM Products");
    activity?.SetTag("products.count", Products.Length);

    // Simulate database call
    Thread.Sleep(50); 
    return Products;
}

  3. Register the Source: Make sure OpenTelemetry is configured to listen to your custom ActivitySource.

    // In Program.cs
.WithTracing(tracing => tracing
    .AddSource(Tracing.MyActivitySource.Name) // Add this line
    .AddAspNetCoreInstrumentation()
    // ... other configuration

Now when you run the request again, you’ll see a third, nested span in your Jaeger trace named FetchingProductsFromDb, complete with the custom tags you added. This provides invaluable, application-specific context for debugging.

Conclusion

Distributed tracing is no longer a “nice-to-have” but a fundamental requirement for building reliable and observable microservices. With OpenTelemetry and its seamless integration into the ASP.NET Core ecosystem, setting it up has never been easier. By leveraging automatic instrumentation and adding custom spans where needed, you can gain deep visibility into your system’s behavior, drastically reducing the time it takes to diagnose and resolve issues.

References