ASP.NET Core middleware is a powerful component of the request pipeline, designed to handle HTTP requests and responses efficiently. But what happens when a request needs to kick off a task that shouldn’t block the response? Think of tasks like sending a confirmation email, generating a detailed audit log, or calling a slow third-party service. Running these directly within the middleware can severely impact your application’s performance and responsiveness.

The solution is to decouple these long-running operations from the request pipeline by offloading them to a background service. In this post, we’ll walk through the robust and recommended way to trigger background jobs from your middleware using IHostedService and an in-memory queue.

The Problem: Blocking the Pipeline

The request pipeline in ASP.NET Core is synchronous at its core, even with async/await. Each middleware component must complete its work before passing control to the next one. If your middleware performs a long-running operation, the client is left waiting until that task finishes.

For example, imagine a piece of middleware that logs detailed request information to a slow, remote data store:

// DO NOT DO THIS
app.Use(async (context, next) =>
{
    // This blocks the pipeline until the log is written
    await LogRequestDetailsToSlowStoreAsync(context.Request); 

    await next(context);
});

This approach creates a bottleneck. As traffic increases, requests will pile up waiting for these slow operations to complete, leading to high latency and a poor user experience.

The Solution: A Hosted Service and a Job Queue

The correct way to handle this is to create a “fire-and-forget” mechanism that is safe and managed by the application’s lifetime. We can achieve this by combining two key components:

  1. A Job Queue: A singleton service that holds the tasks to be executed. The middleware’s only job is to quickly add a work item to this queue. We’ll use System.Threading.Channels.Channel<T> for a high-performance, thread-safe queue.
  2. A Background Worker Service (IHostedService): A long-running service that continuously monitors the queue. When a new item appears, it dequeues it and processes it on a separate thread, completely outside the request pipeline.

This architecture ensures the middleware remains lightweight and fast, immediately passing control to the next component while the background service handles the heavy lifting.

Step-by-Step Implementation

Let’s build this system from the ground up.

1. Define the Job Queue

First, we’ll create an interface for our queue and its implementation. Using an interface allows for better testability and future flexibility (e.g., swapping the in-memory queue for a persistent one).

Create a new file IBackgroundTaskQueue.cs:

public interface IBackgroundTaskQueue
{
    ValueTask EnqueueAsync(Func<CancellationToken, ValueTask> workItem);

    ValueTask<Func<CancellationToken, ValueTask>> DequeueAsync(CancellationToken cancellationToken);
}

Now, let’s implement this using Channel<T>. A channel is a modern synchronization primitive perfect for producer/consumer scenarios like this.

Create BackgroundTaskQueue.cs:

using System.Threading.Channels;

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
    private readonly Channel<Func<CancellationToken, ValueTask>> _queue;

    public BackgroundTaskQueue(int capacity)
    {
        var options = new BoundedChannelOptions(capacity)
        {
            FullMode = BoundedChannelFullMode.Wait
        };
        _queue = Channel.CreateBounded<Func<CancellationToken, ValueTask>>(options);
    }

    public async ValueTask EnqueueAsync(Func<CancellationToken, ValueTask> workItem)
    {
        ArgumentNullException.ThrowIfNull(workItem);
        await _queue.Writer.WriteAsync(workItem);
    }

    public async ValueTask<Func<CancellationToken, ValueTask>> DequeueAsync(
        CancellationToken cancellationToken)
    {
        var workItem = await _queue.Reader.ReadAsync(cancellationToken);
        return workItem;
    }
}

Expert Insight: Using a BoundedChannel is a good practice for preventing memory exhaustion. If jobs are produced faster than they can be consumed, the Wait mode will apply back-pressure, slowing down the producer (the middleware) rather than letting the queue grow indefinitely and potentially causing an OutOfMemoryException.

2. Create the Background Worker Service

Next, we need the worker that consumes items from the queue. We’ll inherit from BackgroundService, which is the recommended base class for implementing IHostedService.

Create QueuedHostedService.cs:

public class QueuedHostedService : BackgroundService
{
    private readonly ILogger<QueuedHostedService> _logger;
    private readonly IBackgroundTaskQueue _taskQueue;

    public QueuedHostedService(
        IBackgroundTaskQueue taskQueue,
        ILogger<QueuedHostedService> logger)
    {
        _taskQueue = taskQueue;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Queued Hosted Service is running.");
        await ProcessQueueAsync(stoppingToken);
    }

    private async Task ProcessQueueAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                // DequeueAsync will wait until an item is available or the token is cancelled.
                var workItem = await _taskQueue.DequeueAsync(stoppingToken);
                await workItem(stoppingToken);
            }
            catch (OperationCanceledException)
            {
                // Prevent throwing if stoppingToken was signaled
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error occurred executing work item.");
            }
        }
    }

    public override async Task StopAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Queued Hosted Service is stopping.");
        await base.StopAsync(stoppingToken);
    }
}

This service runs a continuous loop. It waits for a work item from the queue, executes it, and then logs any errors without crashing the entire service.

3. Build the Middleware

Now we create the middleware that adds jobs to the queue. Let’s make a simple audit logging middleware.

Create AuditLoggingMiddleware.cs:

public class AuditLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<AuditLoggingMiddleware> _logger;

    public AuditLoggingMiddleware(RequestDelegate next, ILogger<AuditLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context, IBackgroundTaskQueue queue)
    {
        await _next(context);

        // Don't wait for the task to complete
        await queue.EnqueueAsync(token => 
            LogAuditAsync(context.Request.Path, context.Response.StatusCode, token));
    }

    private async ValueTask LogAuditAsync(string path, int statusCode, CancellationToken token)
    {
        // Simulate a slow I/O operation
        await Task.Delay(2000, token); 
        _logger.LogInformation("AUDIT LOG: Path: {Path}, StatusCode: {StatusCode}", path, statusCode);
    }
}

Notice that the InvokeAsync method calls EnqueueAsync and immediately continues. The actual LogAuditAsync work, with its Task.Delay, will be executed later by our QueuedHostedService.

4. Register Everything in Program.cs

Finally, we need to register our services and wire up the middleware in the dependency injection container and request pipeline.

Update your Program.cs:

var builder = WebApplication.CreateBuilder(args);

// 1. Add services to the container.
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

// 2. Register the background task queue and the hosted service
builder.Services.AddSingleton<IBackgroundTaskQueue>(_ => new BackgroundTaskQueue(100));
builder.Services.AddHostedService<QueuedHostedService>();

var app = builder.Build();

// 3. Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

// 4. Add the custom middleware to the pipeline
app.UseMiddleware<AuditLoggingMiddleware>();

app.UseAuthorization();

app.MapControllers();

app.Run();

Now, when you run your application and make a request, you’ll see an immediate response. A couple of seconds later, the audit log message will appear in your console, confirming that the job was executed in the background.

Why Not Just Use Task.Run?

You might be tempted to simplify things by just using Task.Run in your middleware:

// DANGEROUS: Avoid this pattern
app.Use((context, next) => 
{
    _ = Task.Run(() => LogSomethingSlow());
    return next(context);
});

This is a dangerous anti-pattern in ASP.NET Core for several reasons:

  • Unhandled Exceptions: If LogSomethingSlow() throws an exception, it will be unhandled on a background thread pool thread. This can tear down your entire application process.
  • Application Shutdown: The application host has no knowledge of this background task. If the app shuts down, your task will be terminated abruptly, potentially leaving data in a corrupt state.
  • Dependency Injection Scope: Task.Run doesn’t flow the HttpContext or its associated DI scope. Trying to access a scoped service (like an DbContext) from inside the task will likely result in an ObjectDisposedException.

The hosted service pattern avoids all these issues by providing a managed, long-running context for your background work that is properly integrated with the application’s lifetime and logging.

Conclusion

Triggering background jobs from middleware is a common requirement for building responsive and scalable applications. By leveraging an in-memory queue and a custom IHostedService, you can safely offload work from the request pipeline, ensuring your API remains fast and reliable. This pattern provides a solid foundation for handling background tasks within your application, and for more complex needs like persistence and retries, you can extend it or integrate dedicated libraries like Hangfire or Quartz.NET.

References