Two rate limiters in same pipeline

[lonix1]

I need to throttle requests to some external API: 14 requests per second and 50k requests per day.

I tried to chain two FixedWindowRateLimiter policies:

var resiliencePipeline = new ResiliencePipelineBuilder()
    .AddRateLimiter(new FixedWindowRateLimiter(new FixedWindowRateLimiterOptions {
        PermitLimit          = 14,
        QueueLimit           = 0,
        Window               = TimeSpan.FromSeconds(1),
        QueueProcessingOrder = QueueProcessingOrder.OldestFirst
    }))
    .AddRateLimiter(new FixedWindowRateLimiter(new FixedWindowRateLimiterOptions {
        PermitLimit          = 50_000,
        QueueLimit           = 0,
        Window               = TimeSpan.FromDays(1),
        QueueProcessingOrder = QueueProcessingOrder.OldestFirst
    }))
    .Build();

var result = await resiliencePipeline.ExecuteAsync(
    token => DoWork(args),
    cancellationToken);

That seemed to prefer the second one; I guess the latter configuration overrides the former.

Can Polly somehow keep track of both limits for me, or is that something I must handle manually?

[martincostello]

You need to create a chained rate limiter and give that to Polly. Then it's an implementation detail of the rate limiter itself, and Polly just needs one rate limit.

[lonix1]

Thanks.

After looking at that example, I'm still a little confused. I tried this:

var chainedRateLimiter = PartitionedRateLimiter.CreateChained(           // <--- missing <TResource>
    new FixedWindowRateLimiter(new FixedWindowRateLimiterOptions { /* ... */ }),
    new FixedWindowRateLimiter(new FixedWindowRateLimiterOptions { /* ... */ })
);

var resiliencePipeline = new ResiliencePipelineBuilder()
    .AddRateLimiter(chainedRateLimiter)
    .Build();

var result = await resiliencePipeline.ExecuteAsync(
    token => DoWork(args),
    cancellationToken);

But of course that won't compile as PartitionedRateLimiter.CreateChained complains that

The type arguments for method 'PartitionedRateLimiter.CreateChained' cannot be inferred from the usage.

I am not using HttpContext as in that linked example. I'm not sure what "resource" I'm encapsulating here.

Also PartitionedRateLimiter (Object <- PartitionedRateLimiter) and FixedWindowRateLimiter (Object <- RateLimiter <- ReplenishingRateLimiter <- FixedWindowRateLimiter) have separate class hierarchies, so I'm unsure whether they can be used this way.

What is the recommended approach?

[martincostello]

Something like this should work for you:

var args = Array.Empty<string>();
var cancellationToken = CancellationToken.None;
var key = "partition-key";

var first = PartitionedRateLimiter.Create<ResilienceContext, string>((context) =>
{
    return RateLimitPartition.GetFixedWindowLimiter(
        key,
        (_) => new() { /* ... */ });
});

var second = PartitionedRateLimiter.Create<ResilienceContext, string>((context) =>
{
    return RateLimitPartition.GetFixedWindowLimiter(
        key,
        (_) => new() { /* ... */ });
});

var chainedRateLimiter = PartitionedRateLimiter.CreateChained(first, second);

var resiliencePipeline = new ResiliencePipelineBuilder()
    .AddRateLimiter(new RateLimiterStrategyOptions
    {
        RateLimiter = (context) => chainedRateLimiter.AcquireAsync(context.Context),
    })
    .Build();

var result = await resiliencePipeline.ExecuteAsync(
    token => DoWork(args),
    cancellationToken);

If that works for you, I'll look at add an example to our documentation. It's broadly similar to the partitioned rate limiter example.

[lonix1]

Thanks I just tried that and it works!

I have the pipeline in a try-catch which catches RateLimiterRejectedException and then waits for the retry delay to elapse before continuing. Watching both rate limiters in action separately and yet simultaneously is really cool.

I'd have never figured our the above on my own, one needs to be a Polly insider. Adding it to the docs would probably help many others too.

[lonix1]

In case it may be useful to others, I have something like this:

var requests = await PrepareRequests();

while (requests.Any()) {
  try {
    var request = requests.First();
    await _resiliencePipeline.ExecuteAsync(
      cancellationToken => DoWork(request, cancellationToken),
      cancellationToken);
    requests.Remove(request);
  }
  catch (RateLimiterRejectedException e) {
    if (e.RetryAfter is TimeSpan retryAfter) {
      Console.WriteLine($"Throttled; retrying in {retryAfter}...");
      await Task.Delay(retryAfter);
    }
    else {
      Console.Error.WriteLine("Error");
      Environment.Exit(1);      
    }
  }
}

BTW, I got the if (e.RetryAfter is TimeSpan retryAfter) from the docs. Why would that ever be null? UPDATE: if one uses a different rate limiter, eg. ConcurrencyLimiter.

I might combine that with another policy that gives up after some time, just as a precaution (if the rate limit is per-day, I don't want some background process pending for that long, rather just timeout). If I add that I'll update this post.