Skip to main content

​
BaseRateLimiter

from langchain.rate_limiters import BaseRateLimiter
Base class for rate limiters. Re-exported from langchain_core.rate_limiters. Usage of the base limiter is through the acquire and aacquire methods depending on whether running in a sync or async context. Implementations are free to add a timeout parameter to their initialize method to allow users to specify a timeout for acquiring the necessary tokens when using a blocking call.

​
Current limitations

  • Rate limiting information is not surfaced in tracing or callbacks. This means that the total time it takes to invoke a chat model will encompass both the time spent waiting for tokens and the time spent making the request.

​
Methods

​
acquire

def acquire(self, *, blocking: bool = True) -> bool
Attempt to acquire the necessary tokens for the rate limiter. This method blocks until the required tokens are available if blocking is set to True. If blocking is set to False, the method will immediately return the result of the attempt to acquire the tokens.
​
blocking
bool
default:"True"
If True, the method will block until the tokens are available. If False, the method will return immediately with the result of the attempt.
​
returns
bool
True if the tokens were successfully acquired, False otherwise.

​
aacquire

async def aacquire(self, *, blocking: bool = True) -> bool
Attempt to acquire the necessary tokens for the rate limiter (async version). This method blocks until the required tokens are available if blocking is set to True. If blocking is set to False, the method will immediately return the result of the attempt to acquire the tokens.
​
blocking
bool
default:"True"
If True, the method will block until the tokens are available. If False, the method will return immediately with the result of the attempt.
​
returns
bool
True if the tokens were successfully acquired, False otherwise.

​
InMemoryRateLimiter

from langchain.rate_limiters import InMemoryRateLimiter

rate_limiter = InMemoryRateLimiter(
    requests_per_second=0.1,
    check_every_n_seconds=0.1,
    max_bucket_size=10,
)
An in-memory rate limiter based on a token bucket algorithm. This is an in-memory rate limiter, so it cannot rate limit across different processes. It is thread safe and can be used in either a sync or async context. The in-memory rate limiter is based on a token bucket. The bucket is filled with tokens at a given rate. Each request consumes a token. If there are not enough tokens in the bucket, the request is blocked until there are enough tokens. These tokens have nothing to do with LLM tokens. They are just a way to keep track of how many requests can be made at a given time.

​
Current limitations

  • The rate limiter is not designed to work across different processes. It is an in-memory rate limiter, but it is thread safe.
  • The rate limiter only supports time-based rate limiting. It does not take into account the size of the request or any other factors.
​
requests_per_second
float
default:"1"
The number of tokens to add per second to the bucket. The tokens represent “credit” that can be used to make requests.
​
check_every_n_seconds
float
default:"0.1"
Check whether the tokens are available every this many seconds. Can be a float to represent fractions of a second.
​
max_bucket_size
float
default:"1"
The maximum number of tokens that can be in the bucket. Must be at least 1. Used to prevent bursts of requests.

​
Example

import time

from langchain.rate_limiters import InMemoryRateLimiter

rate_limiter = InMemoryRateLimiter(
    requests_per_second=0.1,  # Can only make a request once every 10 seconds
    check_every_n_seconds=0.1,  # Wake up every 100 ms to check whether allowed to make a request
    max_bucket_size=10,  # Controls the maximum burst size
)

from langchain_anthropic import ChatAnthropic

model = ChatAnthropic(
    model_name="claude-sonnet-4-5-20250929", rate_limiter=rate_limiter
)

for _ in range(5):
    tic = time.time()
    model.invoke("hello")
    toc = time.time()
    print(toc - tic)

​
Methods

Inherits acquire and aacquire methods from BaseRateLimiter.