Why API Rate limiting sucks

Introduction

The growth in APIs (Application Programming Interfaces) plays a pivotal role in enabling seamless communication between various applications and services. However, with the increasing use of APIs, the risk of security threats and abuse by malicious actors also rises. Old fashioned security and bot management of APIs rely on API rate limiting alone.

Why? Fingerprint bot detection, which looks for signs of bot activity from the actual device fingerprint, doesn’t work on API’s as all the traffic to the API is automated. No humans are present. So the only option is to degrade the API service for all users.

VerifiedVisitors has state-of-the-art Machine Learning based behavioural detection, which instead of just crippling the API service for all your users, instead detects the abusive traffic that is data mining or looking to scrape the entire API service systematically. This allows legitimate users to have access on demand without rate limiting, but ensures the abusers of the service are stopped.

API Rate Limiting What is API Rate Limiting?

API Rate Limiting is a mechanism that restricts the number of requests an application or user can make to an API within a specific time frame. By implementing rate limits, you ensure that a single entity cannot overwhelm the API server with excessive requests, thus preventing potential abuse, denial-of-service attacks, and server overloads. Typically the rate limit is set across the entire API.

Why API Rate Limiting doesn’t work

1. Security Enhancement

API Rate Limiting itself just acts as a bottleneck for all your customers. What is needed is a behavioural bot detector, that can distinguish fair and normal usage, for systematic data scrapping and API abuse of service. Behavioural based bot detection acts as a powerful deterrent against malicious bots attempting to access sensitive data or exploit vulnerabilities in your system. Identifying and blocking suspicious traffic patterns, means that you can punish the API abusers, leaving the vast majority of legitimate users free to use the service without artificial restrictions.

CASE STUDY

• Brent is a hardworking developer who is in charge of a large e-commerce API service. The service was very popular, which was great, but he quickly became concerned about large spikes in the API traffic which started to slow down overall performance. The spikes became heavier, and finally started to create service outages which was affecting customers. Brent knew the API was well designed (he'd built it himself) and he's taken a lot of trouble to scale the service, design the schema, and ensure the queries were optimised for users. However, the data source was a legacy on-premise database that couldn't be migrated as an elastic cloud service.
• Brent struggled for months as automated bot traffic constantly farmed the entire data set. The API crashed, regularly, the on-premise service really began to struggle. Brent blocked IPs; the attacks just rotated. Brent tried to block user agents; again they just rotated, or hid in common user agents that would have blocked legitimate traffic. Brent reduced the API query allowance per month; then rate limited the entire service. Whatever he tried, the automated bots quickly adapted. The automated bots simply went under the rate limiting threshold and just took days to systematically harvest all the data, while constantly affecting legitimate users. Sometimes he even had to take the API offline.
• Frustrated, Brent searched for a better way and found the VerifiedVisitors service.
• The VerifiedVisitors ML platform takes a radically different approach. We monitored Brent's API endpoint across a 5 day period, and our API endpoint detector went to work learning from the patterns of visitor behaviour. All the traffic to the API is auto-mated, so traditional fingerprints, digital provenance or IP reputation solutions just don't work as the malicious bots rotate IPs and User Agents constantly. Instead the ML used the actual behaviour of the bots themselves to pick up patterns of sustained data farming over the entire API. Using intelligent Service Workers at the network edge VerifiedVisitors could catch the automated traffic and block it before it hit the API.
• A 12 month headache solved. Once and for all.
  

How behavioural API Rate Limiting Works

Examining the paths through the API and establishing the behaviour of each automated visitor in Machine Learning, allows us to develop a high level graph that shows the relationships between each of the behavioural elements. In the diagram below, we can follow the clusters of normal behaviour and detect the outliers. Typically on APIs what were looking to detect is the difference between systematic data abuse of the service, and genuine API looks on that are on-demand. The advantage of doing this at the edge of network is huge. The API doesn't need any additional software layers, or integrated with the API abuse detectors. It all happens in the cloud, so no additional engineering on the API is needed.

Stability and Performance

By preventing API abuse, bot behavioural detection ensures a stable and consistent performance of your services. It allocates fair resources for your legitimate users, avoiding service degradation due to excessive demands on the API servers, and blocks the service to abusers.

Cost Optimization

With API behavioural bot detection in place, you can optimize your server infrastructure costs by efficiently utilizing resources based on the actual needs of genuine users. This prevents unnecessary bandwidth consumption and infrastructure scaling costs from the data miners and serial abusers of your service.

Implementing API Rate Limiting

Taking out the abusive API traffic usually leads to a much more balanced traffic pattern. However, if you really do need to apply API Rate Limiting, because of hardware, cost restraints or simply because you don’t want to spend any time and money on additional development, then rate limiting alone is a good option, as long as your happy to degrade the overall service for all users accordingly.

Three Common methods of API rate limiting

• 1. Token Bucket Algorithm
• The Token Bucket Algorithm is a widely-used method for rate limiting. It assigns tokens to each user, representing the number of allowed requests within a certain time window. When a user makes a request, a token is consumed. Once the tokens are depleted, the user must wait until new tokens are generated to make additional requests.
• 2. Fixed Window Counters
• Fixed Window Counters impose a strict limit on the number of requests a user can make within a predefined time interval. If the limit is exceeded, subsequent requests are blocked until the window resets.
• 3. Sliding Window Log
• The Sliding Window Log approach considers a rolling time window to calculate the number of requests made by a user. It provides a more granular control over rate limiting by allowing certain bursts of requests as long as the average rate is within the defined limit.

Conclusion

API Behavioural bot detection is a fundamental aspect of bot management and web security that should not be overlooked. By effectively implementing dynamic behavioural bot protection, you can bolster your online defences, enhance performance, and foster a secure and efficient environment for your API users. Embracing behavioural bot detection as part of your broader security strategy is a proactive step towards safeguarding your digital assets in an ever-evolving threat landscape.

‍