Users can search for any place or nearby attractions like restaurants, theaters as well as events. Other similar services include TripAdvisor (for travel), Zomato (for Restaurants)

Functional Requirements

1. Business Profile - Yelp provides dedicated pages for each business, business owners or customers should be able to create a page for Businesses.

2. Search by Location or GPS coordinates - Users should be able to search for attractions by providing location and also able to search nearby places.

3. Reviews and Rating System - Users should be able to provide reviews for the places they have visited, services they have availed. Users should be able to provide ratings in terms of stars out of 5.

4. Photo and Video upload - Users should be able to add photos and videos related to the place.

Non Functional Requirement

1. High Availability: Our search service should be highly available.

2. Scalable: There will be peak demand on the Holiday season or certain tourist attractions.

3. Low Latency: Users should be able to access information as fast as possible.

Additional Requirements

1. Fake Reviews Detection: The system should be able to detect fake reviews by business owners that try to increase their ratings unethically.

2. Image and Video Processing: These services process the images and videos uploaded to provide the best photos to its users.

Estimations

We need to store location entries, Assuming that we will be storing around 500M places. Let’s also assume a 20% growth in the number of places each year, we can safely assume that we don't need more than 1MB(excluding photos and videos) to store the metadata related to a location.

Our system will be read-heavy, we can assume a 1:1000 write to read ratio with around a peak of 100k concurrent users.

Database Schema

We can store the following data in a relational database. We can have a place/location table to store the place information

Location Table :

id : integer
name : varchar(256)
geohash : varchar(12)
description: varchar (512 bytes)
address : varchar (1024)
category_id: integer

Similarly, we will have tables for users, reviews, ratings, photos, and videos. We can store the photos and videos on the cloud blob storage like AWS S3 and maintain a reference in the photo and video table respectively.

API's

Our service can expose the following REST API for searching nearby places.

search(String searchTerm, Map<String, Object> search params, String apiKey):

1. searchTerm(String) : The term which the user is searching for, it could be the name or location of the business.

2. search params(Map) : search criteria, includes the location for which search needs to be performed, the search radius, additional filter params like categories, number of results to return, sorting order, etc.

    sample searchParams :

    {
        "location": [34.3, -118.243],
        "categories" : ["restuarent", "bar", "dine-out"],
        "searchRadius": 5,
        "maxResults": 100,
        "sortCritera": 1,
        "sortAscending" true,
    }

3. apiKey(String) : The API key for the client making the API request, used for access check, rate limiting, and analytics purpose.

API will provide a JSON response for the list of places matching the search criteria.

We can expose additional API to Add/Modify new listing, add reviews, add ratings, upload photo/video to the listing, or a particular review.

High-Level Design

At a high level, The key feature of the Yelp system design is to find the nearby places/businesses with minimum latency and sort them by distance, because everyone wants to do business with the nearest store.

How to store the geographical context in the database.

Naive solution - A naive solution will be to store latitude and longitude data for every POI, same for the user, and then compare the information of the user's location with the database. Given a certain range query, the database where latitude is between xi and xj and longitude is between yi and yj.

This approach will not be scalable given the huge number of places, that users can query for.

GeoHash: Sequence of characters that identify a particular region to design a scalable system that provides geolocation data, we can use the concept of Geohash.

GeoHash essentially encodes the latitude and longitude information about a place to a String. The world map is divided into a rectangular grid system. Each rectangle is identified by a hash string and is further divided into 32 rectangles following a hierarchical structure, each level having an extra character than the hash of the parent rectangle. The "Geohash alphabet" (32ghs) uses all digits 0-9 and almost all lower case letters except "a", "i", "l" and "o".

Geohash represents a boundary and not a point. Since our mission is to calculate the nearby places, it is well suited for the application. Also, there will be some corner scenarios that we need to consider while determining the neighbors as shown in the below picture.

Geohashes are popular for spatial indexing and search applications thought initially were used only as part of URL-shortening service. If you have never heard the concept behind Geohash. I highly recommend viewing the following video.

https://www.youtube.com/watch?v=UaMzra18TD8

Let's return to our Yelp System Design scenario for finding nearby places and restaurants.

The naive algorithm to find out the nearest restaurants and businesses would be to find out the points with the same geohash as the location of the user. As discussed above due to corner cases, some of the nearby points of interest, but depending on the distance we want to search, likely leave out nearby points of interest that are in neighboring geohash bin.

To get correct and a more comprehensive set of points of interest, we have to get all the points in the current bin as well as all the points in the surrounding geohash bins i.e 8 immediate neighboring bins. Then we calculate the distance between all the points around the center.

Using Geohash based approach to find a point on the map, we can handle a very large dataset since on entering a particular geohash bin, we are essentially discarding the remaining bins, reducing the size of the problem exponentially. Users of a Yelp like service needs to see the results in real-time, hence we need to store and index the data about geohash of the places and associated reviews and ratings. Since the location data will not change, we need to implement the indexing for reading efficiency.

We can use a reasonable Geohash size of 12 characters to represent any place on earth with good precision. We can store the Geohash for locations in the database.

Storing GeoHash

Let’s see what are different ways to store this data and find out which method will suit best for our use cases.

Custom Approach

Since this is hierarchical data, we can store the data in the form of a tree structure. Each node will represent a geohash and will 32 children for each of the nodes to model the geohash like structure.
We will have additional tables to store the metadata for any place corresponding to each geohash.

Building the tree structure

We will start with the root node that would represent the whole map. The root node will have 32 children. We will insert take the 12 character geohash of the place we want to insert into the tree and will create a node for each character. For example to insert a place represented by the geohash "9q8zhuyj1ccb1" we would create a node under root node to represent the grid "9", under that node we will create a child node to represent "9q" and so on.

Querying the grid for a place

We can start from the root node and search downward for the required node. At each node, we will move towards the child node with the next character in the geohash, Once we reach the leaf node, that is the required node representing the point of interest. If at any point we don't find the required node, then the point of interest does not exist in the grid.

Finding out the neighbors

To find out the neighbors of any point, we can go to the parent node, or grandparent node depending upon the radius for which we want to search for neighbors, find out all the points under them, Calculate their distance from the Point of Interest and then sort according to the given criteria.

We can optimize the data structure for finding the neighbors, by connecting the leaf nodes with a double linked list to allow quick forward and backward iterations among the neighboring places.

After getting the nearby geohashes, we can query our metadata table to find the details corresponding to those places.

Memory Requirements
Since each leaf node represents a place using a Geohash of 12 characters, Total memory needed to store the Tree Structure for 500M places will be :

12 * ~1Byte * 500M = 6GB

Since we also have internal nodes in the tree, assuming around 1/3 internal nodes with 32 children, they will occupy:

500M * 1/3 * 32 * ~1 Byte = ~5GB

Use NoSQL offerings like Redis, ElasticSearch, MongoDB to store the Geospatial data

Instead of writing custom logic to implement Geospatial search, we can use some popular NoSQL databases like Redis, Elasticsearch to achieve the same functionality.

For example, We can use the Geospatial indexing feature of the Redis database. to quickly implement this feature offloading the indexing, searching, and sorting work to Redis using very few lines of code.
We can use the Geo Set to work with spatial data in Redis, Redis provides commands like GEOADD, GEODIST, GEORADIUS, and GEORADIUSBYMEMBER.

More information can be found using the below linked
Working with Geospatial Data in Redis

Similar support is provided by ElasticSearch and MongoDB databases. Going with elastic search gives full-text search as well as we need to search by the geographic tuple.

Working with Geospatial Data in ElasticSearch

Working with Geospatial Data in MongoDB

Data Partitioning

If we are using NoSQL offerings like Redis, MongoDB, Elasticsearch, etc. We can easily scale horizontally as we scale and add new places.

If we are building a custom application using the Tree-based structure. We can use the following schemes for Partitioning our data.

• Partitioning based on regions: We can divide the complete tree into multiple parts based upon the region. Places in the same region will be stored on the same node. Before storing a new location, we need to first find out which nodes store the data for a particular region, and then we can store a new location on those nodes, Similarly to querying the data.

  The problem with this approach is that there will be certain regions like in San Francisco which will be densely populated and will have more restaurants and other Points of Interest. This will result in an uneven distribution of data and would cause more loads on the server than others. To prevent this problem, we might partition these hot regions further or use consistent hashing.

• Partitioning based on Hash: We can use consistent hashing to decide which server will hold the data of a region, we can use a hash function to hash the geohash of a place and then distribute the data uniformly across multiple servers.
  to fetch all the nearby places, we will get the data from all servers and aggregate the data.

Processing the photos and videos Yelp stores the photos and videos and applies deep learning models to enhance them on the go.
More information can be found on the Yelp Engineering Blog.

Yelp Engineering Blog

Spam detection : Similarly Yelp like services uses machine learning models to prevent false reviews and ratings.

Caching

As our service is read-heavy, there will be around 20% of places which are searched frequently. We can use a caching solution like Memcache or Redis to store this data. This will improve the performance of 80% search queries.
To deal with hot Places, we can introduce a cache in front of our database. We can use an off-the-shelf solution like Memcache, which can store all data about hot places. Application servers before hitting the backend database can quickly check if the cache has that Place. Based on clients’ usage patterns, we can adjust how many cache servers we need. For cache eviction policy, Least Recently Used (LRU) seems suitable for our system.

WebSockets allow exchanging messages using any protocol as long as both client and server agree on the same for example JSON, XML, and any more.

Before the WebSockets, achieving the same functionalities would require constant polling the server which is generally high latency and overloads the backend servers.

Starting from around 2010, WebSockets are supported on all platforms including web and mobile devices. The standard for the WebSocket Protocol (RFC 6455 – The WebSocket Protocol) was published by IETF in 2011.
Backend servers can go for any client authentication mechanism for example cookie-based, HTTP, or TLS authentication.

WebSocets Usecases

Websockets are used in a myraid of applications which demands low latency, realtime connection for exchanging data between client and server.

• Notification/ Chat messages

• Multiplayer online games

• Live Sports commentary / ticker

• Realtime Social Network updates.

• Realtime monitoring

Considerations while using WebSockets

• Security Issues: WebSockets are prone to common security vulnerabilities linked to the HTTP Protocol such as DDos, Authentication and Authorization issues, Sniffing, Cross-Site Websocket Hijacking (Similar to CSRF (Cross-Site Request Forgery))

• Connection Issues : WebSocket connection, once terminated don't recover automatically and need to be handled. However, Reconnection logic is generally handled by the available client side libraries.

WebSockets Implementations

There are a plenty of popular WebSocket implemenations/libs available in the most popular languages, which makes it easy to get up and running easily.

• Javascript : WS, Sockets.io, SockJS

• Java : javax.websocket-api, java.net.Socket, Jetty

• Ruby : EventMachine, websocket-client-simple, em-websocket

• Python : pywebsocket, Tornado

A Database is used to persist information that will be useful later on. Broadly there are two types of database categories available to store the data and those are SQL or NoSQL. Which one to use depends on the needs of the system. For many years in the past SQL or Relational databases were the standard to store information. As the systems became more and more complex and the data grew many-fold NoSQL databases have gained their ground over the last decade and emerged as the primary data store for many applications. To understand which database is better for your needs to let's understand the difference between these two.

SQL or Relational Databases

A SQL or Relational database stores data in an organized set of tables with rows and columns. All the data related to a particular entity is stored in one table or more tables. A row stores all the related values for a particular instance of entity or object and each column stores one attribute of that object. Queries data using SQL syntax and JOINS. CRUD uses SCHEMA and transactions.

NoSQL Databases

NoSQL Database refers to Non Relational database, In contrast to a relational database where data is stored in well-defined tabular relations and Structured Query Language is used to access data, a NoSQL database uses different mechanisms for storage and retrieval. Different flavors of NoSQL use proprietary storage methods. Based on the problem they are solving NoSQL Databases store data in a wide variety of forms like key-value pairs, documents, columnar forms, graphs, or special time-sequenced events.

SQL v/s NoSQL

Schema

SQL: Data is nicely organized in appropriate tables according to the predefined schema, which reduces redundant information. Data conforms to the applied constraints. Requires a lot of initial thought to minimize changes later which might require migration of existing data and related downtime. Structured data prevents developers from sloppily adding data, and constraints prevent the corruption of data due to software bugs. This translates to more effort for developers initially.

NoSQL: schema is flexible, depending upon the type of NoSQL database, we might add or skip different columns and data in each document. It provides the flexibility of the data model to the developers allowing for easy iterations through the development process, without any downtime. On the flip side, it can lead to data corruption, if constraints and checks are not implemented correctly at the application layer.

Storage

SQL: Data is typically stored across multiple tables in normalized forms to prevent duplication of data.

NoSQL: Data is stored in a nested form with everything related to an entity stored in a single document. Though it introduces data redundancy but provides very fast access.

Data Retrieval:

SQL: Due to structured data, the Relational database provides a powerful SQL(Structured Query Language) interface for Data definition, Data Control, and Data Manipulation. The split structure allows us to join data in any way. Numerous join types are available and can be done with any number of tables in any way.

NoSQL: Since data for a particular entity is stored together, data access is simple and does not require any complex queries.

Performance :

SQL: With proper use of indexes and query tuning, it is possible to query very high volumes of data with reasonable performance.

NoSQL: Depending upon the type of NoSQL certain queries and access patterns are extremely optimized and fast. for example, the Key-Value database provides fast lookup, Column based database provides fast aggregation capabilities.

Scalability:

SQL: SQL Database is powerful and flexible but constrained in terms of scaling. Scaling SQL Database systems require expensive hardware to scale vertically. If we distribute SQL over multiple servers using data partitioning, however, performance suffers for certain queries and joins. This can also be attributed to the fact that relational databases came into existence in the early 1970s, in those times the scale and traffic that we witness today were unheard of.

NoSQL: These are built for web-scale applications and are easily scalable horizontally. queries. NoSQL does not use custom expensive hardware that allows for faster retrieval of data at a high scale. Since most of the NoSQL databases were created in recent times, they were developed cloud-native and with a distributed approach in mind. Almost all popular cloud vendors provide managed solutions that scale very easily according to growing traffic.

Data Integrity:

SQL: Majority of SQL databases offer strong ACID compliance and ensure consistency and reliability of data that is stored, thus making them relevant for transactional data even today.

NoSQL:, Unlike SQL counterparts, most NoSQL databases offer vague interpretations of ACID compliance and guarantees. These are majorly designed for scale and hence focus on Availability and Partition Tolerance by compromising on the consistency of the data. Always dig deep about the guarantees offered by the database that you are using before making any assumptions.

After discussing all the differences between SQL vs NoSQL, it is essential to note that in recent times, the lines between the SQL vs NoSQL has been blurring.

SQL providing NoSQL like Features

SQL databases like PostgreSQL and MySQL now provide support for storing, manipulating, and querying JSON Data. Similarly we can use PostgreSQL HStore to store key-value pairs.

NoSQL providing SQL like Features

MongoDB provides support for Joins and Transactions, although the underlying implementation and guarantees can be very different.

Choosing between SQL vs NoSQL

While designing a system or during a system design interview, it is of utmost importance to decide on the right database for storing data as it will allow you to get good performance out of your system. We need to choose the database that is the best fit for a particular use case.

For example: If you are designing a Banking application or other financial services-related application, It is a no-brainer to choose a Relational Database because of the data guarantees and security it has to offer which is a prime concern for this use case.

On the other hand, If you are designing a web crawler or search engine application, it is not advisable to use a Relational database due to the unstructured nature of web data and the scale at which you need to operate.

It is a common source of confusion for the new and in-experienced developers on whether to choose SQL or NoSQL while doing the required system design. So how to approach this?

1. Always think in terms of the access patterns or operations that you will need, does the database supports that as a core feature ?.

2. What kind of Guarantees and Isolation level (for transactions) do you need for your application?

Here is a list of Popular Use cases along with the most suitable Database types for each use case.

Sno Use case Choice of Database 1 Cache NoSQL inmemory datastore : Redis, Memcached 2 Banking Application SQL Database 3 Social Network NoSQL Graph Database: Neo4j 4 Facebook Ad Platform NoSQL columnar Databse: Cassandra, BigTable

Below is Genius Cheatsheet created by Satish Chandra Gupta, that can help you in picking the correct DB for your use case.

Here is a list of all the Popular Database and their rankings - DB Ranking

Need for Scaling

If our system is currently handling 100 requests/min and suddenly the traffic increases to 1000 requests/min, even though our system might be able to handle the same the traffic but due to increased load, the performance might become unacceptable, hence we will need to scale our system to be able to support that additional load with acceptable performance.

The performance of a system can be measured using throughput
and response time.

Type of Scaling

Vertical Scaling (Scaling Up)

Increasing the capacity to the current machine by improving its specifications like attaching more RAM, adding better CPU, upgrading HDD to SSD or adding more storage capacity, increasing network bandwidth etc.

Advantages:

• Architecture - Relatively simple, Less issues to worry about like inter service dependencies, network partitions etc.

• Maintenance - Easy to maintain due to less moving parts.

Disadvantages:

• Cost - Adding more resources leads to exponential increase in cost.

• Scope - Limited scope of Scaling due to physical limitation of size.

• Downtime - Risk of fault or complete downtime due to hardware failure.

Horizontal Scaling (Scaling Out)

Adding more machines and distributing the load among them.

Advantages:

• Cost - Cost efficient due to the use of commodity hardware.

• Scope - Theoretically unlimited scope of scaling.

• Resilience and Fault Tolerant - Due to redundancy, if few nodes go down, other nodes take up the requests.

Disadvantages:

• Architecture - Complex to design, due to fallacies of distributed systems.

• Maintenance - Difficult to observe and maintain due to moving parts.

Which approach is the best ?

There is no single correct answer to this, the choice of how to scale a particular system depends specifically on our system. The bottleneck for the system may be the number of requests, the volume of data , the access patterns or something else.
Depending upon whether you are targeting sub 200ms response time on your API’S or trying to manage petabytes of data, The System design and scaling strategies will be entirely different.
It is not possible to find an approach that fits all.

In general, it is easy to distribute a stateless system across multiple machines. Managing and synchronizing state within shared systems can introduce a lot of additional complexity and becomes hard to design and maintain.

It is a good practice to use a hybrid approach. To keep it simple, We can first scale vertically by using relatively powerful machines till we reach a point where it makes sense to use a large number of smaller machines.

Also, it is an ongoing process, a system designed for a particular load may not be able to handle 5x or 10x the load. At each milestone, we need to optimize, monitor and constantly re-architect on every magnitude of increased load.

Rate Limiting means to throttle the number requests to your service from a particular source (user, device, IP, location, etc) to some maximum limit. The requests submitted over the limit are either immediately rejected or they are delayed. Rate limiting allows us to create resilient services, that can handle various scenarios discussed below.

The simplest example of rate limiting is that you are allowed to enter 3 incorrect passwords before your bank account is locked for online transactions for a day. Similarly, the Github API allows around 5000 requests from a user account per hour, after that, you will get an error to wait for some time before sending another request.
Services typically send 503 (service unavailable) or 429(Too many requests) Http status code when the limit is exceeded.

Advantages of Rate Limiting

• Security: Rate Limiting is implemented to prevent DDoS attacks which overwhelms the servers which are a costly affair for any company both in terms of money and customer experience. Rate limiting your APIs will also limit the amount of data that gets exposed when security is somehow compromised.

• Protection against faulty Clients: Rate Limiting is also useful to make your system foolproof against any faulty or malicious client software, which might be sending a lot of requests due to a software bug.

• Cost Optimization: Rate Limiting Computationally intensive API will help optimize the cost of infrastructure.

• Maintain Quality: If you have a public API, implementing rate limit offers a better experience for all the users by distributing the number of requests that can be served. It prevents the Noisy Neighbour problem when one user utilizes too much-shared resources, such that it causes higher latency or higher failure rates.

• Maintain Priority: Several API's have paid version and free version, by assigning lower rate limits to the free users, you can make sure that Paid users get a better experience for their money.

Types of Rate Limiting

Due to ever-increasing loads on the servers for popular services, almost all companies use rate-limiting in their services, If you are also creating such services, it is very important that you add some form of rate limiting to prevent your infrastructure and customer base.

We can implement rate limiting based upon various methods and parameters that can be defined when setting rate limits. Based on the security and business requirements, we can choose one of the following criteria.

• User rate limiting: This is the most popular criteria used for rate limiting. Based on the number of requests from a particular User's IP or API Key, requests will be throttled once the limit is reached. Users will be shown the appropriate status code to reflect that their requests are throttled.

• Geographic rate limiting: Based upon the security or business requirements, We can set rate limits based on the geographic regions, This could reduce the likelihood of DDOS attacks and other suspicious activity.

• Resource Based Rate limiting: We can also employ flexible Rate Limits based upon the amount of resources available like CPU, network bandwidth, etc. When resources are constrained, we can reduce the rate limits, Once the resources are available, we can increase the rate limits.

• Hybrid rate limiting: We can combine certain characteristics of the above rate limiting algorithm to achieve optimum results, like a user can send 100 requests per second from a particular IP address in a geographic region.

Requirement Gathering.

Function Requirements

• For a given request, return a boolean value of whether the request is throttled or not.

NonFunction Requirements

• Low Latency - We want the rate limiting module to be as fast as possible, otherwise it adds latency to the processing time of the request.

• Accurate - Our rate limiting module should be as accurate as possible. It should not be throttling requests which should have actually gone through.

• Scalable - Our service should be highly scalable to support more number of requests.

In case the rate limiting service is not available, the system should not block the processing of the request.

Rate Limiting Algorithms

• Token Bucket Algorithm - In this algorithm, we place n tokens in a bucket, every bucket has a maximum capacity and is filled at a constant rate, ex 10 tokens pers second. When a request for our API is received, tokens are withdrawn from the bucket if available. If required number of tokens are not available then the request is rejected or delayed. Eventually, the bucket is refilled with tokens and the client can make more requests to our API. Please note that depending upon the operations in the API request, one request might need more than one token.

• Leaky Bucket Algorithm - It is one of the simplest rate limiting algorithms, It implements rate limiting using a bucket concept per user (depending upon the factor on which rate limiting is applied) which holds all the incoming requests of that user. The bucket is of the fixed size corresponding to the number of requests we want to allow. When a request is received, it is put into the bucket. Requests are picked up from the other end of the bucket (more like a queue than a bucket) for processing. If the number of requests at any point in time exceeds the size of the bucket, those will leak and will not be considered. It can lead to queueing up of old requests if they take long time to process and prevent recent requests from processing.

• Fixed Window Algorithm - As the name suggests, in this algorithm, we fix a time window over which the rate limiting is imposed. Irrespective of the actual time the request came in, the count is maintained for fixed time windows of x seconds.

Consider the example in the above image where the rate limit is 3 requests per second, the first request came in after 500 ms. It is clearly visible from the previous example, that from 500ms to 1500 ms, the server actually serves 4 requests this might lead to a rate limiter allowing extra of requests if more such requests come near the boundary of the time window. The next algorithm will overcome this issue with Fixed Window Algorithm.

• Rolling Window Algorithm - In the Fixed Window Algorithm, the reference time window was fixed, however in case of a rolling window, the time window starts when the first request is received. The rate limit is imposed based upon the number of requests that came in from the start of the window to the end of the window. This will improve the performance around the boundary of the time zone.

As shown in the image above, now the time window starts at 500 ms when the first request comes in, and it allows us to serve only 3 requests R1, R2, R3 for that second. Thus request R4 is throttled, after 1 second is complete, the new time window starts once a new request comes in.

High Level Design for Rate Limiter.

The above diagram shows the High level design for a rate limiting system. When the Webserver receives the request it would ask the Rate Limiter service, whether the request should be served or throttled. If the request is not throttled, it will be forwarded to one of the API servers.

Distributed Rate Limiter Design

The true benefits of rate limiters are predominant in a distributed system, where we are operating on a large scale, and we want to have efficient use of our resources. In a distributed system, there will be multiple API servers serving requests from users.

Due to the number of requests, we would need to scale our rate limiter service by distributing over multiple nodes. In each of the algorithms, described above we store some kind of count that constraints the number of requests that will be served. How can we achieve this if our counters/buckets are distributed over multiple nodes?

Communication between the hosts is the key here. We want to implement the rate limit at a global level, hence we need a way to synchronize among different cluster nodes.

We can use a global data store to maintain the counters for each window and user. But having a global data store is a bottleneck as all the nodes will flock to the data store to get the count. The worst cases could lead to race conditions. If one of the nodes has read the value of the counter before it could update the counter, another node could read the same value and the final value will be incremented by only one instead of two.

We could avoid race conditions by introducing locks. However, that could degrade the performance of the system and it will not scale well. Querying the central data store will add the additional overhead of several milliseconds for each request.

Can we do better ?
Instead of querying the centralized data store for each request, we can maintain the counters locally. Let's take an example: If our rate limit is 4 requests per second per user. We can have a limit of 4 requests per second per user on each server as shown in examples below.

Global Limit : 4 req/sec/user

Let's say Server1 received 2 requests in a particular second, Server2 received 1 request and Server3 received 1 request. Since each server is maintaining the counters locally, If 2 more requests come to Server1, it would allow the API call. Thus our rate limiter has served 6 requests per second for the user.

This scheme will lead to relaxed rate limits. However, each node can synchronize the counters with a central data store eventually as a part of the synchronization cycle. If a particular node had served more requests in a time window it would receive a negative counter value for the future time window and hence the average number of requests served per second per user will still be the same as the original limit. We can configure the interval between synchronization cycles to achieve optimal results.

References and interesting reads.

Rate Limiting Techniques

Alternative Approach to Rate Limiting

We can do this via direct communication via TCP connection between producer and consumer. However, such a system allows communication between exactly one producer and one consumer. Also what happens if producers are producing messages faster than the consumers can consume, And what happens if consumer nodes go down?

Depending upon the application, a popular way to send a message is via a message broker or a queue.

Message Brokers

Message brokers (or Queues) run as servers sitting between producers and consumers. Producers can produce messages to the message broker. On the other hand, consumers can receive messages from the broker. Thus it helps to decouple different interacting systems providing them an asynchronous way of messaging.

One or more producers can communicate with one or more consumers using such messaging systems.

Producers can now produce data at any rate without worrying about the rate at how data is consumed or who is consuming the data.

These systems can tolerate consumers going down and coming back and messages can be persisted in the broker from the time they are produced by the producers till the time they are consumed or even later depending upon the implementation.

Messaging Patterns

Two common ways of handling the Messages are Load balancing and Fan-out

Load Balancing

Each message in the queue is delivered to exactly one of the consumers. In general, multiple consumers will be present in the system parallelizing the processing of the messages from the queue. Once the message is extracted by the consumer, it is removed from the queue. This pattern is generally used to process messages that result in long-running tasks and you want to achieve some parallel processing.

For example, Your credit card company generates a statement of transactions at the end of the billing cycle and sends an email with the statement. Such a system can be implemented by a scheduler that fetches user information from the DB and generates a message in the queue for processing data for each customer. A separate application receives this information, fetches data about the user transactions, crunches numbers, generates the statement, and delivers the email in our inbox.

Fan-out

Each message in the broker is delivered to all the consumers, allowing multiple consumers to listen to the same message and perform different tasks. This pattern is used to process messages that have to be handled in multiple ways by different systems.

For example: While logging in to some applications, we receive the same passcode via text and email. Such a system can be implemented by sending all passcode messages through a messaging system and having Text messaging application and Email messaging application listen to the same message.

Hybrid

We can combine the two patterns by using two consumer groups ( containing multiple consumers) listening to the same messages, such that each consumer group receives all messages but within a consumer group only one of the nodes receives each message. Kafka uses such an implementation.

Benefits of using a Messaging System:

Buffering: Messaging Systems provide a way to buffer the messages, persisting them in the broker (in memory or disc) while they wait to be processed by the consumers. Allowing our systems to deal with sudden traffic spikes up to some extent.
Message Delivery Guarantees: Using techniques described above messages are guaranteed to be delivered and processed by the consumers before they are marked as processed allowing retries in failure scenarios.
Easy Scaling: Because of the asynchronous nature of messaging systems and fault tolerance and configurability, it is rather easy to scale software systems that use messaging systems for communication, allowing multiple producers to communicate with multiple consumers.
Separation of concerns: Using a messaging system allows us flexibility in architecture by making the producers and consumers independent.

Consideration while using a Messaging System:

Back Pressure.

What happens if our system is under heavy sustained load, There are large amounts of messages flowing into the broker and it starts to grow significantly. Depending upon the server configuration, if the size becomes larger than the memory, it will result in cache misses and expensive disk reads. Having the entire system degrade under such circumstances is not the way to build scalable and fault-tolerant systems. Hence if the arrival rate is more than what the system can cope up with, can use backpressure in such scenarios by helping limit the queue size preventing throughput degradation, and maintaining good response times for the message processing. Once the broker gets filled, it can return a service unavailable using the HTTP status code 503, indicating to try at a later time.

Here is an interesting read on the this topic :

Apply back pressure

Acknowledgement and redelivery

If consumers crash while processing a message, which may result in loss of message. However, message brokers implement acknowledgments wherein the consumer must assure the message broker that it has completed processing the message and that broker can mark it as consumed and remove it from the queue.

If the broker does not receive the acknowledgment, it will assume that the message did not get processed successfully and it will redeliver the message again until it is processed with an acknowledgment. We will learn more about this when we discuss implementations of different messaging systems like Kafka, RabbitMQ, etc.

Apart from routing, Load balancers might also do other activities like SSL termination, which means SSL traffic is decrypted by the load balancer and then passed on to the server. This saves the work of SSL decryption on the web servers improving performance and reducing latency.

Load balancing can happen at Layer 4 (L4- Transport) or Layer 7 (L7- Application) of the OSI Model. Also, there are different kinds of Load Balancers.

Hardware Load Balancer or Software Load Balancer

Hardware-based load balancers are high-performance solutions where specialized processors are used to achieving optimum performance. To handle increased loads you need to buy more machines. Hence they are generally a costly solution and require specialized maintenance. Due to the above limitations, these are not very popular. The most widely used ones are Citrix Netscaler and F5.

Software-based load balancers uses generic hardware to run and are easy to scale and flexible in configuration. Some of the popular ones come as installable software and others are provided as a Service called Load Balancer as a Service (LBaas). Popular software-based load balancers included Nginx, Varnish, HAProxy, and LVS. Popular LBaas include AWS ELB and Stratoscale LBaas. LBaas are managed by the cloud vendors and handle fault tolerance and elasticity for the users.

Check out the following video of how Facebook handles billions of request using their unique load balancing techniques.

Load Balancing Layer

Load Balancing can be introduced at any layer in the application increase scalability depending upon the number of requests served.

• Frontend Layer - The most common use case of the load balancers is at the frontend of the application, between the client and the webserver. This helps to increase the number of requests that can be served by the system. Also, SSL Termination is done here to save CPU cycles of the application server.

• Application Layer - Load Balancers are placed between the webserver which is taking the requests and the application servers which are doing CPU intensive tasks. This helps in the proper utilization of the application servers without overloading them.

• Persistance Layer - Load Balancers are placed between the Application Layer and the Persistence Layer to serve more data requests without overloading Database servers.

Load Balancing Algorithms

Several popular load balancing algorithms are used by Software-based load balancers. Depending upon the type of system and workloads, a particular scheme may be well suited than others.

1. Round Robin: In this scheme, the load balancer forwards requests to different nodes sequentially in a round-robin fashion, This is simpler to implement, however, it is not very efficient as it doesn't account for the current load on the server, which may already be loaded when a new request is assigned. Hence is mostly suited for simpler use cases where loads are not processor intensive.

2. Least Connection: The requests will be forwarded to a server that has the least number of connections or serving the least number of requests at that point in time. This delivers better performance than the round-robin, but again suited not suited for workloads where the single request might load the server.

3. Least Response Time: This scheme is based upon the time taken by the server to respond to a dummy request sent to a server selected by an algorithm based on the least connections and response time.

4. Hash-Based: Requests are distributed based upon a defined key example: Request URL, Request IP address.

5. Custom Load Algorithm: In this scheme, load balancer determines which server to forward the request to based on a combination of server metrics like Memory usage, CPU Usage, Response Time.

Summary

Load balancer has become an essential component of any large scale system as it helps to balance loads across multiple machines. As the systems become more complex, increasingly popular and traffic volume surges, Load balancers act as a traffic cop to route the loads systematically, preventing uneven loads and performance issues on the systems. We discussed different types of Load Balancers and algorithms, in a system design interview based upon the problem at hand a particular configuration may be well suited than the other.

Popular CDN offerings include Cloudflare, Akamai, and AWS Cloudfront.

Working of CDN

When the user requests a webpage or any other content, the content will be delivered from a CDN server close to the user. The more the distance from the CDN server the more time content will take to reach the user and the slower will be load time for the content.

If the content is not available on the CDN, it can be fetched from the backend servers. Content on the CDN can have an expiry set by the application or configuration. When the particular content is requested, it is returned with a TTL (time-to-live) header providing the value of expiry.

For example, if you are located in Asia, data fetched from a server in Europe will take more time than the same data fetched from a server in Asia itself due to the distance the data has to travel.

Similar to static content delivery, nowadays CDNs are also capable of serving dynamic content. To generate the dynamic content, scripts are run at the CDN server rather than the backend server, these scripts generate the content based on some variables and events like time, location, or input data from APIs and cache the content. For example, Cloudflare Workers offer Serverless Javascript functions that are executed on the Cloudflare CDN.

Advantages of using CDN

• Faster Load times - Since files are fetched from a server closer to the user, load times are drastically improved. This leads to a better user experience overall, which in turn increases user engagement and reduces bounce rates.

• Reduced Infrastructure cost - Since the request does not receive the backend servers, it decreases the load on the servers, hence reducing the infrastructure cost.

• Increased Efficiency - By using the techniques like minification and file compression, CDNs reduce the size of files that are transferred to the client. CDN can speed up the TLS/SSL-based sites by connection reuse and TLS false start.

• Security - If configured correctly, CDN might help in mitigating security issues like DDoS attacks.

Though there are so many advantages of using CDN, we need to be careful while using a CDN and configuring it. Here are some of the aspects we should keep in mind.

• Need - Do we need a CDN?, In case our content is not accessed frequently or refreshed too quickly, then CDN might not be of any use.

• Cost - The major CDN providers mentioned in the introduction section have pricing based on the number of requests or the amount of data in/out. Hence we should be careful about the cost to prevent massive billings.

• Fallback - Clients should be coded in such a way that if CDN is not available, they should be able to connect to backend servers for the content.

• Cache Expiry - We should be mindful while setting the expiry for the content, Have too long of expiry, the content might be stale. Have too short of Expiry and there might be unnecessary reloading of the content from the origin servers to the CDN.