System Design Basics - Apache Kafka vs RabbitMQ vs ActiveMQ

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

image_credit - Design Guru

Hello devs, if you are preparing for System Design interviews then along with popular software design questions like API Gateway vs Load Balancer and Horizontal vs Vertical Scaling, Forward proxy vs reverse proxy, you should also prepare about things like messaging brokers, kafka, rabbitmq, and activemq like what is difference between Kafka, RabbitMQ, and ActiveMQ?, which is also one of the popular questions on Java interviews.

In my last article, I shared about 50 System Design Interview Questions and REST vs GraphQL vs gRPC, and in this article, I am going to share my thoughts on Kafka, RabbitMQ, and ActiveMQ, three popular message brokers used for asynchronous communication.

Messaging systems and Message brokers play a crucial role in modern distributed architectures, where applications and services communicate with each other over a network.

The messaging systems allow decoupling of the sender and receiver, thereby enabling asynchronous communication. RabbitMQ, Apache Kafka, and ActiveMQ are three popular messaging systems used in the industry.

In this article, we will discuss the differences between RabbitMQ, Apache Kafka, and ActiveMQ.

By the way, If you are preparing for System design interviews and want to learn System Design in depth then you can also checkout sites like ByteByteGo, Design Guru, Exponent, Educative , bugfree.ai, and Udemy which have many great System design courses and if you need free system design courses you can also see the below article.

And, if you are in a hurry, here is a table from ByteByteGo which compares Kafka with RabbitMQ on different parameters like architecture, structure, working, etc

Rabbit MQ vs Kafka

---

What is Apache Kafka and where is it used?

Apache Kafka is an open-source distributed event streaming platform that was originally developed by LinkedIn. Kafka is written in Scala and Java and is designed to handle large-scale streaming data flows.

Kafka uses a publish/subscribe messaging model and is optimized for high throughput, low latency, and fault-tolerance.

Kafka has a durable messaging model, which means that messages are stored on disk and can be replayed multiple times.

If you want to learn more about Kafka, particularly from a system design point of view, you can also join ByteByteGo, a great platform to learn essential system design concepts

---

What is RabbitMQ and where is it used?

RabbitMQ is an open-source message broker that implements the Advanced Message Queuing Protocol (AMQP) standard.

It is written in Erlang and has a pluggable architecture that allows for easy extensibility.

RabbitMQ supports multiple messaging patterns such as publish/subscribe, request/reply, and point-to-point, and it has a robust set of features such as message acknowledgment, routing, and queuing.

Arslan Ahmend has explained about RabbitMQ in his classic Grokking the System design interview course, if you are preparing for a tech interview, you can also see that resource for better preparation.

---

What is ActiveMQ? where does it used?

Apache ActiveMQ is an open-source message broker that implements the Java Message Service (JMS) API. ActiveMQ is written in Java and has a pluggable architecture that allows for easy extensibility.

ActiveMQ supports multiple messaging patterns such as point-to-point, publish/subscribe, and request/reply, and it has a robust set of features such as message acknowledgment, routing, and queuing.

---

Differences between RabbitMQ, Apache Kafka, and ActiveMQ?

Now that you have a fair idea of what is RabbitMQ, ActiveMQ, and Apache Kafka, it's time to find out the difference between them from messaging model to performance.Here are key differences between Apache Kafka, RabbitMQ and ActiveMQ:

1. Messaging Model

RabbitMQ and ActiveMQ both support the JMS API, which means that they follow a traditional messaging model where messages are sent to a queue or a topic and consumed by one or more consumers.

On the other hand, Kafka uses a publish/subscribe messaging model, where messages are published to a topic and consumed by one or more subscribers.

The traditional messaging model used by RabbitMQ and ActiveMQ is well-suited for applications that require strict ordering and reliable delivery of messages.

On the other hand, the publish/subscribe messaging model used by Kafka is better suited for streaming data scenarios, where real-time processing of data is required.

Here is a nice diagram which highlight the architecture difference between Kafka and RabbitMQ

---

2. Scalability

Scalability is an essential requirement for messaging systems, especially when dealing with large volumes of data. RabbitMQ and ActiveMQ are both designed to be scalable, but they have different approaches to achieving scalability.

RabbitMQ uses a clustering approach to achieve scalability, where multiple RabbitMQ brokers are connected to form a cluster. Messages are distributed across the cluster, and consumers can connect to any broker in the cluster to consume messages.

RabbitMQ also supports federation, which allows multiple RabbitMQ clusters to be connected together.

ActiveMQ uses a network of brokers approach to achieve scalability, where multiple ActiveMQ brokers are connected to form a network.

Messages are distributed across the network, and consumers can connect to any broker in the network to consume messages. ActiveMQ also supports master/slave replication, which provides high availability for the message broker.

Kafka, on the other hand, is designed to be highly scalable out of the box. Kafka uses a partitioning approach to achieve scalability, where messages are partitioned across multiple Kafka brokers.

Each partition is replicated across multiple brokers for fault tolerance. This approach allows Kafka to handle large volumes of data while maintaining low latency and high throughput.

---

3. Performance

Performance is another critical factor to consider when choosing a messaging system. RabbitMQ, Kafka, and ActiveMQ all have different performance characteristics.

RabbitMQ is designed to be a reliable messaging system, which means that it prioritizes message delivery over performance.

RabbitMQ can handle moderate message rates and is suitable for applications that require strict ordering and reliable delivery of messages.

Kafka, on the other hand, is designed for high-performance and can handle large volumes of data with low latency. Kafka achieves this performance by using a distributed architecture and optimizing for sequential I/O.

ActiveMQ is also designed for high-performance and can handle high message rates. ActiveMQ achieves this performance by using an asynchronous architecture and optimizing for message batching.

Here is a chart from confluent which compares performance of Apache Kafka, Pulsar and RabbitMQ

Benchmarking Apache Kafka, Apache Pulsar, and RabbitMQ: Which is the Fastest?

---

4. Data Persistence

Data persistence is an important feature of messaging systems, as it allows messages to be stored and retrieved even if the messaging system goes down. RabbitMQ, Kafka, and ActiveMQ all have different approaches to data persistence.

RabbitMQ stores messages on disk by default, which allows messages to be persisted even if the broker goes down.

RabbitMQ also supports different storage backends, including in-memory storage, which provides better performance at the cost of data durability.

Kafka stores messages on disk by default and uses a log-based architecture to achieve high durability and reliability. Kafka retains messages for a configurable period, which allows messages to be replayed if necessary.

ActiveMQ also stores messages on disk by default and supports different storage backends, including JDBC and file-based storage. ActiveMQ can store messages in a database, which provides better data durability at the cost of performance.

Here is a nice diagram from IBM that shows a Kafka architecture:

image --- https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-overview/

---

5. Integration with Other Systems

Integration with other systems is an important factor to consider when choosing a messaging system. RabbitMQ, Kafka, and ActiveMQ all have different integration capabilities.

RabbitMQ integrates well with different programming languages, including Java, Python, Ruby, and .NET. RabbitMQ also has plugins that allow it to integrate with different systems, including databases, web servers, and message brokers.

Kafka integrates well with different data processing systems, including Apache Spark, Apache Storm, and Apache Flink. Kafka also has a connector framework that allows it to integrate with different databases and data sources.

ActiveMQ integrates well with different JMS clients, including Java, .NET, and C++. ActiveMQ also has plugins that allow it to integrate with different systems, including Apache Camel and Apache CXF.

Here is also a nice table to highlight the difference between Kafka, RabbitMQ, and ActiveMQ

---

System Design Interviews Resources:

And, here are curated list of the best system design books, online courses, and practice websites which you can check to better prepare for System design interviews. Most of these courses also answer questions I have shared here.

1. ByteByteGo: A live book and course by Alex Xu for System design interview preparation. It contains all the content of the System Design Interview book volumes 1 and 2, and will be updated with volume 3, which is coming soon.

2. Codemia.io: This is another great platform to practice System design problems for interviews. It has more than 120+ System design problems, many of which are free and also a proper structure to solve them.

3. Bugfree.ai: Bugfree.ai is a popular platform for technical interview preparation. The System Design sections and interview experience include a variety of questions to practice.

4. Exponent: A specialized site for interview pre,p especially for FAANG companies like Amazon and Google. They also have a great system design course and many other materials that can help you crack FAAN interviews.

5. Educative's System Design Course: An interactive learning platform with hands-on exercises and real-world scenarios to strengthen your system design skills.

6. DesignGuru's Grokking System Design Course: An interactive learning platform with hands-on exercises and real-world scenarios to strengthen your system design skills.

7. "System Design Interview" book by Alex Xu: This book provides an in-depth exploration of system design concepts, strategies, and interview preparation tips.

8. "Designing Data-Intensive Applications" by Martin Kleppmann: A comprehensive guide that covers the principles and practices for designing scalable and reliable systems.

9. "System Design Primer" on GitHub: A curated list of resources, including articles, books, and videos, to help you prepare for system design interviews.

10. High Scalability Blog: A blog that features articles and case studies on the architecture of high-traffic websites and scalable systems.

11. YouTube Channels: Check out channels like "Gaurav Sen" and "Tech Dummies" for insightful videos on system design concepts and interview preparation.

image_credit - ByteByteGo

Remember to combine theoretical knowledge with practical application by working on real-world projects and participating in mock interviews.

Conclusion

That's all about the difference between Apache Kafka, RabbitMQ, and ActiveMQ. RabbitMQ, Apache Kafka, and ActiveMQ are three popular messaging systems that have different features and capabilities.

RabbitMQ and ActiveMQ follow a traditional messaging model, while Kafka uses a publish/subscribe messaging model.

RabbitMQ and ActiveMQ use clustering and a network of brokers approach to achieve scalability, while Kafka uses partitioning. RabbitMQ prioritizes message delivery over performance, while Kafka and ActiveMQ prioritize performance. RabbitMQ, Kafka, and ActiveMQ all have different data persistence and integration capabilities.

When choosing a messaging system, it is essential to consider the specific requirements of the application or system.

RabbitMQ and ActiveMQ are suitable for applications that require strict ordering and reliable delivery of messages, while Kafka is suitable for streaming data scenarios.

RabbitMQ and ActiveMQ are suitable for applications that require moderate to high message rates, while Kafka is suitable for applications that require high message rates.

Similarly, RabbitMQ and ActiveMQ are suitable for applications that require high data durability, while Kafka is suitable for applications that require high performance.

Top 10 Essential Tools for DevOps Engineers to Learn in 2026

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

Hello friends, In the last few articles, I talked about System Design interview questions, and WHERE vs HAVING in SQL, and today, I will talk about essential DevOps tools that both developers and DevOps should know.

In today's fast-paced software development landscape, DevOps practices and tools have become essential for efficient Software development and delivery.

DevOps is all about breaking down the traditional silos between development and operations teams, fostering collaboration, and automating key processes.

To achieve these goals, a wide array of DevOps tools and technologies has emerged, each addressing specific aspects of the software delivery lifecycle.

In this article, I am going to share the top 10 DevOps tools that play an important role in the way organizations build, test, deploy, and monitor software.

These tools span a range of categories, from version control and continuous integration to container orchestration and monitoring.

Whether you're a DevOps Engineer or a Senior developer looking to expand your toolkit or an organization seeking to adopt DevOps practices, these tools can help streamline your software development and operations processes.

10 Essential DevOps Tools You Can Learn in 2026

Let's dive into the world of DevOps and discover the top 10 tools that can empower your team to achieve faster, more reliable software delivery.

1. Git

Git revolutionized version control, making it one of the foundational tools in DevOps. It allows developers to track changes in their codebase, collaborate seamlessly, and manage multiple code branches effectively.

Git hosts like GitHub, GitLab, and Bitbucket have further enhanced their capabilities, providing a platform for distributed version control, code review, and project management.

And if you want to learn Git, you can start with Git Complete: The definitive, step-by-step guide to Git, one of the most comprehensive courses on Udemy.

If you need more choices, then you can also see these best Git online courses for beginners in 2026. It contains git courses and tutorials for both beginners and experienced DevOps engineers.

---

2. Jenkins: Continuous Integration and Continuous Delivery (CI/CD)

Jenkins is an open-source automation server that plays a crucial role in automating the CI/CD pipeline. It allows developers to build, test, and deploy code continuously, ensuring that changes are integrated smoothly and errors are detected early in the development process.

With a vast library of plugins, Jenkins can be customized to suit the specific needs of your development environment.

And if you want to learn Jenkins in depth then you can start with Jenkins, From Zero To Hero: Become a DevOps Jenkins Master course, its a nice course to learn Jenkins.

If you need more choices, then you can also see these best Jenkins courses for 2026.

---

3. Docker: Containerization for Portability

Docker has revolutionized how applications are packaged and deployed. With Docker containers, you can bundle your application and its dependencies into a single, lightweight unit that runs consistently across different environments.

This portability and isolation make Docker a key tool for DevOps teams aiming to achieve consistency from development to production.

And if you want to learn Docker in 2026, you can start with Docker and Kubernetes: The Complete Guide course, it's a nice course to learn Docker from scratch.

If you need more choices, then you can also see these best Docker courses for beginners in 2026 to start with.

---

4. Kubernetes

Kubernetes has emerged as the de facto standard for container orchestration. It simplifies the management of containerized applications, automating tasks such as scaling, load balancing, and fault tolerance.

Kubernetes provides the foundation for building resilient, Microservices-based applications, and it's a must-have tool for modern DevOps teams.

And if you want to learn Kubernetes in depth, you can start with this beginner-level hands-on course Kubernetes for the Absolute Beginners --- Hands-on by Mumshad Mannambeth on Udemy.

If you need more choices, here are the best Kubernetes courses for DevOps Engineers to join in 2026.

---

5. Ansible

Ansible is a powerful open-source tool for automating configuration management and application deployment. It allows you to define infrastructure as code, making it easier to provision and manage servers and services.

Ansible's simplicity and agentless architecture make it a favorite among DevOps professionals for automating repetitive tasks.

And if you want to learn Ansible in depth, you can start with the Ansible for the Absolute Beginner — Hands-On — DevOps course by KodeCloud Training on Udemy. It's a nice hands-on course to learn Ansible.

If you need more options, you can always check these best Ansible online courses in 2026. It contains Ansible courses for both beginner and intermediate DevOps engineers.

---

6. Terraform

Terraform is another key tool for infrastructure as code. It enables you to define and provision infrastructure resources across various cloud providers and on-premises environments.

Terraform's declarative syntax and modular design make it a versatile choice for managing infrastructure at scale.

And if you want to learn Terraform in depth, then you can start with the Terraform for the Absolute Beginners with Labs course by Kodecloud training.

If you need more choices, then you can also check out these best Terraform courses for 2026.

---

7. Prometheus

Prometheus is an open-source monitoring and alerting toolkit designed for reliability and scalability. It can collect metrics from various sources, allowing you to gain insight into the health and performance of your applications and infrastructure.

With its flexible query language and robust alerting capabilities, Prometheus empowers DevOps teams to proactively identify and address issues.

If you need a course to learn Prometheus, you can start with Prometheus | The Complete Hands-On for Monitoring & Alerting course on Udemy. I took this course last month, its quite nice.

---

8. ELK Stack

The ELK Stack, which consists of Elasticsearch, Logstash, and Kibana, provides a comprehensive solution for log management and analysis, particularly in the microservices world.

It allows DevOps teams to collect, parse, store, and visualize log data from various sources.

This stack is invaluable for troubleshooting, performance optimization, and security monitoring.

And, if you want to learn more about the ELK stack, you can start with the Complete Elasticsearch Masterclass with Logstash and Kibana course from Udemy. It's a nice beginner-level course for the ELK stack.

And, if you need more choices, you can always take a look at these best ELK stack courses for Beginners in 2026

---

9. Jenkins X

Jenkins X is a Kubernetes-native CI/CD solution that brings automation and GitOps principles to the forefront. It simplifies the process of building, testing, and deploying cloud-native applications on Kubernetes clusters.

Jenkins X streamlines the development workflow and promotes best practices for containerized applications.

---

10. Grafana

Grafana is a popular open-source platform for data visualization and monitoring. It can integrate with various data sources, including Prometheus, to create dynamic dashboards and alerts.

DevOps teams use Grafana to gain real-time insights into application and infrastructure performance, facilitating data-driven decision-making. If you want to learn Grafana in depth, you can start with the Grafana course on Udemy. It's a nice course to learn Grafana from scratch.

If you need more choices, then you can also check out these best Grafana online courses in 2026.

That's all about the 10 essential tools DevOps can learn in 2026. The DevOps landscape is continually evolving, and the tools mentioned above are just a snapshot of the vast ecosystem available to DevOps practitioners.

Each tool plays a crucial role in different aspects of the software delivery pipeline, from version control and continuous integration to container orchestration and monitoring.

The key to successful DevOps adoption is selecting the right tools that align with your organization's needs and goals.

By embracing these DevOps tools, your organization can streamline its development and operations processes, reduce manual effort, improve collaboration, and deliver high-quality software at a faster pace. 

Top 50 Easy, Medium, and Hard System Design Interview Questions for 2026

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

image_credit - Exponent

Hello friends, if you are preparing for Tech interviews, then you must prepare for System design questions because this is where most of the people struggle.

Even experienced programmers struggle to solve common questions like how to design WhatsApp or YouTube, or answer the difference between API Gateway vs Load Balancer and Horizontal vs Vertical Scaling, Forward proxy vs reverse proxy.

In today's increasingly distributed world, the ability to architect robust and scalable systems is a fundamental skill sought after by top-tier tech companies.

System design interviews have become a crucial component in evaluating a candidate's capacity to solve real-world challenges, assess trade-offs, and design systems that can handle complex requirements.

In the past, I have also shared about Database Sharding, System design topics, Microservice Architecture, and System design algorithms, and today, I am going to share system design questions for interviews.

In this article, I have 50+ system design interview questions carefully crafted to guide candidates from the foundational concepts to intricate design scenarios.

Whether you're a beginner aiming to grasp the essentials or an experienced engineer seeking to refine your skills, these questions will not only prepare you for interviews but also improve your knowledge about system design and software architecture.

By the way, if you are preparing for System design interviews and want to learn System Design in depth then you can also checkout sites like ByteByteGo, Design Guru, Exponent, Educative, Codemia.io, Bugfree.ai and Udemy which have many great System design courses

P.S. Keep reading until the end. I have a free bonus for you.

---

50 System Design Interview Questions for 2026

Here is a list of 50 popular System design interview questions for beginners and experienced developers, which you can solve to start your preparation.

In this list, I have not only shared easy, medium, and hard system design problems but also concept-based questions like API Gateway vs Load Balancer or Microservice vs Monolithic. You can practice these system design problems and questions for interviews.

System Design Concept-based Questions

1. What is the difference between API Gateway and Load Balancer? [solution]
2. What is the difference between Reverse Proxy and Forward Proxy? (answer)
3. What is the difference between Horizontal scaling and vertical scaling? (answer)
4. What is difference between Microservices and Monolithic architecture? (Answer)
5. What is difference between vertical and horizontal partitioning ?
6. What is Rate Limiter? How does it work? (answer)
7. How does Single Sign On (SSO) works? (answer)
8. How does Apache Kafka works? why it so fast? (answer)
9. Difference between Kafka, ActiveMQ, and RabbitMQ? (answer)
10. Difference between JWT, OAuth, and SAML? (answer)

Here is a nice diagram from DesignGuru.io which explains difference between vertical and horizontal database partition

---

𝐄𝐚𝐬𝐲 System Design Problems

Now, let's jump into easy system design problems. These are common question where you need to design small utility which is used everywhere like URL shortner:

1. How to Design URL Shortener like TinyURL [solution]
2. How to Design Text Storage Service like Pastebin? [solution]
3. Design Content Delivery Network (CDN) ? [solution]
4. Design Parking Garage [solution]
5. Design Vending Machine [solution]
6. How to Design Distributed Key-Value Store
7. Design Distributed Cache
8. Design Distributed Job Scheduler
9. How to Design Authentication System
10. How to Design Unified Payments Interface (UPI)

And, here is a high level design of YouTube from Educative.io for your reference:

---

𝐌𝐞𝐝𝐢𝐮𝐦 System Design Problems

Now, is the time to see medium difficulty of System design problems. These questions are neither easy nor very tough but you need good knowledge of various software architecture component and system design concepts to answer them.

11. Design Instagram [solution]
12. How to Design Tinder
13. Design WhatsApp (solution)
14. How to Design Facebook
15. Design Twitter
16. Design Reddit
17. Design Netflix [solution]
18. Design Youtube [solution]
19. Design Google Search
20. Design E-commerce Store like Amazon
21. Design Spotify
22. Design TikTok
23. Design Shopify
24. Design Airbnb
25. Design Autocomplete for Search Engines
26. Design Rate Limiter
27. Design Distributed Message Queue like Kafka
28. Design Flight Booking System
29. Design Online Code Editor
30. Design Stock Exchange System
31. Design an Analytics Platform (Metrics & Logging)
32. Design Notification Service
33. Design Payment System

And, here is a high level system design of Netflix from DesignGurus, one of my favorite place for learning system design

---

𝐇𝐚𝐫𝐝 System Design Problems

Now, let's see some hard questions which demand more effort from you. You may feel uncomfortable solving these questions but by doing this you become better.

34. How to Design Location Based Service like Yelp
35. Design Uber
36. Design Food Delivery App like Doordash
37. Design Google Docs
38. How to Design Google Maps
39. Design Zoom
40. How to Design File Sharing System like Dropbox
41. How to Design Ticket Booking System like BookMyShow
42. Design Distributed Web Crawler
43. How to Design Code Deployment System
44. Design Distributed Cloud Storage like S3
45. How to Design Distributed Locking Service

Here is high level design of Google Map by Educative.io

And, if you need solutions then they are available in this GitHub repository by @ Ashish Pratap Singh: https://github.com/ashishps1/awesome-system-design-resources/blob/main/README.md#system-design-interview-problems

And, now see a few more resources for System design interview preparation

---

Best System Design Interview Resources

And, here are curated list of the best system design books, online courses, and practice websites which you can check to better prepare for System design interviews. Most of these courses also answer questions I have shared here.

1. ByteByteGo: A live book and course by Alex Xu for System design interview preparation. It contains all the content of the System Design Interview book volumes 1 and 2, and will be updated with volume 3, which is coming soon.

2. Codemia.io: This is another great platform to practice System design problems for interviews. It has more than 120+ System design problems, many of which are free, and also a proper structure to solve them.

3. Bugfree.ai: This is another popular platform for technical interview preparation. It contains AI-based mock interviews as well as Interview experience and more than 3200+ real questions on System Design, Machine Learning, and other topics for practice =.

4. DesignGuru's Grokking System Design Course: An interactive learning platform with hands-on exercises and real-world scenarios to strengthen your system design skills.

5. "System Design Interview" book  by Alex Xu: This book provides an in-depth exploration of system design concepts, strategies, and interview preparation tips.

6. "System Design Primer" on GitHub: A curated list of resources, including articles, books, and videos, to help you prepare for system design interviews.

7. Educative's System Design Course: An interactive learning platform with hands-on exercises and real-world scenarios to strengthen your system design skills.

8. High Scalability Blog: A blog that features articles and case studies on the architecture of high-traffic websites and scalable systems.

9. YouTube Channels: Check out channels like "Gaurav Sen" (ex-Google engineer and founder of InterviewReddy.io and "Tech Dummies" for insightful videos on system design concepts and interview preparation.

10. "Designing Data-Intensive Applications" by Martin Kleppmann: A comprehensive guide that covers the principles and practices for designing scalable and reliable systems.

11. Exponent: A specialized site for interview prep, especially for FAANG companies like Amazon and Google. They also have a great system design course and many other materials that can help you crack FAANG interviews.

image_credit - ByteByteGo

Remember to combine theoretical knowledge with practical application by working on real-world projects and participating in mock interviews. Continuous practice and learning will undoubtedly enhance your proficiency in system design interviews.

That's all about 50 System design interview questions for 2026. If you are preparing for technical interviews, then most likely you can solve these questions, but if you struggle, you can see the answer links, which go to free tutorials and YouTube videos, as well as the online courses and books I have shared.

Whether you're a candidate preparing for a technical interview or a seasoned professional looking to refine your skills, mastering system design is a pivotal step in advancing your career in the ever-evolving tech industry, and these questions will help you.

Bonus

As promised, here is the bonus for you, a free book. I just found a new free book to learn Distributed System Design, you can also read it here on Microsoft --- https://info.microsoft.com/rs/157-GQE-382/images/EN-CNTNT-eBook-DesigningDistributedSystems.pdf

Difference between WHERE vs HAVING Clause in SQL

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

Hello friends, SQL questions are quite common on programming interviews and one of the popular SQL question is "WHERE vs HAVING clause"? When it comes to filtering records in SQL query, there are two main options, either by using WHERE clause or by using HAVING clause.

While both WHERE and HAVING are used for filtering rows, condition in WHERE clause is applied before grouping of data and condition on HAVING is applied after grouping

I mean, the main difference between them is that you can use WHERE if you want to filter data before grouping, I mean before you group records using GROUP BY clause and use HAVING when you want to filter groups.

This distinction is made by Query Engine on most of popular database like MySQL, Microsoft SQL Server, Oracle, and PostgreSQL

For example,

SELECT *
FROM BOOK
WHERE author="Joshua Bloch"

will only show books where author is "Joshua Bloch", here we have used WHERE clause because there is no grouping.

In case, we need grouping like authors with number of books we can use group by and having clause together and it will print only authors which have more than one book.

SELECT author, count(*) as NumberOfBooks
FROM BOOK
GROUP BY author
HAVING NumberOfBooks > 1

You can also use WHERE and HAVING clause together in one query and in that case WHERE clause will filter before grouping and HAVING clause will filter after grouping as shown in following example:

SELECT author, count(*) as NumberOfBooks
FROM BOOK
WHERE title like '%SQL%'
GROUP BY author
HAVING NumberOfBooks > 1

This will only print author which have multiple books with title 'SQL in them.

By the way, if you are new to SQL, then you can also use websites like Udemy, Coursera, Educative, ZTM Academy, freeCodeCamp, and VladMihalcea' SQL course to learn SQL in depth.

Difference between WHERE and HAVING clause in SQL?

Now that you know what is WHERE and HAVING clause in SQL and what the do, here are more useful difference between WHERE and HAVING clause in SQL :

1. WHERE clause can be used with SELECT, UPDATE and DELETE statements and clauses but HAVING clause can only be used with SELECT statements.

e.g.

SELECT * FROM Employee WHERE EmployeeId=3

This query will print details of employee with id = 3.

Similarly,

SELECT EmployeeName, COUNT(EmployeeName) AS NumberOfEmployee
FROM Employee
HAVING COUNT(EmployeeName) > 2;

this query will print duplicate employees from table.

2. We can't use aggregate functions in the where clause unless it is in a sub query contained in a HAVING clause whereas we can use aggregate function in Having clause. We can use column name in Having clause but the column must be contained in the group by clause.

3. WHERE clause is used on the individual records whereas Having Clause in conjunction with Group By Clause work on the record sets ( group of records ).

And, if you need more SQL questions like this then you can also see, Grokking the SQL Interview book which covers key topics for SQL interviews

That's all about difference between WHERE and HAVING clause in SQL. This is one of the important SQL questions and if you are preparing for Java developer interview, you should know the answer of this question.
While its a very common concept and we used it on daily basis, not many people can answer it correctly on interview.

Mentioning about keywords like filtering and before and after grouping is key here.

By the way, this is also a common SQL question on Java interviews and if you are preparing for Java interviews, you can also see my earlier articles like 35 Java Questions, 15 Spring Framework Questions and 6 System Design Problem to prepare other topics.

All the best !!

Top 10 Data Structures and Algorithms for System Design Interviews

Disclosure: This post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

Hi there, if you are preparing for a System Design Interview, then one thing you should focus on is learning different System Design Algorithms and what problems they solve in Distributed Systems and Microservices.

In the past, I have shared 6 System Design Problems and 10 Essential System Design topics and in this article, I am going to tell you 10 System Design algorithms and distributed data structures which every developer should learn.

Without any further ado, here are the 10 System Design algorithms and distributed Data Structures you can use to solve large-scale distributed system problems:

1. Consistent Hashing
2. MapReduce
3. Distributed Hash Tables (DHT)
4. Bloom Filters
5. Two-phase commit (2PC)
6. Paxos
7. Raft
8. Gossip protocol
9. Chord:
10. CAP theorem

These algorithms and distributed data structures are just a few examples of the many techniques that can be used to solve large-scale distributed system problems.

By the way, if you are preparing for System design interviews and want to learn System Design in depth then you can also checkout sites like ByteByteGo, Design Guru, Exponent, Educative, Codemia.io, bugfree.ai and Udemy which have many great System design courses, and these popular System design YouTube channels, which have many great System design courses and tutorials.

10 Distributed Data Structure and System Design Algorithms for Programmers

It's important to have a good understanding of these algorithms and how to apply them effectively in different scenarios.

So, let's deep dive into each of them and find out what they are, how they work, and when to use them.

1. Consistent Hashing

Consistent hashing is a technique used in distributed systems to efficiently distribute data among multiple nodes.

It is used to minimize the amount of data that needs to be transferred between nodes when a node is added or removed from the system.

The basic idea behind consistent hashing is to use a hash function to map each piece of data to a node in the system. Each node is assigned a range of hash values, and any data that maps to a hash value within that range is assigned to that node.

When a node is added or removed from the system, only the data that was assigned to that node needs to be transferred to another node. This is achieved by using a concept called virtual nodes.

Instead of assigning each physical node a range of hash values, multiple virtual nodes are assigned to each physical node.

Each virtual node is assigned a unique range of hash values, and any data that maps to a hash value within that range is assigned to the corresponding physical node.

When a node is added or removed from the system, only the virtual nodes that are affected need to be reassigned, and any data that was assigned to those virtual nodes is transferred to another node.

This allows the system to scale dynamically and efficiently, without requiring a full redistribution of data each time a node is added or removed.

Overall, consistent hashing provides a simple and efficient way to distribute data among multiple nodes in a distributed system. It is commonly used in large-scale distributed systems, such as content delivery networks and distributed databases, to provide high availability and scalability.

---

2. Map reduce

MapReduce is a programming model and framework for processing large datasets in a distributed system. It was originally developed by Google and is now widely used in many big data processing systems, such as Apache Hadoop.

The basic idea behind MapReduce is to break a large dataset into smaller chunks, distribute them across multiple nodes in a cluster, and process them in parallel. The processing is divided into two phases: a Map phase and a Reduce phase.

In the Map phase, the input dataset is processed by a set of Map functions in parallel. Each Map function takes a key-value pair as input and produces a set of intermediate key-value pairs as output.

These intermediate key-value pairs are then sorted and partitioned by key, and sent to the Reduce phase.

In the Reduce phase, the intermediate key-value pairs are processed by a set of Reduce functions in parallel. Each Reduce function takes a key and a set of values as input, and produces a set of output key-value pairs.

Here is an example of how MapReduce can be used to count the frequency of words in a large text file:

1. Map phase: Each Map function reads a chunk of the input file and outputs a set of intermediate key-value pairs, where the key is a word and the value is the number of occurrences of that word in the chunk.
2. Shuffle phase: The intermediate key-value pairs are sorted and partitioned by key, so that all the occurrences of each word are grouped together.
3. Reduce phase: Each Reduce function takes a word and a set of occurrences as input, and outputs a key-value pair where the key is the word and the value is the total number of occurrences of that word in the input file.

The MapReduce framework takes care of the parallel processing, distribution, and fault tolerance of the computation. This allows it to process large datasets efficiently and reliably, even on clusters of commodity hardware.

---

3. Distributed Hash Tables (DHT)

A Distributed Hash Table (DHT) is a distributed system that provides a decentralized key-value store. It is used in peer-to-peer (P2P) networks to store and retrieve information in a scalable and fault-tolerant manner.

In a DHT, each participating node stores a subset of the key-value pairs, and a mapping function is used to assign keys to nodes.

This allows nodes to locate the value associated with a given key by querying only a small subset of nodes, typically those responsible for keys close to the given key in the mapping space.

DHTs provide several desirable properties, such as self-organization, fault-tolerance, load-balancing, and efficient routing. They are commonly used in P2P file sharing systems, content distribution networks, and distributed databases.

One popular DHT algorithm is the Chord protocol, which uses a ring-based topology and a consistent hashing function to assign keys to nodes. Another widely used DHT is the Kademlia protocol, which uses a binary tree-like structure to locate nodes responsible for a given key.

---

4. Bloom Filters

Bloom Filters are a probabilistic data structure used for efficient set membership tests. They were introduced by Burton Howard Bloom in 1970.

A Bloom Filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set or not. It uses a bit array and a set of hash functions to store and check for the presence of an element in a set.

The process of adding an element to a Bloom Filter involves passing the element through a set of hash functions which returns a set of indices in the bit array. These indices are then set to 1 in the bit array.

To check whether an element is present in the set or not, the same hash functions are applied to the element and the resulting indices are checked in the bit array.

If all the bits at the indices are set to 1, then the element is considered to be present in the set. However, if any of the bits are not set, the element is considered to be absent from the set.

Since Bloom Filters use hash functions to index the bit array, there is a possibility of false positives, i.e., the filter may incorrectly indicate that an element is present in the set when it is not.

However, the probability of a false positive can be controlled by adjusting the size of the bit array and the number of hash functions used.

The false negative rate, i.e., the probability of a Bloom filter failing to identify an element that is actually present in the set, is zero.

Bloom Filters are widely used in various fields such as networking, databases, and web caching to perform efficient set membership tests.

---

5. 2 Phase Commit

Two-phase commit (2PC) is a protocol used to ensure the atomicity and consistency of transactions in distributed systems. It is a way to guarantee that all nodes participating in a transaction either commit or rollback together.

The two-phase commit protocol works in two phases:

1. Prepare Phase: In the prepare phase, the coordinator node sends a message to all participating nodes, asking them to prepare to commit the transaction.

Each participant responds with a message indicating whether it is prepared to commit or not. If any participant cannot prepare, it responds with a message indicating that it cannot participate in the transaction.

1. Commit Phase: If all participants are prepared to commit, the coordinator sends a message to all nodes asking them to commit. Each participant commits the transaction and sends an acknowledgement to the coordinator.

If any participant cannot commit, it rolls back the transaction and sends a message to the coordinator indicating that it has rolled back.

If the coordinator receives acknowledgements from all participants, it sends a message to all nodes indicating that the transaction has been committed.

If the coordinator receives a rollback message from any participant, it sends a message to all nodes indicating that the transaction has been rolled back.

The two-phase commit protocol ensures that all nodes in a distributed system agree on the outcome of a transaction, even in the presence of failures.

However, it has some drawbacks, including increased latency and the possibility of deadlock. Additionally, it requires a coordinator node, which can be a single point of failure.

---

6. Paxos

Paxos is a distributed consensus algorithm that allows a group of nodes to agree on a common value, even in the presence of failures. It was introduced by Leslie Lamport in 1998 and has become a fundamental algorithm for distributed systems.

The Paxos algorithm is designed to handle a variety of failure scenarios, including message loss, duplication, reordering, and node failures.

The algorithm proceeds in two phases: the prepare phase and the accept phase. In the prepare phase, a node sends a prepare message to all other nodes, asking them to promise not to accept any proposal with a number less than a certain value.

Once a majority of nodes have responded with promises, the node can proceed to the accept phase. In the accept phase, the node sends an accept message to all other nodes, proposing a certain value.

If a majority of nodes respond with an acceptance message, the value is considered accepted.

Paxos is a complex algorithm, and there are several variations and optimizations of it, such as Multi-Paxos, Fast Paxos, and others.

These variations aim to reduce the number of messages exchanged, optimize the latency of the algorithm, and reduce the number of nodes that need to participate in the consensus. Paxos is widely used in distributed databases, file systems, and other distributed systems where a high degree of fault tolerance is required.

---

7. Raft

Raft is a consensus algorithm designed to ensure fault-tolerance in distributed systems. It is used to maintain a replicated log that stores a sequence of state changes across multiple nodes in a cluster.

Raft achieves consensus by electing a leader, which coordinates the communication among the nodes and ensures that the log is consistent across the cluster.

The Raft algorithm consists of three main components: leader election, log replication, and safety. In the leader election phase, nodes in the cluster elect a leader using a randomized timeout mechanism.

The leader then coordinates the log replication by receiving state changes from clients and replicating them across the nodes in the cluster. Nodes can also request entries from the leader to ensure consistency across the cluster.

The safety component of Raft ensures that the algorithm is resilient to failures and ensures that the log is consistent across the cluster.

Raft achieves safety by ensuring that only one node can be the leader at any given time and by enforcing a strict ordering of log entries across the cluster.

Raft is widely used in distributed systems to provide fault-tolerance and high availability. It is often used in systems that require strong consistency guarantees, such as distributed databases and key-value stores.

---

8. Gossip

The gossip protocol is a peer-to-peer communication protocol used in distributed systems to disseminate information quickly and efficiently.

It is a probabilistic protocol that allows nodes to exchange information about their state with their neighbors in a decentralized manner.

The protocol gets its name from the way it spreads information like a rumor or gossip.

In a gossip protocol, nodes randomly select a set of other nodes to exchange information with. When a node receives information from another node, it then forwards that information to a subset of its neighbors, and the process continues.

Over time, the entire network becomes aware of the information as it spreads from node to node.

One of the key benefits of the gossip protocol is its fault-tolerance. Since the protocol relies on probabilistic communication rather than a central authority, it can continue to function even if some nodes fail or drop out of the network.

This makes it a useful tool in distributed systems where reliability is a critical concern.

Gossip protocols have been used in a variety of applications, including distributed databases, peer-to-peer file sharing networks, and large-scale sensor networks.

They are particularly well-suited to applications that require fast and efficient dissemination of information across a large number of nodes.

---

9. Chrod

Chord is a distributed hash table (DHT) protocol used for decentralized peer-to-peer (P2P) systems. It provides an efficient way to locate a node (or a set of nodes) in a P2P network given its identifier.

Chord allows P2P systems to scale to very large numbers of nodes while maintaining low overhead.

In a Chord network, each node is assigned an identifier, which can be any m-bit number. The nodes are arranged in a ring, where the nodes are ordered based on their identifiers in a clockwise direction.

Each node is responsible for a set of keys, which can be any value in the range of 0 to 2^m-1.

To find a key in the network, a node first calculates its hash value and then contacts the node whose identifier is the first clockwise successor of that hash value.

If the successor node does not have the desired key, it forwards the request to its successor, and so on, until the key is found. This process is known as a finger lookup, and it typically requires a logarithmic number of messages to find the desired node.

To maintain the consistency of the network, Chord uses a protocol called finger tables, which store information about other nodes in the network.

Each node maintains a finger table that contains the identifiers of its successors at increasing distances in the ring. This allows nodes to efficiently locate other nodes in the network without having to maintain a complete list of all nodes.

Chord also provides mechanisms for maintaining consistency when nodes join or leave the network. When a node joins the network, it notifies its immediate successor, which updates its finger table accordingly.

When a node leaves the network, its keys are transferred to its successor node, and the successor node updates its finger table to reflect the departure.

Overall, Chord provides an efficient and scalable way to locate nodes in a P2P network using a simple and decentralized protocol.

---

10. CAP Theorem

The CAP theorem, also known as Brewer's theorem, is a fundamental concept in distributed systems that states that it is impossible for a distributed system to simultaneously guarantee all of the following three properties:

1. Consistency: Every read receives the most recent write or an error.
2. Availability: Every request receives a response, without guarantee that it contains the most recent version of the information.
3. Partition tolerance: The system continues to function and provide consistent and available services even when network partitions occur.

In other words, a distributed system can only provide two out of the three properties mentioned above.

This theorem implies that in the event of a network partition, a distributed system must choose between consistency and availability.

For example, in a partitioned system, if one node cannot communicate with another node, it must either return an error or provide a potentially stale response.

The CAP theorem has significant implications for designing distributed systems, as it requires developers to make trade-offs between consistency, availability, and partition tolerance.

Conclusion

That's all about the essential System Design Data Structure, Algorithms and Protocol You can learn in 2023. In conclusion, system design is an essential skill for software engineers, especially those working on large-scale distributed systems.

These ten algorithms, data structure, and protocols provide a solid foundation for tackling complex problems and building scalable, reliable systems. By understanding these algorithms and their trade-offs, you can make informed decisions when designing and implementing systems.

Additionally, learning these algorithms can help you prepare for system design interviews and improve their problem-solving skills. However, it's important to note that these algorithms are just a starting point, and you should continue to learn and adapt as technology evolves.

By the way, if you are preparing for System design interviews and want to learn System Design in depth then you can also checkout sites like ByteByteGo, Design Guru, Exponent, Educative, Codemia.io, bugfree.ai and Udemy which have many great System design courses, and these popular System design YouTube channels, which have many great System design courses and tutorials.

Also, here is a nice System design template from DesignGuru which you can use to answer any System design question on interviews. It highlights key software architecture components and allows you to express your knowledge well.

All the best for your System design interviews!!