Get to know software patterns using queues to perform distributed processing of workloads.
Context
In a queue-centric workflow pattern, work is placed in a queue and processed by multiple workers. Workers can retrieve work from the queue whenever they are ready to process it. Scaling is often based on the length of the queue or the average waiting time of a job in the queue (J). To scale, workers can be added or removed on demand.
Competing Consumers - Enterprise Integration Patterns
Queues are scalable cloud services like the AWS Simple Queue Service (SQS):
From https://aws.amazon.com/sqs/
Fully Managed Message Queuing – Amazon Simple Queue Service – Amazon Web Services
In a publish-subscribe architectural style, clients only communicate with a single server - also known as the middleware or broker. Clients do not know anything about other clients - they are referentially decoupled.
Clients communicate by publishing messages to the server. To receive messages, clients can subscribe to certain types of messages. The server ensures that each published message is delivered to all clients that have subscribed - if the subscription criteria is met.
Therefore, the communication is fully asynchronous. A client does not know if a message it published was received by any other client.
Introduction to MQTT Publish-Subscribe Pattern
Queueing is a software pattern where tasks or messages are placed in a queue and processed by workers in a sequential manner. In distributed systems, queues enable asynchronous communication where publishers/producers and subscribers/consumers are referentially decoupled, meaning they don't need to know about each other.
A practical example would be a cloud service like AWS SQS, where multiple workers can process tasks from a queue, allowing for scalable and flexible workload management.
Some of the most used servers and brokers for queueing are:
RabbitMQ: One broker to queue them all | RabbitMQ
Message Queuing Service - Amazon Simple Queue Service - AWS
Redis - The Real-time Data Platform