[MICROSERVICES] Message Queues and REST – An Approach with Go, NodeJS and Clean Architecture

Table of Contents

• 1. A Brief Context
• 2. Technologies and General Architecture
• 2.1. Microservices Architecture
• 2.2. Communication between Microservices
• 2.2.1. Message Queues (Asynchronous Communication)
• 2.2.2. REST APIs (Synchronous Communication)
• 2.3. Clean Architecture
• 3. The Project Ecosystem
• 4. Challenges and Final Considerations
• 4.1. Challenges in Implementation
• 4.2. Next Steps
• 4.3. Conclusion
• 5. References
• 6. Disclaimer

1. A brief context

In recent years, microservices architecture has become a popular choice for building scalable, flexible, and maintainable systems. By dividing an application into smaller, independent services, it is possible to maintain, test and upgrade each service autonomously, facilitating scalability and the inclusion of new technologies.

In this article, we will explore the creation of a microservices architecture using Go and NodeJS, two widely used languages ​​and, in this context, with complementary characteristics. Additionally, we will apply the principles of Clean Architecture, a design approach that aims to keep code clean, modular, and easy to maintain and test, ensuring that business logic is isolated from infrastructure concerns and/or dependencies.

The objective of this project is to practice Go, a language that I have been studying recently, and revisit fundamental concepts of microservices. At the same time, we will use TypeScript in the development of services, applying the principles of Clean Architecture to reinforce good software design practices.

Based on these premises, we will have the opportunity to explore both the positive points and challenges of this approach. After all, not every business requires such a complex structure, and a practical project is the best way to understand its real needs and implications.

2. Technologies and General Architecture

2.1 Microservices Architecture

Microservices architecture divides an application into smaller, independent services, each responsible for a specific part of the functionality. These services communicate through well-defined APIs, which facilitates maintenance, scalability and adoption of new technologies.

Benefits:

Modularity: Facilitates the maintenance and independent development of each service.
Scalability: Allows individual scalability of each service according to demand.
Resilience: Isolates failures and reduces the impact of problems in one service on others.
Comparison with Monoliths:

Monoliths: Applications integrated into a single code base. Although simple initially, they can become difficult to maintain and scale over time.

Microservices: They offer greater flexibility and scalability, but can create additional complexity in management and communication between services.

2.2 Communication between Microservices

In a microservices architecture, communication between services can be done in two main ways: asynchronous communication, using message queues for examples, and synchronous communication, through REST APIs. It is worth highlighting that there are other forms of communication besides queue and rest.

2.2.1 Message Queues (Asynchronous Communication)

Message queues are used to enable asynchronous communication between microservices. They allow services to send and receive messages without requiring an immediate response, which helps improve system resilience and scalability.

Role of Message Queues:

Asynchronous Communication: Facilitates the exchange of information between services without the need for instant response.
Resilience: Manages load spikes and temporary failures, ensuring that messages are eventually processed.

Implementation:

Tools: RabbitMQ and Kafka are popular options for managing message queues.
Integration: Implement message queues for communication between services written in Go and NodeJS, ensuring efficient and scalable data exchange.

2.2.2 REST APIs (Synchronous Communication)

RESTful APIs are used for synchronous communication between services. They are based on HTTP principles and allow services to interact in a standardized and efficient way.

2.3 Clean Architecture

Clean Architecture is a design approach that aims to create systems with a well-organized code base that is easy to maintain and/or test. It emphasizes separation of concerns and independence of layers.

Clean Architecture Principles:

Layer Separation: Divide the code into distinct layers (domain, application, infrastructure) to isolate business logic from technical concerns.
Independence from Frameworks and Libraries: Ensure that business logic is not dependent on specific frameworks or technologies.
Application in Microservices:

Code Organization: Structure each microservice following the principles of Clean Architecture to ensure modular, testable and easy to maintain code.
Maintenance and Evolution: Facilitate the addition of new features and the modification of existing ones without compromising the integrity of the system.

3. The project ecosystem

In the microservices ecosystem, an HTTP endpoint plays a crucial role in orchestrating the document workflow, in this context, it is where it all begins. This endpoint is responsible for receiving and processing the request to create a new document. Upon receiving a request, it queues the document to the Go worker, which will take care of generating and processing the document. Additionally, the endpoint issues a notification to the document service via a Message Queue, informing that a new resource, that is, a document, has entered the queue for processing. This approach ensures efficient integration between system components, allowing the endpoint to manage the creation and tracking of documents in a coordinated and asynchronous manner, while the Go worker takes care of the actual creation of documents and the document service is updated on new items in the queue.

In addition to the HTTP endpoint, the system has two workers with different roles. The first, implemented in Go, is responsible for generating documents. It consumes tasks from a message queue, processes the data, and upon completion of processing, notifies a specific endpoint of completion. Go's efficiency ensures fast and robust processing. The second worker, developed in NodeJS, handles the creation of the initial state of the documents, defining them as "unprocessed" when inserted into the system. The agility of NodeJS allows for quick and efficient management of document states from the beginning of the workflow.

In summary, the system described demonstrates a well-coordinated flow for document management. The HTTP endpoint, together with Go and NodeJS workers, provides an integrated and efficient solution, guaranteeing document processing from creation to completion. The interaction between workers and REST is reflected in the architecture image below, which illustrates how the architecture promotes scalability and modularity, ensuring a robust and coordinated workflow. This approach not only improves operational efficiency, but can also be adapted for different usage scenarios, offering flexibility and future growth.

The final drawing:

The project repository: https://github.com/williamMDsilva/microservice-poc

4. Challenges and Final Considerations

4.1 Challenges in Implementation

Implementing a microservices architecture with Clean Architecture can present several challenges. One of the main challenges is a deep understanding of the business to create code that is truly scalable and maintains the integrity of the business logic. Clean Architecture requires that code be structured in a way that clearly separates concerns, which requires detailed knowledge of the domain for abstractions and separations to be effective.

In addition, knowledge of SOLID principles is crucial. These principles help create more cohesive, less coupled code, and a solid understanding of them can save significant time researching and troubleshooting. Applying SOLID principles not only improves code quality, but also facilitates system maintenance and scalability.

In the specific case of Go, a solid foundation in the language can improve code readability and maintainability. Go offers tools and practices that help keep code clean and efficient, and deeper knowledge can make a difference when implementing complex services.

Finally, using a good boilerplate can be extremely beneficial. Well-designed Boilerplates for Clean Architecture not only speed up the start of development, but also ensure that new features are added within the initially proposed standard. They provide a structure that helps maintain code consistency and quality throughout the project.

4.2 Next Steps

To advance and improve the described architecture, some next steps are recommended:

1. Include Monitoring and Observability Resources: Implementing monitoring and observability tools is essential to ensuring system health and performance. The inclusion of metrics, logs and tracing helps identify problems and analyze system behavior in production.

2. Add Treatments for Unavailability: It is crucial to include mechanisms to deal with failures and unavailability, such as retries, circuit breakers and fallback strategies, to increase the resilience of the system and guarantee the continuity of services.

3. Perform Unit and Integration Tests: Tests are essential to ensure the quality of the code and the correct integration of components. Unit tests verify the functioning of isolated parts of the code, while integration tests ensure that the different components of the system work correctly together.

4. Refactor Services and Modules: Reviewing and refactoring existing services and modules to ensure that the code remains clean, readable and aligned with Clean Architecture principles is an ongoing task that improves the maintainability and scalability of the system.

5. Proof of Concept with Kafka: Considering a proof of concept to replace RabbitMQ with Kafka can offer insight into how different tools impact the project. Impact analysis on service design and overall architecture can provide valuable insights for future technology decisions.

4.3 Conclusion

This project demonstrated the effectiveness of well-planned architecture and the importance of careful implementation of Clean Architecture principles. The use of Go and NodeJS, combined with practices such as SOLID and the use of message queues and REST APIs, contributed to a robust and scalable system. However, developing and maintaining a microservices architecture presents challenges that require in-depth knowledge of the business and technology. Addressing these challenges with adequate planning and the adoption of good practices helps ensure the construction of efficient and sustainable systems. The path forward involves incorporating continuous improvements, such as advanced monitoring and new proofs of concept, to keep the architecture aligned with the needs and evolution of the business.

5. References

Martin, R. C. (2008). Clean Code: Practical Agile Software Skills. Alta Books.

Martin, R. C. (2017). Clean Architecture: Frameworks and Principles for Software Design. Alta Books.

RabbitMQ. (n.d.). Tutorial Two - JavaScript. Retrieved from https://www.rabbitmq.com/tutorials/tutorial-two-javascript

RabbitMQ. (n.d.). Tutorial Two - Go. Retrieved from https://www.rabbitmq.com/tutorials/tutorial-two-go

Disclaimer

Due to the academic and proof-of-concept nature of this project, some features were not implemented and remain as technical debt for future studies. Areas such as automated testing, error handling, resource streaming, service authentication and observability are topics that still need to be explored. All constructive criticism is welcome and encouraged, as it contributes to continuous improvement and deepening knowledge of these important areas.

The above is the detailed content of [MICROSERVICES] Message Queues and REST – An Approach with Go, NodeJS and Clean Architecture. For more information, please follow other related articles on the PHP Chinese website!