Dynamic Configuration Management in Distributed Systems — Part 1

This article is the fruit of the work and the commitment of us; three computer networks and telecommunications engineering students at the National Institute of Applied Sciences and Technology, Tunisia. Namely, Abdelmoula Eya, Med Ali Jardak and Ghaida Bouchâala. As part of our end of year project.

Not to forget to mention, the continuous support of our supervisor, Meher Assel, whose insightful comments and valuable expertise pushed us to refine our thinking e in formulating the research questions and methodology.… Thus, we had the opportunity to step out of our comfort zone while working on this original topic, and for that, we are very grateful.

Our work will be divided into 2 parts, each in an article, the first part will be covering the following:

• A general introduction of Configuration Management
• Configuration Management in Monolithic vs Distributed systems
• The configuration management classes
• The best practices of Configuration management
• The options to store or update a service configuration.
• The push and pull models as the two types of approaches to configuration management.

The second part will explore the implementation of dynamic configuration management by creating a simple proof of concept.

What was our main research driver for this topic?

In 2015, Facebook released a white paper that introduced Facebook’s holistic configuration management solution.

This was our starting point and that’s where it all began.

With this paper, we came across this golden rule: how to make a change to your application without changing the source code, in other terms, without having to restart your application to reflect changes?

This question made us know that configuration management is a crucial piece that can help manage every part of your application. Especially that Web and mobile applications are exposed to many online configuration changes.

The Opening :

Dynamic configuration management is an extremely important part of a modern distributed system, allowing the system to be tuned and its behaviour and functionality to be changed without having to redeploy the entire system.

An effective dynamic configuration system allows software developers to view and update configurations easily, and to deliver configuration updates to applications effectively and reliably.

It allows companies to iterate rapidly on new features and gives them tools to reduce the risk associated with modifying existing systems.

Now let’s get to the heart of the matter!

Configuration management is an essential part of modern applications. It can be challenging for monolithic systems, but not as challenging as for distributed systems where many services interact with each other, have many dependency relationships and may have shared configurations between two or more services.

Monolithic vs Distributed Systems Architecture

1. The Monolith approach

The architecture of a monolithic system is designed as a single, self-contained unit, like a big black box where everything is connected in one piece. This simple approach has a limit in terms of size and complexity.

A bug in one module can have the potential to bring down the entire workflow. In addition, since all the instances of the app are identical, that bug will impact the availability of the entire application.

2. The Distributed system approach

A distributed based system is one in which the different components of an application operate independently of one another and are only weakly coupled.

The distributed architecture has a number of benefits over monolithic systems.
With a distributed approach, for example, organizations are able to test and troubleshoot more efficiently, grow their technology stack and add new features without extensive development and testing.

Configuration Management in Monolithic vs Distributed systems:

1. Configuration Management in Monolithic system

Monolithic systems resemble a piece of a puzzle. This puzzle contains all the features of our application. We put the entire puzzle into a continuous build step. A smart build manager makes decisions around the software configuration of the application like what code to update or libraries to include.

However, the configuration file is linked directly to the whole system which means each feature is updated via one configuration file or many ones.

2. Configuration Management in a Distributed System

Distributed systems have multiple independent components which make each one has its own life cycle; its own configuration management and its own pipeline.

In a microservices environment for example, each micro-service has its own configuration file which makes it clear to update and modify dependencies but it becomes difficult to manage and apply patches. It seems to be obvious now to keep your configuration outside of the code so that anyone can update it without the struggle to find a proper line to be modified.

The Benefits of a good Configuration Management :

More and more companies are adopting configuration management tools and practices because the benefits they offer are truly game-changing. So let’s take a look at the most important benefits of configuration management.

• An easier way to manage the site.
• Prevent changes that cause downtime.
• No need to reinvent the wheel every time major changes are applied.
• Reduced performance and scaling requirements

Moving on, let’s look at a couple of the challenges that come with it.

The Challenges of configuration management in a distributed environment:

With modern software and systems being complex, configuration management can be a delicate undertaking to implement. It’s not just a question of opting to try configuration management and getting great results the next day.
When starting to implement configuration management, there will be a few key challenges:

• Prevent configuration errors
• Keeping consistency across several different teams and environments (e.g., test, production and staging) can be very tricky.
• The more complex your applications are, the more complicated they will be to manage.
• Having to work with several different vendors in various geographies and time zones makes it even more of a struggle to manage your systems.
• It’s much tougher to implement configuration management in companies that don’t embrace agile and DevOps principles.

Luckily, with the right approach to configuration management and the right tools in place, these challenges can be overcome.

Configuration Management Classes:

Static vs. Dynamic Configuration Management process

The task of maintaining configuration settings is one of the most vital parts of the process. It demands careful analysis and monitoring, planning and deployment. In addition, it requires you to be mindful of system requirements and the integrity of your configuration files. Reading variables such as parameterized variables, whether from global or environment-specific variables during system development, is the most fundamental part of any application, there are conditions based on these values that you must cover in a programmatic way and manage them in a separated configuration file.

1. Static Configuration Management

In the static configuration, we have to restart the whole application with even a small change in our configuration file.

In other words, all system components are configured at the same time.
So, if a single part is modified, the entire system must be shut down and restarted according to a new configuration specification.

The default way to use configuration files is to have them loaded at startup; this means that you have to restart your application to reflect the changes.

Modifying or extending the system by modifying existing components or introducing arbitrary new types of components will also require reprogramming and restarting the system.

2. Dynamic Configuration Management

Admittedly, using a static configuration is useful for several cases, but, sometimes we can not afford to take down the cache service for example nor afford to take down our database, and that’s solved by dynamic configuration. With dynamic configuration, It is possible to make changes and extensions without restarting the whole system, at any time while our application is running. And, can be done without stopping the unaffected parts of the system.

Dynamic configurations need more sophisticated and complex software development designs, and they also require a mechanism to control configuration management.

Consequently, throughout the application life cycle, the main idea of configuration management is to separate the application’s configurations from the application’s state, which means that we need to design our application so that it can read configurations dynamically from an external source. This can be done via a module that loads the configurations and applies them to the application state.

The Options to update a Service Configuration:

Here we will describe the different ways we can use to store/update a service configuration.

Option 1: Using Environment Variables

Environment variables are dynamic values that can affect the application. They can be created, modified, saved and deleted and give information about the system behaviour. The downside of this option is that it is not possible to cleanly update the environment variable of a running process.

Option 2: Using Config Files

Here, each service can read its configuration from a config file. This option has multiple advantages, such as the fact that it is simple to implement, easy for local development and can monitor file updates. However, in the case where we have a hundred services running, every update of the configuration requires updating all instances that use that configuration.

Option 3: Using KV store

With this option, a service reads its configuration from a remote store, which can either be a key-value store or any other sort of database. The good thing about this option is that the control plane needs to update the configuration value in only one place. Nevertheless, it places an additional responsibility on the services, which makes them more complex and the local development more challenging, as the programmers need to have the configuration store up and running.

Option 4: Using KV store combined with Config Files

With this approach, we can use a tool that catches configuration changes and then writes them to a configuration file alongside the service. This is perhaps the best option, although it normally has its advantages and disadvantages. The challenge is that there is a delay before you can see the updated config which depends on the technique used by the one updating the configuration. Plus, config files are automatically modified when values from the KV store change.

Option 4 Existing Tools

1. Consul Template

The Consul template tool gives a programmatic way to render configuration files from a wide variety of locations, including Consul KV. This is an excellent option for replacing the complicated API requests that often require customized formatting. The template tool is built on the Go templates and it shares many of the same attributes.
The Consul template is a simple, yet very powerful tool.

2. Kubernetes ConfigMaps

A ConfigMap is a Kubernetes API object that can be used to store data as key-value pairs. Kubernetes pods can use the created ConfigMaps as a:

• Configuration file • Environment variable • Command-line argument

ConfigMaps bind configuration files, command-line arguments, surroundings variables, port numbers, and alternative configuration artefacts to your Pods containers and system parts at run-time.

Pull vs Push Models

Two types of approaches to configuration management exist.

And well, configuration management tools generally tend to implement one (or both) of these management models. It’s all about how a given configuration management tool actually does the things it needs to do, like installing packages and writing files.

1. Pull Model

Well suited but difficult to manage. The server being provisioned (node) runs an agent (daemon) that asks a central authority (master) if/when it has updates to run. A daemon must be installed on all machines and a central authority must be configured.

2. Push Model

The centralised server will push configurations to the nodes. Simple to manage and configure, but not scalable. A central server contacts nodes and sends updates as needed. When a change is made to the infrastructure (code), each node is notified and executes the changes.

Which Model to choose?

Both types of configuration management tools can do everything. So the following comparison is based on the default features.

Advantages of Pull based Configuration Management

• Easier to bootstrap new nodes: Once a node is ready, the agent/client running on that node can begin pulling configuration from the main server. This means that a new machine is configured automatically when the server is ready to be configured (as opposed to the ”push” method where you have to check if the node is ready to be configured).
• Easier scalability: Since nodes can be started automatically and independently of other nodes, this means that scaling such a configuration is much easier.
• Easy to implement

Advantages of Push based Configuration Management

• System-wide control: Since the main server (where you store configuration information) pushes the configuration to the nodes, you have more control over which nodes need to be configured, actions are more synchronous, and error handling becomes much easier since you can immediately see and correct problems.
• Easier to use: Push configuration management tools, such as Ansible, are generally much easier to set up and start. Also, because nodes don’t pull information from the main server, development is more straightforward, as you can easily test your scripts without worrying about a node accidentally picking them up or changing agent settings.

Conclusion

In conclusion, configuration management, with its various classes and options, is an essential part of modern applications, as it offers real game-changing benefits.

Now that this part has finally come to an end, what’s next?

Well, in the next section, you will be able to see how to implement a real-world simple example using Flask, Consul and Consul template.

Bibliography

• BMC Blogs. “Creating Using ConfigMaps in Kubernetes.” In: (2021).
• Carlos Casanova. “Configuration Management, Why bother?” In: (2016).
• Facebook Research. “Holistic Configuration Management at Facebook.” In: (2015).
• Gayatri S Ajith. “Beginner Fundamentals: Push Pull Configuration Management Tools”. In: (2019).
• HashiCorp Learn. “Service Configuration with Consul Template — Consul.’” In: (2021)

Part-2, coming soon…Stay Tuned!