10 Jenkins lessons we learned the hard way | DevOps

How do you define DevOps? I am often asked this question, and I don’t have a great answer because DevOps isn’t about any one thing. It’s about people, processes, and tools working together to help deliver software in better ways than they did before—for their organization, their customers, their community, and themselves.

The best way to think of DevOps is as the intersection of software development and IT operations to enable high-velocity value delivery to your customers. It’s not one thing; it’s everything.

Don’t underestimate the time it takes to set up

A lot of time was spent just setting up our Jenkinsfile and configuring it to do what we needed. We also had to set up our cloud environment, which required a lot of trial and error. It’s important to anticipate how much time it will take to get your system running, especially if you’re aiming for high uptime. Don’t be afraid to be safe!

It took about a month for our Jenkinsfile to be complete, and even then, it’s never really done. We’re constantly modifying it to add new functionality and fix bugs, so it’s constantly evolving. You must give yourself enough time before the launch date to debug your script because mistakes will be made along the way. There are often minor errors which are easy to miss if they don’t cause an immediate failure but can become problematic later on.

Use a standard naming convention for your jobs

To write the names of your jobs, you should make sure you and your team members are using some standard naming convention. For example, if you’re running a job that deploys to the cloud, use a word like cloud or Azure in its name. That way, it will be easier to find all these jobs.

If you’re running a job that deploys to an on-premises environment, then it might make sense to use some convention that references your organization’s name. This makes it easy to recognize jobs and have conversations around them. It also lets you quickly find similar jobs for troubleshooting or enhancements.

Be sure to create different jobs when more than one set of steps is required for deployment (e.g., Azure, AWS). Use meaningful descriptions: As much as possible, be descriptive about what each job does by adding details about the type of environment it deploys to (e.g., Azure), what technologies it uses (e.g., Angular), and which stages the job performs (e.g., development).

Set up security early on

Jenkins is an open-source automation server. Before adding any users to a new project, you must ensure that security is enabled and configured. This is because it’s easier to secure an environment when installing it rather than trying to retrofit it later.

Always remember that other people will use your Jenkins code; if something is unsafe, it can be exploited. Even if the data being passed through isn’t sensitive, don’t use unsafe code - even if they seem harmless now, they may not be tomorrow. Get started quickly: It doesn’t matter what kind of Jenkins DevOps projects you have; there will always be some manual work involved which can take up valuable time.

Continuous integration is key

Continuous integration improves quality, reduces risk, and delivers fast changes. Continuous integration (CI) is a development practice where team members integrate their work daily. An automated build verifies each check-in (including a test) to detect problems early. CI aims to reduce the cost of integrating changes by identifying errors as quickly as possible after they were made. As a result, teams who do not use CI are more likely to experience high defect rates because code has not been tested until it’s too late in the process. With CI, bugs are detected during the development cycle rather than later when it’s too late.

On top of that, many organizations have been able to create working software faster through continuous integration. CI can improve productivity and reduce delays by enabling one team to make changes without waiting for other teams to finish their tasks. An added benefit is that developers learn from the feedback provided by unit tests, allowing them to write better code next time. With all these benefits combined, it’s easy to see why CI has become such a valuable part of today’s agile world.

Automate, automate, automate

Automate your processes to save time and money. This is the best lesson you can learn from this project. You’ll have to figure out what’s worth automating, but it’s often worth doing everything you can to make a process repeatable and reduce human error. The more work you can automate, the more focus you can put on value-added tasks and innovation instead of maintenance. When I think about how much time my team has spent automating things like deploys, pipelines, builds etc., I’m really happy that we’ve taken the time to do it right. It’s not an easy decision because it takes up a lot of engineering time, but in the long run, it pays off. It’s also very satisfying when someone comes to us with a question like Hey, could you build me some documentation? Or I need these features, and our response is always Sure! Let me just whip something up in 5 minutes. Done. All with no effort, and then go back to doing real work. Being responsive, and proactive: Try to anticipate problems before they happen by paying attention to critical metrics and being vigilant for warning signs. Taking action after a problem occurs won’t help anyone, so prevent problems from occurring by monitoring and fixing issues before they become too big. One example is writing automated tests to detect regressions in the code base. Be wary of tool lock-in: Using one particular toolset may sound appealing, but beware - it might be detrimental in the long run if you want to switch things up down the road (especially when vendor lock-ins are involved). If you are stuck in a situation where your vendor isn’t listening to feedback or meeting your needs, don’t feel bad about looking elsewhere for tools.

Create Bash Scripts

It is possible to write your own bash scripts that do the minor work that your pipeline requires instead of writing the groovy commands and DSL in the pipeline. It will also be easy for you to test your changes on a local machine if you desire. The great thing about Bash scripts is that they can be executed directly from the CLI or easily be used inside a Groovy script. The downside of using Bash scripts is that it’s not so well documented, but there are plenty of resources online to get up and running quickly. Plus, because this is written in Bash shell script and not Java (which Jenkins uses), you won’t need to compile anything, which means it’s much faster. There’s also some pretty good documentation available with Git Bash, which most developers use anyway when they want to execute shell scripts locally. So far, I have seen the following useful examples: A blog post explaining how to create Linux users, set permissions, create files with specific contents and manipulate time stamps on files and directories… We’re still working through these examples ourselves as part of our team learning process, but once again, I’m pleasantly surprised by how easy these scripts are to understand. They just take a few seconds to figure out, and then you can run them like any other executable file in Bash. They are just plain text files with a .sh extension.

Make sure to use the LTS versions whenever possible

As a general rule, it is a brilliant idea to use Jenkins, which is supported for an extended period of time (LTS), rather than the latest version of Jenkins. The latest version of the application may contain many features and quick enhancements. However, sometimes it may contain bugs or issues affecting the system’s stability.

As much as the LTS version may lag behind in features, it is less prone to bugs and is, therefore, more reliable than the non-LTS version. These updated features are expected to be added to the LTS release at some point in the future. This will allow you to use the upgraded version without worrying about stability issues for a long time to come. If you are still interested in trying the latest Jenkins release, I suggest you use it in your development sandbox first.

Whenever you decide to update a new version, check for any known issues or possible problems before doing so. Only then should you proceed with updating your system. You can check these issues on Stack Overflow and discuss them there. Developers typically report such issues within a few days of a new version’s release. Ensure you download and install stable versions until all known issues are resolved.

In addition, make sure to keep track of each installation’s history by regularly backing up your data. The most important tip is always to keep a backup!

Use Containers

Containers are a convenient way of doing most of what you need to do in development. In addition to minimizing the number of dependencies on the agent, doing the work in containers also provides better development and debugging experiences than doing the work in agents. Finding and maintaining the tool in the container is more accessible than finding and maintaining a plugin for Jenkins to understand another test format.

If you can use containers, do it. The benefits are worth it. Jenkins has been built with containers in mind, so this strategy fits right into the natural workflow.

Not just any Docker container will do - Jenkins requires a specific container configuration: one per job that runs only that job’s code. These containers provide an isolated environment, which means any changes made during the course of a run won’t be reflected back in other jobs or cause side effects like service interruptions or application crashes. As part of this configuration, each job needs its own unique Dockerfile. For example, a Java job might have its own Dockerfile that installs Java and related tools before running the tests; A JavaScript project would install NodeJS before running npm scripts, etc.

Always use the configuration as code

Consider a scenario in which someone on your team changes a configuration, and it breaks everything. There is no indication of why or how to resolve the change. That person tried to revert the change, but the system was already damaged to the point that it couldn’t be reversed. You would have to restore your old backups, which will take considerable time. It is a complete nightmare to suffer through this entire process! When you are already a few days behind schedule, it is very difficult to keep up with your schedule. Having Jenkins configured in code will prevent all of this from happening, and we can avoid all of this altogether. It is, therefore, essential to run through the code review and testing process before every change can be implemented.

Taking advantage of the ‘as code’ paradigm, we can reproduce and/or restore a full environment within a few minutes based on recipes and automation.

When it comes to Jenkins, hundreds of parameters can be configured within the web UI’s manage section.

A very easy-to-use, human-friendly YAML syntax isavailable for Jenkins Configuration as Code that can be used to define all of these configurations in a plain text format utilizing human-friendly YAML syntax. JCasC eliminates the need for manual steps in configuring Jenkins controllers using JCasC. The configuration can then be validated and applied to the controller in a reproducible manner without requiring any manual intervention on the user’s part.

Store credentials somewhere else

Jenkins’ parameters and secrets are not my cup of tea. AWS Secret Manager may be a better alternative based on my observations. For instance, one can store Amazon Web Services (AWS) credentials in Jenkins. Because of this setting, it will be able to access your account secrets. If you wish, you can retrieve secrets directly using a secret manager. This is much better than having to save them first in Jenkins.

You will have to install a plugin that can be used with AWS Secrets Manager Credentials Provider to use this tool.

Don’t forget about testing

Testing is integral to developing a solid, stable, and resilient application. We often assume that our applications are working as expected, but there is no substitute for testing your application before you release it to production. There are many ways to do so: unit testing, functional testing, integration testing, performance testing, etc.

It’s also important to test against changes in infrastructure, such as new operating systems or new browser versions. If we introduce changes in how our code interacts with the environment around it, we need to ensure those changes won’t break anything else in unexpected ways. We may not be able to foresee every potential issue with running our application on the newer version of a software library, new configuration setting, or any other change; however, by writing tests to capture this behaviour, we can identify these errors early on deploying them into production. Once detected in tests, these issues can be addressed immediately instead of after they’ve been deployed into production, where they may affect more users than if discovered earlier.

Use plugins wisely (Keep plugins to a minimum)

The plugins can be configured to update themselves automatically, but this isn’t always desirable. It creates too much instability. If you let them upgrade at will, you may get upgraded versions daily. These updated plugins can have bugs and even break functionality.

There will always be a risk of abandonware or incompatible sets of dependencies when choosing plugins, even when carefully selecting them. In most cases, it is better to stick with the plugin your team has used for years. You can avoid this by using Composer for more complex dependency management.

This means that there will always be some work involved in maintaining your plugin set over time, but with care, it should not become unmanageable. Of course, new features are constantly added, necessitating upgrading to stay up-to-date. But it should be possible to remain current without incurring significant overhead if you are diligent about monitoring security vulnerabilities, watching out for no longer supported systems, etc.

Keep an eye on your disk space

Disk space is often forgotten about until it’s too late. It’s important to monitor disk space usage and set alerts for when you are getting low. This will allow you to take action before your system crashes from running out of space. If a machine runs out of space, Jenkins will stop working, and all builds will fail.

A great tool that we found for monitoring our disk space is df -h or df -i, both of which display a graphical representation of what percentage of storage capacity you have left on your system in real-time.

One other thing to remember, don’t just monitor your home directory! Monitor /var/logs as well, where Jenkins stores its log files by default. Using the following command, you can do this:

- find /var/logs -mtime +30

This command searches all files under the var/logs directory whose last modification date was more than 30 days ago.

Remember to clean up old logs regularly with this command:

-find /var/logs -mtime +30 xargs rm
-find ~/Library/Logs/* -mtime +30 xargs rm
-find ~/.local/share/jenkins/*.log* -mtime +30 xargs rm

Backup, backup, backup (but only the information that you need)

If you back up your entire $JENKINS_HOME directory, you will be able to preserve the whole Jenkins instance. It is enough to copy the backup from the old system to the new one if it needs to be restored.

However, several files within that JENKINS_HOME directory are not required to be backed up. When you select specific directories and files to back up, you can produce smaller backups, but restoring the system may require more significant effort. Different directories can be backed up on different schedules.

The Jenkins-home should be backed up as well as jobs and user folders from the jenkins-home.

jobs: directory contains information related to all the jobs you create in Jenkins.

users: the directory contains information and keys related to the users.

Good:

It backs up only what you want

You can quickly move your files to a different environment

Not-Good:

It requires some effort to restore the required files only. Sometimes it may be necessary to manually edit the configuration file or use some scripts or shell commands.

If you choose to configure automatic backups with built-in tools like Rsync or Bash, make sure they have been tested before implementing them as part of your workflow. Jenkins comes with a set of default tools for data backup, such as DBUtils for backing up databases and Archiver for archiving files

...