DevOps Essentials: Deploy a Jenkins Instance
With a Git repository as well!
Photo by Frederik Schweiger on Unsplash
All of the most wonderful things mankind has created came from small, unassuming, components that make up something worth more than the sum of its parts.
For example, mechanical watches leverage simple gears, springs, and weights to create a practical and elegant tool that we can use in our day-to-day.
Similarly to gears in a watch, automation servers like Jenkins are components that make up a CI/CD pipeline. If implemented well, can also make for a practical and elegant tool that we can use in our day-to-day.
Too Long; Didn't Read...
Although I recommend that people read through this blog to understand the thought process behind the scripts, I understand if people want to dive right into tinkering.
This is the GitHub repo of the entire Jenkins Project: SimpleJenkins
A summary of the blog:
Automation == Good
andManual Labor == Bad
.Use Docker for Jenkins b/c
simple >>> complicated
.Make scripts for Docker commands to make Jenkins go BRRRR.
Jenkins basic setup and configuration how-to.
Enhancements to make the scripts go
Sicko Mode
.
Why do we want Automation Servers?
The more accurate question to ask is "Why do we NOT want automation servers?"
For newcomers, Automation Servers like Jenkins help make complex tasks easier to execute and troubleshoot.
Suppose you're working for a company and your task is to load a DB onto 50 servers (1 hr), and then you need to execute a program afterward (0.5 hr).
Doing this WITHOUT an Automation Server would take around 75 hours to finish.
(This doesn't factor in troubleshooting errors in the operation btw*)
Practically speaking, it would take AT LEAST 2 full weeks to finish the task.
Don't even get me started about Human error...
This is where Automation Servers like Jenkins come in:
Instead of manually operating on 50 servers sequentially.
A server automatically operates on 50 servers in parallel.
So in theory, this 2-week task could be done in by around 5 hours:
1 hr to deploy Jenkins from scratch
1 hr to configure the Jenkins pipeline
1.5 hr to load the DB and run the program on all 50 servers (in parallel)
1 hr in case troubleshooting is needed
0.5 hr to do whatever you want
Building my own Jenkins Server
The previous example is a very textbook use of Jenkins.
Let's move on to a more simple use case. I want something that helps conveniently deploy a Minecraft server when I decide to make a change in its configuration:
Every time I push code onto GitHub and want to deploy it on my server, I need to:
$ docker-compose down
(Brings down the server)$ git pull
(Pull latest configs from GitHub)$ docker-compose up -d
(Brings up the server)
Yes, a bash script can do this quickly. However, it can't properly satisfy these points:
Remote execution of the pipeline (e.g. GitHub Actions)
Convenient and hassle-free deployment on new host machines
Frequent changes in the deployment runbook
This is my reason to deploy my own Jenkins Server.
Start Simple, and Keep It That Way
I will admit, this is my first time setting up a Jenkins server.
So let's start with my go-to option: Search for a docker image of Jenkins.
Luckily, Jenkins already provides documentation on deploying it using Docker.
So let's go ahead and set up a foundation where we put Jenkins on:
Allocate a Virtual Machine
Download and install Docker
Set a static IP address for the VM
Allocate a Virtual Machine
Creation of the VM
Creating the VM is pretty straightforward. There are plenty of guides to set one up.
The most important thing that you need to consider is the VM's hardware specs.
I recommend starting small and increasing the specs if needed.
I'm planning on having a Jenkins server with 1-5 jobs, so the specs are as follows:
OS: Ubuntu Server LTS
CPU: 1 core / 1 thread
RAM: 1024MB
Disk: 16 GB
VRAM: 16 MB (Recommended minimum)
You might have noticed these specs are the same as the t2.micro instance with 16 GB of disk on AWS. This isn't a coincidence, I have my VM specced like this to determine the viability of using Jenkins on a t2.micro instance.
Change your Password
Remember to change your password on your server!
If you're using an existing server, change its default password.
This is highly important because this server will likely connect to the internet.
For Ubuntu Server, use passwd
to set your password.
Download and Install Docker
I followed the install instructions provided in Docker's official docs for Ubuntu.
For instructions for other Operating Systems, I recommend you check this out.
I recommend that you go on the official site to install docker if you are following along. This is because the steps to install can change over time, so it's best if you get the instructions to install Docker from people that develop Docker.
If you don't understand what the wizards in Docker are saying, simply go on YouTube and find a step-by-step guide to help walk you through it.
IMPORTANT:
You may notice that you get a permission denied
error whenever a docker command is executed. It's normal, you just use sudo
to execute specific docker commands.
Since I'm using docker for tasks that I depend on (NOT as a tool to develop and test). I'll keep this sudo
requirement to deter intruders from unauthorized usage of docker.
Set a Static IP Address for the VM
VM Network Adaptor Shenanigans
REMEMBER: You need to have a bridged adapter between the VM and the host!!!
For example, this is what my setup is for my Jenkins VM on VBox:
I've set the "Attached to" section to "Bridged Adaptor" and selected a "Name" from the dropdown button.
If you don't use VBox, or this guide is depreciated:
Please search up "how to connect vm to local network" to get tutorials for this.
Set Static IP addresses in Linux
I have followed this tutorial from Linuxize to set up the static IP:
- Execute
ip link
to check for all the network interfaces.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp123s789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:13:27:2c:43:01 brd ff:ff:ff:ff:ff:ff
- Create a
netcfg
file:sudo vi /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
enp123s789:
dhcp4: no
addresses:
- << INSERT STATIC IP HERE >>/24
routes:
- to: 0.0.0.0/0
via: 192.168.2.1 # <-- This is the most common gateway. If you're unsure, search it up.
nameservers:
addresses: [8.8.8.8, 8.8.4.4]
- Apply the changes:
sudo netplan apply
Sidenote:
If you executed this while you are SSHing into the host, don't panic if it froze, it might be because the host switched to the static IP address. Simply SSH into the host via the static IP.
- Verify the changes:
ip addr show dev enp123s789
Launch Jenkins for the First Time
After all this prep work, let us start up Jenkins using the following command:
docker run -d \
--name "jenkins_test" \
--restart=on-failure \
-p 8080:8080 -p 50000:50000 \
-v jenkins_test_data:/var/jenkins_home \
jenkins/jenkins:lts-jdk11
What each part of this command does:
run -d
Execute docker in the background--name "jenkins_test"
Sets the container name asjenkins_test
-p 8080:8080 -p 50000:50000
Binds ports8080
and50000
used by Jenkins-v jenkins_test_data:/var/jenkins_home
Initialize a volume calledjenkins_test_data
and its directory is/var/jenkins_home
jenkins/jenkins:lts-jdk11
Use the docker image of Jenkins LTS (for JDK11)
After this script is executed, you can execute docker ps
you can see this output:
username@vmhost:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7dbd6cdd8404 jenkins/jenkins:lts-jdk11 "/usr/bin/tini -- /u…" 13 seconds ago Up 12 seconds 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp jenkins_test
As long as the STATUS says that it's "Up", you should be good to go.
Now, go onto your browser, and go to the URL of http://<Your Static IP>:8080
.
You should be able to see this something like this:
Now, I know what you are asking: "Where do I find this? I can't find it on my server!"
Let me give you two ways to get the password:
Method #1: Use Docker Logs
Executing docker logs <CONTAINER NAME>
prints the docker container's log output.
From here, we can see the initial password we can use to log into Jenkins.
username@vmhost:~$ sudo docker logs jenkins_test
... INFO and WARNING logs ...
*************************************************************
*************************************************************
*************************************************************
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
a23be8706d1644baae97dd5c51b5e3bd
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
*************************************************************
*************************************************************
... INFO and WARNING logs ...
Method #2: Bash into the Container
Jenkins said that we can get the password that is within/var/jenkins_home/secrets/initialAdminPassword
However, you can't access it in the normal directory on your server.
Instead, you need to access it in the container's directory using:docker exec -it <<Container Name>> /bin/bash
username@vmhost:~$ sudo docker exec -it jenkins_test /bin/bash
jenkins@7dbd6cdd8404:/$ ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
jenkins@7dbd6cdd8404:/$ cat /var/jenkins_home/secrets/initialAdminPassword
a23be8706d1644baae97dd5c51b5e3bd
jenkins@7dbd6cdd8404:/$ exit
After getting the password, put it into Jenkins and press "Continue".
Pick "Install suggested plugins" (pick this unless you know what you're doing)
You should now see a download page, just get a coffee while it downloads
Then, fill out the fields to create the admin user and press "Save and Continue"
Finally, you should be greeted with this final form:
Put in your static IP and select "Save and Finish"
Congratulations! You have now successfully started up Jenkins!
Finally, I HIGHLY recommend that you shut down and start your instance back up.
This is because we want to make sure that the initial docker command that we used to launcher our container can retain information even when it has been downed.
It's better to take a few minutes to test this than to take a few hours to redo pipelines that had been lost because the command you have used doesn't set a volume to use.
username@vmhost:~$ sudo docker stop jenkins_test
jenkins_test
username@vmhost:~$ sudo docker start jenkins_test
jenkins_test
Then, reload the browser and you should see the login screen.
Put your credentials in and log back into Jenkins.
Fantastic! So far, you have learned:
Why Automation Servers like Jenkins are useful and important to DevOps
How to set up the foundations to set up Jenkins (VM, Docker, and Static IP)
A Jenkins docker command to reliably launch a Jenkins configuration
A few docker commands that you'll be sure to use in the future:
docker ps
docker start
docker stop
docker logs
docker exec -it jenkins_test /bin/bash
Now after all this is said and done, let's set up a more robust framework to help us better work with this Jenkins container.
Building up our Jenkins Framework
First off, you may be asking yourself: "Why do we need to do more? We already have the command to create the container along with its start and stop commands".
To make things clear: Yes, we indeed have all the pieces for a Jenkins environment. However, we would want to still want to help make scripts and additional features to help improve the ergonomics and usability of this Jenkins server.
Enhancement #01: Basic Scripts
To begin, let's create basic scripts for our main actions:
Create <-- Creates the container
Remove <-- Remove the container
Start <-- Starts the container
Stop <-- Stops the container
Log <-- Helps us to quickly get the log output of a container
Bash <-- Helps us explore the container's file directory
# $ vi ./jenkins_config.sh
#!/bin/bash
CONTAINER_NAME="jenkins_McAws" # Rename this to anything you want
VOLUME_NAME="jenkins_McAws_data" # Rename this to anything you want
# $ vi ./create_DockerJenkins.sh
#!/bin/bash
echo "Creating Jenkins Container"
. ./jenkins_config.sh
docker create --name ${CONTAINER_NAME} \
--restart=on-failure \
-p 8080:8080 -p 50000:50000 \
-v ${VOLUME_NAME}:/var/jenkins_home \
jenkins/jenkins:lts-jdk11
# $ ./remove_DockerJenkins.sh
#!/bin/bash
echo "Removing Jenkins Container"
. ./jenkins_config.sh
docker stop ${CONTAINER_NAME}
docker rm ${CONTAINER_NAME}
# $ vi ./start_DockerJenkins.sh
#!/bin/bash
echo "Start Jenkins Container"
. ./jenkins_config.sh
docker start ${CONTAINER_NAME}
# $ vi ./stop_DockerJenkins.sh
#!/bin/bash
echo "Stopping Jenkins Container"
. ./jenkins_config.sh
docker stop ${CONTAINER_NAME}
# $ vi ./log_DockerJenkins.sh
#!/bin/bash
echo "Opening Logs of Jenkins Container"
. ./jenkins_config.sh
docker logs ${CONTAINER_NAME}
# $ vi ./bash_DockerJenkins.sh
#!/bin/bash
echo "Starting Bash of Jenkins Container"
. ./jenkins_config.sh
docker exec -it ${CONTAINER_NAME} /bin/bash
These scripts will help us to execute operations without needing to put in the container name (which can get real annoying, real fast). These scripts can also allow opportunities to bundle multiple commands at once (e.g. remove_DockerJenkins.sh
)
Enhancement #02: Dockerfile (i.e. Custom Image)
Let's enhance this further by creating a Dockerfile for Jenkins.
FROM <any_image_from_docker>
), and then you add whatever toppings you want until you're satisfied (i.e. COPY
, RUN
, ADD
, etc)We create this simple Dockerfile that is a copy of the jenkins/jenkins:lts-jdk11
image without any modifications. (Don't worry, we'll add more to this soon)
# $ vi Dockerfile
# We base our image off the Jenkins image we used on previous commands
FROM jenkins/jenkins:lts-jdk11
We must update the jenkins_config.sh
and create_DockerJenkins.sh
to accommodate this enhancement.
# $ vi ./jenkins_config.sh
#!/bin/bash
IMAGE_NAME="jenkins_image" # Specify our custom IMAGE_NAME
CONTAINER_NAME="jenkins_McAws"
VOLUME_NAME="jenkins_McAws_data"
# $ ./create_DockerJenkins.sh
#!/bin/bash
echo "Creating Jenkins Container"
. ./jenkins_config.sh
# Builds docker image using the Dockerfile and naming it as ${IMAGE_NAME}
docker build -t ${IMAGE_NAME} . # <- Don't forget the '.'!
docker create --name ${CONTAINER_NAME} \
--restart=on-failure \
-p 8080:8080 -p 50000:50000 \
-v ${VOLUME_NAME}:/var/jenkins_home \
${IMAGE_NAME} # Replace the base jenkins image with our custom image
Enhancement #03: Upping our Security
Although upping security doesn't make our lives 'easier' per se, it's still good to secure your hosts so that we have peace of mind that malicious attackers will have a harder time getting access to Jenkins' functionality.
Go to http://<your-static-ip-address:8080/manage/
and click on Security
Then, under Authorization
click on Matrix Based Security
Finally, you should see something like this:
I suggest you have this configuration:
Anonymous
has no rights. (i.e. nothing should be allowed without authentication)Authenticaticated Users
have everything EXCEPT forCredentials
andAdmin
.A named Admin account should have
Admin
rights.
So, it should look like this: (The user "test" is our admin account for this example)
After looking at this picture, you should have some questions:
Why does
Authenticated Users
haveOverall Read
access?- It's because Jenkins can't properly respond to requests (e.g. get URL pages) without it. In other words, it's the minimum level of access users need to have for Jenkins.
Why is the Admin account "test" not have the other checkboxes checked?
- Enabling
Admin
rights grant access to everything to the right of it.
- Enabling
While we're still talking about security, let us try to set up a token for our next enhancement: Safe(r) Jenkins shutdown.
Go to http://<your-static-ip-address:8080/user/<admin-username>/configure/
.
Then, click Add new Token
and type out the name of the token and click Generate
.
This should generate a password for it:
As you can see from the warning, the token will only be displayed ONCE, so save it and put it in a secure location to be used later.
Enhancement #04: Safe(r) Shutdown Sequence
I have a bit of paranoia that I might absent-mindedly forget that I need to shut down the Jenkins container before shutting down the host environment.
So what I'm trying to do is to create a way for the Jenkins container to shut itself down as safely as possible when the host environment is shutting down.
To do this, we need to understand a bit about how to shut down Jenkins manually.
There are a lot of ways to do it, but the one that we're looking for is a CLI method:
Put in http://<your-static-ip-address>:8080/cli/
to navigate to Jenin's CLI page:
And if we scroll down, we can see this command: safe-shutdown
In addition, we should also boost our
So, what this page is telling us is that we need to:
Download
jenkins-cli.jar
Execute:
java -jar jenkins-cli.jar -s <jenkins-URL> safe-shutdown
To meet these requirements, these are the enhancements I have made:
Create the entrypoint_jenkins.sh
file that we
# $ vi entrypoint_jenkins.sh
...
Create the Dockerfile
that does three things:
Put the
entrypoint_jenkins.sh
file into the container.Put the
auth_api_ADMIN_shutdown.txt
file into the container.Execute
entrypoint_jenkins.sh
every time that the container starts up.
# $ vi Dockerfile
FROM jenkins/jenkins:lts-jdk11
COPY entrypoint_jenkins.sh /usr/local/bin/entrypoint_jenkins.sh
COPY auth_api_ADMIN_shutdown.txt /usr/local/bin/auth_api_ADMIN_shutdown.txt
ENTRYPOINT ["/usr/local/bin/entrypoint_jenkins.sh"]
We'll cover the entrypoint_jenkins.sh
and auth_api_ADMIN_shutdown.txt
files below:
How does the entrypoint_jenkins.sh
script work?
We can separate this file into 3 components that work together:
The Execution of Jenkins
The download function that grabs the Jenkins CLI jar.
The graceful shutdown function that shuts down Jenkins as safely as possible.
Execution of Jenkins
Execute the same command that the default Docker Jenkins Image uses:exec /usr/bin/tini -- /usr/local/bin/jenkins.sh
Download Jenkins CLI
The download_jenkins_jar
function pulls the jar file it doesn't exist.
The command sleep 60 && download_jenkins_jar &
can be broken down as such:
${DOWNLOAD_COMMAND} &
(Executes the command in a separate thread)- This is done because the Jenkins startup command stops any further command execution until the Jenkins service stops. So, this is why we can't just put
download_jenkins_jar
after the startup command.
- This is done because the Jenkins startup command stops any further command execution until the Jenkins service stops. So, this is why we can't just put
sleep 60 && download_jenkins_jar
(Waits 60 secs and downloads the jar)- We wait 60 seconds because we want to wait for the Jenkins service to fully start before beginning the download.
We have PID_DOWNLOAD=$!
and wait ${PID_DOWNLOAD}
is to make sure that the download_jenkins_jar
function finishes (no matter if it's a success or failure, we must make sure it properly reaches its conclusion).
Graceful Shutdown
We have the function graceful_shutdown
that attempts to:
Download the Jenkins jar using
download_jenkins_jar
Execute the safe shutdown command:
java -jar ${JENKINS_CLI_JAR} -s http://localhost:8080 -auth "@${JENKINS_AUTH_FILE}" safe-shutdown
Then, we set up a trigger to execute graceful_shutdown
whenever a SIGTERM
is sent to the docker container (This occurs when the environment is shutting down):trap 'graceful_shutdown' SIGTERM
#!/bin/bash
JENKINS_HOME="/var/jenkins_home/"
JENKINS_CLI_JAR="${JENKINS_HOME}/jenkins-cli.jar"
JENKINS_AUTH_FILE="/usr/local/bin/auth_api_ADMIN_shutdown.txt"
download_jenkins_jar() {
echo "Checking if Jenkins CLI jar file exists."
if [ ! -e "${JENKINS_CLI_JAR}" ]; then
echo "Jenkins CLI jar does NOT exist, attempting to grab jenkins-cli."
# Attempt to download the jenkins CLI jar
curl http://localhost:8080/jnlpJars/jenkins-cli.jar -o ${JENKINS_CLI_JAR}
fi
}
# Define the function to handle the graceful shutdown
graceful_shutdown() {
echo "Received SIGTERM signal. Performing graceful shutdown."
download_jenkins_jar
echo "Attempting safe-shutdown of jenkins-cli."
# Stop Jenkins gracefully using the Jenkins CLI
java -jar ${JENKINS_CLI_JAR} -s http://localhost:8080 -auth "@${JENKINS_AUTH_FILE}" safe-shutdown
exit 1
}
# On a new thread: Sleeps for 60 seconds and then attempts to download Jenkins jar
# This 60 second timeout should allow jenkins to fully start up and be able to download the cli jar
sleep 60 && download_jenkins_jar &
PID_DOWNLOAD=$!
# Register the signal handler function
trap 'graceful_shutdown' SIGTERM
# Boot up Jenkins
exec /usr/bin/tini -- /usr/local/bin/jenkins.sh
# Wait for the PID download to finish
wait ${PID_DOWNLOAD}
The file auth_api_ADMIN_shutdown.txt
is the file we use to authenticate.
Needless to say, do not share this file with anyone.
What you are seeing below is for demonstration purposes only.
# $ auth_api_ADMIN_shutdown.txt (DO NOT INCLUDE THIS COMMENT!)
# [username]:[api_token_that_username_owns]
test:11763b252b8fc13e1f115b2c5831cfe59c
With all the enhancements in place, we now have a fully working Jenkins environment that we can work with.
I recommend that you follow the following setup steps:
Remove any and all docker containers/volumes that were used for testing.
Configure the
jenkins_config.sh
file.Create and start up the container using the scripts.
Log into Jenkins, and perform Jenkins setup if needed.
Configure security settings in Jenkins. (Set API token)
Shut down the container
Create and configure the
auth_api_ADMIN_shutdown.txt
fileStart up the environment again.
After all that, you should be good to go!
Of couse, this is the GitHub Repository for you to try out for yourself.