All of the most wonderful things mankind has created came from small, unassuming, components that make up something worth more than the sum of its parts.

For example, mechanical watches leverage simple gears, springs, and weights to create a practical and elegant tool that we can use in our day-to-day.

Similarly to gears in a watch, automation servers like Jenkins are components that make up a CI/CD pipeline. If implemented well, can also make for a practical and elegant tool that we can use in our day-to-day.

Too Long; Didn't Read...

Although I recommend that people read through this blog to understand the thought process behind the scripts, I understand if people want to dive right into tinkering.

This is the GitHub repo of the entire Jenkins Project: SimpleJenkins

A summary of the blog:

Automation == Good and Manual Labor == Bad.
Use Docker for Jenkins b/c simple >>> complicated.
Make scripts for Docker commands to make Jenkins go BRRRR.
Jenkins basic setup and configuration how-to.
Enhancements to make the scripts go Sicko Mode.

Why do we want Automation Servers?

The more accurate question to ask is "Why do we NOT want automation servers?"

For newcomers, Automation Servers like Jenkins help make complex tasks easier to execute and troubleshoot.

Suppose you're working for a company and your task is to load a DB onto 50 servers (1 hr), and then you need to execute a program afterward (0.5 hr).

Doing this WITHOUT an Automation Server would take around 75 hours to finish.
(This doesn't factor in troubleshooting errors in the operation btw*)
Practically speaking, it would take AT LEAST 2 full weeks to finish the task.
Don't even get me started about Human error...

This is where Automation Servers like Jenkins come in:
Instead of manually operating on 50 servers sequentially.
A server automatically operates on 50 servers in parallel.
So in theory, this 2-week task could be done in by around 5 hours:

1 hr to deploy Jenkins from scratch
1 hr to configure the Jenkins pipeline
1.5 hr to load the DB and run the program on all 50 servers (in parallel)
1 hr in case troubleshooting is needed
0.5 hr to do whatever you want

Building my own Jenkins Server

The previous example is a very textbook use of Jenkins.
Let's move on to a more simple use case. I want something that helps conveniently deploy a Minecraft server when I decide to make a change in its configuration:

Every time I push code onto GitHub and want to deploy it on my server, I need to:

$ docker-compose down (Brings down the server)
$ git pull (Pull latest configs from GitHub)
$ docker-compose up -d (Brings up the server)

Yes, a bash script can do this quickly. However, it can't properly satisfy these points:

Remote execution of the pipeline (e.g. GitHub Actions)
Convenient and hassle-free deployment on new host machines
Frequent changes in the deployment runbook

This is my reason to deploy my own Jenkins Server.

Start Simple, and Keep It That Way

I will admit, this is my first time setting up a Jenkins server.
So let's start with my go-to option: Search for a docker image of Jenkins.

Luckily, Jenkins already provides documentation on deploying it using Docker.

So let's go ahead and set up a foundation where we put Jenkins on:

Allocate a Virtual Machine
Download and install Docker
Set a static IP address for the VM

Allocate a Virtual Machine

Creation of the VM

Creating the VM is pretty straightforward. There are plenty of guides to set one up.

The most important thing that you need to consider is the VM's hardware specs.
I recommend starting small and increasing the specs if needed.

I'm planning on having a Jenkins server with 1-5 jobs, so the specs are as follows:

OS: Ubuntu Server LTS
CPU: 1 core / 1 thread
RAM: 1024MB
Disk: 16 GB
VRAM: 16 MB (Recommended minimum)

You might have noticed these specs are the same as the t2.micro instance with 16 GB of disk on AWS. This isn't a coincidence, I have my VM specced like this to determine the viability of using Jenkins on a t2.micro instance.

Change your Password

Remember to change your password on your server!
If you're using an existing server, change its default password.
This is highly important because this server will likely connect to the internet.

For Ubuntu Server, use passwd to set your password.

Download and Install Docker

I followed the install instructions provided in Docker's official docs for Ubuntu.

For instructions for other Operating Systems, I recommend you check this out.

I recommend that you go on the official site to install docker if you are following along. This is because the steps to install can change over time, so it's best if you get the instructions to install Docker from people that develop Docker.

If you don't understand what the wizards in Docker are saying, simply go on YouTube and find a step-by-step guide to help walk you through it.

IMPORTANT:
You may notice that you get a permission denied error whenever a docker command is executed. It's normal, you just use sudo to execute specific docker commands.
Since I'm using docker for tasks that I depend on (NOT as a tool to develop and test). I'll keep this sudo requirement to deter intruders from unauthorized usage of docker.

Set a Static IP Address for the VM

VM Network Adaptor Shenanigans

REMEMBER: You need to have a bridged adapter between the VM and the host!!!

For example, this is what my setup is for my Jenkins VM on VBox:
I've set the "Attached to" section to "Bridged Adaptor" and selected a "Name" from the dropdown button.

If you don't use VBox, or this guide is depreciated:
Please search up "how to connect vm to local network" to get tutorials for this.

Set Static IP addresses in Linux

I have followed this tutorial from Linuxize to set up the static IP:

Execute ip link to check for all the network interfaces.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp123s789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:13:27:2c:43:01 brd ff:ff:ff:ff:ff:ff

Create a netcfg file: sudo vi /etc/netplan/01-netcfg.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp123s789:
      dhcp4: no
      addresses: 
        - << INSERT STATIC IP HERE >>/24
      routes:
        - to: 0.0.0.0/0
          via: 192.168.2.1 # <-- This is the most common gateway. If you're unsure, search it up. 
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

Apply the changes: sudo netplan apply

Sidenote:
If you executed this while you are SSHing into the host, don't panic if it froze, it might be because the host switched to the static IP address. Simply SSH into the host via the static IP.

Verify the changes: ip addr show dev enp123s789

Launch Jenkins for the First Time

After all this prep work, let us start up Jenkins using the following command:

docker run -d \
    --name "jenkins_test" \
    --restart=on-failure \
    -p 8080:8080 -p 50000:50000 \
    -v jenkins_test_data:/var/jenkins_home \
    jenkins/jenkins:lts-jdk11

What each part of this command does:

run -d
Execute docker in the background
--name "jenkins_test"
Sets the container name as jenkins_test
-p 8080:8080 -p 50000:50000
Binds ports 8080 and 50000 used by Jenkins
-v jenkins_test_data:/var/jenkins_home
Initialize a volume called jenkins_test_data and its directory is /var/jenkins_home
jenkins/jenkins:lts-jdk11
Use the docker image of Jenkins LTS (for JDK11)

After this script is executed, you can execute docker ps you can see this output:

username@vmhost:~$ sudo docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                                                                                      NAMES
7dbd6cdd8404   jenkins/jenkins:lts-jdk11   "/usr/bin/tini -- /u…"   13 seconds ago   Up 12 seconds   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp   jenkins_test

As long as the STATUS says that it's "Up", you should be good to go.

Now, go onto your browser, and go to the URL of http://<Your Static IP>:8080.
You should be able to see this something like this:

Now, I know what you are asking: "Where do I find this? I can't find it on my server!"

Let me give you two ways to get the password:

Method #1: Use Docker Logs

Executing docker logs <CONTAINER NAME> prints the docker container's log output.
From here, we can see the initial password we can use to log into Jenkins.

username@vmhost:~$ sudo docker logs jenkins_test
... INFO and WARNING logs ...

*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

a23be8706d1644baae97dd5c51b5e3bd

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

... INFO and WARNING logs ...

Method #2: Bash into the Container

Jenkins said that we can get the password that is within
/var/jenkins_home/secrets/initialAdminPassword
However, you can't access it in the normal directory on your server.
Instead, you need to access it in the container's directory using:
docker exec -it <<Container Name>> /bin/bash

username@vmhost:~$ sudo docker exec -it jenkins_test /bin/bash
jenkins@7dbd6cdd8404:/$ ls
bin  boot  dev    etc  home  lib    lib64  media  mnt  opt    proc  root  run  sbin  srv  sys  tmp  usr  var
jenkins@7dbd6cdd8404:/$ cat /var/jenkins_home/secrets/initialAdminPassword
a23be8706d1644baae97dd5c51b5e3bd
jenkins@7dbd6cdd8404:/$ exit

After getting the password, put it into Jenkins and press "Continue".

Pick "Install suggested plugins" (pick this unless you know what you're doing)

You should now see a download page, just get a coffee while it downloads

Then, fill out the fields to create the admin user and press "Save and Continue"

Finally, you should be greeted with this final form:
Put in your static IP and select "Save and Finish"

Congratulations! You have now successfully started up Jenkins!

Finally, I HIGHLY recommend that you shut down and start your instance back up.
This is because we want to make sure that the initial docker command that we used to launcher our container can retain information even when it has been downed.

It's better to take a few minutes to test this than to take a few hours to redo pipelines that had been lost because the command you have used doesn't set a volume to use.

username@vmhost:~$ sudo docker stop jenkins_test
jenkins_test
username@vmhost:~$ sudo docker start jenkins_test
jenkins_test

Then, reload the browser and you should see the login screen.
Put your credentials in and log back into Jenkins.

Fantastic! So far, you have learned:

Why Automation Servers like Jenkins are useful and important to DevOps
How to set up the foundations to set up Jenkins (VM, Docker, and Static IP)
A Jenkins docker command to reliably launch a Jenkins configuration
A few docker commands that you'll be sure to use in the future:
- docker ps
- docker start
- docker stop
- docker logs
- docker exec -it jenkins_test /bin/bash

Now after all this is said and done, let's set up a more robust framework to help us better work with this Jenkins container.

Building up our Jenkins Framework

First off, you may be asking yourself: "Why do we need to do more? We already have the command to create the container along with its start and stop commands".

To make things clear: Yes, we indeed have all the pieces for a Jenkins environment. However, we would want to still want to help make scripts and additional features to help improve the ergonomics and usability of this Jenkins server.

💭

Improving the ergonomics of an environment is a very worthwhile investment across all levels of a company. Ergonomics help lead to less fatigue and burnout for the personnel in charge of BAU operations. Reduced burnout helps ease turnover and saves the company money by not needing to spend as much hiring and onboarding newcomers to replace lost numbers.

Enhancement #01: Basic Scripts

To begin, let's create basic scripts for our main actions:

Create <-- Creates the container
Remove <-- Remove the container
Start <-- Starts the container
Stop <-- Stops the container
Log <-- Helps us to quickly get the log output of a container
Bash <-- Helps us explore the container's file directory

# $ vi ./jenkins_config.sh

#!/bin/bash

CONTAINER_NAME="jenkins_McAws" # Rename this to anything you want
VOLUME_NAME="jenkins_McAws_data" # Rename this to anything you want

# $ vi ./create_DockerJenkins.sh 

#!/bin/bash

echo "Creating Jenkins Container"

. ./jenkins_config.sh

docker create --name ${CONTAINER_NAME} \
  --restart=on-failure \
  -p 8080:8080 -p 50000:50000 \
  -v ${VOLUME_NAME}:/var/jenkins_home \
  jenkins/jenkins:lts-jdk11

# $ ./remove_DockerJenkins.sh

#!/bin/bash

echo "Removing Jenkins Container"

. ./jenkins_config.sh

docker stop ${CONTAINER_NAME}

docker rm ${CONTAINER_NAME}

# $ vi ./start_DockerJenkins.sh

#!/bin/bash

echo "Start Jenkins Container"

. ./jenkins_config.sh

docker start ${CONTAINER_NAME}

# $ vi ./stop_DockerJenkins.sh

#!/bin/bash

echo "Stopping Jenkins Container"

. ./jenkins_config.sh

docker stop ${CONTAINER_NAME}

# $ vi ./log_DockerJenkins.sh

#!/bin/bash

echo "Opening Logs of Jenkins Container"

. ./jenkins_config.sh

docker logs ${CONTAINER_NAME}

# $ vi ./bash_DockerJenkins.sh

#!/bin/bash

echo "Starting Bash of Jenkins Container"

. ./jenkins_config.sh

docker exec -it ${CONTAINER_NAME} /bin/bash

These scripts will help us to execute operations without needing to put in the container name (which can get real annoying, real fast). These scripts can also allow opportunities to bundle multiple commands at once (e.g. remove_DockerJenkins.sh)

Enhancement #02: Dockerfile (i.e. Custom Image)

Let's enhance this further by creating a Dockerfile for Jenkins.

💡

The Dockerfile is a recipe to create a docker image for containers. Think of it like ordering a pizza, you typically start from a base (i.e. FROM <any_image_from_docker> ), and then you add whatever toppings you want until you're satisfied (i.e. COPY, RUN, ADD, etc)

We create this simple Dockerfile that is a copy of the jenkins/jenkins:lts-jdk11 image without any modifications. (Don't worry, we'll add more to this soon)

# $ vi Dockerfile

# We base our image off the Jenkins image we used on previous commands
FROM jenkins/jenkins:lts-jdk11

We must update the jenkins_config.sh and create_DockerJenkins.sh to accommodate this enhancement.

# $ vi ./jenkins_config.sh

#!/bin/bash

IMAGE_NAME="jenkins_image" # Specify our custom IMAGE_NAME
CONTAINER_NAME="jenkins_McAws"
VOLUME_NAME="jenkins_McAws_data"

# $ ./create_DockerJenkins.sh

#!/bin/bash

echo "Creating Jenkins Container"

. ./jenkins_config.sh

# Builds docker image using the Dockerfile and naming it as ${IMAGE_NAME}
docker build -t ${IMAGE_NAME} . # <- Don't forget the '.'!

docker create --name ${CONTAINER_NAME} \
  --restart=on-failure \
  -p 8080:8080 -p 50000:50000 \
  -v ${VOLUME_NAME}:/var/jenkins_home \
  ${IMAGE_NAME} # Replace the base jenkins image with our custom image

Enhancement #03: Upping our Security

Although upping security doesn't make our lives 'easier' per se, it's still good to secure your hosts so that we have peace of mind that malicious attackers will have a harder time getting access to Jenkins' functionality.

Go to http://<your-static-ip-address:8080/manage/ and click on Security

Then, under Authorization click on Matrix Based Security

Finally, you should see something like this:

I suggest you have this configuration:

Anonymous has no rights. (i.e. nothing should be allowed without authentication)
Authenticaticated Users have everything EXCEPT for Credentials and Admin.
A named Admin account should have Admin rights.

So, it should look like this: (The user "test" is our admin account for this example)

After looking at this picture, you should have some questions:

Why does Authenticated Users have Overall Read access?
- It's because Jenkins can't properly respond to requests (e.g. get URL pages) without it. In other words, it's the minimum level of access users need to have for Jenkins.
Why is the Admin account "test" not have the other checkboxes checked?
- Enabling Admin rights grant access to everything to the right of it.

While we're still talking about security, let us try to set up a token for our next enhancement: Safe(r) Jenkins shutdown.

Go to http://<your-static-ip-address:8080/user/<admin-username>/configure/.
Then, click Add new Token and type out the name of the token and click Generate.

This should generate a password for it:

As you can see from the warning, the token will only be displayed ONCE, so save it and put it in a secure location to be used later.

Enhancement #04: Safe(r) Shutdown Sequence

I have a bit of paranoia that I might absent-mindedly forget that I need to shut down the Jenkins container before shutting down the host environment.

So what I'm trying to do is to create a way for the Jenkins container to shut itself down as safely as possible when the host environment is shutting down.

To do this, we need to understand a bit about how to shut down Jenkins manually.
There are a lot of ways to do it, but the one that we're looking for is a CLI method:

Put in http://<your-static-ip-address>:8080/cli/ to navigate to Jenin's CLI page:

And if we scroll down, we can see this command: safe-shutdown

In addition, we should also boost our

So, what this page is telling us is that we need to:

Download jenkins-cli.jar
Execute: java -jar jenkins-cli.jar -s <jenkins-URL> safe-shutdown

To meet these requirements, these are the enhancements I have made:

Create the entrypoint_jenkins.sh file that we

# $ vi entrypoint_jenkins.sh

...

Create the Dockerfile that does three things:

Put the entrypoint_jenkins.sh file into the container.
Put the auth_api_ADMIN_shutdown.txt file into the container.
Execute entrypoint_jenkins.sh every time that the container starts up.

# $ vi Dockerfile
FROM jenkins/jenkins:lts-jdk11

COPY entrypoint_jenkins.sh /usr/local/bin/entrypoint_jenkins.sh
COPY auth_api_ADMIN_shutdown.txt /usr/local/bin/auth_api_ADMIN_shutdown.txt 

ENTRYPOINT ["/usr/local/bin/entrypoint_jenkins.sh"]

We'll cover the entrypoint_jenkins.sh and auth_api_ADMIN_shutdown.txt files below:

How does the entrypoint_jenkins.sh script work?
We can separate this file into 3 components that work together:

The Execution of Jenkins
The download function that grabs the Jenkins CLI jar.
The graceful shutdown function that shuts down Jenkins as safely as possible.

Execution of Jenkins

Execute the same command that the default Docker Jenkins Image uses:
exec /usr/bin/tini -- /usr/local/bin/jenkins.sh

Download Jenkins CLI

The download_jenkins_jar function pulls the jar file it doesn't exist.

The command sleep 60 && download_jenkins_jar & can be broken down as such:

${DOWNLOAD_COMMAND} & (Executes the command in a separate thread)
- This is done because the Jenkins startup command stops any further command execution until the Jenkins service stops. So, this is why we can't just put download_jenkins_jar after the startup command.
sleep 60 && download_jenkins_jar (Waits 60 secs and downloads the jar)
- We wait 60 seconds because we want to wait for the Jenkins service to fully start before beginning the download.

We have PID_DOWNLOAD=$! and wait ${PID_DOWNLOAD} is to make sure that the download_jenkins_jar function finishes (no matter if it's a success or failure, we must make sure it properly reaches its conclusion).

Graceful Shutdown

We have the function graceful_shutdown that attempts to:

Download the Jenkins jar using download_jenkins_jar
Execute the safe shutdown command:
java -jar ${JENKINS_CLI_JAR} -s http://localhost:8080 -auth "@${JENKINS_AUTH_FILE}" safe-shutdown

Then, we set up a trigger to execute graceful_shutdown whenever a SIGTERM is sent to the docker container (This occurs when the environment is shutting down):
trap 'graceful_shutdown' SIGTERM

#!/bin/bash

JENKINS_HOME="/var/jenkins_home/"
JENKINS_CLI_JAR="${JENKINS_HOME}/jenkins-cli.jar"
JENKINS_AUTH_FILE="/usr/local/bin/auth_api_ADMIN_shutdown.txt"

download_jenkins_jar() {
        echo "Checking if Jenkins CLI jar file exists."
        if [ ! -e "${JENKINS_CLI_JAR}" ]; then
                echo "Jenkins CLI jar does NOT exist, attempting to grab jenkins-cli."
                # Attempt to download the jenkins CLI jar
                curl http://localhost:8080/jnlpJars/jenkins-cli.jar -o ${JENKINS_CLI_JAR}
        fi
}

# Define the function to handle the graceful shutdown
graceful_shutdown() {
        echo "Received SIGTERM signal. Performing graceful shutdown."

        download_jenkins_jar 

        echo "Attempting safe-shutdown of jenkins-cli."
        # Stop Jenkins gracefully using the Jenkins CLI
        java -jar ${JENKINS_CLI_JAR} -s http://localhost:8080 -auth "@${JENKINS_AUTH_FILE}" safe-shutdown

        exit 1
}

# On a new thread: Sleeps for 60 seconds and then attempts to download Jenkins jar
# This 60 second timeout should allow jenkins to fully start up and be able to download the cli jar
sleep 60 && download_jenkins_jar & 
PID_DOWNLOAD=$!

# Register the signal handler function
trap 'graceful_shutdown' SIGTERM

# Boot up Jenkins
exec /usr/bin/tini -- /usr/local/bin/jenkins.sh

# Wait for the PID download to finish
wait ${PID_DOWNLOAD}

The file auth_api_ADMIN_shutdown.txt is the file we use to authenticate.
Needless to say, do not share this file with anyone.
What you are seeing below is for demonstration purposes only.

# $ auth_api_ADMIN_shutdown.txt (DO NOT INCLUDE THIS COMMENT!)
# [username]:[api_token_that_username_owns]
test:11763b252b8fc13e1f115b2c5831cfe59c

With all the enhancements in place, we now have a fully working Jenkins environment that we can work with.

I recommend that you follow the following setup steps:

Remove any and all docker containers/volumes that were used for testing.
Configure the jenkins_config.sh file.
Create and start up the container using the scripts.
Log into Jenkins, and perform Jenkins setup if needed.
Configure security settings in Jenkins. (Set API token)
Shut down the container
Create and configure the auth_api_ADMIN_shutdown.txt file
Start up the environment again.

After all that, you should be good to go!

Of couse, this is the GitHub Repository for you to try out for yourself.

DevOps Essentials: Deploy a Jenkins Instance

Comments

DevOps Essentials

More from this blog

How to host a Minecraft instance using Docker and AWS

Hello World

Too Long; Didn't Read...

Why do we want Automation Servers?

Building my own Jenkins Server

Start Simple, and Keep It That Way

Allocate a Virtual Machine

Creation of the VM

Change your Password

Download and Install Docker

Set a Static IP Address for the VM

VM Network Adaptor Shenanigans

Set Static IP addresses in Linux

Launch Jenkins for the First Time

Method #1: Use Docker Logs

Method #2: Bash into the Container

Building up our Jenkins Framework

Enhancement #01: Basic Scripts

Enhancement #02: Dockerfile (i.e. Custom Image)

Enhancement #03: Upping our Security

Enhancement #04: Safe(r) Shutdown Sequence

Execution of Jenkins

Download Jenkins CLI

Graceful Shutdown

Command Palette

Comments

DevOps Essentials

More from this blog

Too Long; Didn't Read...

Why do we want Automation Servers?

Building my own Jenkins Server

Start Simple, and Keep It That Way

Allocate a Virtual Machine

Creation of the VM

Change your Password

Download and Install Docker

Set a Static IP Address for the VM

VM Network Adaptor Shenanigans

Set Static IP addresses in Linux

Launch Jenkins for the First Time

Method #1: Use Docker Logs

Method #2: Bash into the Container

Building up our Jenkins Framework

Enhancement #01: Basic Scripts

Enhancement #02: Dockerfile (i.e. Custom Image)

Enhancement #03: Upping our Security

Enhancement #04: Safe(r) Shutdown Sequence

Execution of Jenkins

Download Jenkins CLI

Graceful Shutdown