Skip to content

Commit

Permalink
Merge pull request #20 from arminc/refactoring
Browse files Browse the repository at this point in the history
Refactoring
  • Loading branch information
arminc authored Sep 24, 2017
2 parents e70d960 + 634138e commit d894f52
Show file tree
Hide file tree
Showing 12 changed files with 520 additions and 374 deletions.
8 changes: 7 additions & 1 deletion Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 8 additions & 22 deletions Gopkg.toml
Original file line number Diff line number Diff line change
@@ -1,25 +1,4 @@

# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"

# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md for detailed Gopkg.toml documentation.

[[constraint]]
name = "github.com/coreos/clair"
Expand All @@ -31,3 +10,10 @@

[[constraint]]
name = "gopkg.in/yaml.v2"

[[constraint]]
name = "github.com/jawher/mow.cli"
version = "v1.0.1"

[[constraint]]
name = "github.com/mbndr/logo"
71 changes: 56 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,39 +6,39 @@

## Docker containers vulnerability scan

When you work with containers (Docker) you are not only packaging your application but also part of the OS. Therefore it is crucial to know what kind of libraries might be vulnerable in you container. One way to find this information is to use and look at the Docker Hub or Quay.io security scan. The problem whit these scans is that they are only showing you the information but are not part of your CI/CD that actually blocks your container when it contains vulnerabilities.
When you work with containers (Docker) you are not only packaging your application but also part of the OS. It is crucial to know what kind of libraries might be vulnerable in your container. One way to find this information is to look at the Docker registry [Hub or Quay.io] security scan. This means your vulnerable image is already on the Docker registry.

What you want is:
What you want is a scan as a part of CI/CD pipeline that stops the Docker image push on vulnerabilities:

1. Build and test your application
1. Build the container
1. Test the container for vulnerabilities
1. Check the vulnerabilities against allowed ones, if everything is allowed pass, otherwise fail
1. Check the vulnerabilities against allowed ones, if everything is allowed then pass otherwise fail

This straight forward process is not that easy to achieve when using the services like Docker Hub or Quay.io. This is because they work asynchronously which makes it harder to do straight forward CI/CD pipeline.
This straightforward process is not that easy to achieve when using the services like Docker Hub or Quay.io. This is because they work asynchronously which makes it harder to do straightforward CI/CD pipeline.

## Clair to the rescue

CoreOS has created an awesome container scan tool called "clair". Clair is also used by Quay.io. What clair does not have is a simple tool that scans your image and compares the vulnerabilities against a whitelist to see if they are approved or not.
CoreOS has created an awesome container scan tool called Clair. Clair is also used by Quay.io. What clair does not have is a simple tool that scans your image and compares the vulnerabilities against a whitelist to see if they are approved or not.

This is where clair-scanner comes in to place. The clair-scanner does the following:
This is where clair-scanner comes into place. The clair-scanner does the following:

* Scans an image against Clair server
* Compares the vulnerabilities against a whitelist
* Tells you if there are vurnabilities that are not in the whitelist and fails
* Tells you if there are vulnerabilities that are not in the whitelist and fails
* If everything is fine it completes correctly

## Clair server or standalone

For the clair-scanner to work you need a clair server. It is not always convenient to have a dedicated clair server therefore I have created a way to run this standalone. See here <https://github.com/arminc/clair-local-scan>
For the clair-scanner to work, you need a clair server. It is not always convenient to have a dedicated clair server, therefore, I have created a way to run this standalone. See here <https://github.com/arminc/clair-local-scan>

## Credits

The clair-scanner is a copy of the Clair 'analyze-local-images' <https://github.com/coreos/analyze-local-images> with changes/improvments and addition that checks the vulnerabilities against a whitelist.
The clair-scanner is a copy of the Clair 'analyze-local-images' <https://github.com/coreos/analyze-local-images> with changes/improvements and addition that checks the vulnerabilities against a whitelist.

## Build

clair-scanner is build with Go 1.9 and uses `dep` as dependencies manager. Use the Makefile to build and install dependencies.
clair-scanner is built with Go 1.9 and uses `dep` as dependencies manager. Use the Makefile to build and install dependencies.

```bash
make ensure && make build
Expand All @@ -59,16 +59,53 @@ docker run -p 5432:5432 -d --name db arminc/clair-db:2017-09-18
docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan:v2.0.1
```

Now scan a container, that has a whitelisted CVE:
Now scan a container, that has a whitelisted CVE (this is on OSX with Docker for Mac):

```bash
clair-scanner nginx:1.11.6-alpine example-nginx.yaml http://YOUR_LOCAL_IP:6060 YOUR_LOCAL_IP
clair-scanner -w example-alpine.yaml --ip YOUR_LOCAL_IP alpine:3.5
```

Or a container that does not have a whitelisted CVE:
Output:

```bash
clair-scanner nginx:1.11.6-alpine example-whitelist.yaml http://YOUR_LOCAL_IP:6060 YOUR_LOCAL_IP
2017/09/24 11:20:24 [INFO] ▶ Start clair-scanner
2017/09/24 11:20:24 [INFO] ▶ Server listening on port 9279
2017/09/24 11:20:24 [INFO] ▶ Analyzing 693bdf455e7bf0952f8a4539f9f96aa70c489ca239a7dbed0afb481c87cbe131
2017/09/24 11:20:24 [INFO] ▶ Image [alpine:3.5] not vulnerable
```

Or a container that does not have a whitelisted CVE (this is on OSX with Docker for Mac):

```bash
clair-scanner --ip YOUR_LOCAL_IP alpine:3.5
```

Output:

```bash
2017/09/24 11:16:41 [INFO] ▶ Start clair-scanner
2017/09/24 11:16:41 [INFO] ▶ Server listening on port 9279
2017/09/24 11:16:41 [INFO] ▶ Analyzing 693bdf455e7bf0952f8a4539f9f96aa70c489ca239a7dbed0afb481c87cbe131
2017/09/24 11:16:41 [CRIT] ▶ Image contains unapproved vulnerabilities: [CVE-2016-9840 CVE-2016-9841 CVE-2016-9842 CVE-2016-9843]
```

## Help information

```bash
$ ./clair-scanner -h

Usage: clair-scanner [OPTIONS] IMAGE

Scan local Docker images for vulnerabilities with Clair

Arguments:
IMAGE="" Name of the Docker image to scan

Options:
-w, --whitelist="" Path to the whitelist file
-c, --clair="http://127.0.0.1:6060" Clair url
--ip="localhost" IP addres where clair-scanner is running on
-l, --log="" Log to a file
```

## Example whitelist yaml file
Expand All @@ -85,4 +122,8 @@ images:
CVE-2017-5230: XSX
alpine:
CVE-2017-3261: SE
```
```
## Release
To make a release create a tag and push it
57 changes: 57 additions & 0 deletions clair.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package main

import (
"bytes"
"encoding/json"
"io/ioutil"
"net/http"

"github.com/coreos/clair/api/v1"
)

func analyzeLayers(layerIds []string, clairURL string, scannerIP string) {
tmpPath := "http://" + scannerIP + ":" + httpPort

for i := 0; i < len(layerIds); i++ {
Logger.Infof("Analyzing %s", layerIds[i])

if i > 0 {
analyzeLayer(clairURL, tmpPath+"/"+layerIds[i]+"/layer.tar", layerIds[i], layerIds[i-1])
} else {
analyzeLayer(clairURL, tmpPath+"/"+layerIds[i]+"/layer.tar", layerIds[i], "")
}
}
}

func analyzeLayer(clairURL, path, layerName, parentLayerName string) {
payload := v1.LayerEnvelope{
Layer: &v1.Layer{
Name: layerName,
Path: path,
ParentName: parentLayerName,
Format: "Docker",
},
}
jsonPayload, err := json.Marshal(payload)
if err != nil {
Logger.Fatalf("Could not analyze layer, payload is not json %s", err)
}

request, err := http.NewRequest("POST", clairURL+postLayerURI, bytes.NewBuffer(jsonPayload))
if err != nil {
Logger.Fatalf("Could not analyze layer, could not prepare request for Clair %s", err)
}

request.Header.Set("Content-Type", "application/json")
client := &http.Client{}
response, err := client.Do(request)
if err != nil {
Logger.Fatalf("Could not analyze layer, POST to Clair failed %s", err)
}
defer response.Body.Close()

if response.StatusCode != 201 {
body, _ := ioutil.ReadAll(response.Body)
Logger.Fatalf("Could not analyze layer, Clair responded with a failure: Got response %d with message %s", response.StatusCode, string(body))
}
}
75 changes: 75 additions & 0 deletions docker.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
package main

import (
"context"
"encoding/json"
"io"
"os"
"strings"

"github.com/docker/docker/client"
)

// TODO Add support for older version of docker

type manifestJson struct {
Layers []string
}

// saveDockerImage saves Docker image to temorary folder
func saveDockerImage(imageName string, tmpPath string) {
docker := createDockerClient()

imageReader, err := docker.ImageSave(context.Background(), []string{imageName})
if err != nil {
Logger.Fatalf("Could not save Docker image [%v] : %v", imageName, err)
}

defer imageReader.Close()

if err = untar(imageReader, tmpPath); err != nil {
Logger.Fatalf("Could not save Docker image, could not untar [%v] : %v", imageName, err)
}
}

func createDockerClient() client.APIClient {
docker, err := client.NewEnvClient()
if err != nil {
Logger.Fatalf("Could not create a Docker client: %v", err)
}
return docker
}

// TODO make a test
func getImageLayerIds(path string) []string {
manifest := readManifestFile(path)

var layers []string
for _, layer := range manifest[0].Layers {
layers = append(layers, strings.TrimSuffix(layer, "/layer.tar"))
}
return layers
}

func readManifestFile(path string) []manifestJson {
manifestFile := path + "/manifest.json"
mf, err := os.Open(manifestFile)
if err != nil {
Logger.Fatalf("Could not read Docker image layers, could not open [%v]: %v", manifestFile, err)
}
defer mf.Close()

return parseAndValidateManifestFile(mf)
}

func parseAndValidateManifestFile(manifestFile io.Reader) []manifestJson {
var manifest []manifestJson
if err := json.NewDecoder(manifestFile).Decode(&manifest); err != nil {
Logger.Fatalf("Could not read Docker image layers, manifest.json is not json: %v", err)
} else if len(manifest) != 1 {
Logger.Fatalf("Could not read Docker image layers, manifest.json is not valid")
} else if len(manifest[0].Layers) == 0 {
Logger.Fatalf("Could not read Docker image layers, no layers can be found")
}
return manifest
}
6 changes: 6 additions & 0 deletions example-alpine.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
images:
alpine:
CVE-2016-9840: zlib
CVE-2016-9841: zlib
CVE-2016-9842: zlib
CVE-2016-9843: zlib
3 changes: 0 additions & 3 deletions example-nginx.yaml

This file was deleted.

Loading

0 comments on commit d894f52

Please sign in to comment.