Buildpack comes with summon and conjur-api-go pre-installed to speed …

…up retrieval of secrets (#8)

## What does this PR do?

PROBLEM: it's somewhat painful to wait for summon and summon-conjur to install.
SOLUTION: a single binary is a much better proposition. the intention is for each release to carry a conjur-env binary in the vendor directory that can be used with or without the buildpack.
this makes the whole process of secret injection significantly faster i.e. less than a second

The actual buildpack as a tarball can be incorporated as part of the release, it's folder structure looks like so:

```
.
.
├── bin
│   ├── compile
│   ├── decorate
│   └── detect
├── buildpack_vars.sh
├── lib
│   └── 0001_retrieve-secrets.sh
├── upload.sh
└── vendor
    └── conjur-env
```

running `./upload.sh` will upload the buildpack to your cf instance (same flow as before but no need to build)

Interestingly, this also makes it easier for usage with custom buildpacks i.e. download the latest release, and run `./bin/compile <PATH_TO_YOUR_APP>` and you're set :)

## Notes

It's worth inspecting the artefacts over at https://jenkins.conjur.net/job/cyberark--cloudfoundry-conjur-buildpack/job/conjur-env/ to see for yourself.

## Suggestions for improvement

It would be a good idea to add a SHA to each release so users can ensure the binary is legit.

.gitignore

@@ -3,3 +3,5 @@ ci/features/reports/
 buildpack-build/
 .DS_Store
 .idea/
+vendor/
+cloudfoundry_conjur_buildpack/

CHANGELOG.md

@@ -6,6 +6,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+## [1.0.0] - 2018-03-01
+
+### Changed
+Buildpack uses `conjur-env` binary built from the guts of `summon` and `conjur-api-go` instead of installing Summon and Summon-Conjur each time it is invoked.
+
 ## [0.3.0] - 2018-02-13
 
 ### Added
@@ -20,6 +25,7 @@ Added supporting files and documentation for the custom buildpack use case
 
 The first tagged version.
 
-[Unreleased]: https://github.com/cyberark/cloudfoundry-conjur-buildpack/compare/v0.3.0...HEAD
+[Unreleased]: https://github.com/cyberark/cloudfoundry-conjur-buildpack/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/cyberark/cloudfoundry-conjur-buildpack/compare/v0.3.0...v1.0.0
 [0.3.0]: https://github.com/cyberark/cloudfoundry-conjur-buildpack/compare/v0.2.0...v0.3.0
 [0.2.0]: https://github.com/cyberark/cloudfoundry-conjur-buildpack/compare/v0.1.0...v0.2.0

README.md

@@ -1,7 +1,7 @@
 The CyberArk Conjur Buildpack is a [decorator buildpack](https://github.com/cf-platform-eng/meta-buildpack#what-is-a-decorator) that provides convenient and secure access to secrets stored in Conjur.
 
 The buildpack carries out the following:
- 
+
 + Examines your app to determine the secrets to fetch using a [`secrets.yml`](https://cyberark.github.io/summon/#secrets.yml) file in the app root folder.
 + Utilizes credentials stored in your app's [`VCAP_SERVICES`](https://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#VCAP-SERVICES) environment variable to communicate with the bound `cyberark-conjur` service.
 + The buildpack will authenticate using the aforementioned credentials, fetch the relevant secrets and inject them into the session environment variables at the start of the app. This means the secrets are only available to the app process.
@@ -20,16 +20,17 @@ Internally, the Conjur Buildpack uses [Summon](https://cyberark.github.io/summon
 
 ### `meta-buildpack`
 
-`meta-buildpack` makes it possible to "decorate" the application with the ability to retrieve secrets on startup. Please read [`meta-buildpack` documentation](https://github.com/cf-platform-eng/meta-buildpack#how-it-works) for a quick run through of how `meta-buildpack` works. 
+`meta-buildpack` makes it possible to "decorate" the application with the ability to retrieve secrets on startup. Please read [`meta-buildpack` documentation](https://github.com/cf-platform-eng/meta-buildpack#how-it-works) for a quick run through of how `meta-buildpack` works.
 
-`meta-buildpack` relies on automatic detection of the language buildpack. The first language buildpack in buildpack index order to detect and claim the build will be used to build the droplet and run the application. With this in mind, **it is not currently possible to specify custom buildpacks in the manifest or as an argument to `cf push`**. In these instances `meta-buildpack` will not be invoked; consequently the Conjur buildpack will also not be invoked.
+`meta-buildpack` relies on automatic detection of the language buildpack. The first language buildpack in buildpack index order to detect and claim the build will be used to build the droplet and run the application. For `meta-buildpack` to be invoked, it must be at the top
+of the buildpack index and your application must automatically detect the appropriate buildpack on `cf push`. If your application requires you to specify a buildpack on `cf push`, you can follow the [instructions for apps with custom buildpacks](#custom-buildpack-usage) to benefit from the convenience of the Conjur Buildpack without having to rely on the `meta-buildpack` for it to be invoked.
 
 ### Lifecycle scripts
 
 The buildpack is comprised of the [3 lifecycle scripts](https://github.com/cf-platform-eng/meta-buildpack#how-to-write-a-decorator) that are required for decorator buildpacks.
 
 + Detect script always returns a non-zero exit status.
-+ Compile script installs summon, summon-conjur in the app's root directory and a `.profile.d` script at `./.profile.d/0001_retrieve-secrets.sh` relative to the app's root directory.
++ Compile script copies the `conjur-env` binary into your application's `vendor` directory and the `lib/0001_retrieve-secrets.sh` script into your application's `.profile.d` directory.
 + Decorate script returns a 0 when `secrets.yml` exists in the root folder of your app and the "cyberark-conjur" key is present in `VCAP_SERVICES`, otherwise non-zero exit status
 
 The `.profile.d` script is responsible for retrieving secrets and injecting them into the session environment variables at the start of the app.
@@ -50,13 +51,17 @@ cd meta-buildpack
 ./upload
 ```
 
-Install the [Conjur buildpack](https://github.com/conjurinc/cloudfoundry-conjur-buildpack):
+To install the Conjur Buildpack, download a ZIP of a release (at least version 1.x), unzip the release into its own directory, and run the `upload.sh` script:
 ```
-git clone [email protected]:conjurinc/cloudfoundry-conjur-buildpack
-cd cloudfoundry-conjur-buildpack
+mkdir conjur-buildpack
+cd conjur-buildpack/
+curl -L $(curl -s https://api.github.com/repos/cyberark/cloudfoundry-conjur-buildpack/releases/latest | grep browser_download_url | grep zip | awk '{print $NF}' | sed 's/",*//g') > conjur-buildpack.zip
+unzip conjur-buildpack.zip
 ./upload.sh
 ```
 
+Earlier versions of the Conjur Buildpack (v0.x) may be installed by cloning the repository and running `./upload.sh`.
+
 ### Buildpack Usage
 
 #### Create a `secrets.yml` file
@@ -97,8 +102,9 @@ Once you run `cf push`, assuming the app is bound to a Conjur service instance,
 
 To retrieve secrets without using `meta-buildpack` you can simply:
 
-+ copy the contents of the `./lib` directory in this repository to the `./.profile.d` directory relative to your app's root directory.
-
++ copy the contents of the `./lib` directory in a release to the `./.profile.d` directory relative to your app's root directory.
++ copy the contents of the `./vendor` directory in a release to the `./vendor` directory relative to your app's root directory.
+
 When your application starts the secrets specified in the `secrets.yml` file will now be available in the session environment variables at the start of the app.
 
 ## Development
@@ -116,7 +122,20 @@ Before getting started, you should install some developer tools. These are not r
 To test the usage of the Conjur Service Broker within a CF deployment, you can
 follow the demo scripts in the [Cloud Foundry demo repo](https://github.com/conjurinc/cloudfoundry-conjur-demo).
 
-To run the test suite, call `./test.sh` from your local machine - the script will stand up the needed containers and run the full suite of rspec and cucumber tests.
+To prepare to run the test suite on your local machine, call `./package.sh` to build the `summon` and `conjur-api-go` binaries and place them in the `vendor/` directory. Once you have done this, simply call `./test.sh`. The `test.sh` script will stand up the needed containers and run the full suite of rspec and cucumber tests.
+
+When you generate a new release, run `./package.sh` to generate a new release ZIP file and upload the ZIP file to include it with the release.
+
+### Updating the `summon` and `conjur-api-go` binaries
+
+To update the versions of `summon` / `conjur-api-go` that are included in the buildpack, update to the appropriate version number in `conjur-env/Gopkg.toml`. To update `conjur-env/vendor/` and `conjur-env/Gopkg.lock`, start up a `conjur-env` container and run `dep ensure`:
+```
+$ docker-compose -f conjur-env/docker-compose.yml run --entrypoint bash conjur-env-builder
+# dep ensure
+```
+You can verify that the correct dependencies are being used by running `dep status` from within the container before exiting and committing your changes.
+
+Once you have done this, the next time `./package.sh` is run the `vendor/conjur-env` directory will be created with updated dependencies.
 
 ### Contributing
 
@@ -126,6 +145,8 @@ To run the test suite, call `./test.sh` from your local machine - the script wil
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
 
+Make sure your Pull Request includes an update to the [CHANGELOG](https://github.com/cyberark/cloudfoundry-conjur-buildpack/blob/master/CHANGELOG.md) describing your changes.
+
 ## License
 
 Copyright 2018 CyberArk

VERSION

@@ -1 +1 @@
-0.3.0
+1.0.0

bin/compile

@@ -1,14 +1,14 @@
 #!/bin/bash -e
 # bin/compile <build-dir>
 
+echo "[cyberark-conjur-buildpack]: compiling"
+
 BUILD_DIR=$1
-ESCAPED_BUILD_DIR=$(echo $BUILD_DIR | sed -e 's/[\/&]/\\&/g')
 BIN_DIR=$(cd $(dirname $0); pwd)
-ROOT_DIR=$(dirname $BIN_DIR)
-
-$ROOT_DIR/lib/0000_install-summon.sh $BUILD_DIR
+BUILDPACK_DIR=$(dirname $BIN_DIR)
 
-pushd $BUILD_DIR
-  mkdir -p ".profile.d"
-  cp $ROOT_DIR/lib/0001_retrieve-secrets.sh ./.profile.d/0001_retrieve-secrets.sh
+pushd ${BUILD_DIR}
+  mkdir -p .profile.d vendor
+  cp ${BUILDPACK_DIR}/vendor/conjur-env ./vendor/conjur-env
+  cp ${BUILDPACK_DIR}/lib/0001_retrieve-secrets.sh ./.profile.d/0001_retrieve-secrets.sh
 popd

bin/decorate

@@ -8,7 +8,9 @@
 #      "appliance_url": "https://conjur.myorg.com/",
 #      "authn_api_key": "2389fh3hf9283niiejwfhjsb83ydbn23u",
 #      "authn_login": "3F20D12E-A470-4B7B-8778-C8885769887F",
-#      "account": "brokered-services"
+#      "account": "brokered-services",
+#      "ssl_certificate": "-----BEGIN CERTIFICATE----...",
+#      "version": 4
 #    }
 #  }]
 #}

buildpack_vars.sh

@@ -1,6 +1,8 @@
 #!/bin/bash -e
 
+version=$(cat VERSION)
+
 SRC_DIR=$(cd "$(dirname $0)/."  && pwd)
 NAME=$(basename "$SRC_DIR" | sed s/-/_/g)
 TGT_DIR=${SRC_DIR}
-ZIP_FILE="$TGT_DIR/$NAME.zip"
+ZIP_FILE="$TGT_DIR/$NAME-$version.zip"

ci/Dockerfile

@@ -1,16 +1,9 @@
-FROM alpine:3.2
+FROM cloudfoundry/cflinuxfs2
 
-ENV BUILD_PACKAGES bash curl-dev ruby-dev build-base
-ENV RUBY_PACKAGES ruby ruby-io-console ruby-bundler jq zip
+ENV RUBY_PACKAGES bundler
 
 # Update and install all of the required packages.
-# At the end, remove the apk cache
-RUN apk update && \
-    apk upgrade && \
-    apk add $BUILD_PACKAGES && \
-    apk add $RUBY_PACKAGES && \
-    rm -rf /var/cache/apk/*
-
+RUN gem install $RUBY_PACKAGES
 
 ENV WD_PATH /conjurinc/cloudfoundry-conjur-buildpack
 ENV CI_PATH $WD_PATH/ci
@@ -19,5 +12,3 @@ WORKDIR $WD_PATH
 
 COPY Gemfile $CI_PATH
 RUN cd $CI_PATH && bundle install
-
-RUN mkdir /lib64/; ln -s /lib/ld-linux-x86-64.so.2 /lib64

ci/features/buildpackless.feature

@@ -3,7 +3,8 @@ Feature: profile d scripts without the buildpack
 
   @BUILD_DIR
   Scenario: Populates environment with secrets from Conjur
-    Given the .profile.d scripts are copied into the lib directory of the app's root folder
+    Given the compile script is run against the app's root folder
+    And conjur-env is installed
     And a root policy:
     """
     - !variable conjur_single_line_secret_id

ci/features/compile.feature

@@ -1,10 +1,9 @@
 Feature: Compile script
-  Compile script installs summon, summon-conjur and .profile.d script
+  Compile script installs conjur-env and .profile.d script
 
   @BUILD_DIR
-  Scenario: Successfully installs summon, summon-conjur and .profile.d scripts
+  Scenario: Successfully installs conjur-env and .profile.d scripts
     When the 'compile' script is run
     Then the result should have a 0 exit status
-    And summon is installed
-    And summon-conjur is installed
+    And conjur-env is installed
     And the retrieve secrets .profile.d script is installed

ci/features/profile_d.feature

@@ -4,6 +4,7 @@ Feature: profile d script
   @BUILD_DIR
   Scenario: Populates environment with secrets from Conjur
     Given the 'compile' script is run
+    And conjur-env is installed
     And a root policy:
     """
     - !variable conjur_single_line_secret_id

ci/features/step_definitions/common_steps.rb

@@ -26,16 +26,12 @@
   step "the '#{ENV['BUILDPACK_BUILD_DIR']}/bin/#{script} #{@BUILD_DIR}' command is run"
 end
 
-Given(/^the \.profile\.d scripts are copied into the lib directory of the app's root folder$/) do
+Given(/^the compile script is run against the app's root folder$/) do
   step "the following command is run:", <<EOS
-cd #{@BUILD_DIR}
-mkdir -p .profile.d
-cp #{ENV['BUILDPACK_BUILD_DIR']}/lib/0000_install-summon.sh ./.profile.d/0000_install-summon.sh
-cp #{ENV['BUILDPACK_BUILD_DIR']}/lib/0001_retrieve-secrets.sh ./.profile.d/0001_retrieve-secrets.sh
+#{ENV['BUILDPACK_BUILD_DIR']}/bin/compile #{@BUILD_DIR}
 EOS
 end
 
-
 Given(/^the following command is run:$/) do |multiline_text|
   step "the '#{multiline_text}' command is run"
 end
@@ -64,7 +60,7 @@
    "authn_login": "admin",
    "account": "#{Conjur.configuration.account}",
    "ssl_certificate": "",
-   "version": "5"
+   "version": 5
   }
  }]
 }

ci/features/step_definitions/compile_steps.rb

@@ -1,10 +1,5 @@
-Then(/^summon is installed$/) do
-  `#{@BIN_DIR}/summon -v`
-  expect($?.exitstatus).to eq (0)
-end
-
-Then(/^summon-conjur is installed$/) do
-  `#{@BIN_DIR}/summon-conjur -v`
+Then(/^conjur-env is installed$/) do
+  `ls #{@VENDOR_DIR}/conjur-env`
   expect($?.exitstatus).to eq (0)
 end
 

ci/features/step_definitions/profile_d_steps.rb

@@ -12,7 +12,6 @@
 When(/^the \.profile\.d scripts are sourced$/) do
   @commands ||= []
   @commands << <<EOL
-. #{@BUILD_DIR}/.profile.d/0000_install-summon.sh #{@BUILD_DIR}
 . #{@BUILD_DIR}/.profile.d/0001_retrieve-secrets.sh #{@BUILD_DIR}
 EOL
 end

ci/features/support/env.rb

@@ -13,7 +13,7 @@
 
 Before('@BUILD_DIR') do
   @BUILD_DIR = Dir.mktmpdir
-  @BIN_DIR = "#{@BUILD_DIR}/bin"
+  @VENDOR_DIR = "#{@BUILD_DIR}/vendor"
 end
 
 After('@BUILD_DIR') do

conjur-env/.gitignore

@@ -0,0 +1,2 @@
+src
+conjur-env

conjur-env/Dockerfile

@@ -0,0 +1,10 @@
+FROM golang:1.8
+MAINTAINER Conjur Inc.
+
+RUN go get -u github.com/golang/dep/cmd/dep
+
+RUN mkdir -p /go/src/github.com/cyberark/conjur-env
+WORKDIR /go/src/github.com/cyberark/conjur-env
+
+COPY . .
+RUN dep ensure

conjur-env/Gopkg.toml

@@ -0,0 +1,43 @@
+# Gopkg.toml example
+#
+# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
+# for detailed Gopkg.toml documentation.
+#
+# required = ["github.com/user/thing/cmd/thing"]
+# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
+#
+# [[constraint]]
+#   name = "github.com/user/project"
+#   version = "1.0.0"
+#
+# [[constraint]]
+#   name = "github.com/user/project2"
+#   branch = "dev"
+#   source = "github.com/myfork/project2"
+#
+# [[override]]
+#   name = "github.com/x/y"
+#   version = "2.4.0"
+#
+# [prune]
+#   non-go = false
+#   go-tests = true
+#   unused-packages = true
+
+
+[[constraint]]
+  name = "github.com/cyberark/conjur-api-go"
+  version = "0.3.0"
+
+[[constraint]]
+  name = "github.com/cyberark/summon"
+  version = "0.6.6"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/mitchellh/go-homedir"
+
+[prune]
+  go-tests = true
+  unused-packages = true
+  non-go = true