Merge pull request #31 from Trioxis/feature/determine-backup-actions-#27

Determine backup actions (fixes #27)

.travis.yml

@@ -5,7 +5,7 @@ node_js:
 script:
   - npm test
   - npm run cover
-after_success:
+after_script:
   - npm run _send_to_coveralls
   - npm run _send_to_codeclimate
   - npm run _send_to_codecov

README.md

@@ -65,7 +65,7 @@ You can set tests to run automatically with
 ```
 npm run watch
 ```
-Your terminal will display updated test status as soon as save your code.
+Your terminal will display updated test status as soon as save your code. (This doesn't do linting, you can use `npm run watch-lint` for that).
 
 #### Code Coverage
 

docs/API.md

@@ -1,11 +1,90 @@
 # Module APIs
 
+## Object Specifications
+
+These objects are a more useful (to us) representation of EC2 objects and are passed around the Modules below. The values are mapped from a combination of the [EC2 Response](http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html) and the 'Name' and [`backups:config-v0` tags](./BackupTagAPI.md) on the object.
+
+### AWSBM Snapshot Object
+
+```JavaScript
+{
+	Name,
+	// The value of the 'Name' EC2 tag
+	SnapshotId,
+	// The EC2 resource id of the snapshot, e.g. 'snap-1234abcd'
+	FromVolumeId,
+	// The EC2 resource id of the volume that this snapshot was made from, e.g. 'vol-1234abcd'
+	FromVolumeName,
+	// The value of the 'Name' EC2 tag of the volume at creation time (it is possible this could have changed since creation)
+	StartTime,
+	// A moment.js object representing the time the snapshot was created
+	ExpiryDate,
+	// A moment.js object representing the time after which this snapshot has expired. Can be
+	// 'undefined' if no ExpiryDate was specified in the `backups:config-v0` tag
+	BackupType,
+	// The name of the backup type this snapshot is (one of the values from the volume's BackupTypes)
+	Tags: {	}
+	// The EC2 tags on the snapshot, where the property names are the 'Key' and the values are their 'Value'
+}
+```
+
+### AWSBM Volume Object
+
+```JavaScript
+{
+	VolumeId,
+	// The EC2 resource id of the volume, e.g. 'vol-1234abcd'
+	Name,
+	// The value of the 'Name' EC2 tag. If one doesn't exist, it is set to the VolumeId
+	BackupConfig: {
+		BackupTypes: [
+			// An object for every item in the 'backups:config-v0' tag of the volume
+			{
+				Name,
+				// The Alias of the backup type if it exists, otherwise '[Frequency|Expiry]'
+				Frequency,
+				// How often this backup should occur, in hours
+				Expiry
+				// How long snapshots of this time should exist, in hours
+			}
+		]
+	},
+	Tags: { },
+	// The EC2 tags on the volume, where the property name are the 'Key' and the values are their 'Value'
+	Snapshots: { }
+	// [OPTIONAL] Each property name of this object is the name of a 'BackupType'. The property contains
+	// an array of all the snapshots with the same BackupType and that have the same 'FromVolumeName' as this
+	// volume's 'Name'.
+}
+```
+
+### AWSBM Action Objects
+
+#### Make Snapshot Action
+
+```JavaScript
+{
+	Action: 'SNAPSHOT_VOLUME',
+	VolumeId,
+	// EC2 resource id of the volume being backed up
+	VolumeName,
+	// Value of the EC2 Tag with Key 'Name'
+	BackupType,
+	// The type of backup this snapshot will represent
+	ExpiryDate
+	// A moment.js object that represents the point in time that this snapshot expires.
+	// It should be the current time + the number of hours defined in Expiry for the BackupType
+}
+```
+
+## Modules
+
 ## ActionCreator
 
 [`ActionCreator.js`](../src/ActionCreator.js) exports three named functions. They all return objects that represent an action to be used by the `Actioner`
 
 - `makeDeleteAction(snap)` - accepts an object that represents a snapshot in EC2. It returns an object that represents an action that will delete the snapshot.
-- `makeCreationActions(volume, snapList)` - accepts an object that represents an EBS volume and an array of snapshot objects that exist in EC2. Returns an array of actions that will create the required backup snapshots for the EBS volume.
+- `makeCreationActions(volume)` - accepts an AWSBM Volume object (with `Snapshots` defined). Returns an array of `SNAPSHOT_VOLUME` AWSBM Action objects that will create the required backup snapshots for the EBS volume.
 - `determineBackupsNeeded(volume, snapList)` - accepts an EBS volume object and an array of snapshot objects. This simply determines that types of snapshots required and returns them as an array. This should be used by `makeCreateAction` to figure out what creation actions it needs to make.
 
 ## Actioner
@@ -23,40 +102,18 @@ let ec2 = new EC2Store(params);
 ```
 where `params` is an object that configures how the class will contact EC2 (things like availability zone, user account id and/or credentials). Now that we have an `EC2Store` instance called `ec2`, we can use it to get information from EC2. These functions should only return EC2 objects that have a `backups:config-v0` tag and should be mapped to a more useful format (which is described in [tests](../test/_TestEC2Store.js)).
 
-`ec2.listSnapshots` - returns a Promised object of the form `{snapshots, warnings}`. `snapshots` is an array of all snapshots in EC2 with a tag named `backups:config-v0`. The array contains snapshot objects of the form
-```
-{ SnapshotId, Name, StartTime, ExpiryDate, Tags }
-```
-* `SnapshotId` is the AWS EC2 resource id
-* `Name` is the value of the EC2 tag with the key `Name`. If the EC2 tag does not exist, the `SnapshotId` is used instead
-* `StartTime` is the time that the snapshot began being created, in the time format that the AWS API provides
-* `ExpiryDate` is the date and time after which the snapshot should be deleted. It is in `YYYYMMDDHHmmss` format and should be the same as the value given in the `backups:config-v0` parameter as defined in the [tag API doc](./BackupTagAPI.md). If `ExpiryDate` is not specified in `backups:config-v0`, this will be `undefined`.
-* `Tags` is an object mapped from the EC2 tags on the volume. Properties on the `Tags` object are named according to the tag `key` and have the tag value
+`ec2.listSnapshots` - returns a Promised object of the form `{snapshots, warnings}`. `snapshots` is an array of all snapshots in EC2 with a tag named `backups:config-v0`, represented as AWSBM Snapshot objects.
 
 
-`ec2.listEBS` - returns a Promised object of the form `{volumes, warnings}`. `volumes` is a array of all the EBS volumes in EC2 that have a tag named `backups:config-v0`. The array contains volume objects of the form
-```
-{ VolumeId, Name, BackupConfig, Tags }
-```
-* `VolumeId` is the AWS EC2 resource id
-* `Name` is the value of the EC2 tag with key `Name`. If the EC2 tag does not exist, the `VolumeId` is used instead
-* `BackupConfig` is an object with property `BackupTypes`
-    * `BackupTypes` is an array of objects mapped from the `backups:config-v0` tag. They have the form `{Frequency: x, Expiry: y, Alias: 'AliasName'}` (see the [Backup Tag API](./BackupTagAPI.md) for details). The `Alias` property is optional.
-* `Tags` is an object mapped from the EC2 tags on the volume. Properties on the `Tags` object are named according to the tag `key` and have the tag value
+`ec2.listEBS` - returns a Promised object of the form `{volumes, warnings}`. `volumes` is a array of all the EBS volumes in EC2 that have a tag named `backups:config-v0`, represented as AWSBM Volume objects (without `Snapshots` defined).
 
 The `warnings` property of the above objects is an array of strings that describe problems that occurred while parsing EC2 objects.
 
-## SnapshotAnalyser
+## Analyser
 
-[`SnapshotAnalyser.js`](../src/SnapshotAnalyser.js) exports two named functions that examine whether or not snapshots should continue to exist.
+[`Analyser.js`](../src/Analyser.js) exports two named functions that examine whether or not snapshots should continue to exist.
 
+- `matchSnapsToVolumes(volumes, snapList)` - given an array of AWSBM Volumes (with no `Snapshots` property defined) and an array of AWSBM Snapshots, returns `{matchedVolumes, orphanedSnaps}` where `matchedSnapshots` is the array of AWSBM Volumes with `Snapshots` defined and `orphanedSnapshots` are all the AWSBM Snapshots that had no `FromVolumeName` matching `Name` value in the AWSBM Volumes.
+- `sortSnapsByMostRecent(snapList)` - returns an array of snapshots sorted by most recently created first.
 - `findDeadSnapshots(snapshotList)` - accepts an array of snapshot objects and returns an array of snapshots that have expired.
 - `snapshotIsDead(snapshot)` - given a snapshot object, returns true if the snapshot has expired. It determines this based on tags that represent things like expiry dates. If the snapshot is still valid, the function returns false.
-
-# Logging and Metrics
-
-There is a metric manager available at `./metrics`. By giving this manager an object that exposes `log` and/or `pushMetric` functions, you can log to multiple locations easily throughout the code.
-
-## Metric Manager
-
-- `log(message)` - Currently logs to console.

docs/BackupTagAPI.md

@@ -10,12 +10,14 @@ This document refers to **`v0`** of the API. So the tag key should be **`backups
 
 ## Tag API for EBS volumes
 
+**Each volume in an account must have a unique `Name` without commas in it. Otherwise there may be problems determining which snapshots belong to which volume**
+
 The value of the `backups:config-v0` tag defines how often the EBS volume should be backed up and how long these backups are to be retained. It is in the format of a comma delimited list of tuples and aliases.
 
 A tuple takes the form **`[x|y]`**
 
-* **`x`** - backup **frequency** - is an integer denoting the number of hours between successive backups
-* **`y`** - backup **expiry** - is an integer denoting the time to live (TTL) for the backup in hours
+* **`x`** - backup **Frequency** - is an integer denoting the number of hours between successive backups
+* **`y`** - backup **Expiry** - is an integer denoting the time to live (TTL) for the backup in hours
 
 For example, the tuple `[1,12]` means the EBS should be backed up once an hour and these backups should be kept for twelve hours before they are deleted.
 
@@ -42,11 +44,15 @@ You would use the following value for `backups:config-v0`
 
 ## Tag API for Snapshots
 
-The value of the `backups:config-v0` tag defines conditions that must be met for the snapshot to be deleted. Currently the only condition is the expiry date of the snapshot. The value is in the form of a comma delimited list of conditions.
+The value of the `backups:config-v0` tag defines necessary metadata key-value pairs. For example, conditions that must be met for the snapshot to be deleted, or what volume the snapshot belongs to. The tag value is in the form of a comma delimited list of key-value pairs. The key and value are separated by a `:`.
+
+The **`ExpiryDate`** value contains the date after which a snapshot should be deleted. The date value is UTC and in the format `YYYYMMDDHHmmss` (these are the [same tokens as used in moment.js](http://momentjs.com/docs/#/parsing/string-format/)).
+
+The **`FromVolumeName`** value describes the Name tag of the volume that the snapshot was made from. It is used when determining what backup types are necessary.
 
-The **`ExpiryDate`** condition the date after which a snapshot should be deleted. The date value is in the format `YYYYMMDDHHmmss` (these are the [same tokens as used in moment.js](http://momentjs.com/docs/#/parsing/string-format/)).
+The **`BackupType`** value is defined from one of the values of the `backups:config-v0` tag of the volume from which the snapshot was made. It is either the Alias or `[Frequency|Expiry]`.
 
-To delete a snapshot after 11:25:13 PM on the 5th of June 2015, you would use this value for `backups:config-v0`
+For a snapshot that came from a backup type `[4|12]` of the volume with Name `website-data` and is set to be deleted at 11:25:13 PM on the 5th of June 2015, you would use this value for `backups:config-v0`
 ```
-ExpiryDate:20150605232513
+ExpiryDate:20150605232513,FromVolumeName:website-data,BackupType:[4|12]
 ```

package.json

@@ -1,6 +1,6 @@
 {
   "name": "aws-backup-manager",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "Automatically back up EBS volumes using tags",
   "main": "index.js",
   "directories": {
@@ -9,6 +9,7 @@
   "scripts": {
     "test": "npm run _lint && npm run _mocha",
     "watch": "npm run _mocha -- --watch --reporter min",
+    "watch-lint": "node ./node_modules/.bin/esw -w index.js src/**",
     "cover": "babel-node node_modules/isparta/bin/isparta cover --report text --report html --report lcov node_modules/mocha/bin/_mocha -- --reporter dot --recursive",
     "_lint": "node ./node_modules/eslint/bin/eslint.js index.js src/**",
     "_mocha": "node ./node_modules/mocha/bin/mocha --compilers js:babel-register --recursive",
@@ -42,6 +43,7 @@
     "codecov": "^1.0.1",
     "coveralls": "^2.11.6",
     "eslint": "^1.10.3",
+    "eslint-watch": "^2.1.7",
     "expect.js": "^0.3.1",
     "isparta": "^4.0.0",
     "mocha": "^2.3.4",

src/ActionCreator.js

@@ -1,3 +1,5 @@
+import moment from 'moment';
+
 // Given a snapshot object, make an action that will delete the snapshots
 // TODO: What format are actions in?
 let makeDeleteAction = (snap) => {
@@ -6,9 +8,26 @@ let makeDeleteAction = (snap) => {
 
 // Given an EBS volume and a list of snapshots, return a list of actions
 // to create the necessary snapshots that fulful the backup requirements
-let makeCreationActions = (volume, snapList) => {
+let makeCreationActions = (volume) => {
+	let backupTypes = volume.BackupConfig.BackupTypes.filter(type => {
+		let now = moment();
+		if (!volume.Snapshots[type.Name] ||
+				volume.Snapshots[type.Name].length <= 0 ||
+				now.diff(volume.Snapshots[type.Name][0].StartTime, 'hours', true) > type.Frequency
+		) {
+			return true;
+		} else {
+			return false;
+		}
+	});
 
-	return determineBackupsNeeded(volume, snapList).map(backup => backup);
+	return backupTypes.map((backup) => ({
+		Action: 'SNAPSHOT_VOLUME',
+		VolumeId: volume.VolumeId,
+		VolumeName: volume.Name,
+		BackupType: backup.Name,
+		ExpiryDate: moment().add(backup.Expiry, 'hours')
+	}));
 };
 
 // Read the tags on the EBS volume and check if the backup requirements are

src/Analyser.js

@@ -0,0 +1,50 @@
+// Given a list of snapshots, returns a list of snapshots that have expired
+// This can be determined using ExpiryDate tags
+let findDeadSnapshots = (snapshotList) => {
+	return [snapshotList];
+};
+
+// Helper function that checks if a single snapshot has expired or not.
+// Returns true if the snapshot has expired and needs to be deleted.
+let snapshotIsDead = (snapshot) => {
+	return snapshot;
+};
+
+// Given a list of EBS volumes and list of snapshots, matches each snapshot to the
+// EBS's Snapshots[BackupType] array. Returns an object containing the volume list
+// and a list of orphaned snapshots
+// The snapshot arrays are sorted by latest first
+let matchSnapsToVolumes = (volumes, snapList) => {
+	snapList = sortSnapsByMostRecent(snapList);
+	let matchedVolumes = volumes.map((volume) => {
+		volume.Snapshots = {};
+		snapList = snapList.filter((snap) => {
+			if (snap.FromVolumeName === volume.Name) {
+				if (!volume.Snapshots[snap.BackupType]) volume.Snapshots[snap.BackupType] = [];
+				volume.Snapshots[snap.BackupType].push(snap);
+				return false;
+			} else {
+				return true;
+			}
+		});
+		return volume;
+	});
+	let orphanedSnaps = snapList;
+	return {matchedVolumes, orphanedSnaps};
+};
+
+// Sorts a list of snapshots by most recently created first
+let sortSnapsByMostRecent = (snapList) => {
+	// sort in to latest first
+	return snapList.sort((a, b) => {
+		if (a.StartTime.isAfter(b.StartTime)) {
+			return -1;
+		} else if (a.StartTime.isSame(b.StartTime)) {
+			return 0;
+		} else {
+			return 1;
+		}
+	});
+};
+
+export {findDeadSnapshots, snapshotIsDead, matchSnapsToVolumes, sortSnapsByMostRecent};

src/EC2Store.js

@@ -53,13 +53,20 @@ class EC2Store {
 						// Use snapshot id if a Name tag does not exist
 						snap.Name = snapResponse.SnapshotId;
 						snap.SnapshotId = snapResponse.SnapshotId;
-						snap.StartTime = snapResponse.StartTime;
+						snap.FromVolumeId = snapResponse.VolumeId;
+						// NOTE: Ingesting date value with it's time zone (ZZ)
+						snap.StartTime = moment(snapResponse.StartTime, 'ddd MMM DD YYYY HH:mm:ss ZZ');
+						snap.FromVolumeName = undefined;
+						snap.BackupType = undefined;
+						snap.ExpiryDate = undefined;
 
 						// Map EC2 tags to easy to use Tag object
 						snap.Tags = {};
 						snapResponse.Tags.map(tag => {
 							snap.Tags[tag.Key] = tag.Value;
-							if (tag.Key === 'Name') snap.Name = tag.Value;
+							if (tag.Key === 'Name') {
+								snap[tag.Key] = tag.Value;
+							}
 						});
 
 						return snap;
@@ -74,17 +81,19 @@ class EC2Store {
 							let [key, value] = backupParam.split(':');
 
 							// Check the expiry date is in YYYYMMDDHHmmss format (14 digits)
-							snap.ExpiryDate = undefined;
 							if (key === 'ExpiryDate') {
 								if (new RegExp(`^\\d{${EXPIRY_DATE_FORMAT.length}}$`).test(value)) {
-									if (moment(value, EXPIRY_DATE_FORMAT).isValid()) {
-										snap.ExpiryDate = parseInt(value);
+									// NOTE: Ingest date as a UTC date using moment.utc()
+									if (moment.utc(value, EXPIRY_DATE_FORMAT).isValid()) {
+										snap.ExpiryDate = moment.utc(value, EXPIRY_DATE_FORMAT).local();
 									} else {
 										warnings.push(`Snapshot ${prettyPrintSnap(snap)}: ExpiryDate set to undefined because the parsed value '${value}' is not a valid date in ${EXPIRY_DATE_FORMAT} format. Check the ExpiryDate in '${BACKUP_API_TAG}' is valid`);
 									}
 								} else {
 									warnings.push(`Snapshot ${prettyPrintSnap(snap)}: ExpiryDate set to undefined beacuse the parsed value '${value}' is invalid. Check the '${BACKUP_API_TAG}' tag is valid and ExpiryDate is in ${EXPIRY_DATE_FORMAT} format`);
 								}
+							} else if (key === 'FromVolumeName' || key === 'BackupType') {
+								snap[key] = value;
 							} else {
 								warnings.push(`Snapshot ${prettyPrintSnap(snap)}: Unknown '${BACKUP_API_TAG}' parameter: '${backupParam}'`);
 							}
@@ -141,11 +150,13 @@ class EC2Store {
 
 								if (tuple && tuple.length === 3) {
 									volume.BackupConfig.BackupTypes.push({
+										Name: `[${parseInt(tuple[1])}|${parseInt(tuple[2])}]`,
 										Frequency: parseInt(tuple[1]),
 										Expiry: parseInt(tuple[2])
 									});
 								} else if (ALIASES.hasOwnProperty(backupType)) {
 									volume.BackupConfig.BackupTypes.push({
+										Name: backupType,
 										Alias: backupType,
 										Frequency: ALIASES[backupType][0],
 										Expiry: ALIASES[backupType][1]