Implement 0.1.0 (#65)

* Clean up 0.1.0 branch

* 0.1.0 (#64)

.gitignore

@@ -4,3 +4,5 @@ bin
 pkg
 cistern
 !vendor/*
+cistern.json
+data/

Gopkg.toml

@@ -0,0 +1,25 @@
+
+# Gopkg.toml example
+#
+# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
+# for detailed Gopkg.toml documentation.
+#
+# required = ["github.com/user/thing/cmd/thing"]
+# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
+#
+# [[constraint]]
+#   name = "github.com/user/project"
+#   version = "1.0.0"
+#
+# [[constraint]]
+#   name = "github.com/user/project2"
+#   branch = "dev"
+#   source = "github.com/myfork/project2"
+#
+# [[override]]
+#  name = "github.com/x/y"
+#  version = "2.4.0"
+
+[[constraint]]
+  name = "github.com/aws/aws-sdk-go"
+  version = "=1.10.14"

LICENSE

@@ -1,27 +1,19 @@
-Copyright (c) 2014 Preetam Jinka
-All rights reserved.
+Copyright (c) 2017 Cistern Authors
 
-Redistribution and use in source and binary forms, with or without modification,
-are permitted provided that the following conditions are met:
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
 
-1. Redistributions of source code must retain the above copyright notice, this
-list of conditions and the following disclaimer.
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
 
-2. Redistributions in binary form must reproduce the above copyright notice,
-this list of conditions and the following disclaimer in the documentation and/or
-other materials provided with the distribution.
-
-3. Neither the name of the copyright holder nor the names of its contributors
-may be used to endorse or promote products derived from this software without
-specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
-ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
-ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
-(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
-ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
-SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,9 +1,190 @@
-<img src='https://cloud.githubusercontent.com/assets/379404/12572133/3a54ee4e-c3b6-11e5-9756-3e13617b5970.png' width='150px'/>
-
-[![Circle CI](https://circleci.com/gh/Cistern/cistern.svg?style=svg&circle-token=d06bfb21101f58f7492cc3b2c118415c6f3e9b0d)](https://circleci.com/gh/Cistern/cistern) [![Join the chat at https://gitter.im/Preetam/cistern](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Preetam/cistern?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+<p align="center">
+<img src='https://user-images.githubusercontent.com/379404/28300304-b38a05b2-6b4c-11e7-9e80-19bc3c41fa0b.png' width=400/>
+</p>
 
+<p align="center">
 Cistern is a network flow collector.
+</p>
+
+![experimental](https://img.shields.io/badge/status-experimental-orange.svg)
+[![Join the chat at https://gitter.im/Preetam/cistern](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Preetam/cistern?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
-License
 ---
-BSD (see LICENSE)
+
+###### Supported sources
+
+* CloudWatch Logs
+  * VPC Flow Logs
+
+Coming soon:
+* JSON CloudWatch Logs events
+* sFlow v5
+
+## Getting started
+
+Cistern has no external dependencies.
+
+```
+Usage of ./cistern:
+  -api-addr string
+    	API listen address (default "localhost:2020")
+  -config string
+    	Path to config file (default "./cistern.json")
+  -data-dir string
+    	Data directory (default "./data/")
+```
+
+
+
+#### Config file
+
+The config file has two main options:
+
+* cloudwatch_logs: A list of CloudWatch Logs log groups to consume. See [VPC Flow Logs](#vpc-flow-logs) for more details.
+* retention: The retention of events in days.
+
+```json
+{
+  "cloudwatch_logs": [],
+  "retention": 3
+}
+```
+
+### VPC Flow Logs
+
+You can specify the flow log groups to consume in the config file.
+In the `cloudwatch_logs` section, add an object for each log group
+with the name and `flowlog: true`.
+
+**Example**
+
+```json
+{
+  "cloudwatch_logs": [
+    {
+      "name": "flowlogs",
+      "flowlog": true
+    }
+  ],
+  "retention": 3
+}
+```
+
+#### Credentials
+
+Cistern will try to use AWS credentials from the following locations:
+
+* The environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+* The Shared Credentials file (~/.aws/credentials)
+* EC2 Instance Role Credentials
+
+To specify the region, set the AWS_REGION environment variable.
+
+## Querying
+
+You can query Cistern using the API or the CLI, which uses the API.
+Most of the logic is built into the API, so it's not a bad idea to
+learn how the API works in order to make sense of the CLI.
+
+### API
+
+**POST /collections/:collection/compact**
+
+This endpoint enforces the retention policy for the collection.
+
+**POST /collections/:collection/query**
+
+This endpoint queries events within a collection.
+
+The query endpoint accepts a **Query** object, which has the following
+syntax:
+
+```js
+Query: {
+    /* All of the following are optional. */
+    "time_range": TimeRange
+    "columns":    []Column
+    "group_by":   []string
+    "filters":    []Filter
+    "point_size": integer
+    "order_by":   []string
+    "limit":      integer
+    "descending": bool
+}
+
+TimeRange : {
+    "start": string
+    "end":   string
+}
+
+Column: {
+    "name":      string
+    "aggregate": string
+}
+
+Filter: {
+    "column":    string
+    "condition": string
+    "value":     *
+}
+```
+
+#### Filters
+
+Filters are applied as the first stage of query execution. A filter requires
+a column name, a condition, and a value for the condition. The supported
+conditions are:
+
+* **eq**: equal
+* **neq**: not equal
+
+If multiple filters are specified, they are applied in an "AND" condition.
+
+#### Generating time series
+
+Time series can be generated by providing a nonzero `point_size`. This will
+automatically group events by time ranges determined by the provided point size.
+
+### CLI
+
+```
+Usage of ./cistern-cli:
+  -address string
+    	Cistern node address (default "http://localhost:2020")
+  -collection string
+    	Collection to query
+  -columns string
+    	Comma-separated list of columns to aggregate.
+    	Example: 'sum(bytes), sum(packets)'
+  -descending
+    	Sort in descending order.
+  -end int
+    	End Unix timestamp
+  -filters string
+    	Comma-separated list of filters.
+    	Filters have the format '<column> <condition> <value>'.
+    	Possible conditions are [eq,neq].
+    	Values have to be valid JSON values.
+    	Example: 'dest_address neq "172.31.31.192" , packets eq 3'
+  -group string
+    	Comma-separated list of fields to group by.
+    	Example: 'source_address, dest_address'
+  -limit int
+    	Maximum number of events to return.
+  -order-by string
+    	Comma-separated list of columns to order by.
+    	Providing multiple columns means the results are ordered
+    	by the first column, then the next, etc.
+  -point-size duration
+    	Point size of time series. 0 means series will not be generated.
+  -start int
+    	Start Unix timestamp
+  -version
+    	Show version and exit.
+```
+
+The CLI prints the output of the API response in JSON format.
+
+## License
+
+MIT (see LICENSE)