diff --git a/.gitignore b/.gitignore index afd972e..7f9a98c 100644 --- a/.gitignore +++ b/.gitignore @@ -14,7 +14,7 @@ bin/ .idea/ **.tgz .log -.env +.env* *.pem *.key .DS_Store diff --git a/APIGOV-26720-new b/APIGOV-26720-new deleted file mode 160000 index ccc5f31..0000000 --- a/APIGOV-26720-new +++ /dev/null @@ -1 +0,0 @@ -Subproject commit ccc5f31df0c83e5fda56ea897452ff0bd594af3c diff --git a/Makefile b/Makefile index 75e1035..fb5fa64 100644 --- a/Makefile +++ b/Makefile @@ -19,6 +19,10 @@ all: clean test: dep @go vet ${GO_PKG_LIST} @go test -race -v -short -coverprofile=${WORKSPACE}/gocoverage.out -count=1 ${GO_PKG_LIST} + +test-s: dep + @go vet ${GO_PKG_LIST} + @go test -race -short -coverprofile=${WORKSPACE}/gocoverage.out -count=1 ${GO_PKG_LIST} clean: @rm -rf ./bin/ diff --git a/README.md b/README.md index 15231ed..960b3bd 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,13 @@ The Kong agents are used to discover, provision access to, and track usages of Kong Gateway routes. - [Getting started](#getting-started) + - [Discovery process](#discovery-process) + - [Provisioning process](#provisioning-process) + - [Marketplace application](#marketplace-application) + - [Access request](#access-request) + - [Credential](#credential) + - [Traceability process](#traceability-process) + - [Environment variables](#environment-variables) - [Setup](#setup) - [Amplify setup](#amplify-setup) - [Platform - organization ID](#platform---organization-id) @@ -14,6 +21,7 @@ The Kong agents are used to discover, provision access to, and track usages of K - [Local specification path](#local-specification-path) - [URL specification paths](#url-specification-paths) - [Kong Dev Portal](#kong-dev-portal) + - [HTTP Log plugin](#http-log-plugin) - [Kong agents deployment](#kong-agents-deployment) - [Additional information](#additional-information) - [Docker](#docker) @@ -21,16 +29,62 @@ The Kong agents are used to discover, provision access to, and track usages of K - [Deployment](#deployment) - [Helm](#helm) - [Download](#download) + - [Traceability agent stateful set](#traceability-agent-stateful-set) - [Create secrets](#create-secrets) - [Create volume, local specification files only](#create-volume-local-specification-files-only) + - [ConfigMap](#configmap) + - [AWS S3 Synchronization](#aws-s3-synchronization) - [Create overrides](#create-overrides) - [Deploy local helm chart](#deploy-local-helm-chart) - - [Discovery process](#discovery-process) - - [Provisioning process](#provisioning-process) - - [Marketplace application](#marketplace-application) - - [Access request](#access-request) - - [Credential](#credential) - - [Environment variables](#environment-variables) + +## Discovery process + +On startup the Kong discovery agent first validates that it is able to connect to all required services. Once connected to Kong the agent begins looking at the Plugins configured, as the ACL plugin is required for handling Amplify Central provisioning events. Then the agent will determine, from the plugins, which credential types the Kong Gateway has configured and create the Central representation of those types. + +After that initial startup process the discovery agent begins running its main discovery loop. In this loop the agent first gets a list of all Gateway Services. With each service the agent looks for all configured routes. The agent then looks to gather the specification file, see [Specification discovery methods](#specification-discovery-methods), if found the process continues. Using the route the agent checks for plugins to determine the types of credentials to associate with it. After gathering all of this information the agent creates a new API service with the specification file and linking the appropriate credentials. The endpoints associated to the API service are constructed using the **KONG_PROXY_HOST**, **KONG_PROXY_PORTS_HTTP**, and **KONG_PROXY_PORTS_HTTPS** settings. + +## Provisioning process + +As described in the [Discovery process](#discovery-process) section the Kong agent creates all supported credential types on Central at startup. Once API services are published they can be made into Assets and Products via Central itself. The Products can then be published to the Marketplace for consumption. In order to receive access to the service a user must first request access to it and the Kong agent provisioning process will execute based off of that request. + +### Marketplace application + +A Marketplace application is created by a Marketplace user. When a resource within the Kong environment is added to that application Central will create a ManagedApplication resource that the agent will execute off of. This ManagedApplication resource event is captured by the Kong agent and the agent creates a Kong consumer. In addition to the creation of the Consumer the agent adds an ACL Group ID to the Consumer, to be used by the Access Request. + +### Access request + +When a Marketplace user requests access to a resource, within the Kong environment, Central will create an AccessRequest resource in the same Kong environment. The agent receives this event and makes several changes within Kong. First the agent will add, or update, an ACL configuration on the Route being requested. This ACL will allow the Group ID created during the handling of the [Marketplace application](#marketplace-application) access to the route. Additionally, if a quota for this route has been set in Central in the product being handled the agent will add a Rate limiting plugin to reflect the quota that was set in Central for that product. Note: Quotas in Central can have a Weekly amount, this is not supported by Kong and the agent will reject the Access Request. + +### Credential + +Finally, when a Marketplace user requests a credential, within the Kong environment, Central will create a Credential resource in the same Kong environment. The agent receives this event and creates the proper credential type for the Consumer that the [Marketplace application](#marketplace-application) handling created. After successfully creating this credential the necessary details are returned back to the Central to be viewed and used by the Marketplace user. + +## Traceability process + +On startup the Kong traceability agent first validates that it is able to connect to all required services. Once validation is complete the agent begins listening for log events to be sent to it. The agent receives these events and iterates through them to determine if any of the events should be sampled. If it is to be sampled the agent creates a transaction summary and leg sending that the Amplify Central. Regardless of the event being set for sampling the agent will update the proper API Metric and Usage details to be sent to Amplify Central on the interval configured. See [Usage](https://docs.axway.com/bundle/amplify-central/page/docs/connect_manage_environ/connected_agent_common_reference/traceability_usage/index.html). + +## Environment variables + +All Kong specific environment variables available are listed below + +| Name | Description | +| -------------------------------------- | --------------------------------------------------------------------------------------------- | +| Discovery Agent Variables | | +| **KONG_ADMIN_URL** | The Kong admin API URL that the agent will query against | +| **KONG_ADMIN_AUTH_APIKEY_HEADER** | The API Key header name the agent will use when authenticating | +| **KONG_ADMIN_AUTH_APIKEY_VALUE** | The API Key value the agent will use when authenticating | +| **KONG_ADMIN_AUTH_BASICAUTH_USERNAME** | The HTTP Basic username that the agent will use when authenticating | +| **KONG_ADMIN_AUTH_BASICAUTH_PASSWORD** | The HTTP Basic password that the agent will use when authenticating | +| **KONG_PROXY_HOST** | The proxy endpoint that the agent will use in API Services for discovered Kong routes | +| **KONG_PROXY_PORTS_HTTP** | The HTTP port number that the agent will set for discovered APIS | +| **KONG_PROXY_PORTS_HTTPS** | The HTTPs port number that the agent will set for discovered APIS | +| **KONG_SPEC_LOCALPATH** | The local path that the agent will look in for API definitions | +| **KONG_SPEC_URLPATHS** | The URL paths that the agent will query on the gateway service for API definitions | +| **KONG_SPEC_DEVPORTALENABLED** | Set to true if hte agent should look for spec files in the Kong Dev Portal (default: `false`) | +| | | +| Traceability Agent Variables | | +| **KONG_LOGS_HTTP_PATH** | The path endpoint that the Traceability agent will listen on (default: `/requestlogs`) | +| **KONG_LOGS_HTTP_PORT** | The port that the Traceability agent HTTP server will listen on (default: `9000`) | ## Setup @@ -87,15 +141,23 @@ You now have the service account information needed for you Kong Agent installat - Note the *Logical Name* for your new environment --- -**_NOTE:_** +**NOTE:** Don't forget to update your Amplify Central Region specific variables, such as the `CENTRAL_URL` setting. All CENTRAL_* variables listed on [docs.axway.com](https://docs.axway.com/bundle/amplify-central/page/docs/connect_manage_environ/connect_api_manager/agent-variables/index.html) may be used on the Kong Agent. -___ + +--- ### Kong setup +--- +**NOTE:** + +The Discovery agent expects that the Kong Gateway utilizes the [ACL](https://docs.konghq.com/hub/kong-inc/acl/) plugin to control access to the various routes provided in the Kong Gateway. On startup the agent checks that this plugin is in use prior to performing any discovery. The agent then uses this plugin while provisioning access to routes in Kong. [Provisioning Process](#provisioning-process). + +--- + #### Kong admin API secured by Kong Gateway See [Kong - Securing the Admin API](https://docs.konghq.com/gateway/latest/production/running-kong/secure-admin-api/) @@ -162,7 +224,7 @@ KONG_SPEC_URLPATHS=/openapi.json,/swagger.json ##### Kong Dev Portal The Kong Dev Portal discovery method is configured by providing a value for the `KONG_SPEC_DEVPORTALENABLED`, but also the local spec discovery needs to be disabled by setting an empty value for the`KONG_SPEC_LOCALPATH`, otherwise, the local discovery process will be used. - + Ex. Configuration for agent @@ -172,6 +234,38 @@ KONG_SPEC_LOCALPATH="" KONG_SPEC_DEVPORTALENABLED=true ``` +#### HTTP Log plugin + +The Traceability agent utilizes Kong's HTTP log plugin to track transactions. In order to set this up the plugin will have to be added, globally, and configured to send to the endpoint that the Traceability agent will listen on + +- Navigate to the Plugins page +- Click *+ New Plugin* +- In the list of plugins find *HTTP Log* and click *enable* +- Ensure *Global* is selected so the agent receives events for all traffic +- Enter the following, all can be customized as necessary for your infrastructure, [HTTP Log](https://docs.konghq.com/hub/kong-inc/http-log/configuration/) + - An Instance Name (optional) + - Tags (optional) + - content_type - `applicaiton/json` + - custom_fields_by_lua - empty + - flush_timeout - empty + - headers - empty + - http_endpoint - the endpoint the agent will listen on (ie. `http://traceability.host:9000/requestlogs`) + - keepalive - `60000` + - method - `POST` + - queue.initial_retry_delay - `0.01` + - queue.max_batch_size - `1000` + - queue.max_bytes - empty + - queue.max_coalescing_delay - `10` + - queue.max_entries - `100000` + - queue.max_retry_delay - `60` + - queue.max_retry_time - `60` + - queue_size - empty + - retry_count - empty + - timeout - `10000` +- Click *Install* + +Kong is now setup to send transactions to the traceability agent. + ## Kong agents deployment The Kong agents are delivered as containers, kong_discovery_agent and kong_traceability_agent. These containers can be deployed directly to a container server, such as Docker, or using the provided helm chart. In this section you will lean how to deploy the agents directly as containers or within a kubernetes cluster using the helm chart. @@ -180,7 +274,7 @@ The Kong agents are delivered as containers, kong_discovery_agent and kong_trace Before beginning to deploy the agents following information will need to be gathered in addition to the details that were noted in setup. -- The full URL to connect to the Kong admin API, `KONG_ADMIN_URL`. Note that if secured by kong, the URL should look like: https://host:port/secured-route-from-kong +- The full URL to connect to the Kong admin API, `KONG_ADMIN_URL`. Note that if secured by kong, the URL should look like: [https://host:port/secured-route-from-kong] - The host the agent will use when setting the endpoint of a discovered API, (`KONG_PROXY_HOST`) - The HTTP `KONG_PROXY_PORTS_HTTP` and HTTPs `KONG_PROXY_PORTS_HTTPS` ports the agent will use with the endpoint above - The URL paths, hosted by the gateway service, to query for spec files, `KONG_SPEC_URLPATHS` @@ -189,7 +283,7 @@ Before beginning to deploy the agents following information will need to be gath #### Environment variable files -In this section we will use the information gathered within the setup and additional information sections above and create two environment variable files for each agent to use. This is the minimum configuration assuming defaults for all other available settings. Note the setting below expect the use of the API Key authentication method for the [Kong admin api](#kong-admin-api-secured-by-kong-gateway). +In this section we will use the information gathered within the setup and additional information sections above and create two environment variable files for each agent to use. This is the minimum configuration assuming defaults for all other available settings. Note the settings below expect the use of the API Key authentication method for the [Kong admin api](#kong-admin-api-secured-by-kong-gateway). Discovery Agent @@ -213,10 +307,6 @@ AGENTFEATURES_MARKETPLACEPROVISIONING=true Traceability Agent ```shell -KONG_ADMIN_URL=https://kong.url.com:8444 -KONG_ADMIN_AUTH_APIKEY_HEADER="apikey" -KONG_ADMIN_AUTH_APIKEY_VALUE=123456789abcdefghijkl098765432109 - CENTRAL_ORGANIZATIONID=123456789 CENTRAL_AUTH_CLIENTID=kong-agents_123456789-abcd-efgh-ijkl-098765432109 CENTRAL_ENVIRONMENT=kong @@ -243,7 +333,7 @@ docker run -d -v /home/user/keys:/keys -v /home/user/specs:/specs -v /home/user/ Kong Traceability agent ```shell -docker run -d -v /home/user/keys:/keys -v /home/user/traceability/data:/data --env-file traceability-agents.env ghcr.io/axway/kong_traceability_agent:latest +docker run -d -v /home/user/keys:/keys -v /home/user/traceability/data:/data --env-file traceability-agents.env -p 9000:9000 ghcr.io/axway/kong_traceability_agent:latest ``` ### Helm @@ -259,6 +349,10 @@ tar xvf kong-agents.tar.gz --strip-components=2 agents-kong-${tag}/helm/kong-age rm kong-agents.tar.gz # remove the archive ``` +#### Traceability agent stateful set + +The helm deployment of the Traceability agent uses a resource type of Stateful set along with a service to distribute the events to the agent pods. This is to allow scaling of the traceability agent in order to properly handle the load of events being sent through Kong. The agent is expected to be ran in the same kubernetes cluster as the Gateway and the [HTTP Log plugin](#http-log-plugin) should set its endpoint configuration to the [Service](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#services) that is created (ie.`http://kong-traceability-agent.kong-agents.svc.cluster.local:9000` where `kong-traceability-agent` is the service name and `kong-agents` is the namespace for the service) + #### Create secrets Platform service account key secret @@ -288,7 +382,13 @@ stringData: #### Create volume, local specification files only -A volume of with the local specification files is required, given that is the desired [specification discovery method](#specification-discovery-methods). This volume could be of any kubernetes resource type which can be mounted in the Kong agent container. Below is a sample of a ConfigMap that is used for the local specification files. See [Kubernetes Volumes](https://kubernetes.io/docs/concepts/storage/volumes/). +A volume of with the local specification files is required, given that is the desired [specification discovery method](#specification-discovery-methods). This volume could be of any kubernetes resource type which can be mounted in the Kong agent container. See [Kubernetes Volumes](https://kubernetes.io/docs/concepts/storage/volumes/). + +Below are a couple of examples on adding specifications to a volume, of any type, to the agent pod for discovery purposes. + +##### ConfigMap + +Here is a sample of a ConfigMap that is used for the local specification files. ```yaml apiVersion: v1 @@ -306,13 +406,14 @@ If a ConfigMap is being used, the kubectl command provides a utility to create t kubectl create configmap specs --from-file=specs/ ``` -___ -**_NOTE:_** +--- +**NOTE:** An update to the ConfigMap will *NOT* be seen by any running pods, a pod restart would be required to see changes. It is recommended to use a volume type that is more mutable than a ConfigMap. The agent has no knowledge of the volume type being used. -___ + +--- Once a resource with the files is created, which ever resource type is chosen, the overrides file will need to be updated with that resource information for mounting as a volume. @@ -325,19 +426,124 @@ kong: name: my-spec-files # name of the resource created ``` +##### AWS S3 Synchronization + +A kubernetes PersistentVolume resource with a CronJob volume can be set up to regularly synchronize spec files from an S3 bucket to the volume for the agent to utilize. Below you will find the three kubernetes resources that would need to be created as well as the update to the agnet helm chart override file. + +- Create a PersistentVolume - this will store the specification files in the cluster + - In this example a storage class of manual is used with a host path in the kubernetes cluster, however any class type may be used + - [K8S Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) + - [EKS Persistent Volumes](https://aws.amazon.com/blogs/storage/persistent-storage-for-kubernetes/) + +```yaml +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: spec-volume + labels: + type: local +spec: + storageClassName: manual + capacity: + storage: 1Gi + accessModes: + - ReadWriteOnce + hostPath: + path: "/data" +``` + +- Create a PersistentVolumeClaim - this allows pods to mount this volume, needed for the job and the agent + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: spec-volume-claim +spec: + storageClassName: manual + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +``` + +- Create a CronJob - this will run on the specified interval synchronizing the S3 bucket to the volume + - The keys are embedded in this definition, but this can be replaced by a kubernetes secret or service account with the proper role in EKS + - The schedule is to sync the spec files every 15 minutes + - The bucket name is within the command, `specs-bucket` + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: + name: s3-sync +spec: + schedule: "*/15 * * * *" + concurrencyPolicy: Forbid + jobTemplate: + spec: + template: + spec: + containers: + - name: s3-sync + image: public.ecr.aws/aws-cli/aws-cli + env: + - name: AWS_ACCESS_KEY_ID + value: XXXXXXXXXXXXXXXXXXX + - name: AWS_SECRET_ACCESS_KEY + value: XXXXXXXXXXXXXXXXXXX + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - aws s3 sync s3://specs-bucket/ /specs/ + volumeMounts: + - name: specs-mount + mountPath: /specs + volumes: + - name: specs-mount + persistentVolumeClaim: + claimName: spec-volume-claim + restartPolicy: Never +``` + +- Override the agent helm chart accordingly + +```yaml +kong: + ... + spec: + localPath: + persistentVolumeClaim: # type of the resource, provided in the deployment as a volume. + claimName: spec-volume-claim # name of the resource created +``` + #### Create overrides overrides.yaml ```yaml +discovery: + image: + tag: v0.0.1 # update accordingly + +traceability: + image: + tag: v0.0.1 # update accordingly + kong: + enable: + traceability: true # set this to true to deploy the traceability agent stateful set admin: - url: https://kong.url.com:8444 + url: http://kong-gateway-kong-admin.kong.svc.cluster.local:8001 proxy: host: kong.proxy.endpoint.com ports: - http: 8000 - https: 8443 + http: 80 + https: 443 spec: localPath: configMap: @@ -365,42 +571,3 @@ Install the helm chart using the created overrides file. ```shell helm install kong-agents ./kong-agents -f overrides.yaml ``` - -## Discovery process - -On startup the Kong discovery agent first validates that it is able to connect to all required services. Once connected to Kong the agent begins looking at the Plugins configured, as the ACL plugin is required for handling Amplify Central provisioning events. Then the agent will determine, from the plugins, which credential types the Kong Gateway has configured and create the Central representation of those types. - -After that initial startup process the discovery agent begins running its main discovery loop. In this loop the agent first gets a list of all Gateway Services. With each service the agent looks for all configured routes. The agent then looks to gather the specification file, see [Specification discovery methods](#specification-discovery-methods), if found the process continues. Using the route the agent checks for plugins to determine the types of credentials to associate with it. After gathering all of this information the agent creates a new API service with the specification file and linking the appropriate credentials. The endpoints associated to the API service are constructed using the **KONG_PROXY_HOST**, **KONG_PROXY_PORTS_HTTP**, and **KONG_PROXY_PORTS_HTTPS** settings. - -## Provisioning process - -As described in the [Discovery process](#discovery-process) section the Kong agent creates all supported credential types on Central at startup. Once API services are published they can be made into Assets and Products via Central itself. The Products can then be published to the Marketplace for consumption. In order to receive access to the service a user must first request access to it and the Kong agent provisioning process will execute based off of that request. - -### Marketplace application - -A Marketplace application is created by a Marketplace user. When a resource within the Kong environment is added to that application Central will create a ManagedApplication resource that the agent will execute off of. This ManagedApplication resource event is captured by the Kong agent and the agent creates a Kong consumer. In addition to the creation of the Consumer the agent adds an ACL Group ID to the Consumer, to be used by the Access Request. - -### Access request - -When a Marketplace user requests access to a resource, within the Kong environment, Central will create an AccessRequest resource in the same Kong environment. The agent receives this event and makes several changes within Kong. First the agent will add, or update, an ACL configuration on the Route being requested. This ACL will allow the Group ID created during the handling of the [Marketplace application](#marketplace-application) access to the route. Additionally, if a quota for this route has been set in Central in the product being handled the agent will add a Rate limiting plugin to reflect the quota that was set in Central for that product. Note: Quotas in Central can have a Weekly amount, this is not supported by Kong and the agent will reject the Access Request. - -### Credential - -Finally, when a Marketplace user requests a credential, within the Kong environment, Central will create a Credential resource in the same Kong environment. The agent receives this event and creates the proper credential type for the Consumer that the [Marketplace application](#marketplace-application) handling created. After successfully creating this credential the necessary details are returned back to the Central to be viewed and used by the Marketplace user. - -## Environment variables - -All Kong specific environment variables available are listed below - -| Name | Description | -| -------------------------------------- | ------------------------------------------------------------------------------------- | -| **KONG_ADMIN_URL** | The Kong admin API URL that the agent will query against | -| **KONG_ADMIN_AUTH_APIKEY_HEADER** | The API Key header name the agent will use when authenticating | -| **KONG_ADMIN_AUTH_APIKEY_VALUE** | The API Key value the agent will use when authenticating | -| **KONG_ADMIN_AUTH_BASICAUTH_USERNAME** | The HTTP Basic username that the agent will use when authenticating | -| **KONG_ADMIN_AUTH_BASICAUTH_PASSWORD** | The HTTP Basic password that the agent will use when authenticating | -| **KONG_PROXY_HOST** | The proxy endpoint that the agent will use in API Services for discovered Kong routes | -| **KONG_PROXY_PORTS_HTTP** | The HTTP port number that the agent will set for discovered APIS | -| **KONG_PROXY_PORTS_HTTPS** | The HTTPs port number that the agent will set for discovered APIS | -| **KONG_SPEC_LOCALPATH** | The local path that the agent will look in for API definitions | -| **KONG_SPEC_URLPATHS** | The URL paths that the agent will query on the gateway service for API definitions | diff --git a/build/traceability/kong_traceability_agent.yml b/build/traceability/kong_traceability_agent.yml index 5d8d08d..aa70bab 100644 --- a/build/traceability/kong_traceability_agent.yml +++ b/build/traceability/kong_traceability_agent.yml @@ -1,15 +1,10 @@ kong_traceability_agent: # Settings for connecting to Kong kong: - admin: - url: ${KONG_ADMIN_URL} - auth: - apikey: - header: ${KONG_ADMIN_AUTH_APIKEY_HEADER} - value: ${KONG_ADMIN_AUTH_APIKEY_VALUE} - httpLogPlugin: - path: ${KONG_HTTPLOGPLUGIN_PATH} - port: ${KONG_HTTPLOGPLUGIN_PORT} + logs: + http: + path: ${KONG_LOGS_HTTP_PATH} + port: ${KONG_LOGS_HTTP_PORT} # Settings for connecting to Amplify Central central: url: ${CENTRAL_URL:https://apicentral.axway.com} diff --git a/go.mod b/go.mod index b71276b..dbc9b96 100644 --- a/go.mod +++ b/go.mod @@ -3,7 +3,7 @@ module github.com/Axway/agents-kong go 1.18 require ( - github.com/Axway/agent-sdk v1.1.68-0.20231120204404-5fa2c5b231e1 + github.com/Axway/agent-sdk v1.1.69 github.com/elastic/beats/v7 v7.17.15 github.com/google/uuid v1.3.1 github.com/kong/go-kong v0.47.0 diff --git a/go.sum b/go.sum index 5cdef8d..f0ef2a2 100644 --- a/go.sum +++ b/go.sum @@ -36,8 +36,8 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo= dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU= -github.com/Axway/agent-sdk v1.1.68-0.20231120204404-5fa2c5b231e1 h1:UPRwbD7Hb/p4CpSHv0hDwOmjbSqYy5Fk7pSnnatDwes= -github.com/Axway/agent-sdk v1.1.68-0.20231120204404-5fa2c5b231e1/go.mod h1:Iuv9KlWksVTbTKdfs4bKVYMDc33ZTLYoHt572z2CbbI= +github.com/Axway/agent-sdk v1.1.69 h1:k9YHhoXfGoZtBw6hJf3TrRNPdmLTY07eXewY5Odxx+M= +github.com/Axway/agent-sdk v1.1.69/go.mod h1:Iuv9KlWksVTbTKdfs4bKVYMDc33ZTLYoHt572z2CbbI= github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 h1:w+iIsaOQNcT7OZ575w+acHgRric5iCyQh+xv+KJ4HB8= github.com/Azure/go-autorest v14.2.0+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24= github.com/Azure/go-autorest/autorest v0.11.12/go.mod h1:eipySxLmqSyC5s5k1CLupqet0PSENBEDP93LQ9a8QYw= diff --git a/helm/kong-agents/Chart.yaml b/helm/kong-agents/Chart.yaml index 9c86696..fb7679d 100644 --- a/helm/kong-agents/Chart.yaml +++ b/helm/kong-agents/Chart.yaml @@ -19,10 +19,10 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.0.0 +version: 0.0.1 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: "0.0.0" +appVersion: "v0.0.1" diff --git a/helm/kong-agents/templates/_helpers.tpl b/helm/kong-agents/templates/_helpers.tpl index 9c0638d..9db6e43 100644 --- a/helm/kong-agents/templates/_helpers.tpl +++ b/helm/kong-agents/templates/_helpers.tpl @@ -42,6 +42,14 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} +{{/* +Traceability selector labels +*/}} +{{- define "kong-agents.traceability.selectorLabels" -}} +{{ include "kong-agents.selectorLabels" . }} +app.agent.type: traceability +{{- end }} + {{/* Selector labels */}} diff --git a/helm/kong-agents/templates/discovery-deployment.yaml b/helm/kong-agents/templates/discovery-deployment.yaml index d6c07f9..2647233 100644 --- a/helm/kong-agents/templates/discovery-deployment.yaml +++ b/helm/kong-agents/templates/discovery-deployment.yaml @@ -121,7 +121,10 @@ spec: "KONG_PROXY_PORTS_HTTP" "KONG_PROXY_PORTS_HTTPS" "KONG_SPEC_LOCALPATH" - "KONG_SPEC_URLPATHS"))) + "KONG_SPEC_URLPATHS" + "KONG_LOGS_HTTP_SERVER_PATH" + "KONG_LOGS_HTTP_SERVER_PORT" + "STATUS_PORT"))) }} - name: {{ $key }} value: {{ $value | quote }} @@ -132,6 +135,8 @@ spec: value: "/keys/private_key.pem" - name: CENTRAL_AUTH_PUBLICKEY value: "/keys/public_key.pem" + - name: STATUS_PORT + value: "{{ .Values.statusPort }}" volumeMounts: - name: "kong-agent-keys" mountPath: "/keys" diff --git a/helm/kong-agents/templates/service.yaml b/helm/kong-agents/templates/service.yaml new file mode 100644 index 0000000..023a238 --- /dev/null +++ b/helm/kong-agents/templates/service.yaml @@ -0,0 +1,17 @@ +{{- if .Values.kong.enable.traceability }} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "kong-agents.fullname" . }}-traceability + labels: + {{- include "kong-agents.labels" . | nindent 4 }} +spec: + selector: + {{- include "kong-agents.selectorLabels" . | nindent 4 }} + app.agent.type: traceability + ports: + - name: logs + protocol: TCP + port: {{ .Values.kong.logs.http.port }} + targetPort: logs +{{- end -}} \ No newline at end of file diff --git a/helm/kong-agents/templates/traceability-deployment.yaml b/helm/kong-agents/templates/traceability-statefulset.yaml similarity index 76% rename from helm/kong-agents/templates/traceability-deployment.yaml rename to helm/kong-agents/templates/traceability-statefulset.yaml index c1a85d0..2d42835 100644 --- a/helm/kong-agents/templates/traceability-deployment.yaml +++ b/helm/kong-agents/templates/traceability-statefulset.yaml @@ -1,6 +1,6 @@ {{- if .Values.kong.enable.traceability }} apiVersion: apps/v1 -kind: Deployment +kind: StatefulSet metadata: name: {{ include "kong-agents.fullname" . }}-traceability labels: @@ -9,7 +9,7 @@ spec: replicas: {{ .Values.traceability.replicaCount }} selector: matchLabels: - {{- include "kong-agents.selectorLabels" . | nindent 6 }} + {{- include "kong-agents.traceability.selectorLabels" . | nindent 6 }} {{- with .Values.additionalLabels }} {{- range $key, $value := . }} {{ default "none" $key }}: {{ default "none" $value | quote }} @@ -22,7 +22,7 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} labels: - {{- include "kong-agents.selectorLabels" . | nindent 8 }} + {{- include "kong-agents.traceability.selectorLabels" . | nindent 8 }} {{- with .Values.additionalLabels }} {{- range $key, $value := . }} {{ default "none" $key }}: {{ default "none" $value | quote }} @@ -48,6 +48,9 @@ spec: - name: probe-port containerPort: {{ .Values.statusPort }} protocol: TCP + - name: logs + containerPort: {{ .Values.kong.logs.http.port }} + protocol: TCP livenessProbe: httpGet: path: /status @@ -67,20 +70,10 @@ spec: {{- end }} {{- end }} env: - - name: KONG_ADMIN_URL - value: "{{ .Values.kong.admin.url }}" - {{- if .Values.kong.admin.auth.apikey.value }} - - name: KONG_ADMIN_AUTH_APIKEY_VALUE - valueFrom: - secretKeyRef: - name: kong-admin-auth-apikey - key: value - - name: KONG_ADMIN_AUTH_APIKEY_HEADER - valueFrom: - secretKeyRef: - name: kong-admin-auth-apikey - key: header - {{- end }} + - name: KONG_LOGS_HTTP_SERVER_PATH + value: "{{ .Values.kong.logs.http.path }}" + - name: KONG_LOGS_HTTP_SERVER_PORT + value: "{{ .Values.kong.logs.http.port }}" {{- with .Values.env }} {{- range $key, $value := . }} {{- if and (not (eq (toString $value) "")) @@ -92,19 +85,36 @@ spec: "KONG_PROXY_PORTS_HTTP" "KONG_PROXY_PORTS_HTTPS" "KONG_SPEC_LOCALPATH" - "KONG_SPEC_URLPATHS"))) + "KONG_SPEC_URLPATHS" + "KONG_LOGS_HTTP_SERVER_PATH" + "KONG_LOGS_HTTP_SERVER_PORT" + "STATUS_PORT"))) }} - name: {{ $key }} value: {{ $value | quote }} {{- end }} {{- end }} {{- end }} + - name: CENTRAL_AUTH_PRIVATEKEY + value: "/keys/private_key.pem" + - name: CENTRAL_AUTH_PUBLICKEY + value: "/keys/public_key.pem" + - name: STATUS_PORT + value: "{{ .Values.statusPort }}" + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name volumeMounts: + - name: beat-storage + mountPath: /data - name: "kong-agent-keys" mountPath: "/keys" resources: {{- toYaml .Values.resources | nindent 12 }} volumes: + - name: beat-storage + emptyDir: {} - name: kong-agent-keys secret: secretName: {{ .Values.secrets.keys }} diff --git a/helm/kong-agents/values.yaml b/helm/kong-agents/values.yaml index 43c19d1..01004bd 100644 --- a/helm/kong-agents/values.yaml +++ b/helm/kong-agents/values.yaml @@ -41,6 +41,10 @@ kong: filter: urlPaths: [] localPath: + logs: + http: + path: /requestlogs + port: 9000 # Add any environment variable overrides here env: {} diff --git a/pkg/kong/provisioning.go b/pkg/kong/provisioning.go index a8397b8..1239b72 100644 --- a/pkg/kong/provisioning.go +++ b/pkg/kong/provisioning.go @@ -360,15 +360,15 @@ func getSpecificPlugin(plugins []*klib.Plugin, serviceID, routeID, consumerID, p continue } - if consumerID == "" || plugin.Consumer == nil || (plugin.Consumer != nil && *plugin.Consumer.ID == consumerID) { + if (consumerID == "" && plugin.Consumer == nil) || (consumerID != "" && plugin.Consumer != nil && *plugin.Consumer.ID == consumerID) { consumerMatch = true } - if routeID == "" || plugin.Route == nil || (plugin.Route != nil && *plugin.Route.ID == routeID) { + if (routeID == "" && plugin.Route == nil) || (routeID != "" && plugin.Route != nil && *plugin.Route.ID == routeID) { routeMatch = true } - if serviceID == "" || plugin.Service == nil || (plugin.Service != nil && *plugin.Service.ID == serviceID) { + if (serviceID == "" && plugin.Service == nil) || (serviceID != "" && plugin.Service != nil && *plugin.Service.ID == serviceID) { serviceMatch = true } diff --git a/pkg/kong/provisioning_test.go b/pkg/kong/provisioning_test.go index 9903c7f..810b6af 100644 --- a/pkg/kong/provisioning_test.go +++ b/pkg/kong/provisioning_test.go @@ -584,6 +584,9 @@ func TestAddQuota(t *testing.T) { ID: klib.String("routeID"), }, Enabled: klib.Bool(true), + Consumer: &klib.Consumer{ + ID: klib.String("consumerID"), + }, }, }, "next": "null", @@ -606,6 +609,9 @@ func TestAddQuota(t *testing.T) { Route: &klib.Route{ ID: klib.String("routeID"), }, + Consumer: &klib.Consumer{ + ID: klib.String("consumerID"), + }, Enabled: klib.Bool(false), }, }, diff --git a/pkg/traceability/beater/beater.go b/pkg/traceability/beater/beater.go index 2907d43..47214a1 100644 --- a/pkg/traceability/beater/beater.go +++ b/pkg/traceability/beater/beater.go @@ -5,11 +5,15 @@ import ( "fmt" "io" "net/http" + "os" + "sync" "github.com/elastic/beats/v7/libbeat/beat" "github.com/elastic/beats/v7/libbeat/common" "github.com/google/uuid" + "github.com/Axway/agent-sdk/pkg/agent" + "github.com/Axway/agent-sdk/pkg/transaction/metric" agentErrors "github.com/Axway/agent-sdk/pkg/util/errors" hc "github.com/Axway/agent-sdk/pkg/util/healthcheck" "github.com/Axway/agent-sdk/pkg/util/log" @@ -19,15 +23,19 @@ import ( ) type httpLogBeater struct { - client beat.Client - logger log.FieldLogger - server http.Server + client beat.Client + logger log.FieldLogger + server http.Server + processing sync.WaitGroup + shutdownDone sync.WaitGroup } // New creates an instance of kong_traceability_agent. func New(*beat.Beat, *common.Config) (beat.Beater, error) { b := &httpLogBeater{ - logger: log.NewFieldLogger().WithComponent("httpLogBeater").WithPackage("beater"), + logger: log.NewFieldLogger().WithComponent("httpLogBeater").WithPackage("beater"), + processing: sync.WaitGroup{}, + shutdownDone: sync.WaitGroup{}, } // Validate that all necessary services are up and running. If not, return error @@ -48,18 +56,23 @@ func (b *httpLogBeater) Run(beater *beat.Beat) error { if err != nil { return err } + agent.RegisterShutdownHandler(b.shutdownHandler) mux := http.NewServeMux() - mux.HandleFunc(config.GetAgentConfig().HttpLogPluginConfig.Path, b.HandleHello) + mux.HandleFunc(config.GetAgentConfig().KongGatewayCfg.Logs.HTTP.Path, b.HandleEvent) // other handlers can be assigned to separate paths - b.server = http.Server{Handler: mux, Addr: fmt.Sprintf(":%d", config.GetAgentConfig().HttpLogPluginConfig.Port)} - b.logger.Fatal(b.server.ListenAndServe()) + b.server = http.Server{Handler: mux, Addr: fmt.Sprintf(":%d", config.GetAgentConfig().KongGatewayCfg.Logs.HTTP.Port)} + b.server.ListenAndServe() + + // wait for the shutdown process to finish prior to exit + b.shutdownDone.Add(1) + b.shutdownDone.Wait() return nil } -func (b *httpLogBeater) HandleHello(w http.ResponseWriter, r *http.Request) { +func (b *httpLogBeater) HandleEvent(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { b.logger.Trace("received a non post request") w.WriteHeader(http.StatusMethodNotAllowed) @@ -74,19 +87,37 @@ func (b *httpLogBeater) HandleHello(w http.ResponseWriter, r *http.Request) { return } + b.processing.Add(1) go func(data []byte) { + defer b.processing.Done() ctx := context.WithValue(context.Background(), processor.CtxTransactionID, uuid.NewString()) eventProcessor, err := processor.NewEventsHandler(ctx, data) if err == nil { - eventsToPublish, err := eventProcessor.Handle() - if err == nil { - b.client.PublishAll(eventsToPublish) - } + eventsToPublish := eventProcessor.Handle() + b.client.PublishAll(eventsToPublish) } }(logData) } +func (b *httpLogBeater) shutdownHandler() { + b.logger.Info("waiting for current processing to finish") + defer b.shutdownDone.Done() + + // wait for all processing to finish + b.processing.Wait() + + // publish the metrics and usage + b.logger.Info("publishing cached metrics and usage") + metric.GetMetricCollector().ShutdownPublish() + + // remove the agent resource in k8s clusters + pod_name := os.Getenv("POD_NAME") + if pod_name != "" { + agent.GetCentralClient().DeleteResourceInstance(agent.GetAgentResource()) + } +} + // Stop stops kong_traceability_agent. func (b *httpLogBeater) Stop() { b.server.Shutdown(context.Background()) diff --git a/pkg/traceability/cmd/cmd.go b/pkg/traceability/cmd/cmd.go index 674c973..0765528 100644 --- a/pkg/traceability/cmd/cmd.go +++ b/pkg/traceability/cmd/cmd.go @@ -50,8 +50,8 @@ func initConfig(centralConfig corecfg.CentralConfig) (interface{}, error) { rootProps := TraceCmd.GetProperties() agentConfig := &config.AgentConfig{ - CentralCfg: centralConfig, - HttpLogPluginConfig: config.ParseProperties(rootProps), + CentralCfg: centralConfig, + KongGatewayCfg: config.ParseProperties(rootProps), } config.SetAgentConfig(agentConfig) diff --git a/pkg/traceability/config/config.go b/pkg/traceability/config/config.go index 254c08b..7039895 100644 --- a/pkg/traceability/config/config.go +++ b/pkg/traceability/config/config.go @@ -6,22 +6,32 @@ import ( ) const ( - cfgKongHTTPLogPluginPath = "kong.httpLogPlugin.path" - cfgKongHTTPLogPluginPort = "kong.httpLogPlugin.port" + cfgKongHTTPLogsPath = "kong.logs.http.path" + cfgKongHTTPLogsPort = "kong.logs.http.port" ) func AddKongProperties(rootProps properties.Properties) { - rootProps.AddStringProperty(cfgKongHTTPLogPluginPath, "/requestlogs", "Path on which the HTTP Log plugin sends request logs") - rootProps.AddIntProperty(cfgKongHTTPLogPluginPort, 9000, "Port that listens for request logs from HTTP Log plugin") + rootProps.AddStringProperty(cfgKongHTTPLogsPath, "/requestlogs", "Path on which the HTTP Log plugin sends request logs") + rootProps.AddIntProperty(cfgKongHTTPLogsPort, 9000, "Port that listens for request logs from HTTP Log plugin") } // AgentConfig - represents the config for agent type AgentConfig struct { - CentralCfg corecfg.CentralConfig `config:"central"` - HttpLogPluginConfig *KongHttpLogPluginConfig `config:"httpLogPlugin"` + CentralCfg corecfg.CentralConfig `config:"central"` + KongGatewayCfg KongGatewayConfig `config:"kong"` } -type KongHttpLogPluginConfig struct { +// KongGatewayConfig - represents the config for gateway +type KongGatewayConfig struct { + corecfg.IConfigValidator + Logs KongLogsConfig `config:"logs"` +} + +type KongLogsConfig struct { + HTTP KongLogsHTTPConfig `config:"http"` +} + +type KongLogsHTTPConfig struct { Path string `config:"path"` Port int `config:"port"` } @@ -36,10 +46,14 @@ func GetAgentConfig() *AgentConfig { return agentConfig } -func ParseProperties(rootProps properties.Properties) *KongHttpLogPluginConfig { +func ParseProperties(rootProps properties.Properties) KongGatewayConfig { // Parse the config from bound properties and setup gateway config - return &KongHttpLogPluginConfig{ - Path: rootProps.StringPropertyValue(cfgKongHTTPLogPluginPath), - Port: rootProps.IntPropertyValue(cfgKongHTTPLogPluginPort), + return KongGatewayConfig{ + Logs: KongLogsConfig{ + HTTP: KongLogsHTTPConfig{ + Path: rootProps.StringPropertyValue(cfgKongHTTPLogsPath), + Port: rootProps.IntPropertyValue(cfgKongHTTPLogsPort), + }, + }, } } diff --git a/pkg/traceability/main/agent.go b/pkg/traceability/main/agent.go index f274969..0fba253 100644 --- a/pkg/traceability/main/agent.go +++ b/pkg/traceability/main/agent.go @@ -11,6 +11,13 @@ import ( func main() { os.Setenv("AGENTFEATURES_VERSIONCHECKER", "false") + + // use the pod name as the agent name + pod_name := os.Getenv("POD_NAME") + if pod_name != "" { + os.Setenv("CENTRAL_AGENTNAME", pod_name) + } + if err := traceability.TraceCmd.Execute(); err != nil { fmt.Println(err) os.Exit(1) diff --git a/pkg/traceability/processor/definitions.go b/pkg/traceability/processor/definitions.go index a264d70..cdb82c3 100644 --- a/pkg/traceability/processor/definitions.go +++ b/pkg/traceability/processor/definitions.go @@ -1,6 +1,9 @@ package processor -import "github.com/Axway/agent-sdk/pkg/util/log" +import ( + "github.com/Axway/agent-sdk/pkg/transaction/metric" + "github.com/Axway/agent-sdk/pkg/util/log" +) const ( CtxTransactionID log.ContextField = "transactionID" @@ -31,19 +34,19 @@ type Latencies struct { } type Request struct { - QueryString map[string]string `json:"querystring"` - Size int `json:"size"` - URI string `json:"uri"` - URL string `json:"url"` - Headers map[string]string `json:"headers"` - Method string `json:"method"` - TLS *TLS `json:"tls"` + QueryString map[string]string `json:"querystring"` + Size int `json:"size"` + URI string `json:"uri"` + URL string `json:"url"` + Headers map[string]interface{} `json:"headers"` + Method string `json:"method"` + TLS *TLS `json:"tls"` } type Response struct { - Headers map[string]string `json:"headers"` - Status int `json:"status"` - Size int `json:"size"` + Headers map[string]interface{} `json:"headers"` + Status int `json:"status"` + Size int `json:"size"` } type Route struct { @@ -89,9 +92,13 @@ type TLS struct { } type Consumer struct { - CustomID string `json:"custom_id"` - CreatedAt int64 `json:"created_at"` - ID string `json:"id"` - Tags []string `json:"tags"` - Username string `json:"username"` + CustomID string `json:"custom_id"` + CreatedAt int64 `json:"created_at"` + ID string `json:"id"` + Tags interface{} `json:"tags"` + Username string `json:"username"` +} + +type metricCollector interface { + AddMetricDetail(metricDetail metric.Detail) } diff --git a/pkg/traceability/processor/handler.go b/pkg/traceability/processor/handler.go index d4c411d..2099fa4 100644 --- a/pkg/traceability/processor/handler.go +++ b/pkg/traceability/processor/handler.go @@ -4,22 +4,29 @@ import ( "context" "encoding/json" + "github.com/Axway/agent-sdk/pkg/transaction" "github.com/Axway/agent-sdk/pkg/util/log" "github.com/elastic/beats/v7/libbeat/beat" ) // EventsHandler - type EventsHandler struct { - ctx context.Context - logger log.FieldLogger - logEntries []TrafficLogEntry + ctx context.Context + logger log.FieldLogger + metrics MetricsProcessor + logEntries []TrafficLogEntry + eventGenerator func() transaction.EventGenerator + collectorGetter func() metricCollector } // NewEventsHandler - return a new EventProcessor func NewEventsHandler(ctx context.Context, logData []byte) (*EventsHandler, error) { p := &EventsHandler{ - ctx: ctx, - logger: log.NewLoggerFromContext(ctx).WithComponent("eventsHandler").WithPackage("processor"), + ctx: ctx, + logger: log.NewLoggerFromContext(ctx).WithComponent("eventsHandler").WithPackage("processor"), + metrics: NewMetricsProcessor(ctx), + eventGenerator: transaction.NewEventGenerator, + collectorGetter: getMetricCollector, } err := json.Unmarshal(logData, &p.logEntries) @@ -32,21 +39,37 @@ func NewEventsHandler(ctx context.Context, logData []byte) (*EventsHandler, erro } // Handle - processes the batch of events from the http request -func (p *EventsHandler) Handle() ([]beat.Event, error) { +func (p *EventsHandler) Handle() []beat.Event { events := make([]beat.Event, 0) p.logger.WithField("numEvents", len(p.logEntries)).Info("handling events in request") + p.metrics.setCollector(p.collectorGetter()) for i, entry := range p.logEntries { - log := p.logger.WithField(string(ctxEntryIndex), i) - processor, _ := NewTransactionProcessor(context.WithValue(p.ctx, ctxEntryIndex, i), entry) + ctx := context.WithValue(p.ctx, ctxEntryIndex, i) - // Map the log entry to log event structure expected by AMPLIFY Central Observer - newEvents, err := processor.process() + sample, err := p.metrics.process(entry) if err != nil { - log.WithError(err).Error("creating event") + p.logger.WithError(err).Error("handling event for metric") + continue + } + if !sample { continue } - events = append(events, newEvents...) + + // Map the log entry to log event structure expected by AMPLIFY Central Observer + events = append(events, p.handleTransaction(ctx, entry)...) + } + + return events +} + +func (p *EventsHandler) handleTransaction(ctx context.Context, entry TrafficLogEntry) []beat.Event { + log := p.logger.WithField(string(ctxEntryIndex), ctx.Value(ctxEntryIndex)) + + newEvents, err := NewTransactionProcessor(ctx).setEventGenerator(p.eventGenerator()).setEntry(entry).process() + if err != nil { + log.WithError(err).Error("creating transaction event") + return []beat.Event{} } - return events, nil + return newEvents } diff --git a/pkg/traceability/processor/metrics.go b/pkg/traceability/processor/metrics.go new file mode 100644 index 0000000..c0f8652 --- /dev/null +++ b/pkg/traceability/processor/metrics.go @@ -0,0 +1,80 @@ +package processor + +import ( + "context" + "fmt" + + "github.com/Axway/agent-sdk/pkg/traceability/sampling" + "github.com/Axway/agent-sdk/pkg/transaction/metric" + "github.com/Axway/agent-sdk/pkg/transaction/models" + "github.com/Axway/agent-sdk/pkg/transaction/util" + sdkUtil "github.com/Axway/agent-sdk/pkg/transaction/util" + "github.com/Axway/agent-sdk/pkg/util/log" +) + +// MetricsProcessor - +type MetricsProcessor struct { + ctx context.Context + logger log.FieldLogger + collector metricCollector +} + +func NewMetricsProcessor(ctx context.Context) MetricsProcessor { + return MetricsProcessor{ + ctx: ctx, + logger: log.NewLoggerFromContext(ctx).WithComponent("eventMapper").WithPackage("processor"), + } +} + +func (m *MetricsProcessor) setCollector(collector metricCollector) { + m.collector = collector +} + +// process - receives the log event and returns if the transaction should be sampled +func (m *MetricsProcessor) process(entry TrafficLogEntry) (bool, error) { + details := sampling.TransactionDetails{} + if entry.Response != nil { + details.Status = util.GetTransactionSummaryStatus(entry.Response.Status) + } + if entry.Service != nil { + details.APIID = entry.Route.ID + } + if entry.Consumer != nil { + details.SubID = entry.Consumer.ID + } + + sample, err := sampling.ShouldSampleTransaction(details) + if err != nil { + return false, err + } + m.updateMetric(entry) + + return sample, nil +} + +func (m *MetricsProcessor) updateMetric(entry TrafficLogEntry) { + apiDetails := models.APIDetails{ + ID: entry.Route.ID, + Name: entry.Service.Name, + Stage: entry.Route.Name, + } + + statusCode := entry.Response.Status + duration := entry.Latencies.Request + appDetails := models.AppDetails{} + if entry.Consumer != nil { + appDetails.Name = entry.Consumer.Username + appDetails.ID = sdkUtil.FormatApplicationID(entry.Consumer.ID) + } + + if m.collector != nil { + metricDetail := metric.Detail{ + APIDetails: apiDetails, + StatusCode: fmt.Sprint(statusCode), + Duration: int64(duration), + Bytes: int64(entry.Request.Size), + AppDetails: appDetails, + } + m.collector.AddMetricDetail(metricDetail) + } +} diff --git a/pkg/traceability/processor/mock/collector_mock.go b/pkg/traceability/processor/mock/collector_mock.go new file mode 100644 index 0000000..1ab9d1a --- /dev/null +++ b/pkg/traceability/processor/mock/collector_mock.go @@ -0,0 +1,29 @@ +package mock + +import ( + "fmt" + "sync" + + "github.com/Axway/agent-sdk/pkg/transaction/metric" +) + +var collector *CollectorMock + +func GetMockCollector() *CollectorMock { + return collector +} + +func SetMockCollector(c *CollectorMock) { + collector = c +} + +type CollectorMock struct { + sync.WaitGroup + Details []metric.Detail +} + +func (c *CollectorMock) AddMetricDetail(metricDetail metric.Detail) { + fmt.Printf("%v\n", metricDetail) + c.Details = append(c.Details, metricDetail) + c.Done() +} diff --git a/pkg/traceability/processor/mock/eventgenerator_mock.go b/pkg/traceability/processor/mock/eventgenerator_mock.go new file mode 100644 index 0000000..2003a87 --- /dev/null +++ b/pkg/traceability/processor/mock/eventgenerator_mock.go @@ -0,0 +1,68 @@ +package mock + +import ( + "encoding/json" + "time" + + "github.com/Axway/agent-sdk/pkg/transaction" + "github.com/elastic/beats/v7/libbeat/beat" + "github.com/elastic/beats/v7/libbeat/common" +) + +// EventGeneratorMock - mock event generator +type EventGeneratorMock struct { + shouldUseTrafficForAggregation bool +} + +// NewEventGeneratorMock - Create a new mock event generator +func NewEventGeneratorMock() transaction.EventGenerator { + return &EventGeneratorMock{} +} + +// CreateEvent - Creates a new mocked event for tests +func (c *EventGeneratorMock) CreateEvent(logEvent transaction.LogEvent, eventTime time.Time, metaData, eventFields common.MapStr, privateData interface{}) (event beat.Event, err error) { + serializedLogEvent, _ := json.Marshal(logEvent) + eventData := make(map[string]interface{}) + eventData["message"] = string(serializedLogEvent) + event = beat.Event{ + Timestamp: eventTime, + Meta: metaData, + Private: privateData, + Fields: eventData, + } + return +} + +// CreateEvents - Creates a new mocked event for tests +func (c *EventGeneratorMock) CreateEvents(summaryEvent transaction.LogEvent, detailEvents []transaction.LogEvent, eventTime time.Time, metaData, eventFields common.MapStr, privateData interface{}) ([]beat.Event, error) { + serializedSumEvent, _ := json.Marshal(summaryEvent) + sumEventData := make(map[string]interface{}) + sumEventData["message"] = string(serializedSumEvent) + events := []beat.Event{ + { + Timestamp: eventTime, + Meta: metaData, + Private: privateData, + Fields: sumEventData, + }, + } + + for _, detailEvent := range detailEvents { + serializedEvent, _ := json.Marshal(detailEvent) + eventData := make(map[string]interface{}) + eventData["message"] = string(serializedEvent) + events = append(events, beat.Event{ + Timestamp: eventTime, + Meta: metaData, + Private: privateData, + Fields: eventData, + }) + } + + return events, nil +} + +// SetUseTrafficForAggregation - set the flag to use traffic events for aggregation. +func (c *EventGeneratorMock) SetUseTrafficForAggregation(useTrafficForAggregation bool) { + c.shouldUseTrafficForAggregation = useTrafficForAggregation +} diff --git a/pkg/traceability/processor/processor.go b/pkg/traceability/processor/processor.go deleted file mode 100644 index 858e2f8..0000000 --- a/pkg/traceability/processor/processor.go +++ /dev/null @@ -1,170 +0,0 @@ -package processor - -import ( - "bytes" - "context" - "encoding/json" - "fmt" - "net/http" - "strconv" - "time" - - "github.com/elastic/beats/v7/libbeat/beat" - "github.com/google/uuid" - - "github.com/Axway/agent-sdk/pkg/agent" - "github.com/Axway/agent-sdk/pkg/transaction" - sdkUtil "github.com/Axway/agent-sdk/pkg/transaction/util" - "github.com/Axway/agent-sdk/pkg/util/log" -) - -const ( - host = "host" - userAgent = "user-agent" - leg0 = "leg0" - inbound = "inbound" -) - -// TransactionProcessor - -type TransactionProcessor struct { - ctx context.Context - logger log.FieldLogger - eventGenerator transaction.EventGenerator - event TrafficLogEntry -} - -func NewTransactionProcessor(ctx context.Context, entry TrafficLogEntry) (*TransactionProcessor, bool) { - p := &TransactionProcessor{ - ctx: ctx, - logger: log.NewLoggerFromContext(ctx).WithComponent("eventMapper").WithPackage("processor"), - eventGenerator: transaction.NewEventGenerator(), - event: entry, - } - - return p, true -} - -func (m *TransactionProcessor) process() ([]beat.Event, error) { - centralCfg := agent.GetCentralConfig() - txnID := uuid.New().String() - - // leg 0 - transactionLogEvent, err := m.createTransactionEvent(m.event, txnID) - if err != nil { - m.logger.WithError(err).Error("building transaction leg event") - return nil, err - } - - // summary - summaryLogEvent, err := m.createSummaryEvent(m.event, centralCfg.GetTeamID(), txnID) - if err != nil { - m.logger.WithError(err).Error("building transaction summary event") - return nil, err - } - - // create Central log events - events, err := m.eventGenerator.CreateEvents(*summaryLogEvent, []transaction.LogEvent{*transactionLogEvent}, time.Unix(m.event.StartedAt, 0), nil, nil, nil) - if err != nil { - m.logger.WithError(err).Error("building Central events") - return nil, err - } - - return events, nil -} - -func (m *TransactionProcessor) getTransactionEventStatus(code int) transaction.TxEventStatus { - if code >= 400 { - return transaction.TxEventStatusFail - } - return transaction.TxEventStatusPass -} - -func (m *TransactionProcessor) getTransactionSummaryStatus(statusCode int) transaction.TxSummaryStatus { - transSummaryStatus := transaction.TxSummaryStatusUnknown - if statusCode >= http.StatusOK && statusCode < http.StatusBadRequest { - transSummaryStatus = transaction.TxSummaryStatusSuccess - } else if statusCode >= http.StatusBadRequest && statusCode < http.StatusInternalServerError { - transSummaryStatus = transaction.TxSummaryStatusFailure - } else if statusCode >= http.StatusInternalServerError && statusCode < http.StatusNetworkAuthenticationRequired { - transSummaryStatus = transaction.TxSummaryStatusException - } - return transSummaryStatus -} - -func (m *TransactionProcessor) buildHeaders(headers map[string]string) string { - jsonHeader, err := json.Marshal(headers) - if err != nil { - log.Error(err.Error()) - } - - return string(jsonHeader) -} - -func (m *TransactionProcessor) buildSSLInfoIfAvailable(ktle TrafficLogEntry) (string, string, string) { - if ktle.Request.TLS != nil { - return ktle.Request.TLS.Version, - ktle.Request.URL, - ktle.Request.URL // Using SSL server name as SSL subject name for now - } - return "", "", "" -} - -func (m *TransactionProcessor) processQueryArgs(args map[string]string) string { - b := new(bytes.Buffer) - for key, value := range args { - fmt.Fprintf(b, "%s=\"%s\",", key, value) - } - return b.String() -} - -func (m *TransactionProcessor) createTransactionEvent(ktle TrafficLogEntry, txnid string) (*transaction.LogEvent, error) { - - httpProtocolDetails, err := transaction.NewHTTPProtocolBuilder(). - SetURI(ktle.Request.URI). - SetMethod(ktle.Request.Method). - SetArgs(m.processQueryArgs(ktle.Request.QueryString)). - SetStatus(ktle.Response.Status, http.StatusText(ktle.Response.Status)). - SetHost(ktle.Request.Headers[host]). - SetHeaders(m.buildHeaders(ktle.Request.Headers), m.buildHeaders(ktle.Response.Headers)). - SetByteLength(ktle.Request.Size, ktle.Response.Size). - SetLocalAddress(ktle.ClientIP, 0). // Could not determine local port for now - SetRemoteAddress("", "", ktle.Service.Port). - SetSSLProperties(m.buildSSLInfoIfAvailable(ktle)). - SetUserAgent(ktle.Request.Headers[userAgent]). - Build() - - if err != nil { - log.Errorf("Error while filling protocol details for transaction event: %s", err) - return nil, err - } - - return transaction.NewTransactionEventBuilder(). - SetTimestamp(ktle.StartedAt). - SetTransactionID(txnid). - SetID(leg0). - SetSource(ktle.ClientIP). - SetDestination(ktle.Request.Headers[host]). - SetDuration(ktle.Latencies.Request). - SetDirection(inbound). - SetStatus(m.getTransactionEventStatus(ktle.Response.Status)). - SetProtocolDetail(httpProtocolDetails). - Build() -} - -func (m *TransactionProcessor) createSummaryEvent(ktle TrafficLogEntry, teamID string, txnid string) (*transaction.LogEvent, error) { - - builder := transaction.NewTransactionSummaryBuilder(). - SetTimestamp(ktle.StartedAt). - SetTransactionID(txnid). - SetStatus(m.getTransactionSummaryStatus(ktle.Response.Status), strconv.Itoa(ktle.Response.Status)). - SetTeam(teamID). - SetEntryPoint(ktle.Service.Protocol, ktle.Request.Method, ktle.Request.URI, ktle.Request.URL). - SetDuration(ktle.Latencies.Request). - SetProxy(sdkUtil.FormatProxyID(ktle.Route.ID), ktle.Service.Name, 1) - - if ktle.Consumer != nil { - builder.SetApplication(sdkUtil.FormatApplicationID(ktle.Consumer.ID), ktle.Consumer.Username) - } - - return builder.Build() -} diff --git a/pkg/traceability/processor/processor_test.go b/pkg/traceability/processor/processor_test.go new file mode 100644 index 0000000..3da1e35 --- /dev/null +++ b/pkg/traceability/processor/processor_test.go @@ -0,0 +1,111 @@ +package processor + +import ( + "context" + "testing" + + "github.com/Axway/agent-sdk/pkg/traceability/redaction" + "github.com/Axway/agent-sdk/pkg/traceability/sampling" + "github.com/Axway/agent-sdk/pkg/transaction/metric" + "github.com/Axway/agents-kong/pkg/traceability/processor/mock" + "github.com/stretchr/testify/assert" +) + +var testData = []byte(`[{ + "service": {"host": "httpbin.org","created_at": 1614232642,"connect_timeout": 60000,"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260","protocol": "http","read_timeout": 60000,"port": 80,"path": "/anything","updated_at": 1614232642,"write_timeout": 60000,"retries": 5,"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25"}, + "route": {"id": "78f79740-c410-4fd9-a998-d0a60a99dc9b","paths": ["/log"],"protocols": ["http"],"strip_path": true,"created_at": 1614232648,"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25","request_buffering": true,"updated_at": 1614232648,"preserve_host": false,"regex_priority": 0,"response_buffering": true,"https_redirect_status_code": 426,"path_handling": "v0","service": {"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260"}}, + "request": {"querystring": {},"size": 138,"uri": "/log","url": "http://localhost:8000/log","headers": {"host": "localhost:8000","accept-encoding": "gzip, deflate","user-agent": "HTTPie/2.4.0","accept": "*/*","connection": "keep-alive"},"method": "GET"}, + "response": {"headers": {"content-type": "application/json","date": "Thu, 25 Feb 2021 05:57:48 GMT","connection": "close","access-control-allow-credentials": "true","content-length": "503","server": "gunicorn/19.9.0","via": "kong/2.2.1.0-enterprise-edition","x-kong-proxy-latency": "57","x-kong-upstream-latency": "457","access-control-allow-origin": "*"},"status": 200,"size": 827}, + "latencies": {"request": 515,"kong": 58,"proxy": 457}, + "tries": [{"balancer_latency": 0,"port": 80,"balancer_start": 1614232668399,"ip": "18.211.130.98"}], + "client_ip": "192.168.144.1", + "workspace": "54baa5a9-23d6-41e0-9c9a-02434b010b25", + "workspace_name": "default", + "upstream_uri": "/anything", + "authenticated_entity": {"id": "c62c1455-9b1d-4f2d-8797-509ba83b8ae8"}, + "consumer": {"id": "ae974d6c-0f8a-4dc5-b701-fa0aa38592bd","created_at": 1674035962,"username_lower": "foo","username": "foo","type": 0}, + "started_at": 1614232668342 +},{ + "service": {"host": "httpbin.org","created_at": 1614232642,"connect_timeout": 60000,"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260","protocol": "http","read_timeout": 60000,"port": 80,"path": "/anything","updated_at": 1614232642,"write_timeout": 60000,"retries": 5,"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25"}, + "route": {"id": "78f79740-c410-4fd9-a998-d0a60a99dc9b","paths": ["/log"],"protocols": ["http"],"strip_path": true,"created_at": 1614232648,"ws_id": "54baa5a9-23d6-41e0-9c9a-02434b010b25","request_buffering": true,"updated_at": 1614232648,"preserve_host": false,"regex_priority": 0,"response_buffering": true,"https_redirect_status_code": 426,"path_handling": "v0","service": {"id": "167290ee-c682-4ebf-bdea-e49a3ac5e260"}}, + "request": {"querystring": {},"size": 138,"uri": "/log","url": "http://localhost:8000/log","headers": {"host": "localhost:8000","accept-encoding": "gzip, deflate","user-agent": "HTTPie/2.4.0","accept": "*/*","connection": "keep-alive"},"method": "GET"}, + "response": {"headers": {"content-type": "application/json","date": "Thu, 25 Feb 2021 05:57:48 GMT","connection": "close","access-control-allow-credentials": "true","content-length": "503","server": "gunicorn/19.9.0","via": "kong/2.2.1.0-enterprise-edition","x-kong-proxy-latency": "57","x-kong-upstream-latency": "457","access-control-allow-origin": "*"},"status": 200,"size": 827}, + "latencies": {"request": 515,"kong": 58,"proxy": 457}, + "tries": [{"balancer_latency": 0,"port": 80,"balancer_start": 1614232668399,"ip": "18.211.130.98"}], + "client_ip": "192.168.144.1", + "workspace": "54baa5a9-23d6-41e0-9c9a-02434b010b25", + "workspace_name": "default", + "upstream_uri": "/anything", + "authenticated_entity": {"id": "c62c1455-9b1d-4f2d-8797-509ba83b8ae8"}, + "consumer": {"id": "ae974d6c-0f8a-4dc5-b701-fa0aa38592bd","created_at": 1674035962,"username_lower": "foo","username": "foo","type": 0}, + "started_at": 1614232668342 +}]`) + +func TestNewHandler(t *testing.T) { + cases := map[string]struct { + data []byte + constructorErr bool + setupSampling bool + expectedEvents int + expectedMetricDetails int + }{ + "expect error creating handler, when no data sent into handler": { + data: []byte{}, + constructorErr: true, + }, + "expect no error when empty array data sent into handler": { + data: []byte("[]"), + }, + "handle data without sampling setup": { + data: testData, + }, + "handle data with sampling setup": { + data: testData, + setupSampling: true, + expectedEvents: 4, + expectedMetricDetails: 2, + }, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + ctx := context.WithValue(context.Background(), "test", name) + + redaction.SetupGlobalRedaction(redaction.DefaultConfig()) + if tc.setupSampling { + sampling.SetupSampling(sampling.DefaultConfig(), false) + } + + // create the handler + h, err := NewEventsHandler(ctx, tc.data) + if tc.constructorErr { + assert.NotNil(t, err) + assert.Nil(t, h) + return + } + assert.Nil(t, err) + assert.NotNil(t, h) + + // setup collector + collector := &mock.CollectorMock{Details: make([]metric.Detail, 0)} + mock.SetMockCollector(collector) + h.collectorGetter = func() metricCollector { + return mock.GetMockCollector() + } + + // setup event generator + h.eventGenerator = mock.NewEventGeneratorMock + + // if metric details are expected + if tc.expectedMetricDetails > 1 { + collector.Add(tc.expectedMetricDetails) + } + + // execute the handler + events := h.Handle() + collector.Wait() + assert.Nil(t, err) + assert.Len(t, events, tc.expectedEvents) + assert.Equal(t, tc.expectedMetricDetails, len(mock.GetMockCollector().Details)) + }) + } +} diff --git a/pkg/traceability/processor/transaction.go b/pkg/traceability/processor/transaction.go new file mode 100644 index 0000000..7ec8fcc --- /dev/null +++ b/pkg/traceability/processor/transaction.go @@ -0,0 +1,142 @@ +package processor + +import ( + "context" + "fmt" + "net/http" + "strconv" + "time" + + "github.com/elastic/beats/v7/libbeat/beat" + "github.com/google/uuid" + + "github.com/Axway/agent-sdk/pkg/transaction" + sdkUtil "github.com/Axway/agent-sdk/pkg/transaction/util" + "github.com/Axway/agent-sdk/pkg/util/log" +) + +const ( + host = "host" + userAgent = "user-agent" + leg0 = "leg0" + inbound = "inbound" +) + +// TransactionProcessor - +type TransactionProcessor struct { + ctx context.Context + logger log.FieldLogger + eventGenerator transaction.EventGenerator + event TrafficLogEntry +} + +func NewTransactionProcessor(ctx context.Context) *TransactionProcessor { + p := &TransactionProcessor{ + ctx: ctx, + logger: log.NewLoggerFromContext(ctx).WithComponent("eventMapper").WithPackage("processor"), + } + return p +} + +func (p *TransactionProcessor) setEntry(entry TrafficLogEntry) *TransactionProcessor { + p.event = entry + return p +} + +func (p *TransactionProcessor) setEventGenerator(eventGenerator transaction.EventGenerator) *TransactionProcessor { + p.eventGenerator = eventGenerator + p.eventGenerator.SetUseTrafficForAggregation(false) + return p +} + +func (p *TransactionProcessor) process() ([]beat.Event, error) { + if p.eventGenerator == nil { + return nil, fmt.Errorf("an event generator is required") + } + txnID := uuid.New().String() + + // leg 0 + transactionLogEvent, err := createTransactionEvent(p.event, txnID) + if err != nil { + p.logger.WithError(err).Error("building transaction leg event") + return nil, err + } + legEvent, err := p.eventGenerator.CreateEvent(*transactionLogEvent, time.Unix(p.event.StartedAt, 0), nil, nil, nil) + if err != nil { + p.logger.WithError(err).Error("creating transaction leg event") + return nil, err + } + + // summary + summaryLogEvent, err := createSummaryEvent(p.event, "id", txnID) + if err != nil { + p.logger.WithError(err).Error("building transaction summary event") + return nil, err + } + summaryEvent, err := p.eventGenerator.CreateEvent(*summaryLogEvent, time.Unix(p.event.StartedAt, 0), nil, nil, nil) + if err != nil { + p.logger.WithError(err).Error("creating transaction summary event") + return nil, err + } + + return []beat.Event{summaryEvent, legEvent}, nil +} + +func createTransactionEvent(ktle TrafficLogEntry, txnid string) (*transaction.LogEvent, error) { + requestHost := "" + if value, found := ktle.Request.Headers[host]; found { + requestHost = fmt.Sprintf("%v", value) + } + + userAgentVal := "" + if value, found := ktle.Request.Headers[userAgent]; found { + userAgentVal = fmt.Sprintf("%v", value) + } + + httpProtocolDetails, err := transaction.NewHTTPProtocolBuilder(). + SetURI(ktle.Request.URI). + SetMethod(ktle.Request.Method). + SetArgs(processQueryArgs(ktle.Request.QueryString)). + SetStatus(ktle.Response.Status, http.StatusText(ktle.Response.Status)). + SetHost(requestHost). + SetHeaders(buildHeaders(ktle.Request.Headers), buildHeaders(ktle.Response.Headers)). + SetByteLength(ktle.Request.Size, ktle.Response.Size). + SetLocalAddress(ktle.ClientIP, 0). // Could not determine local port for now + SetRemoteAddress("", "", ktle.Service.Port). + SetSSLProperties(buildSSLInfoIfAvailable(ktle)). + SetUserAgent(userAgentVal). + Build() + + if err != nil { + return nil, err + } + + return transaction.NewTransactionEventBuilder(). + SetTimestamp(ktle.StartedAt). + SetTransactionID(txnid). + SetID(leg0). + SetSource(ktle.ClientIP). + SetDestination(requestHost). + SetDuration(ktle.Latencies.Request). + SetDirection(inbound). + SetStatus(getTransactionEventStatus(ktle.Response.Status)). + SetProtocolDetail(httpProtocolDetails). + Build() +} + +func createSummaryEvent(ktle TrafficLogEntry, teamID string, txnid string) (*transaction.LogEvent, error) { + builder := transaction.NewTransactionSummaryBuilder(). + SetTimestamp(ktle.StartedAt). + SetTransactionID(txnid). + SetStatus(getTransactionSummaryStatus(ktle.Response.Status), strconv.Itoa(ktle.Response.Status)). + SetTeam(teamID). + SetEntryPoint(ktle.Service.Protocol, ktle.Request.Method, ktle.Request.URI, ktle.Request.URL). + SetDuration(ktle.Latencies.Request). + SetProxyWithStage(sdkUtil.FormatProxyID(ktle.Route.ID), ktle.Service.Name, ktle.Route.Name, 1) + + if ktle.Consumer != nil { + builder.SetApplication(sdkUtil.FormatApplicationID(ktle.Consumer.ID), ktle.Consumer.Username) + } + + return builder.Build() +} diff --git a/pkg/traceability/processor/util.go b/pkg/traceability/processor/util.go new file mode 100644 index 0000000..8c72c37 --- /dev/null +++ b/pkg/traceability/processor/util.go @@ -0,0 +1,65 @@ +package processor + +import ( + "bytes" + "encoding/json" + "fmt" + "net/http" + + "github.com/Axway/agent-sdk/pkg/transaction" + "github.com/Axway/agent-sdk/pkg/transaction/metric" + "github.com/Axway/agent-sdk/pkg/util/log" +) + +func getTransactionEventStatus(code int) transaction.TxEventStatus { + if code >= 400 { + return transaction.TxEventStatusFail + } + return transaction.TxEventStatusPass +} + +func getTransactionSummaryStatus(statusCode int) transaction.TxSummaryStatus { + transSummaryStatus := transaction.TxSummaryStatusUnknown + if statusCode >= http.StatusOK && statusCode < http.StatusBadRequest { + transSummaryStatus = transaction.TxSummaryStatusSuccess + } else if statusCode >= http.StatusBadRequest && statusCode < http.StatusInternalServerError { + transSummaryStatus = transaction.TxSummaryStatusFailure + } else if statusCode >= http.StatusInternalServerError && statusCode < http.StatusNetworkAuthenticationRequired { + transSummaryStatus = transaction.TxSummaryStatusException + } + return transSummaryStatus +} + +func buildHeaders(headers map[string]interface{}) string { + newHeaders := make(map[string]string) + for key, val := range headers { + newHeaders[key] = fmt.Sprintf("%v", val) + } + + jsonHeader, err := json.Marshal(newHeaders) + if err != nil { + log.Error(err.Error()) + } + return string(jsonHeader) +} + +func buildSSLInfoIfAvailable(ktle TrafficLogEntry) (string, string, string) { + if ktle.Request.TLS != nil { + return ktle.Request.TLS.Version, + ktle.Request.URL, + ktle.Request.URL // Using SSL server name as SSL subject name for now + } + return "", "", "" +} + +func processQueryArgs(args map[string]string) string { + b := new(bytes.Buffer) + for key, value := range args { + fmt.Fprintf(b, "%s=\"%s\",", key, value) + } + return b.String() +} + +func getMetricCollector() metricCollector { + return metric.GetMetricCollector() +} diff --git a/png/KongAgentHTTP.png b/png/KongAgentHTTP.png deleted file mode 100644 index f9298fc..0000000 Binary files a/png/KongAgentHTTP.png and /dev/null differ diff --git a/png/KongFilePlugin.png b/png/KongFilePlugin.png deleted file mode 100644 index 919fa76..0000000 Binary files a/png/KongFilePlugin.png and /dev/null differ diff --git a/png/KongHTTPPlugin.png b/png/KongHTTPPlugin.png deleted file mode 100644 index 881457f..0000000 Binary files a/png/KongHTTPPlugin.png and /dev/null differ diff --git a/specs/petstore.json b/specs/petstore.json deleted file mode 100644 index c00d4bf..0000000 --- a/specs/petstore.json +++ /dev/null @@ -1,1054 +0,0 @@ -{ - "swagger": "2.0", - "info": { - "description": "This is a sample server Petstore server. You can find out more about Swagger at [http://swagger.io](http://swagger.io) or on [irc.freenode.net, #swagger](http://swagger.io/irc/). For this sample, you can use the api key `special-key` to test the authorization filters.", - "version": "1.0.5", - "title": "Swagger Petstore", - "termsOfService": "http://swagger.io/terms/", - "contact": { - "email": "apiteam@swagger.io" - }, - "license": { - "name": "Apache 2.0", - "url": "http://www.apache.org/licenses/LICENSE-2.0.html" - } - }, - "host": "petstore.swagger.io", - "basePath": "/v2", - "tags": [ - { - "name": "pet", - "description": "Everything about your Pets", - "externalDocs": { - "description": "Find out more", - "url": "http://swagger.io" - } - }, - { - "name": "store", - "description": "Access to Petstore orders" - }, - { - "name": "user", - "description": "Operations about user", - "externalDocs": { - "description": "Find out more about our store", - "url": "http://swagger.io" - } - } - ], - "schemes": [ - "https", - "http" - ], - "paths": { - "/pet/{petId}/uploadImage": { - "post": { - "tags": [ - "pet" - ], - "summary": "uploads an image", - "description": "", - "operationId": "uploadFile", - "consumes": [ - "multipart/form-data" - ], - "produces": [ - "application/json" - ], - "parameters": [ - { - "name": "petId", - "in": "path", - "description": "ID of pet to update", - "required": true, - "type": "integer", - "format": "int64" - }, - { - "name": "additionalMetadata", - "in": "formData", - "description": "Additional data to pass to server", - "required": false, - "type": "string" - }, - { - "name": "file", - "in": "formData", - "description": "file to upload", - "required": false, - "type": "file" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "$ref": "#/definitions/ApiResponse" - } - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - } - }, - "/pet": { - "post": { - "tags": [ - "pet" - ], - "summary": "Add a new pet to the store", - "description": "", - "operationId": "addPet", - "consumes": [ - "application/json", - "application/xml" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "Pet object that needs to be added to the store", - "required": true, - "schema": { - "$ref": "#/definitions/Pet" - } - } - ], - "responses": { - "405": { - "description": "Invalid input" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - }, - "put": { - "tags": [ - "pet" - ], - "summary": "Update an existing pet", - "description": "", - "operationId": "updatePet", - "consumes": [ - "application/json", - "application/xml" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "Pet object that needs to be added to the store", - "required": true, - "schema": { - "$ref": "#/definitions/Pet" - } - } - ], - "responses": { - "400": { - "description": "Invalid ID supplied" - }, - "404": { - "description": "Pet not found" - }, - "405": { - "description": "Validation exception" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - } - }, - "/pet/findByStatus": { - "get": { - "tags": [ - "pet" - ], - "summary": "Finds Pets by status", - "description": "Multiple status values can be provided with comma separated strings", - "operationId": "findPetsByStatus", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "status", - "in": "query", - "description": "Status values that need to be considered for filter", - "required": true, - "type": "array", - "items": { - "type": "string", - "enum": [ - "available", - "pending", - "sold" - ], - "default": "available" - }, - "collectionFormat": "multi" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "type": "array", - "items": { - "$ref": "#/definitions/Pet" - } - } - }, - "400": { - "description": "Invalid status value" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - } - }, - "/pet/findByTags": { - "get": { - "tags": [ - "pet" - ], - "summary": "Finds Pets by tags", - "description": "Multiple tags can be provided with comma separated strings. Use tag1, tag2, tag3 for testing.", - "operationId": "findPetsByTags", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "tags", - "in": "query", - "description": "Tags to filter by", - "required": true, - "type": "array", - "items": { - "type": "string" - }, - "collectionFormat": "multi" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "type": "array", - "items": { - "$ref": "#/definitions/Pet" - } - } - }, - "400": { - "description": "Invalid tag value" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ], - "deprecated": true - } - }, - "/pet/{petId}": { - "get": { - "tags": [ - "pet" - ], - "summary": "Find pet by ID", - "description": "Returns a single pet", - "operationId": "getPetById", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "petId", - "in": "path", - "description": "ID of pet to return", - "required": true, - "type": "integer", - "format": "int64" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "$ref": "#/definitions/Pet" - } - }, - "400": { - "description": "Invalid ID supplied" - }, - "404": { - "description": "Pet not found" - } - }, - "security": [ - { - "api_key": [] - } - ] - }, - "post": { - "tags": [ - "pet" - ], - "summary": "Updates a pet in the store with form data", - "description": "", - "operationId": "updatePetWithForm", - "consumes": [ - "application/x-www-form-urlencoded" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "petId", - "in": "path", - "description": "ID of pet that needs to be updated", - "required": true, - "type": "integer", - "format": "int64" - }, - { - "name": "name", - "in": "formData", - "description": "Updated name of the pet", - "required": false, - "type": "string" - }, - { - "name": "status", - "in": "formData", - "description": "Updated status of the pet", - "required": false, - "type": "string" - } - ], - "responses": { - "405": { - "description": "Invalid input" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - }, - "delete": { - "tags": [ - "pet" - ], - "summary": "Deletes a pet", - "description": "", - "operationId": "deletePet", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "api_key", - "in": "header", - "required": false, - "type": "string" - }, - { - "name": "petId", - "in": "path", - "description": "Pet id to delete", - "required": true, - "type": "integer", - "format": "int64" - } - ], - "responses": { - "400": { - "description": "Invalid ID supplied" - }, - "404": { - "description": "Pet not found" - } - }, - "security": [ - { - "petstore_auth": [ - "write:pets", - "read:pets" - ] - } - ] - } - }, - "/store/order": { - "post": { - "tags": [ - "store" - ], - "summary": "Place an order for a pet", - "description": "", - "operationId": "placeOrder", - "consumes": [ - "application/json" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "order placed for purchasing the pet", - "required": true, - "schema": { - "$ref": "#/definitions/Order" - } - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "$ref": "#/definitions/Order" - } - }, - "400": { - "description": "Invalid Order" - } - } - } - }, - "/store/order/{orderId}": { - "get": { - "tags": [ - "store" - ], - "summary": "Find purchase order by ID", - "description": "For valid response try integer IDs with value >= 1 and <= 10. Other values will generated exceptions", - "operationId": "getOrderById", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "orderId", - "in": "path", - "description": "ID of pet that needs to be fetched", - "required": true, - "type": "integer", - "maximum": 10, - "minimum": 1, - "format": "int64" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "$ref": "#/definitions/Order" - } - }, - "400": { - "description": "Invalid ID supplied" - }, - "404": { - "description": "Order not found" - } - } - }, - "delete": { - "tags": [ - "store" - ], - "summary": "Delete purchase order by ID", - "description": "For valid response try integer IDs with positive integer value. Negative or non-integer values will generate API errors", - "operationId": "deleteOrder", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "orderId", - "in": "path", - "description": "ID of the order that needs to be deleted", - "required": true, - "type": "integer", - "minimum": 1, - "format": "int64" - } - ], - "responses": { - "400": { - "description": "Invalid ID supplied" - }, - "404": { - "description": "Order not found" - } - } - } - }, - "/store/inventory": { - "get": { - "tags": [ - "store" - ], - "summary": "Returns pet inventories by status", - "description": "Returns a map of status codes to quantities", - "operationId": "getInventory", - "produces": [ - "application/json" - ], - "parameters": [], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "type": "object", - "additionalProperties": { - "type": "integer", - "format": "int32" - } - } - } - }, - "security": [ - { - "api_key": [] - } - ] - } - }, - "/user/createWithArray": { - "post": { - "tags": [ - "user" - ], - "summary": "Creates list of users with given input array", - "description": "", - "operationId": "createUsersWithArrayInput", - "consumes": [ - "application/json" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "List of user object", - "required": true, - "schema": { - "type": "array", - "items": { - "$ref": "#/definitions/User" - } - } - } - ], - "responses": { - "default": { - "description": "successful operation" - } - } - } - }, - "/user/createWithList": { - "post": { - "tags": [ - "user" - ], - "summary": "Creates list of users with given input array", - "description": "", - "operationId": "createUsersWithListInput", - "consumes": [ - "application/json" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "List of user object", - "required": true, - "schema": { - "type": "array", - "items": { - "$ref": "#/definitions/User" - } - } - } - ], - "responses": { - "default": { - "description": "successful operation" - } - } - } - }, - "/user/{username}": { - "get": { - "tags": [ - "user" - ], - "summary": "Get user by user name", - "description": "", - "operationId": "getUserByName", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "username", - "in": "path", - "description": "The name that needs to be fetched. Use user1 for testing. ", - "required": true, - "type": "string" - } - ], - "responses": { - "200": { - "description": "successful operation", - "schema": { - "$ref": "#/definitions/User" - } - }, - "400": { - "description": "Invalid username supplied" - }, - "404": { - "description": "User not found" - } - } - }, - "put": { - "tags": [ - "user" - ], - "summary": "Updated user", - "description": "This can only be done by the logged in user.", - "operationId": "updateUser", - "consumes": [ - "application/json" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "username", - "in": "path", - "description": "name that need to be updated", - "required": true, - "type": "string" - }, - { - "in": "body", - "name": "body", - "description": "Updated user object", - "required": true, - "schema": { - "$ref": "#/definitions/User" - } - } - ], - "responses": { - "400": { - "description": "Invalid user supplied" - }, - "404": { - "description": "User not found" - } - } - }, - "delete": { - "tags": [ - "user" - ], - "summary": "Delete user", - "description": "This can only be done by the logged in user.", - "operationId": "deleteUser", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "username", - "in": "path", - "description": "The name that needs to be deleted", - "required": true, - "type": "string" - } - ], - "responses": { - "400": { - "description": "Invalid username supplied" - }, - "404": { - "description": "User not found" - } - } - } - }, - "/user/login": { - "get": { - "tags": [ - "user" - ], - "summary": "Logs user into the system", - "description": "", - "operationId": "loginUser", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "name": "username", - "in": "query", - "description": "The user name for login", - "required": true, - "type": "string" - }, - { - "name": "password", - "in": "query", - "description": "The password for login in clear text", - "required": true, - "type": "string" - } - ], - "responses": { - "200": { - "description": "successful operation", - "headers": { - "X-Expires-After": { - "type": "string", - "format": "date-time", - "description": "date in UTC when token expires" - }, - "X-Rate-Limit": { - "type": "integer", - "format": "int32", - "description": "calls per hour allowed by the user" - } - }, - "schema": { - "type": "string" - } - }, - "400": { - "description": "Invalid username/password supplied" - } - } - } - }, - "/user/logout": { - "get": { - "tags": [ - "user" - ], - "summary": "Logs out current logged in user session", - "description": "", - "operationId": "logoutUser", - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [], - "responses": { - "default": { - "description": "successful operation" - } - } - } - }, - "/user": { - "post": { - "tags": [ - "user" - ], - "summary": "Create user", - "description": "This can only be done by the logged in user.", - "operationId": "createUser", - "consumes": [ - "application/json" - ], - "produces": [ - "application/json", - "application/xml" - ], - "parameters": [ - { - "in": "body", - "name": "body", - "description": "Created user object", - "required": true, - "schema": { - "$ref": "#/definitions/User" - } - } - ], - "responses": { - "default": { - "description": "successful operation" - } - } - } - } - }, - "securityDefinitions": { - "api_key": { - "type": "apiKey", - "name": "api_key", - "in": "header" - }, - "petstore_auth": { - "type": "oauth2", - "authorizationUrl": "https://petstore.swagger.io/oauth/authorize", - "flow": "implicit", - "scopes": { - "read:pets": "read your pets", - "write:pets": "modify pets in your account" - } - } - }, - "definitions": { - "ApiResponse": { - "type": "object", - "properties": { - "code": { - "type": "integer", - "format": "int32" - }, - "type": { - "type": "string" - }, - "message": { - "type": "string" - } - } - }, - "Category": { - "type": "object", - "properties": { - "id": { - "type": "integer", - "format": "int64" - }, - "name": { - "type": "string" - } - }, - "xml": { - "name": "Category" - } - }, - "Pet": { - "type": "object", - "required": [ - "name", - "photoUrls" - ], - "properties": { - "id": { - "type": "integer", - "format": "int64" - }, - "category": { - "$ref": "#/definitions/Category" - }, - "name": { - "type": "string", - "example": "doggie" - }, - "photoUrls": { - "type": "array", - "xml": { - "wrapped": true - }, - "items": { - "type": "string", - "xml": { - "name": "photoUrl" - } - } - }, - "tags": { - "type": "array", - "xml": { - "wrapped": true - }, - "items": { - "xml": { - "name": "tag" - }, - "$ref": "#/definitions/Tag" - } - }, - "status": { - "type": "string", - "description": "pet status in the store", - "enum": [ - "available", - "pending", - "sold" - ] - } - }, - "xml": { - "name": "Pet" - } - }, - "Tag": { - "type": "object", - "properties": { - "id": { - "type": "integer", - "format": "int64" - }, - "name": { - "type": "string" - } - }, - "xml": { - "name": "Tag" - } - }, - "Order": { - "type": "object", - "properties": { - "id": { - "type": "integer", - "format": "int64" - }, - "petId": { - "type": "integer", - "format": "int64" - }, - "quantity": { - "type": "integer", - "format": "int32" - }, - "shipDate": { - "type": "string", - "format": "date-time" - }, - "status": { - "type": "string", - "description": "Order Status", - "enum": [ - "placed", - "approved", - "delivered" - ] - }, - "complete": { - "type": "boolean" - } - }, - "xml": { - "name": "Order" - } - }, - "User": { - "type": "object", - "properties": { - "id": { - "type": "integer", - "format": "int64" - }, - "username": { - "type": "string" - }, - "firstName": { - "type": "string" - }, - "lastName": { - "type": "string" - }, - "email": { - "type": "string" - }, - "password": { - "type": "string" - }, - "phone": { - "type": "string" - }, - "userStatus": { - "type": "integer", - "format": "int32", - "description": "User Status" - } - }, - "xml": { - "name": "User" - } - } - }, - "externalDocs": { - "description": "Find out more about Swagger", - "url": "http://swagger.io" - } -} \ No newline at end of file