diff --git a/docs/HTTP-action.md b/docs/HTTP-action.md
new file mode 100644
index 0000000..3073d02
--- /dev/null
+++ b/docs/HTTP-action.md
@@ -0,0 +1,119 @@
+# HTTP Action
+
+Description
+-----------
+This plugin executing HTTP/HTTPS requests
+
+Properties
+----------
+
+### General
+
+**URL:** Url to fetch to the first page.
+The url must start with a protocol (e.g. http://).
+
+**HTTP Method:** HTTP request method.
+
+**Headers:** Headers to send with each HTTP request.
+
+**Request body:** Body to send with each HTTP request.
+
+
+### Basic Authentication
+
+**Username:** Username for basic authentication.
+
+**Password:** Password for basic authentication.
+
+### HTTP Proxy
+
+**Proxy URL:** Proxy URL. Must contain a protocol, address and port.
+
+**Username:** Proxy username.
+
+**Password:** Proxy password.
+
+### Error Handling
+
+**HTTP Errors Handling:** Defines the error handling strategy to use for certain HTTP response codes.
+The left column contains a regular expression for HTTP status code. The right column contains an action which
+is done in case of match. If HTTP status code matches multiple regular expressions, the first specified in mapping
+is matched.
+
+Example:
+
+| HTTP Code Regexp | Error Handling |
+| ----------------- |:-----------------------:|
+| 2.. | Success |
+| 401 | Retry and fail |
+| 4.. | Fail |
+| 5.. | Retry and send to error |
+| .* | Fail |
+
+Note: pagination types "Link in response header", "Link in response body", "Token in response body" do not support
+"Send to error", "Skip", "Retry and send to error", "Retry and skip" options.
+
+**Non-HTTP Error Handling:** Error handling strategy to use when the HTTP response cannot be transformed to an output record.
+Possible values are:
+Stop on error - Fails pipeline due to erroneous record.
+Send to error - Sends erroneous record's text to error port and continues.
+Skip on error - Ignores erroneous records.
+
+**Connect Timeout:** Maximum time in seconds connection initialization is allowed to take.
+
+### OAuth2
+
+**OAuth2 Enabled:** If true, plugin will perform OAuth2 authentication.
+
+**Auth URL:** Endpoint for the authorization server used to retrieve the authorization code.
+
+**Token URL:** Endpoint for the resource server, which exchanges the authorization code for an access token.
+
+**Client ID:** Client identifier obtained during the Application registration process.
+
+**Client Secret:** Client secret obtained during the Application registration process.
+
+**Scopes:** Scope of the access request, which might have multiple space-separated values.
+
+**Refresh Token:** Token used to receive accessToken, which is end product of OAuth2.
+
+### SSL/TLS
+
+**Verify HTTPS Trust Certificates:** If false, untrusted trust certificates (e.g. self signed), will not lead to an
+error. Do not disable this in production environment on a network you do not entirely trust. Especially public internet.
+
+**Keystore File:** A path to a file which contains keystore.
+
+**Keystore Type:** Format of a keystore.
+
+**Keystore Password:** Password for a keystore. If a keystore is not password protected leave it empty.
+
+**Keystore Key Algorithm:** An algorithm used for keystore.
+
+**Keystore Cert Alias**
+
+ Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores.
+
+Below is an example how the store need to be prepared:
+ ```
+ cat client.crt client.key > client-bundle.pem
+
+ openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS}
+
+ keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \
+ -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS}
+ ```
+
+**TrustStore File:** A path to a file which contains truststore.
+
+**TrustStore Type:** Format of a truststore.
+
+**TrustStore Password:** Password for a truststore. If a truststore is not password protected leave it empty.
+
+**TrustStore Key Algorithm:** An algorithm used for truststore.
+
+**Transport Protocols:** Transport protocols which are allowed for connection.
+
+**Cipher Suites:** Cipher suites which are allowed for connection.
+Colons, commas or spaces are also acceptable separators.
+
diff --git a/docs/HTTP-batchsource.md b/docs/HTTP-batchsource.md
index 37ed1e2..1e22786 100644
--- a/docs/HTTP-batchsource.md
+++ b/docs/HTTP-batchsource.md
@@ -403,6 +403,20 @@ error. Do not disable this in production environment on a network you do not ent
**Keystore Key Algorithm:** An algorithm used for keystore.
+**Keystore Cert Alias**
+
+ Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores.
+
+Below is an example how the store need to be prepared:
+ ```
+ cat client.crt client.key > client-bundle.pem
+
+ openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS}
+
+ keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \
+ -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS}
+ ```
+
**TrustStore File:** A path to a file which contains truststore.
**TrustStore Type:** Format of a truststore.
diff --git a/docs/HTTP-streamingsource.md b/docs/HTTP-streamingsource.md
index 4cad538..eafe35f 100644
--- a/docs/HTTP-streamingsource.md
+++ b/docs/HTTP-streamingsource.md
@@ -410,6 +410,20 @@ error. Do not disable this in production environment on a network you do not ent
**Keystore Key Algorithm:** An algorithm used for keystore.
+**Keystore Cert Alias**
+
+Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores.
+
+Below is an example how the store need to be prepared:
+ ```
+ cat client.crt client.key > client-bundle.pem
+
+ openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS}
+
+ keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \
+ -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS}
+ ```
+
**TrustStore File:** A path to a file which contains truststore.
**TrustStore Type:** Format of a truststore.
diff --git a/icons/HTTP-action.png b/icons/HTTP-action.png
new file mode 100644
index 0000000..933bf62
Binary files /dev/null and b/icons/HTTP-action.png differ
diff --git a/pom.xml b/pom.xml
index d49ef2e..87d37be 100644
--- a/pom.xml
+++ b/pom.xml
@@ -21,7 +21,7 @@
HTTP Plugins
io.cdap
http-plugins
- 1.4.0-SNAPSHOT
+ 1.4.2-SNAPSHOT
@@ -79,10 +79,12 @@
6.1.1
3.9
1.12
+ 1.2
+ 1.2.17
2.8.5
2.3.0
4.5.9
- 2.4.0-SNAPSHOT
+ 2.4.0
2.9.9
4.11
2.7.1
@@ -93,6 +95,20 @@
+
+
+ commons-logging
+ commons-logging
+ ${common.logging.version}
+ compile
+
+
+ log4j
+ log4j
+ ${log4j.version}
+ compile
+
+
io.cdap.cdap
cdap-api
diff --git a/src/main/java/io/cdap/plugin/http/action/HttpAction.java b/src/main/java/io/cdap/plugin/http/action/HttpAction.java
new file mode 100644
index 0000000..e611809
--- /dev/null
+++ b/src/main/java/io/cdap/plugin/http/action/HttpAction.java
@@ -0,0 +1,101 @@
+/*
+ * Copyright © 2021 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.http.action;
+
+import io.cdap.cdap.api.annotation.Description;
+import io.cdap.cdap.api.annotation.Name;
+import io.cdap.cdap.api.annotation.Plugin;
+import io.cdap.cdap.etl.api.PipelineConfigurer;
+import io.cdap.cdap.etl.api.action.Action;
+import io.cdap.cdap.etl.api.action.ActionContext;
+import io.cdap.plugin.http.common.RetryPolicy;
+import io.cdap.plugin.http.common.error.HttpErrorHandler;
+import io.cdap.plugin.http.common.error.RetryableErrorHandling;
+import io.cdap.plugin.http.common.http.HttpClient;
+import io.cdap.plugin.http.common.http.HttpConstants;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import org.awaitility.Awaitility;
+import org.awaitility.core.ConditionTimeoutException;
+import org.awaitility.pollinterval.FixedPollInterval;
+import org.awaitility.pollinterval.IterativePollInterval;
+import org.awaitility.pollinterval.PollInterval;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Objects;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * HTTP Action Plugin
+ */
+@Plugin(type = Action.PLUGIN_TYPE)
+@Name(HttpConstants.HTTP_PLUGIN_NAME)
+@Description("Action that runs a HTTP command")
+public class HttpAction extends Action {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HttpAction.class);
+ private final HttpActionConfig config;
+
+ public HttpAction(HttpActionConfig config) {
+ this.config = config;
+ }
+
+ private boolean makeRequest(HttpClient httpClient, String uri, HttpErrorHandler errorHandler) throws IOException {
+ try (HttpResponse response = new HttpResponse(httpClient.executeHTTP(uri))) {
+ RetryableErrorHandling errorHandlingStrategy = errorHandler.getErrorHandlingStrategy(response.getStatusCode());
+
+ if (errorHandlingStrategy == RetryableErrorHandling.FAIL) {
+ String body = "null";
+ try {
+ body = response.getBody();
+ } catch (Exception ignore) {
+ }
+ LOG.warn(
+ String.format("Request to the url '%s' failed with error code %s and message: %s",
+ uri,
+ response.getStatusCode(),
+ body
+ ));
+ }
+ return !errorHandlingStrategy.shouldRetry();
+ }
+ }
+
+ @Override
+ public void run(ActionContext actionContext) throws Exception {
+ try (HttpClient httpClient = new HttpClient(config)) {
+ HttpErrorHandler httpErrorHandler = new HttpErrorHandler(config);
+ PollInterval pollInterval = (config.getRetryPolicy().equals(RetryPolicy.LINEAR))
+ ? FixedPollInterval.fixed(Objects.requireNonNull(config.getLinearRetryInterval()), TimeUnit.SECONDS)
+ : IterativePollInterval.iterative(duration -> duration.multiply(2));
+
+ Awaitility
+ .await().with()
+ .pollInterval(pollInterval)
+ .timeout(config.getMaxRetryDuration(), TimeUnit.SECONDS)
+ .until(() -> makeRequest(httpClient, config.getUrl(), httpErrorHandler));
+ } catch (ConditionTimeoutException e) {
+ LOG.warn(String.format("Request to the url '%s' failed due to timeout", config.getUrl()));
+ }
+ }
+
+ @Override
+ public void configurePipeline(PipelineConfigurer pipelineConfigurer) {
+ super.configurePipeline(pipelineConfigurer);
+ config.validate();
+ }
+}
diff --git a/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java b/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java
new file mode 100644
index 0000000..101b8c9
--- /dev/null
+++ b/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java
@@ -0,0 +1,495 @@
+/*
+ * Copyright © 2021 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.http.action;
+
+import io.cdap.cdap.api.annotation.Description;
+import io.cdap.cdap.api.annotation.Macro;
+import io.cdap.cdap.api.annotation.Name;
+import io.cdap.cdap.api.plugin.PluginConfig;
+import io.cdap.cdap.etl.api.validation.InvalidConfigPropertyException;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.RetryPolicy;
+import io.cdap.plugin.http.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity;
+import io.cdap.plugin.http.common.error.RetryableErrorHandling;
+import io.cdap.plugin.http.common.http.HttpConstants;
+import io.cdap.plugin.http.common.http.IHttpConfig;
+import io.cdap.plugin.http.common.http.KeyStoreType;
+
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import java.util.regex.PatternSyntaxException;
+
+import javax.annotation.Nullable;
+
+/**
+ * Http Action plugin Config
+ */
+public class HttpActionConfig extends PluginConfig implements IHttpConfig {
+
+ @Name(HttpConstants.PROPERTY_URL)
+ @Description("Url to fetch to the first page. The url must start with a protocol (e.g. http://).")
+ @Macro
+ protected String url;
+
+ @Name(HttpConstants.PROPERTY_HTTP_METHOD)
+ @Description("HTTP request method.")
+ @Macro
+ protected String httpMethod;
+
+ @Name(HttpConstants.PROPERTY_HEADERS)
+ @Nullable
+ @Description("Headers to send with each HTTP request.")
+ @Macro
+ protected String headers;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_REQUEST_BODY)
+ @Description("Body to send with each HTTP request.")
+ @Macro
+ protected String requestBody;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_USERNAME)
+ @Description("Username for basic authentication.")
+ @Macro
+ protected String username;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_PASSWORD)
+ @Description("Password for basic authentication.")
+ @Macro
+ protected String password;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_PROXY_URL)
+ @Description("Proxy URL. Must contain a protocol, address and port.")
+ @Macro
+ protected String proxyUrl;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_PROXY_USERNAME)
+ @Description("Proxy username.")
+ @Macro
+ protected String proxyUsername;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_PROXY_PASSWORD)
+ @Description("Proxy password.")
+ @Macro
+ protected String proxyPassword;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)
+ @Description("Defines the error handling strategy to use for certain HTTP response codes." +
+ "The left column contains a regular expression for HTTP status code. The right column contains an action which" +
+ "is done in case of match. If HTTP status code matches multiple regular expressions, " +
+ "the first specified in mapping is matched.")
+ protected String httpErrorsHandling;
+
+ @Name(HttpConstants.PROPERTY_ERROR_HANDLING)
+ @Description("Error handling strategy to use when the HTTP response cannot be transformed to an output record.")
+ protected String errorHandling;
+
+
+ @Name(HttpConstants.PROPERTY_CONNECT_TIMEOUT)
+ @Description("Maximum time in seconds connection initialization is allowed to take.")
+ @Macro
+ protected Integer connectTimeout;
+
+ @Name(HttpConstants.PROPERTY_READ_TIMEOUT)
+ @Description("Maximum time in seconds fetching data from the server is allowed to take.")
+ @Macro
+ protected Integer readTimeout;
+
+ @Name(HttpConstants.PROPERTY_OAUTH2_ENABLED)
+ @Description("If true, plugin will perform OAuth2 authentication.")
+ protected String oauth2Enabled;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_AUTH_URL)
+ @Description("Endpoint for the authorization server used to retrieve the authorization code.")
+ @Macro
+ protected String authUrl;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TOKEN_URL)
+ @Description("Endpoint for the resource server, which exchanges the authorization code for an access token.")
+ @Macro
+ protected String tokenUrl;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_CLIENT_ID)
+ @Description("Client identifier obtained during the Application registration process.")
+ @Macro
+ protected String clientId;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_CLIENT_SECRET)
+ @Description("Client secret obtained during the Application registration process.")
+ @Macro
+ protected String clientSecret;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_SCOPES)
+ @Description("Scope of the access request, which might have multiple space-separated values.")
+ @Macro
+ protected String scopes;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_REFRESH_TOKEN)
+ @Description("Token used to receive accessToken, which is end product of OAuth2.")
+ @Macro
+ protected String refreshToken;
+
+ @Name(HttpConstants.PROPERTY_VERIFY_HTTPS)
+ @Description("If false, untrusted trust certificates (e.g. self signed), will not lead to an" +
+ "error. Do not disable this in production environment on a network you do not entirely trust. " +
+ "Especially public internet.")
+ @Macro
+ protected String verifyHttps;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_KEYSTORE_FILE)
+ @Description("A path to a file which contains keystore.")
+ @Macro
+ protected String keystoreFile;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_KEYSTORE_TYPE)
+ @Description("Format of a keystore.")
+ @Macro
+ protected String keystoreType;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_KEYSTORE_PASSWORD)
+ @Description("Password for a keystore. If a keystore is not password protected leave it empty.")
+ @Macro
+ protected String keystorePassword;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_KEYSTORE_KEY_ALGORITHM)
+ @Description("An algorithm used for keystore.")
+ @Macro
+ protected String keystoreKeyAlgorithm;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_FILE)
+ @Description("A path to a file which contains truststore.")
+ @Macro
+ protected String trustStoreFile;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_TYPE)
+ @Description("Format of a truststore.")
+ @Macro
+ protected String trustStoreType;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_PASSWORD)
+ @Description("Password for a truststore. If a truststore is not password protected leave it empty.")
+ @Macro
+ protected String trustStorePassword;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_KEY_ALGORITHM)
+ @Description("An algorithm used for truststore.")
+ @Macro
+ protected String trustStoreKeyAlgorithm;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS)
+ @Description("Transport protocols which are allowed for connection.")
+ @Macro
+ protected String transportProtocols;
+
+ @Nullable
+ @Name(HttpConstants.PROPERTY_CIPHER_SUITES)
+ @Description("Cipher suites which are allowed for connection. " +
+ "Colons, commas or spaces are also acceptable separators.")
+ @Macro
+ protected String cipherSuites;
+
+ @Name(HttpConstants.PROPERTY_KEYSTORE_CERT_ALIAS)
+ @Macro
+ @Nullable
+ @Description("Alias of the key in the keystore to be used for communication")
+ protected String keystoreCertAliasName;
+
+ @Override
+ public String getUrl() {
+ return url;
+ }
+
+ @Override
+ public String getHttpMethod() {
+ return httpMethod;
+ }
+
+ @Override
+ @Nullable
+ public String getHeaders() {
+ return headers;
+ }
+
+ @Override
+ @Nullable
+ public String getRequestBody() {
+ return requestBody;
+ }
+
+ @Override
+ @Nullable
+ public String getUsername() {
+ return username;
+ }
+
+ @Override
+ @Nullable
+ public String getPassword() {
+ return password;
+ }
+
+ @Override
+ @Nullable
+ public String getProxyUrl() {
+ return proxyUrl;
+ }
+
+ @Override
+ @Nullable
+ public String getProxyUsername() {
+ return proxyUsername;
+ }
+
+ @Override
+ @Nullable
+ public String getProxyPassword() {
+ return proxyPassword;
+ }
+
+ @Override
+ public ErrorHandling getErrorHandling() {
+ return BaseHttpSourceConfig.getEnumValueByString(
+ ErrorHandling.class,
+ errorHandling,
+ HttpConstants.PROPERTY_ERROR_HANDLING
+ );
+ }
+
+ @Override
+ public List getHttpErrorHandlingEntries() {
+ Map httpErrorsHandlingMap = BaseHttpSourceConfig.getMapFromKeyValueString(httpErrorsHandling);
+ List results = new ArrayList<>(httpErrorsHandlingMap.size());
+
+ for (Map.Entry entry : httpErrorsHandlingMap.entrySet()) {
+ String regex = entry.getKey();
+ try {
+ results.add(new HttpErrorHandlerEntity(Pattern.compile(regex),
+ BaseHttpSourceConfig.getEnumValueByString(RetryableErrorHandling.class,
+ entry.getValue(), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)));
+ } catch (PatternSyntaxException e) {
+ // We embed causing exception message into this one. Since this message is shown on UI when validation fails.
+ throw new InvalidConfigPropertyException(
+ String.format("Error handling regex '%s' is not valid. %s", regex, e.getMessage()),
+ HttpConstants.PROPERTY_HTTP_ERROR_HANDLING
+ );
+ }
+ }
+ return results;
+ }
+
+ @Override
+ public Long getLinearRetryInterval() {
+ return 0L;
+ }
+
+ @Override
+ public Long getMaxRetryDuration() {
+ return 1L;
+ }
+
+ @Override
+ public RetryPolicy getRetryPolicy() {
+ return RetryPolicy.LINEAR;
+ }
+
+ @Override
+ public Integer getConnectTimeout() {
+ return connectTimeout;
+ }
+
+ @Override
+ public Integer getReadTimeout() {
+ return 30;
+ }
+
+ @Override
+ public Boolean getOauth2Enabled() {
+ return Boolean.parseBoolean(oauth2Enabled);
+ }
+
+ @Override
+ @Nullable
+ public String getAuthUrl() {
+ return authUrl;
+ }
+
+ @Override
+ @Nullable
+ public String getTokenUrl() {
+ return tokenUrl;
+ }
+
+ @Override
+ @Nullable
+ public String getClientId() {
+ return clientId;
+ }
+
+ @Override
+ @Nullable
+ public String getClientSecret() {
+ return clientSecret;
+ }
+
+ @Override
+ @Nullable
+ public String getRefreshToken() {
+ return refreshToken;
+ }
+
+ @Override
+ public Boolean getVerifyHttps() {
+ return Boolean.parseBoolean(verifyHttps);
+ }
+
+ @Override
+ @Nullable
+ public String getKeystoreFile() {
+ return keystoreFile;
+ }
+
+ @Override
+ @Nullable
+ public KeyStoreType getKeystoreType() {
+ return BaseHttpSourceConfig.getEnumValueByString(
+ KeyStoreType.class,
+ keystoreType,
+ HttpConstants.PROPERTY_KEYSTORE_TYPE
+ );
+ }
+
+ @Override
+ @Nullable
+ public String getKeystorePassword() {
+ return keystorePassword;
+ }
+
+ @Override
+ @Nullable
+ public String getKeystoreKeyAlgorithm() {
+ return keystoreKeyAlgorithm;
+ }
+
+ @Override
+ @Nullable
+ public String getTrustStoreFile() {
+ return trustStoreFile;
+ }
+
+ @Override
+ @Nullable
+ public KeyStoreType getTrustStoreType() {
+ return BaseHttpSourceConfig.getEnumValueByString(
+ KeyStoreType.class,
+ trustStoreType,
+ HttpConstants.PROPERTY_TRUSTSTORE_TYPE
+ );
+ }
+
+ @Override
+ @Nullable
+ public String getTrustStorePassword() {
+ return trustStorePassword;
+ }
+
+ @Override
+ @Nullable
+ public String getTrustStoreKeyAlgorithm() {
+ return trustStoreKeyAlgorithm;
+ }
+
+ @Override
+ @Nullable
+ public String getCipherSuites() {
+ return cipherSuites;
+ }
+
+ @Override
+ @Nullable
+ public String getKeystoreCertAliasName() {
+ return keystoreCertAliasName;
+ }
+
+ @Override
+ @Nullable
+ public Map getHeadersMap() {
+ return BaseHttpSourceConfig.getMapFromKeyValueString(headers);
+ }
+
+ @Override
+ public List getTransportProtocolsList() {
+ return BaseHttpSourceConfig.getListFromString(transportProtocols);
+ }
+
+ @Override
+ public void validate() {
+ // Validate URL
+ if (!containsMacro(HttpConstants.PROPERTY_URL)) {
+ try {
+ // replace with placeholder with anything just during pagination
+ new URI(getUrl());
+ } catch (URISyntaxException e) {
+ throw new InvalidConfigPropertyException(
+ String.format("URL value is not valid: '%s'", getUrl()), e, HttpConstants.PROPERTY_URL);
+ }
+ }
+
+ // Validate OAuth2 properties
+ if (!containsMacro(HttpConstants.PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) {
+ String reasonOauth2 = "OAuth2 is enabled";
+ BaseHttpSourceConfig.assertIsSet(getAuthUrl(), HttpConstants.PROPERTY_AUTH_URL, reasonOauth2);
+ BaseHttpSourceConfig.assertIsSet(getTokenUrl(), HttpConstants.PROPERTY_TOKEN_URL, reasonOauth2);
+ BaseHttpSourceConfig.assertIsSet(getClientId(), HttpConstants.PROPERTY_CLIENT_ID, reasonOauth2);
+ BaseHttpSourceConfig.assertIsSet(getClientSecret(), HttpConstants.PROPERTY_CLIENT_SECRET, reasonOauth2);
+ BaseHttpSourceConfig.assertIsSet(getRefreshToken(), HttpConstants.PROPERTY_REFRESH_TOKEN, reasonOauth2);
+ }
+
+ if (!containsMacro(HttpConstants.PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) {
+ BaseHttpSourceConfig.assertIsNotSet(
+ getTrustStoreFile(),
+ HttpConstants.PROPERTY_TRUSTSTORE_FILE,
+ String.format("trustore settings are ignored due to disabled %s", HttpConstants.PROPERTY_VERIFY_HTTPS)
+ );
+ }
+ }
+
+}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java b/src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java
similarity index 82%
rename from src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java
rename to src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java
index a554dd6..6933ce9 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java
+++ b/src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common;
+package io.cdap.plugin.http.common;
import com.google.common.base.Strings;
import io.cdap.cdap.api.annotation.Description;
@@ -23,13 +23,15 @@
import io.cdap.cdap.etl.api.validation.InvalidConfigPropertyException;
import io.cdap.cdap.etl.api.validation.InvalidStageException;
import io.cdap.plugin.common.ReferencePluginConfig;
-import io.cdap.plugin.http.source.common.error.ErrorHandling;
-import io.cdap.plugin.http.source.common.error.HttpErrorHandlerEntity;
-import io.cdap.plugin.http.source.common.error.RetryableErrorHandling;
-import io.cdap.plugin.http.source.common.http.KeyStoreType;
-import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory;
-import io.cdap.plugin.http.source.common.pagination.PaginationType;
-import io.cdap.plugin.http.source.common.pagination.page.PageFormat;
+import io.cdap.plugin.http.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity;
+import io.cdap.plugin.http.common.error.RetryableErrorHandling;
+import io.cdap.plugin.http.common.http.HttpConstants;
+import io.cdap.plugin.http.common.http.IHttpConfig;
+import io.cdap.plugin.http.common.http.KeyStoreType;
+import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory;
+import io.cdap.plugin.http.common.pagination.PaginationType;
+import io.cdap.plugin.http.common.pagination.page.PageFormat;
import java.io.IOException;
import java.net.URI;
@@ -49,28 +51,13 @@
/**
* Base configuration for HTTP Streaming and Batch plugins.
*/
-public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
- public static final String PROPERTY_REFERENCE_NAME = "referenceName";
- public static final String PROPERTY_URL = "url";
- public static final String PROPERTY_HTTP_METHOD = "httpMethod";
- public static final String PROPERTY_HEADERS = "headers";
- public static final String PROPERTY_REQUEST_BODY = "requestBody";
+public abstract class BaseHttpSourceConfig extends ReferencePluginConfig implements IHttpConfig {
public static final String PROPERTY_FORMAT = "format";
public static final String PROPERTY_RESULT_PATH = "resultPath";
public static final String PROPERTY_FIELDS_MAPPING = "fieldsMapping";
public static final String PROPERTY_CSV_SKIP_FIRST_ROW = "csvSkipFirstRow";
- public static final String PROPERTY_USERNAME = "username";
- public static final String PROPERTY_PASSWORD = "password";
- public static final String PROPERTY_PROXY_URL = "proxyUrl";
- public static final String PROPERTY_PROXY_USERNAME = "proxyUsername";
- public static final String PROPERTY_PROXY_PASSWORD = "proxyPassword";
- public static final String PROPERTY_HTTP_ERROR_HANDLING = "httpErrorsHandling";
- public static final String PROPERTY_ERROR_HANDLING = "errorHandling";
- public static final String PROPERTY_RETRY_POLICY = "retryPolicy";
public static final String PROPERTY_LINEAR_RETRY_INTERVAL = "linearRetryInterval";
public static final String PROPERTY_MAX_RETRY_DURATION = "maxRetryDuration";
- public static final String PROPERTY_CONNECT_TIMEOUT = "connectTimeout";
- public static final String PROPERTY_READ_TIMEOUT = "readTimeout";
public static final String PROPERTY_PAGINATION_TYPE = "paginationType";
public static final String PROPERTY_START_INDEX = "startIndex";
public static final String PROPERTY_MAX_INDEX = "maxIndex";
@@ -80,47 +67,27 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
public static final String PROPERTY_NEXT_PAGE_URL_PARAMETER = "nextPageUrlParameter";
public static final String PROPERTY_CUSTOM_PAGINATION_CODE = "customPaginationCode";
public static final String PROPERTY_WAIT_TIME_BETWEEN_PAGES = "waitTimeBetweenPages";
- public static final String PROPERTY_OAUTH2_ENABLED = "oauth2Enabled";
- public static final String PROPERTY_AUTH_URL = "authUrl";
- public static final String PROPERTY_TOKEN_URL = "tokenUrl";
- public static final String PROPERTY_CLIENT_ID = "clientId";
- public static final String PROPERTY_CLIENT_SECRET = "clientSecret";
- public static final String PROPERTY_SCOPES = "scopes";
- public static final String PROPERTY_REFRESH_TOKEN = "refreshToken";
- public static final String PROPERTY_VERIFY_HTTPS = "verifyHttps";
- public static final String PROPERTY_KEYSTORE_FILE = "keystoreFile";
- public static final String PROPERTY_KEYSTORE_TYPE = "keystoreType";
- public static final String PROPERTY_KEYSTORE_PASSWORD = "keystorePassword";
- public static final String PROPERTY_KEYSTORE_KEY_ALGORITHM = "keystoreKeyAlgorithm";
- public static final String PROPERTY_TRUSTSTORE_FILE = "trustStoreFile";
- public static final String PROPERTY_TRUSTSTORE_TYPE = "trustStoreType";
- public static final String PROPERTY_TRUSTSTORE_PASSWORD = "trustStorePassword";
- public static final String PROPERTY_TRUSTSTORE_KEY_ALGORITHM = "trustStoreKeyAlgorithm";
- public static final String PROPERTY_TRANSPORT_PROTOCOLS = "transportProtocols";
- public static final String PROPERTY_CIPHER_SUITES = "cipherSuites";
- public static final String PROPERTY_SCHEMA = "schema";
-
public static final String PAGINATION_INDEX_PLACEHOLDER_REGEX = "\\{pagination.index\\}";
public static final String PAGINATION_INDEX_PLACEHOLDER = "{pagination.index}";
- @Name(PROPERTY_URL)
+ @Name(HttpConstants.PROPERTY_URL)
@Description("Url to fetch to the first page. The url must start with a protocol (e.g. http://).")
@Macro
protected String url;
- @Name(PROPERTY_HTTP_METHOD)
+ @Name(HttpConstants.PROPERTY_HTTP_METHOD)
@Description("HTTP request method.")
@Macro
protected String httpMethod;
- @Name(PROPERTY_HEADERS)
+ @Name(HttpConstants.PROPERTY_HEADERS)
@Nullable
@Description("Headers to send with each HTTP request.")
@Macro
protected String headers;
@Nullable
- @Name(PROPERTY_REQUEST_BODY)
+ @Name(HttpConstants.PROPERTY_REQUEST_BODY)
@Description("Body to send with each HTTP request.")
@Macro
protected String requestBody;
@@ -153,48 +120,48 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
protected String csvSkipFirstRow;
@Nullable
- @Name(PROPERTY_USERNAME)
+ @Name(HttpConstants.PROPERTY_USERNAME)
@Description("Username for basic authentication.")
@Macro
protected String username;
@Nullable
- @Name(PROPERTY_PASSWORD)
+ @Name(HttpConstants.PROPERTY_PASSWORD)
@Description("Password for basic authentication.")
@Macro
protected String password;
@Nullable
- @Name(PROPERTY_PROXY_URL)
+ @Name(HttpConstants.PROPERTY_PROXY_URL)
@Description("Proxy URL. Must contain a protocol, address and port.")
@Macro
protected String proxyUrl;
@Nullable
- @Name(PROPERTY_PROXY_USERNAME)
+ @Name(HttpConstants.PROPERTY_PROXY_USERNAME)
@Description("Proxy username.")
@Macro
protected String proxyUsername;
@Nullable
- @Name(PROPERTY_PROXY_PASSWORD)
+ @Name(HttpConstants.PROPERTY_PROXY_PASSWORD)
@Description("Proxy password.")
@Macro
protected String proxyPassword;
@Nullable
- @Name(PROPERTY_HTTP_ERROR_HANDLING)
+ @Name(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)
@Description("Defines the error handling strategy to use for certain HTTP response codes." +
"The left column contains a regular expression for HTTP status code. The right column contains an action which" +
"is done in case of match. If HTTP status code matches multiple regular expressions, " +
"the first specified in mapping is matched.")
protected String httpErrorsHandling;
- @Name(PROPERTY_ERROR_HANDLING)
+ @Name(HttpConstants.PROPERTY_ERROR_HANDLING)
@Description("Error handling strategy to use when the HTTP response cannot be transformed to an output record.")
protected String errorHandling;
- @Name(PROPERTY_RETRY_POLICY)
+ @Name(HttpConstants.PROPERTY_RETRY_POLICY)
@Description("Policy used to calculate delay between retries.")
protected String retryPolicy;
@@ -209,12 +176,12 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
@Macro
protected Long maxRetryDuration;
- @Name(PROPERTY_CONNECT_TIMEOUT)
+ @Name(HttpConstants.PROPERTY_CONNECT_TIMEOUT)
@Description("Maximum time in seconds connection initialization is allowed to take.")
@Macro
protected Integer connectTimeout;
- @Name(PROPERTY_READ_TIMEOUT)
+ @Name(HttpConstants.PROPERTY_READ_TIMEOUT)
@Description("Maximum time in seconds fetching data from the server is allowed to take.")
@Macro
protected Integer readTimeout;
@@ -276,47 +243,47 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
@Macro
protected Long waitTimeBetweenPages;
- @Name(PROPERTY_OAUTH2_ENABLED)
+ @Name(HttpConstants.PROPERTY_OAUTH2_ENABLED)
@Description("If true, plugin will perform OAuth2 authentication.")
protected String oauth2Enabled;
@Nullable
- @Name(PROPERTY_AUTH_URL)
+ @Name(HttpConstants.PROPERTY_AUTH_URL)
@Description("Endpoint for the authorization server used to retrieve the authorization code.")
@Macro
protected String authUrl;
@Nullable
- @Name(PROPERTY_TOKEN_URL)
+ @Name(HttpConstants.PROPERTY_TOKEN_URL)
@Description("Endpoint for the resource server, which exchanges the authorization code for an access token.")
@Macro
protected String tokenUrl;
@Nullable
- @Name(PROPERTY_CLIENT_ID)
+ @Name(HttpConstants.PROPERTY_CLIENT_ID)
@Description("Client identifier obtained during the Application registration process.")
@Macro
protected String clientId;
@Nullable
- @Name(PROPERTY_CLIENT_SECRET)
+ @Name(HttpConstants.PROPERTY_CLIENT_SECRET)
@Description("Client secret obtained during the Application registration process.")
@Macro
protected String clientSecret;
@Nullable
- @Name(PROPERTY_SCOPES)
+ @Name(HttpConstants.PROPERTY_SCOPES)
@Description("Scope of the access request, which might have multiple space-separated values.")
@Macro
protected String scopes;
@Nullable
- @Name(PROPERTY_REFRESH_TOKEN)
+ @Name(HttpConstants.PROPERTY_REFRESH_TOKEN)
@Description("Token used to receive accessToken, which is end product of OAuth2.")
@Macro
protected String refreshToken;
- @Name(PROPERTY_VERIFY_HTTPS)
+ @Name(HttpConstants.PROPERTY_VERIFY_HTTPS)
@Description("If false, untrusted trust certificates (e.g. self signed), will not lead to an" +
"error. Do not disable this in production environment on a network you do not entirely trust. " +
"Especially public internet.")
@@ -324,72 +291,78 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig {
protected String verifyHttps;
@Nullable
- @Name(PROPERTY_KEYSTORE_FILE)
+ @Name(HttpConstants.PROPERTY_KEYSTORE_FILE)
@Description("A path to a file which contains keystore.")
@Macro
protected String keystoreFile;
@Nullable
- @Name(PROPERTY_KEYSTORE_TYPE)
+ @Name(HttpConstants.PROPERTY_KEYSTORE_TYPE)
@Description("Format of a keystore.")
@Macro
protected String keystoreType;
@Nullable
- @Name(PROPERTY_KEYSTORE_PASSWORD)
+ @Name(HttpConstants.PROPERTY_KEYSTORE_PASSWORD)
@Description("Password for a keystore. If a keystore is not password protected leave it empty.")
@Macro
protected String keystorePassword;
@Nullable
- @Name(PROPERTY_KEYSTORE_KEY_ALGORITHM)
+ @Name(HttpConstants.PROPERTY_KEYSTORE_KEY_ALGORITHM)
@Description("An algorithm used for keystore.")
@Macro
protected String keystoreKeyAlgorithm;
@Nullable
- @Name(PROPERTY_TRUSTSTORE_FILE)
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_FILE)
@Description("A path to a file which contains truststore.")
@Macro
protected String trustStoreFile;
@Nullable
- @Name(PROPERTY_TRUSTSTORE_TYPE)
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_TYPE)
@Description("Format of a truststore.")
@Macro
protected String trustStoreType;
@Nullable
- @Name(PROPERTY_TRUSTSTORE_PASSWORD)
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_PASSWORD)
@Description("Password for a truststore. If a truststore is not password protected leave it empty.")
@Macro
protected String trustStorePassword;
@Nullable
- @Name(PROPERTY_TRUSTSTORE_KEY_ALGORITHM)
+ @Name(HttpConstants.PROPERTY_TRUSTSTORE_KEY_ALGORITHM)
@Description("An algorithm used for truststore.")
@Macro
protected String trustStoreKeyAlgorithm;
@Nullable
- @Name(PROPERTY_TRANSPORT_PROTOCOLS)
+ @Name(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS)
@Description("Transport protocols which are allowed for connection.")
@Macro
protected String transportProtocols;
@Nullable
- @Name(PROPERTY_CIPHER_SUITES)
+ @Name(HttpConstants.PROPERTY_CIPHER_SUITES)
@Description("Cipher suites which are allowed for connection. " +
"Colons, commas or spaces are also acceptable separators.")
@Macro
protected String cipherSuites;
- @Name(PROPERTY_SCHEMA)
+ @Name(HttpConstants.PROPERTY_SCHEMA)
@Macro
@Nullable
@Description("Output schema. Is required to be set.")
protected String schema;
+ @Name(HttpConstants.PROPERTY_KEYSTORE_CERT_ALIAS)
+ @Macro
+ @Nullable
+ @Description("Alias of the key in the keystore to be used for communication")
+ protected String keystoreCertAliasName;
+
protected BaseHttpSourceConfig(String referenceName) {
super(referenceName);
}
@@ -461,11 +434,11 @@ public String getHttpErrorsHandling() {
}
public ErrorHandling getErrorHandling() {
- return getEnumValueByString(ErrorHandling.class, errorHandling, PROPERTY_ERROR_HANDLING);
+ return getEnumValueByString(ErrorHandling.class, errorHandling, HttpConstants.PROPERTY_ERROR_HANDLING);
}
public RetryPolicy getRetryPolicy() {
- return getEnumValueByString(RetryPolicy.class, retryPolicy, PROPERTY_RETRY_POLICY);
+ return getEnumValueByString(RetryPolicy.class, retryPolicy, HttpConstants.PROPERTY_RETRY_POLICY);
}
@Nullable
@@ -574,7 +547,7 @@ public String getKeystoreFile() {
@Nullable
public KeyStoreType getKeystoreType() {
- return getEnumValueByString(KeyStoreType.class, keystoreType, PROPERTY_KEYSTORE_TYPE);
+ return getEnumValueByString(KeyStoreType.class, keystoreType, HttpConstants.PROPERTY_KEYSTORE_TYPE);
}
@Nullable
@@ -594,7 +567,7 @@ public String getTrustStoreFile() {
@Nullable
public KeyStoreType getTrustStoreType() {
- return getEnumValueByString(KeyStoreType.class, trustStoreType, PROPERTY_TRUSTSTORE_TYPE);
+ return getEnumValueByString(KeyStoreType.class, trustStoreType, HttpConstants.PROPERTY_TRUSTSTORE_TYPE);
}
@Nullable
@@ -623,10 +596,15 @@ public Schema getSchema() {
return Strings.isNullOrEmpty(schema) ? null : Schema.parseJson(schema);
} catch (IOException e) {
throw new InvalidConfigPropertyException("Unable to parse output schema: " +
- schema, e, PROPERTY_SCHEMA);
+ schema, e, HttpConstants.PROPERTY_SCHEMA);
}
}
+ @Nullable
+ public String getKeystoreCertAliasName() {
+ return keystoreCertAliasName;
+ }
+
@Nullable
public Map getHeadersMap() {
return getMapFromKeyValueString(headers);
@@ -641,12 +619,13 @@ public List getHttpErrorHandlingEntries() {
try {
results.add(new HttpErrorHandlerEntity(Pattern.compile(regex),
getEnumValueByString(RetryableErrorHandling.class,
- entry.getValue(), PROPERTY_HTTP_ERROR_HANDLING)));
+ entry.getValue(), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)));
} catch (PatternSyntaxException e) {
// We embed causing exception message into this one. Since this message is shown on UI when validation fails.
throw new InvalidConfigPropertyException(
- String.format(
- "Error handling regex '%s' is not valid. %s", regex, e.getMessage()), PROPERTY_HTTP_ERROR_HANDLING);
+ String.format("Error handling regex '%s' is not valid. %s", regex, e.getMessage()),
+ HttpConstants.PROPERTY_HTTP_ERROR_HANDLING
+ );
}
}
return results;
@@ -672,18 +651,18 @@ public List getTransportProtocolsList() {
public void validate() {
// Validate URL
- if (!containsMacro(PROPERTY_URL)) {
+ if (!containsMacro(HttpConstants.PROPERTY_URL)) {
try {
// replace with placeholder with anything just during pagination
new URI(getUrl().replaceAll(PAGINATION_INDEX_PLACEHOLDER_REGEX, "0"));
} catch (URISyntaxException e) {
throw new InvalidConfigPropertyException(
- String.format("URL value is not valid: '%s'", getUrl()), e, PROPERTY_URL);
+ String.format("URL value is not valid: '%s'", getUrl()), e, HttpConstants.PROPERTY_URL);
}
}
// Validate HTTP Error Handling Map
- if (!containsMacro(PROPERTY_HTTP_ERROR_HANDLING)) {
+ if (!containsMacro(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)) {
List httpErrorsHandlingEntries = getHttpErrorHandlingEntries();
boolean supportsSkippingPages = PaginationIteratorFactory
.createInstance(this, null).supportsSkippingPages();
@@ -695,7 +674,8 @@ public void validate() {
postRetryStrategy.equals(ErrorHandling.SKIP)) {
throw new InvalidConfigPropertyException(
String.format("Error handling strategy '%s' is not support in combination with pagination type",
- httpErrorsHandlingEntry.getStrategy(), getPaginationType()), PROPERTY_HTTP_ERROR_HANDLING);
+ httpErrorsHandlingEntry.getStrategy(), getPaginationType()
+ ), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING);
}
}
}
@@ -703,7 +683,7 @@ public void validate() {
// Validate Linear Retry Interval
- if (!containsMacro(PROPERTY_RETRY_POLICY) && getRetryPolicy() == RetryPolicy.LINEAR) {
+ if (!containsMacro(HttpConstants.PROPERTY_RETRY_POLICY) && getRetryPolicy() == RetryPolicy.LINEAR) {
assertIsSet(getLinearRetryInterval(), PROPERTY_LINEAR_RETRY_INTERVAL, "retry policy is linear");
}
@@ -745,7 +725,7 @@ public void validate() {
throw new InvalidConfigPropertyException(
String.format("Url '%s' must contain '%s' placeholder when pagination type is '%s'", getUrl(),
PAGINATION_INDEX_PLACEHOLDER, getPaginationType()),
- PROPERTY_URL);
+ HttpConstants.PROPERTY_URL);
}
break;
case CUSTOM:
@@ -785,18 +765,21 @@ PAGINATION_INDEX_PLACEHOLDER, getPaginationType()),
}
// Validate OAuth2 properties
- if (!containsMacro(PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) {
+ if (!containsMacro(HttpConstants.PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) {
String reasonOauth2 = "OAuth2 is enabled";
- assertIsSet(getAuthUrl(), PROPERTY_AUTH_URL, reasonOauth2);
- assertIsSet(getTokenUrl(), PROPERTY_TOKEN_URL, reasonOauth2);
- assertIsSet(getClientId(), PROPERTY_CLIENT_ID, reasonOauth2);
- assertIsSet(getClientSecret(), PROPERTY_CLIENT_SECRET, reasonOauth2);
- assertIsSet(getRefreshToken(), PROPERTY_REFRESH_TOKEN, reasonOauth2);
+ assertIsSet(getAuthUrl(), HttpConstants.PROPERTY_AUTH_URL, reasonOauth2);
+ assertIsSet(getTokenUrl(), HttpConstants.PROPERTY_TOKEN_URL, reasonOauth2);
+ assertIsSet(getClientId(), HttpConstants.PROPERTY_CLIENT_ID, reasonOauth2);
+ assertIsSet(getClientSecret(), HttpConstants.PROPERTY_CLIENT_SECRET, reasonOauth2);
+ assertIsSet(getRefreshToken(), HttpConstants.PROPERTY_REFRESH_TOKEN, reasonOauth2);
}
- if (!containsMacro(PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) {
- assertIsNotSet(getTrustStoreFile(), PROPERTY_TRUSTSTORE_FILE,
- String.format("trustore settings are ignored due to disabled %s", PROPERTY_VERIFY_HTTPS));
+ if (!containsMacro(HttpConstants.PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) {
+ assertIsNotSet(
+ getTrustStoreFile(),
+ HttpConstants.PROPERTY_TRUSTSTORE_FILE,
+ String.format("trustore settings are ignored due to disabled %s", HttpConstants.PROPERTY_VERIFY_HTTPS)
+ );
}
}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java b/src/main/java/io/cdap/plugin/http/common/EnumWithValue.java
similarity index 94%
rename from src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java
rename to src/main/java/io/cdap/plugin/http/common/EnumWithValue.java
index 653a12d..562caaf 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java
+++ b/src/main/java/io/cdap/plugin/http/common/EnumWithValue.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common;
+package io.cdap.plugin.http.common;
/**
* Enum which has string values as a part of it.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java b/src/main/java/io/cdap/plugin/http/common/RetryPolicy.java
similarity index 95%
rename from src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java
rename to src/main/java/io/cdap/plugin/http/common/RetryPolicy.java
index 2d0157d..c2cb8cb 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java
+++ b/src/main/java/io/cdap/plugin/http/common/RetryPolicy.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common;
+package io.cdap.plugin.http.common;
/**
* An enum which represents a retry policy.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java b/src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java
similarity index 91%
rename from src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java
rename to src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java
index 47f06b5..7bea805 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java
+++ b/src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.error;
+package io.cdap.plugin.http.common.error;
-import io.cdap.plugin.http.source.common.EnumWithValue;
+import io.cdap.plugin.http.common.EnumWithValue;
/**
* Indicates error handling strategy which will be used during reading HTTP records.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java
similarity index 68%
rename from src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java
rename to src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java
index c6a802f..172d947 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java
+++ b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java
@@ -13,9 +13,10 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.error;
+package io.cdap.plugin.http.common.error;
+
+import io.cdap.plugin.http.common.http.IHttpConfig;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -29,23 +30,31 @@ public class HttpErrorHandler {
private static final Logger LOG = LoggerFactory.getLogger(HttpErrorHandler.class);
private List httpErrorsHandlingEntries;
+ private boolean showWarnings = true;
- public HttpErrorHandler(BaseHttpSourceConfig config) {
+ public HttpErrorHandler(IHttpConfig config) {
this.httpErrorsHandlingEntries = config.getHttpErrorHandlingEntries();
}
+ public HttpErrorHandler(IHttpConfig config, boolean showWarnings) {
+ this.httpErrorsHandlingEntries = config.getHttpErrorHandlingEntries();
+ this.showWarnings = showWarnings;
+ }
+
public RetryableErrorHandling getErrorHandlingStrategy(int httpCode) {
String httpCodeString = Integer.toString(httpCode);
- for (HttpErrorHandlerEntity httpErrorsHandlingEntry : httpErrorsHandlingEntries) {
+ for (HttpErrorHandlerEntity httpErrorsHandlingEntry: httpErrorsHandlingEntries) {
Matcher matcher = httpErrorsHandlingEntry.getPattern().matcher(httpCodeString);
if (matcher.matches()) {
return httpErrorsHandlingEntry.getStrategy();
}
}
- LOG.warn(String.format("No error handling strategy defined for HTTP status code '%d'. " +
- "Please correct httpErrorsHandling.", httpCode));
+ if (showWarnings) {
+ LOG.warn(String.format("No error handling strategy defined for HTTP status code '%d'. " +
+ "Please correct httpErrorsHandling.", httpCode));
+ }
return RetryableErrorHandling.FAIL;
}
}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java
similarity index 96%
rename from src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java
rename to src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java
index 09a803a..8e4e417 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java
+++ b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.error;
+package io.cdap.plugin.http.common.error;
import java.util.Objects;
import java.util.regex.Pattern;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java b/src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java
similarity index 94%
rename from src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java
rename to src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java
index 0efa839..e910ca3 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java
+++ b/src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.error;
+package io.cdap.plugin.http.common.error;
-import io.cdap.plugin.http.source.common.EnumWithValue;
+import io.cdap.plugin.http.common.EnumWithValue;
/**
* Indicates error handling strategy which will be used to handle unexpected http status codes.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java b/src/main/java/io/cdap/plugin/http/common/http/HttpClient.java
similarity index 84%
rename from src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java
rename to src/main/java/io/cdap/plugin/http/common/http/HttpClient.java
index bb41293..aa6b151 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/HttpClient.java
@@ -13,11 +13,11 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
import com.google.common.base.Charsets;
import com.google.common.base.Strings;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
@@ -45,11 +45,11 @@
*/
public class HttpClient implements Closeable {
private final Map headers;
- private final BaseHttpSourceConfig config;
+ private final IHttpConfig config;
private final StringEntity requestBody;
private CloseableHttpClient httpClient;
- public HttpClient(BaseHttpSourceConfig config) {
+ public HttpClient(IHttpConfig config) {
this.config = config;
this.headers = config.getHeadersMap();
@@ -96,29 +96,31 @@ private CloseableHttpClient createHttpClient() throws IOException {
httpClientBuilder.setSSLSocketFactory(new SSLConnectionSocketFactoryCreator(config).create());
// set timeouts
- Long connectTimeoutMillis = TimeUnit.SECONDS.toMillis(config.getConnectTimeout());
- Long readTimeoutMillis = TimeUnit.SECONDS.toMillis(config.getReadTimeout());
+ int connectTimeoutMillis = (int) TimeUnit.SECONDS.toMillis(config.getConnectTimeout());
+ int readTimeoutMillis = (int) TimeUnit.SECONDS.toMillis(config.getReadTimeout());
RequestConfig.Builder requestBuilder = RequestConfig.custom();
- requestBuilder.setSocketTimeout(readTimeoutMillis.intValue());
- requestBuilder.setConnectTimeout(connectTimeoutMillis.intValue());
- requestBuilder.setConnectionRequestTimeout(connectTimeoutMillis.intValue());
+ requestBuilder.setSocketTimeout(readTimeoutMillis);
+ requestBuilder.setConnectTimeout(connectTimeoutMillis);
+ requestBuilder.setConnectionRequestTimeout(connectTimeoutMillis);
httpClientBuilder.setDefaultRequestConfig(requestBuilder.build());
// basic auth
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
if (!Strings.isNullOrEmpty(config.getUsername()) && !Strings.isNullOrEmpty(config.getPassword())) {
- AuthScope authScope = new AuthScope(HttpHost.create(config.getUrl()));
- credentialsProvider.setCredentials(authScope,
- new UsernamePasswordCredentials(config.getUsername(), config.getPassword()));
+ credentialsProvider.setCredentials(
+ new AuthScope(HttpHost.create(config.getUrl())),
+ new UsernamePasswordCredentials(config.getUsername(), config.getPassword())
+ );
}
// proxy and proxy auth
if (!Strings.isNullOrEmpty(config.getProxyUrl())) {
HttpHost proxyHost = HttpHost.create(config.getProxyUrl());
if (!Strings.isNullOrEmpty(config.getProxyUsername()) && !Strings.isNullOrEmpty(config.getProxyPassword())) {
- credentialsProvider.setCredentials(new AuthScope(proxyHost),
- new UsernamePasswordCredentials(
- config.getProxyUsername(), config.getProxyPassword()));
+ credentialsProvider.setCredentials(
+ new AuthScope(proxyHost),
+ new UsernamePasswordCredentials(config.getProxyUsername(), config.getProxyPassword())
+ );
}
httpClientBuilder.setProxy(proxyHost);
}
diff --git a/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java b/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java
new file mode 100644
index 0000000..0e10acb
--- /dev/null
+++ b/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java
@@ -0,0 +1,63 @@
+/*
+ * Copyright © 2017 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.http.common.http;
+
+/**
+ * HTTP Plugin Constants
+ */
+public final class HttpConstants {
+ private HttpConstants() {
+
+ }
+
+ public static final String HTTP_PLUGIN_NAME = "HTTP";
+
+ // Field Name Constants
+ public static final String PROPERTY_URL = "url";
+ public static final String PROPERTY_HTTP_METHOD = "httpMethod";
+ public static final String PROPERTY_HEADERS = "headers";
+ public static final String PROPERTY_REQUEST_BODY = "requestBody";
+ public static final String PROPERTY_USERNAME = "username";
+ public static final String PROPERTY_PASSWORD = "password";
+ public static final String PROPERTY_PROXY_URL = "proxyUrl";
+ public static final String PROPERTY_PROXY_USERNAME = "proxyUsername";
+ public static final String PROPERTY_PROXY_PASSWORD = "proxyPassword";
+ public static final String PROPERTY_HTTP_ERROR_HANDLING = "httpErrorsHandling";
+ public static final String PROPERTY_ERROR_HANDLING = "errorHandling";
+ public static final String PROPERTY_RETRY_POLICY = "retryPolicy";
+ public static final String PROPERTY_CONNECT_TIMEOUT = "connectTimeout";
+ public static final String PROPERTY_READ_TIMEOUT = "readTimeout";
+ public static final String PROPERTY_OAUTH2_ENABLED = "oauth2Enabled";
+ public static final String PROPERTY_AUTH_URL = "authUrl";
+ public static final String PROPERTY_TOKEN_URL = "tokenUrl";
+ public static final String PROPERTY_CLIENT_ID = "clientId";
+ public static final String PROPERTY_CLIENT_SECRET = "clientSecret";
+ public static final String PROPERTY_SCOPES = "scopes";
+ public static final String PROPERTY_REFRESH_TOKEN = "refreshToken";
+ public static final String PROPERTY_VERIFY_HTTPS = "verifyHttps";
+ public static final String PROPERTY_KEYSTORE_FILE = "keystoreFile";
+ public static final String PROPERTY_KEYSTORE_TYPE = "keystoreType";
+ public static final String PROPERTY_KEYSTORE_PASSWORD = "keystorePassword";
+ public static final String PROPERTY_KEYSTORE_KEY_ALGORITHM = "keystoreKeyAlgorithm";
+ public static final String PROPERTY_TRUSTSTORE_FILE = "trustStoreFile";
+ public static final String PROPERTY_TRUSTSTORE_TYPE = "trustStoreType";
+ public static final String PROPERTY_TRUSTSTORE_PASSWORD = "trustStorePassword";
+ public static final String PROPERTY_TRUSTSTORE_KEY_ALGORITHM = "trustStoreKeyAlgorithm";
+ public static final String PROPERTY_TRANSPORT_PROTOCOLS = "transportProtocols";
+ public static final String PROPERTY_CIPHER_SUITES = "cipherSuites";
+ public static final String PROPERTY_SCHEMA = "schema";
+ public static final String PROPERTY_KEYSTORE_CERT_ALIAS = "keystoreCertAlias";
+}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java b/src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java
similarity index 98%
rename from src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java
rename to src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java
index f8d5254..21c2cc9 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java
@@ -14,7 +14,7 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
diff --git a/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java b/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java
new file mode 100644
index 0000000..09d9375
--- /dev/null
+++ b/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java
@@ -0,0 +1,124 @@
+/*
+ * Copyright © 2019 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.http.common.http;
+
+import io.cdap.plugin.http.common.RetryPolicy;
+import io.cdap.plugin.http.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+
+/**
+ * Interface for the HTTP Plugins Config
+ */
+public interface IHttpConfig {
+
+ String getUrl();
+
+ String getHttpMethod();
+
+ @Nullable
+ String getHeaders();
+
+ @Nullable
+ String getRequestBody();
+
+ @Nullable
+ String getUsername();
+
+ @Nullable
+ String getPassword();
+
+ @Nullable
+ String getProxyUrl();
+
+ @Nullable
+ String getProxyUsername();
+
+ @Nullable
+ String getProxyPassword();
+
+ ErrorHandling getErrorHandling();
+
+ List getHttpErrorHandlingEntries();
+
+ Long getLinearRetryInterval();
+
+ Long getMaxRetryDuration();
+
+ RetryPolicy getRetryPolicy();
+
+ Integer getConnectTimeout();
+
+ Integer getReadTimeout();
+
+ Boolean getOauth2Enabled();
+
+ @Nullable
+ String getAuthUrl();
+
+ @Nullable
+ String getTokenUrl();
+
+ @Nullable
+ String getClientId();
+
+ @Nullable
+ String getClientSecret();
+
+ @Nullable
+ String getRefreshToken();
+
+ Boolean getVerifyHttps();
+
+ @Nullable
+ String getKeystoreFile();
+
+ @Nullable
+ KeyStoreType getKeystoreType();
+
+ @Nullable
+ String getKeystorePassword();
+
+ @Nullable
+ String getKeystoreKeyAlgorithm();
+
+ @Nullable
+ String getTrustStoreFile();
+
+ @Nullable
+ KeyStoreType getTrustStoreType();
+
+ @Nullable
+ String getTrustStorePassword();
+
+ @Nullable
+ String getTrustStoreKeyAlgorithm();
+
+ @Nullable
+ String getCipherSuites();
+
+ @Nullable
+ String getKeystoreCertAliasName();
+
+ @Nullable
+ Map getHeadersMap();
+
+ List getTransportProtocolsList();
+
+ void validate();
+}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java b/src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java
similarity index 91%
rename from src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java
rename to src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java
index 6acbf12..5d607af 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
-import io.cdap.plugin.http.source.common.EnumWithValue;
+import io.cdap.plugin.http.common.EnumWithValue;
/**
* An enum which represent a type of keystore or truststore.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java b/src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java
similarity index 94%
rename from src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java
rename to src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java
index 27ac280..1928683 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java
@@ -13,10 +13,10 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
import com.google.gson.JsonElement;
-import io.cdap.plugin.http.source.common.pagination.page.JSONUtil;
+import io.cdap.plugin.http.common.pagination.page.JSONUtil;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java b/src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java
similarity index 72%
rename from src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java
rename to src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java
index 1883f9b..729c852 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java
@@ -14,14 +14,10 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
import com.google.common.base.Strings;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
@@ -42,11 +38,9 @@
* Class which creates an SSLConnectionSocketFactory.
*/
public class SSLConnectionSocketFactoryCreator {
- private static final Logger LOG = LoggerFactory.getLogger(SSLConnectionSocketFactoryCreator.class);
-
- private final BaseHttpSourceConfig config;
+ private final IHttpConfig config;
- public SSLConnectionSocketFactoryCreator(BaseHttpSourceConfig config) {
+ public SSLConnectionSocketFactoryCreator(IHttpConfig config) {
this.config = config;
}
@@ -59,6 +53,7 @@ public SSLConnectionSocketFactory create() {
SSLContext sslContext = SSLContext.getInstance("TLS"); // "TLS" means rely system properties
sslContext.init(getKeyManagers(), getTrustManagers(), null);
+
return new SSLConnectionSocketFactory(sslContext, config.getTransportProtocolsList().toArray(new String[0]),
cipherSuites, SSLConnectionSocketFactory.getDefaultHostnameVerifier());
} catch (KeyManagementException | CertificateException | NoSuchAlgorithmException | KeyStoreException
@@ -70,27 +65,30 @@ public SSLConnectionSocketFactory create() {
private KeyManager[] getKeyManagers() throws CertificateException, NoSuchAlgorithmException,
KeyStoreException, IOException, UnrecoverableKeyException {
- KeyStore keystore = loadKeystore(config.getKeystoreFile(), config.getKeystoreType().name(),
- config.getKeystorePassword());
-
String keyStorePassword = config.getKeystorePassword();
+ KeyStore keystore = loadKeystore(config.getKeystoreFile(), config.getKeystoreType().name(), keyStorePassword);
// we have to manually fall back to default keystore. SSLContext won't provide such a functionality.
if (keystore == null) {
String keyStore = System.getProperty("javax.net.ssl.keyStore");
String keyStoreType = System.getProperty("javax.net.ssl.keyStoreType", KeyStore.getDefaultType());
keyStorePassword = System.getProperty("javax.net.ssl.keyStorePassword", "");
-
keystore = loadKeystore(keyStore, keyStoreType, keyStorePassword);
}
- String keystoreAlgorithm =
- (Strings.isNullOrEmpty(config.getKeystoreKeyAlgorithm())) ? KeyManagerFactory.getDefaultAlgorithm()
+ String keystoreAlgorithm = (Strings.isNullOrEmpty(config.getKeystoreKeyAlgorithm()))
+ ? KeyManagerFactory.getDefaultAlgorithm()
: config.getKeystoreKeyAlgorithm();
+
KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(keystoreAlgorithm);
- char[] passwordArr = (keyStorePassword == null) ? null : keyStorePassword.toCharArray();
- keyManagerFactory.init(keystore, passwordArr);
- return keyManagerFactory.getKeyManagers();
+ keyManagerFactory.init(
+ keystore,
+ (keyStorePassword == null) ? null : keyStorePassword.toCharArray()
+ );
+
+ return (Strings.isNullOrEmpty(config.getKeystoreCertAliasName()))
+ ? keyManagerFactory.getKeyManagers()
+ : X509KeyManagerAliasWrapper.getKeyManagers(keyManagerFactory, config.getKeystoreCertAliasName());
}
private TrustManager[] getTrustManagers()
@@ -100,13 +98,17 @@ private TrustManager[] getTrustManagers()
return new TrustManager[] { new TrustAllTrustManager() };
}
- KeyStore trustStore = loadKeystore(config.getTrustStoreFile(), config.getTrustStoreType().name(),
- config.getTrustStorePassword());
+ KeyStore trustStore = loadKeystore(
+ config.getTrustStoreFile(),
+ config.getTrustStoreType().name(),
+ config.getTrustStorePassword()
+ );
+
TrustManager[] trustManagers = null;
if (trustStore != null) {
- String trustStoreAlgorithm =
- (Strings.isNullOrEmpty(config.getTrustStoreKeyAlgorithm())) ? TrustManagerFactory.getDefaultAlgorithm()
- : config.getTrustStoreKeyAlgorithm();
+ String trustStoreAlgorithm = (Strings.isNullOrEmpty(config.getTrustStoreKeyAlgorithm()))
+ ? TrustManagerFactory.getDefaultAlgorithm()
+ : config.getTrustStoreKeyAlgorithm();
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(trustStoreAlgorithm);
trustManagerFactory.init(trustStore);
trustManagers = trustManagerFactory.getTrustManagers();
@@ -117,13 +119,15 @@ private TrustManager[] getTrustManagers()
private static KeyStore loadKeystore(String keystoreFile, String type, String password)
throws IOException, CertificateException, NoSuchAlgorithmException, KeyStoreException {
- KeyStore keystore = null;
- if (keystoreFile != null) {
- keystore = KeyStore.getInstance(type);
- char[] passwordArr = (password == null) ? null : password.toCharArray();
- try (InputStream is = Files.newInputStream(Paths.get(keystoreFile))) {
- keystore.load(is, passwordArr);
- }
+ if (keystoreFile == null) {
+ return null;
+ }
+
+ KeyStore keystore = KeyStore.getInstance(type);
+ char[] passwordArr = (password == null) ? null : password.toCharArray();
+
+ try (InputStream is = Files.newInputStream(Paths.get(keystoreFile))) {
+ keystore.load(is, passwordArr);
}
return keystore;
}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java b/src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java
similarity index 95%
rename from src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java
rename to src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java
index c90743e..86f0b93 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java
+++ b/src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java
@@ -14,7 +14,7 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.http;
+package io.cdap.plugin.http.common.http;
import java.security.cert.X509Certificate;
import javax.net.ssl.X509TrustManager;
diff --git a/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java b/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java
new file mode 100644
index 0000000..c1a2f8a
--- /dev/null
+++ b/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java
@@ -0,0 +1,87 @@
+/*
+ * Copyright © 2021 Cask Data, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package io.cdap.plugin.http.common.http;
+
+import java.net.Socket;
+import java.security.Principal;
+import java.security.PrivateKey;
+import java.security.cert.X509Certificate;
+
+import javax.net.ssl.KeyManager;
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.X509KeyManager;
+
+
+/**
+ * This is just wrapper over SunX509KeyManagerImpl with possibility to provide specific alias
+ *
+ * Usage example:
+ * X509KeyManagerAliasWrapper.getKeyManagers(keyManagerFactory, CERT_ALIAS);
+ */
+public class X509KeyManagerAliasWrapper implements X509KeyManager {
+
+ private final X509KeyManager originalKeyManager;
+ private final String certAlias;
+
+ public X509KeyManagerAliasWrapper(X509KeyManager originalKeyManager, String certAlias) {
+ this.originalKeyManager = originalKeyManager;
+ this.certAlias = certAlias;
+ }
+
+ public static KeyManager[] getKeyManagers(KeyManagerFactory keyManagerFactory, String certAlias) {
+ KeyManager[] keyManagers = keyManagerFactory.getKeyManagers();
+
+ // Current implementation only support X509 Certificates
+ if (keyManagers.length != 1) {
+ return keyManagers;
+ }
+ if (!(keyManagers[0] instanceof X509KeyManager)) {
+ return keyManagers;
+ }
+
+ return new KeyManager[]{ new X509KeyManagerAliasWrapper((X509KeyManager) keyManagers[0], certAlias) };
+ };
+
+ @Override
+ public String chooseClientAlias(String[] strings, Principal[] principals, Socket socket) {
+ return certAlias;
+ }
+
+ @Override
+ public String[] getClientAliases(String s, Principal[] principals) {
+ return originalKeyManager.getClientAliases(s, principals);
+ }
+
+ @Override
+ public String[] getServerAliases(String s, Principal[] principals) {
+ return originalKeyManager.getServerAliases(s, principals);
+ }
+
+ @Override
+ public String chooseServerAlias(String s, Principal[] principals, Socket socket) {
+ return originalKeyManager.chooseServerAlias(s, principals, socket);
+ }
+
+ @Override
+ public X509Certificate[] getCertificateChain(String s) {
+ return originalKeyManager.getCertificateChain(s);
+ }
+
+ @Override
+ public PrivateKey getPrivateKey(String s) {
+ return originalKeyManager.getPrivateKey(s);
+ }
+}
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java
similarity index 89%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java
index ece3486..b05cd89 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java
@@ -13,19 +13,19 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
-
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.RetryPolicy;
-import io.cdap.plugin.http.source.common.error.ErrorHandling;
-import io.cdap.plugin.http.source.common.error.HttpErrorHandler;
-import io.cdap.plugin.http.source.common.error.RetryableErrorHandling;
-import io.cdap.plugin.http.source.common.http.HttpClient;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.page.PageFactory;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
-import io.cdap.plugin.http.source.common.pagination.state.UrlPaginationIteratorState;
+package io.cdap.plugin.http.common.pagination;
+
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.RetryPolicy;
+import io.cdap.plugin.http.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.HttpErrorHandler;
+import io.cdap.plugin.http.common.error.RetryableErrorHandling;
+import io.cdap.plugin.http.common.http.HttpClient;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.page.PageFactory;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.pagination.state.UrlPaginationIteratorState;
import org.awaitility.Awaitility;
import org.awaitility.core.ConditionTimeoutException;
import org.awaitility.pollinterval.FixedPollInterval;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java
similarity index 85%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java
index ad1e832..f7a9dea 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.apache.http.Header;
import java.io.IOException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java
similarity index 84%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java
index 69084fa..4dd41f1 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java
@@ -13,13 +13,13 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.IndexPaginationIteratorState;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.IndexPaginationIteratorState;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java b/src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java
similarity index 98%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java
index 077ce11..7d0157d 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java
@@ -14,7 +14,7 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
import com.google.common.base.Strings;
import org.python.core.PyException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java
similarity index 86%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java
index cdc815d..296cb9d 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java
similarity index 85%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java
index 5e59b35..78e8985 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.apache.http.Header;
import org.apache.http.HeaderElement;
import org.slf4j.Logger;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java
similarity index 78%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java
index 80f48e3..386048f 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java
similarity index 89%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java
index c2cad0d..8eec5c7 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java
@@ -13,10 +13,10 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
/**
* A factory which creates instance of {@BaseHttpPaginationIterator} in accordance to pagination type configured in
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java
similarity index 91%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java
index d567f58..984c4c8 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.EnumWithValue;
+import io.cdap.plugin.http.common.EnumWithValue;
/**
* An enum which represent a type of pagination.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java
similarity index 85%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java
index d3f8804..5788553 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.apache.http.client.utils.URIBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java
similarity index 90%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java
index f6958e0..9d3ab6b 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.Closeable;
import java.util.Arrays;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java
similarity index 91%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java
index 5905196..17fef12 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.api.data.schema.Schema;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.IOException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java
similarity index 90%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java
index 393e675..3c7ed25 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java
@@ -14,12 +14,12 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.format.StructuredRecordStringConverter;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.IOException;
import java.util.NoSuchElementException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java
similarity index 88%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java
index 657dee3..3590b3f 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java
@@ -13,14 +13,14 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.etl.api.InvalidEntry;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.error.ErrorHandling;
-import io.cdap.plugin.http.source.common.error.HttpErrorHandler;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.HttpErrorHandler;
+import io.cdap.plugin.http.common.http.HttpResponse;
/**
* Represents a page with an error
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java
similarity index 97%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java
index f700aa4..8354ec2 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.api.data.schema.Schema;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java
similarity index 99%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java
index a0dc95e..b48b8e0 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import com.google.common.annotations.VisibleForTesting;
import com.google.gson.JsonArray;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java
similarity index 97%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java
index b89d5af..b50fff7 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
@@ -21,8 +21,8 @@
import io.cdap.cdap.api.data.schema.Schema;
import io.cdap.cdap.etl.api.InvalidEntry;
import io.cdap.cdap.format.StructuredRecordStringConverter;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import org.apache.commons.lang3.StringUtils;
import java.util.ArrayList;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java
similarity index 92%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java
index f686f98..bc01e37 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java
@@ -13,11 +13,11 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.etl.api.InvalidEntry;
-import io.cdap.plugin.http.source.common.error.ErrorHandling;
+import io.cdap.plugin.http.common.error.ErrorHandling;
/**
* Represents a single entry found on page. The entry can either be an {@link InvalidEntry} or {@link StructuredRecord}.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java
similarity index 87%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java
index 60b7e9c..a1690a2 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java
@@ -13,11 +13,11 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.error.HttpErrorHandler;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.error.HttpErrorHandler;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.IOException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java
similarity index 89%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java
index 527cf3f..c1f0296 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java
@@ -13,9 +13,9 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
-import io.cdap.plugin.http.source.common.EnumWithValue;
+import io.cdap.plugin.http.common.EnumWithValue;
/**
* An enum which represent format of the page.
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java
similarity index 93%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java
index a3f18b6..50be7ab 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java
@@ -13,12 +13,12 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.api.data.schema.Schema;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.BufferedReader;
import java.io.IOException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java
similarity index 87%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java
index b95564a..d5d7839 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java
@@ -13,11 +13,11 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import io.cdap.cdap.api.data.format.StructuredRecord;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import java.io.IOException;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java
similarity index 95%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java
index c50c990..52cab5d 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.google.gson.JsonArray;
@@ -25,8 +25,8 @@
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.api.data.schema.Schema;
import io.cdap.cdap.format.StructuredRecordStringConverter;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpResponse;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpResponse;
import org.w3c.dom.Document;
import java.util.Iterator;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java
similarity index 98%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java
index b39a89b..71ef81c 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.TextNode;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java
similarity index 89%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java
index 1ad9f9f..563f430 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java
@@ -14,9 +14,9 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.state;
+package io.cdap.plugin.http.common.pagination.state;
-import io.cdap.plugin.http.source.common.pagination.IncrementAnIndexPaginationIterator;
+import io.cdap.plugin.http.common.pagination.IncrementAnIndexPaginationIterator;
import java.util.Objects;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java
similarity index 92%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java
index 4ae7093..ece1900 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java
@@ -14,7 +14,7 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.state;
+package io.cdap.plugin.http.common.pagination.state;
import java.io.Serializable;
diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java
similarity index 95%
rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java
rename to src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java
index b549be5..4b3804f 100644
--- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java
+++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java
@@ -14,7 +14,7 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.state;
+package io.cdap.plugin.http.common.pagination.state;
import java.util.Objects;
diff --git a/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java b/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java
index 4f15644..27db419 100644
--- a/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java
+++ b/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java
@@ -30,6 +30,8 @@
import io.cdap.cdap.etl.api.batch.BatchSink;
import io.cdap.cdap.etl.api.batch.BatchSinkContext;
import io.cdap.cdap.format.StructuredRecordStringConverter;
+import io.cdap.plugin.http.common.http.HttpConstants;
+
import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -60,7 +62,7 @@
* Sink plugin to send the messages from the pipeline to an external http endpoint.
*/
@Plugin(type = BatchSink.PLUGIN_TYPE)
-@Name("HTTP")
+@Name(HttpConstants.HTTP_PLUGIN_NAME)
@Description("Sink plugin to send the messages from the pipeline to an external http endpoint.")
public class HTTPSink extends BatchSink {
diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java
index dc3be1b..72df8ca 100644
--- a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java
+++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java
@@ -31,8 +31,9 @@
import io.cdap.cdap.etl.api.batch.BatchSource;
import io.cdap.cdap.etl.api.batch.BatchSourceContext;
import io.cdap.plugin.common.LineageRecorder;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.page.PageEntry;
+import io.cdap.plugin.http.common.http.HttpConstants;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.page.PageEntry;
import org.apache.hadoop.io.NullWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -43,11 +44,9 @@
* Plugin returns records from HTTP source specified by link. Pagination via APIs is supported.
*/
@Plugin(type = BatchSource.PLUGIN_TYPE)
-@Name(HttpBatchSource.NAME)
+@Name(HttpConstants.HTTP_PLUGIN_NAME)
@Description("Read data from HTTP endpoint.")
public class HttpBatchSource extends BatchSource {
- static final String NAME = "HTTP";
-
private static final Logger LOG = LoggerFactory.getLogger(HttpBatchSource.class);
private final HttpBatchSourceConfig config;
diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java
index 1b02eec..74d3c81 100644
--- a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java
+++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java
@@ -15,7 +15,7 @@
*/
package io.cdap.plugin.http.source.batch;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
/**
* Provides all the configurations required for configuring the {@link HttpBatchSource} plugin.
diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java
index fa9b5e1..c0d82d8 100644
--- a/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java
+++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java
@@ -15,7 +15,7 @@
*/
package io.cdap.plugin.http.source.batch;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.InputSplit;
diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java
index 993c20f..1f172d3 100644
--- a/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java
+++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java
@@ -17,9 +17,9 @@
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
-import io.cdap.plugin.http.source.common.pagination.BaseHttpPaginationIterator;
-import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.BaseHttpPaginationIterator;
+import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.InputSplit;
diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java
index 2585818..a424cb2 100644
--- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java
+++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java
@@ -18,10 +18,10 @@
import com.google.common.base.Throwables;
import io.cdap.cdap.api.data.format.StructuredRecord;
-import io.cdap.plugin.http.source.common.pagination.BaseHttpPaginationIterator;
-import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState;
+import io.cdap.plugin.http.common.pagination.BaseHttpPaginationIterator;
+import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.rdd.RDD;
import org.apache.spark.streaming.StreamingContext;
diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java
index 4d9367d..7e3d5d8 100644
--- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java
+++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java
@@ -26,7 +26,9 @@
import io.cdap.cdap.etl.api.streaming.StreamingSource;
import io.cdap.plugin.common.Constants;
import io.cdap.plugin.common.IdUtils;
+import io.cdap.plugin.http.common.http.HttpConstants;
import org.apache.spark.streaming.api.java.JavaDStream;
+
import scala.reflect.ClassTag;
import scala.reflect.ClassTag$;
@@ -35,10 +37,9 @@
* For paginated APIs once the last page is reached it waits for the next pages.
*/
@Plugin(type = StreamingSource.PLUGIN_TYPE)
-@Name(HttpStreamingSource.NAME)
+@Name(HttpConstants.HTTP_PLUGIN_NAME)
@Description(HttpStreamingSource.DESCRIPTION)
public class HttpStreamingSource extends StreamingSource {
- static final String NAME = "HTTP";
static final String DESCRIPTION = "Read data from HTTP endpoint periodically waiting for updates";
private HttpStreamingSourceConfig config;
diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java
index 333afb8..9d3be85 100644
--- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java
+++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java
@@ -19,7 +19,7 @@
import io.cdap.cdap.api.annotation.Description;
import io.cdap.cdap.api.annotation.Macro;
import io.cdap.cdap.api.annotation.Name;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
import javax.annotation.Nullable;
diff --git a/src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java b/src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java
similarity index 96%
rename from src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java
rename to src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java
index 16d22b5..ae67e14 100644
--- a/src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java
+++ b/src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java
@@ -14,17 +14,17 @@
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination;
+package io.cdap.plugin.http.common.pagination;
import io.cdap.cdap.api.data.format.StructuredRecord;
import io.cdap.cdap.api.data.schema.Schema;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpClient;
+import io.cdap.plugin.http.common.pagination.page.BasePage;
+import io.cdap.plugin.http.common.pagination.page.JSONUtil;
+import io.cdap.plugin.http.common.pagination.page.PageEntry;
+import io.cdap.plugin.http.common.pagination.page.PageFormat;
import io.cdap.plugin.http.source.batch.HttpBatchSourceConfig;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
-import io.cdap.plugin.http.source.common.http.HttpClient;
-import io.cdap.plugin.http.source.common.pagination.page.BasePage;
-import io.cdap.plugin.http.source.common.pagination.page.JSONUtil;
-import io.cdap.plugin.http.source.common.pagination.page.PageEntry;
-import io.cdap.plugin.http.source.common.pagination.page.PageFormat;
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.StatusLine;
diff --git a/src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java b/src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java
similarity index 98%
rename from src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java
rename to src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java
index 5401db9..c94679e 100644
--- a/src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java
+++ b/src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java
@@ -13,7 +13,7 @@
* License for the specific language governing permissions and limitations under
* the License.
*/
-package io.cdap.plugin.http.source.common.pagination.page;
+package io.cdap.plugin.http.common.pagination.page;
import com.google.gson.JsonObject;
import org.junit.Assert;
diff --git a/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java b/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java
index c001c64..6c27b09 100644
--- a/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java
+++ b/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java
@@ -37,8 +37,9 @@
import io.cdap.cdap.test.DataSetManager;
import io.cdap.cdap.test.TestConfiguration;
import io.cdap.cdap.test.WorkflowManager;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpConstants;
import io.cdap.plugin.http.source.batch.HttpBatchSource;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
@@ -85,20 +86,20 @@ public static void setupTestClass() throws Exception {
public List getPipelineResults(Map sourceProperties) throws Exception {
Map allProperties = new ImmutableMap.Builder()
.put("referenceName", name.getMethodName())
- .put(BaseHttpSourceConfig.PROPERTY_HTTP_METHOD, "GET")
- .put(BaseHttpSourceConfig.PROPERTY_OAUTH2_ENABLED, "false")
- .put(BaseHttpSourceConfig.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail")
- .put(BaseHttpSourceConfig.PROPERTY_ERROR_HANDLING, "stopOnError")
- .put(BaseHttpSourceConfig.PROPERTY_RETRY_POLICY, "linear")
+ .put(HttpConstants.PROPERTY_HTTP_METHOD, "GET")
+ .put(HttpConstants.PROPERTY_OAUTH2_ENABLED, "false")
+ .put(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail")
+ .put(HttpConstants.PROPERTY_ERROR_HANDLING, "stopOnError")
+ .put(HttpConstants.PROPERTY_RETRY_POLICY, "linear")
.put(BaseHttpSourceConfig.PROPERTY_MAX_RETRY_DURATION, "10")
.put(BaseHttpSourceConfig.PROPERTY_LINEAR_RETRY_INTERVAL, "1")
.put(BaseHttpSourceConfig.PROPERTY_WAIT_TIME_BETWEEN_PAGES, "0")
- .put(BaseHttpSourceConfig.PROPERTY_CONNECT_TIMEOUT, "60")
- .put(BaseHttpSourceConfig.PROPERTY_READ_TIMEOUT, "120")
- .put(BaseHttpSourceConfig.PROPERTY_VERIFY_HTTPS, "true")
- .put(BaseHttpSourceConfig.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)")
- .put(BaseHttpSourceConfig.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)")
- .put(BaseHttpSourceConfig.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2")
+ .put(HttpConstants.PROPERTY_CONNECT_TIMEOUT, "60")
+ .put(HttpConstants.PROPERTY_READ_TIMEOUT, "120")
+ .put(HttpConstants.PROPERTY_VERIFY_HTTPS, "true")
+ .put(HttpConstants.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)")
+ .put(HttpConstants.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)")
+ .put(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2")
.putAll(sourceProperties)
.build();
diff --git a/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java b/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java
index 07a594e..5f52575 100644
--- a/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java
+++ b/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java
@@ -22,7 +22,8 @@
import io.cdap.cdap.api.data.schema.Schema;
import io.cdap.cdap.etl.mock.test.HydratorTestBase;
import io.cdap.cdap.test.TestConfiguration;
-import io.cdap.plugin.http.source.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.BaseHttpSourceConfig;
+import io.cdap.plugin.http.common.http.HttpConstants;
import org.junit.Assert;
import org.junit.ClassRule;
import org.junit.Rule;
@@ -70,7 +71,7 @@ public void testIncrementAnIndex() throws Exception {
);
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL,
+ .put(HttpConstants.PROPERTY_URL,
getServerAddress() + "/rest/api/2/search?maxResults=2&startAt={pagination.index}")
//"https://issues.cask.co/rest/api/2/search?maxResults=2&startAt={pagination.index}")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json")
@@ -79,7 +80,7 @@ public void testIncrementAnIndex() throws Exception {
"type:/fields/issuetype/name,description:/fields/description," +
"projectCategory:/fields/project/projectCategory/name,isSubtask:/fields/issuetype/subtask," +
"fixVersions:/fields/fixVersions")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Increment an index")
.put(BaseHttpSourceConfig.PROPERTY_START_INDEX, "0")
.put(BaseHttpSourceConfig.PROPERTY_INDEX_INCREMENT, "2")
@@ -124,14 +125,14 @@ public void testIncrementAnIndexXml() throws Exception {
// https://services.odata.org
// /V3/Northwind/Northwind.svc/Customers?$inlinecount=allpages&$top=5&$skip={pagination.index}
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/V3/Northwind/Northwind.svc" +
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/V3/Northwind/Northwind.svc" +
"/Customers?$inlinecount=allpages&$top=5&$skip={pagination.index}")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "xml")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/feed/entry")
.put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING,
"companyName:content/properties/CompanyName,postalCode:content/properties/PostalCode," +
"country:content/properties/Country,phone:content/properties/Phone,fax:content/properties/Fax/text()")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Increment an index")
.put(BaseHttpSourceConfig.PROPERTY_START_INDEX, "0")
.put(BaseHttpSourceConfig.PROPERTY_INDEX_INCREMENT, "20")
@@ -177,12 +178,12 @@ public void testLinkInResponseBody() throws Exception {
Schema.of(Schema.Type.STRING))
);
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL,
+ .put(HttpConstants.PROPERTY_URL,
// http://confluence.atlassian.com/rest/api/space/ADMINJIRASERVER0710/content/page?limit=100&start=0
getServerAddress() + "/rest/api/space/ADMINJIRASERVER0710/content/page?limit=2&start=0")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Link in response body")
.put(BaseHttpSourceConfig.PROPERTY_NEXT_PAGE_FIELD_PATH, "/_links/next")
.build();
@@ -222,13 +223,13 @@ public void testLinkInResponseHeader() throws Exception {
))
);
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL,
+ .put(HttpConstants.PROPERTY_URL,
//"https://api.github.com/search/code?q=Salesforce+user:data-integrations")
getServerAddress() + "/search/code?q=Salesforce+user:data-integrations&per_page=2")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/items")
.put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "path:/path,score:/score,repository:/repository")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Link in response header")
.build();
@@ -282,11 +283,11 @@ public void testNonePagination() throws Exception {
Map properties = new ImmutableMap.Builder()
// https://searchcode.com/api/codesearch_I/?q=cdap
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results")
.put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "repo:repo,language:/language,file:/filename,url:/url")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None")
.build();
@@ -307,9 +308,9 @@ public void testNonePaginationBlob() throws Exception {
);
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/blob")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/blob")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "blob")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None")
.build();
@@ -347,9 +348,9 @@ public void testNonePaginationCSV() throws Exception {
Map properties = new ImmutableMap.Builder()
// https://s3.theeventscalendar.com/uploads/2014/09/test-data-venues11.csv
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/uploads/2014/09/test-data-venues11.csv")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/uploads/2014/09/test-data-venues11.csv")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "csv")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None")
.build();
@@ -370,9 +371,9 @@ public void testNonePaginationText() throws Exception {
);
//"http://www.columbia.edu/~fdc/sample.html")
Map properties = new ImmutableMap.Builder()
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/sample.html")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/sample.html")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "text")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None")
.build();
@@ -395,11 +396,11 @@ public void testNonePaginationXml() throws Exception {
Map properties = new ImmutableMap.Builder()
// https://www.w3.org/2003/05/soap-envelope/
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/2003/05/soap-envelope/")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/2003/05/soap-envelope/")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "xml")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/schema/complexType[@name='Fault']/sequence/element")
.put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "name:@name,type:@type")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None")
.build();
@@ -449,11 +450,11 @@ public void testPaginationCustom() throws Exception {
Map properties = new ImmutableMap.Builder()
// https://searchcode.com/api/codesearch_I/?q=cdap&per_page=2
- .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap&per_page=2")
+ .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap&per_page=2")
.put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json")
.put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results")
.put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "repo:repo,language:/language,file:/filename,url:/url")
- .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString())
+ .put(HttpConstants.PROPERTY_SCHEMA, schema.toString())
.put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Custom")
.put(BaseHttpSourceConfig.PROPERTY_CUSTOM_PAGINATION_CODE, paginationCode)
.build();
@@ -485,20 +486,20 @@ public void testPaginationCustom() throws Exception {
protected Map getProperties(Map sourceProperties) {
return new ImmutableMap.Builder()
.put("referenceName", testName.getMethodName())
- .put(BaseHttpSourceConfig.PROPERTY_HTTP_METHOD, "GET")
- .put(BaseHttpSourceConfig.PROPERTY_OAUTH2_ENABLED, "false")
- .put(BaseHttpSourceConfig.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail")
- .put(BaseHttpSourceConfig.PROPERTY_ERROR_HANDLING, "stopOnError")
- .put(BaseHttpSourceConfig.PROPERTY_RETRY_POLICY, "linear")
+ .put(HttpConstants.PROPERTY_HTTP_METHOD, "GET")
+ .put(HttpConstants.PROPERTY_OAUTH2_ENABLED, "false")
+ .put(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail")
+ .put(HttpConstants.PROPERTY_ERROR_HANDLING, "stopOnError")
+ .put(HttpConstants.PROPERTY_RETRY_POLICY, "linear")
.put(BaseHttpSourceConfig.PROPERTY_MAX_RETRY_DURATION, "10")
.put(BaseHttpSourceConfig.PROPERTY_LINEAR_RETRY_INTERVAL, "1")
.put(BaseHttpSourceConfig.PROPERTY_WAIT_TIME_BETWEEN_PAGES, "0")
- .put(BaseHttpSourceConfig.PROPERTY_CONNECT_TIMEOUT, "60")
- .put(BaseHttpSourceConfig.PROPERTY_READ_TIMEOUT, "120")
- .put(BaseHttpSourceConfig.PROPERTY_VERIFY_HTTPS, "true")
- .put(BaseHttpSourceConfig.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)")
- .put(BaseHttpSourceConfig.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)")
- .put(BaseHttpSourceConfig.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2")
+ .put(HttpConstants.PROPERTY_CONNECT_TIMEOUT, "60")
+ .put(HttpConstants.PROPERTY_READ_TIMEOUT, "120")
+ .put(HttpConstants.PROPERTY_VERIFY_HTTPS, "true")
+ .put(HttpConstants.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)")
+ .put(HttpConstants.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)")
+ .put(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2")
.putAll(sourceProperties)
.build();
}
diff --git a/widgets/HTTP-action.json b/widgets/HTTP-action.json
new file mode 100644
index 0000000..4f2ab93
--- /dev/null
+++ b/widgets/HTTP-action.json
@@ -0,0 +1,384 @@
+{
+ "metadata": {
+ "spec-version": "1.5"
+ },
+ "configuration-groups": [
+ {
+ "label": "General",
+ "properties": [
+ {
+ "widget-type": "textbox",
+ "label": "URL",
+ "name": "url"
+ },
+ {
+ "widget-type": "select",
+ "label": "HTTP Method",
+ "name": "httpMethod",
+ "widget-attributes": {
+ "values": [
+ "GET",
+ "POST",
+ "PUT",
+ "DELETE",
+ "HEAD"
+ ],
+ "default": "GET"
+ }
+ },
+ {
+ "widget-type": "keyvalue",
+ "label": "Headers",
+ "name": "headers",
+ "widget-attributes": {
+ "showDelimiter": "false"
+ }
+ },
+ {
+ "widget-type": "textarea",
+ "name": "requestBody",
+ "label": "Request Body",
+ "widget-attributes": {
+ "rows": "5"
+ }
+ }
+ ]
+ },
+ {
+ "label": "OAuth2",
+ "properties": [
+ {
+ "widget-type": "toggle",
+ "label": "OAuth2 Enabled",
+ "name": "oauth2Enabled",
+ "widget-attributes": {
+ "default": "false",
+ "on": {
+ "label": "True",
+ "value": "true"
+ },
+ "off": {
+ "label": "False",
+ "value": "false"
+ }
+ }
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Auth URL",
+ "name": "authUrl"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Token URL",
+ "name": "tokenUrl"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Client ID",
+ "name": "clientId"
+ },
+ {
+ "widget-type": "password",
+ "label": "Client Secret",
+ "name": "clientSecret"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Scopes",
+ "name": "scopes"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Refresh Token",
+ "name": "refreshToken"
+ }
+ ]
+ },
+ {
+ "label": "Basic Authentication",
+ "properties": [
+ {
+ "widget-type": "textbox",
+ "label": "Username",
+ "name": "username"
+ },
+ {
+ "widget-type": "password",
+ "label": "Password",
+ "name": "password"
+ }
+ ]
+ },
+ {
+ "label": "HTTP Proxy",
+ "properties": [
+ {
+ "widget-type": "textbox",
+ "label": "Proxy URL",
+ "name": "proxyUrl"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Username",
+ "name": "proxyUsername"
+ },
+ {
+ "widget-type": "password",
+ "label": "Password",
+ "name": "proxyPassword"
+ }
+ ]
+ },
+ {
+ "label": "Error Handling",
+ "properties": [
+ {
+ "widget-type": "keyvalue-dropdown",
+ "label": "HTTP Errors Handling",
+ "name": "httpErrorsHandling",
+ "widget-attributes": {
+ "default": "2..:Success,.*:Fail",
+ "showDelimiter": "false",
+ "dropdownOptions": [
+ "Success",
+ "Fail"
+ ],
+ "key-placeholder": "HTTP Status Code Regex"
+ }
+ },
+ {
+ "widget-type": "radio-group",
+ "label": "Non-HTTP Error Handling",
+ "name": "errorHandling",
+ "widget-attributes": {
+ "layout": "inline",
+ "default": "stopOnError",
+ "options": [
+ {
+ "id": "stopOnError",
+ "label": "Stop on error"
+ }
+ ]
+ }
+ },
+ {
+ "widget-type": "number",
+ "label": "Connect Timeout",
+ "name": "connectTimeout",
+ "widget-attributes": {
+ "min": "0",
+ "default": "120"
+ }
+ },
+ {
+ "widget-type": "number",
+ "label": "Read Timeout",
+ "name": "readTimeout",
+ "widget-attributes": {
+ "min": "0",
+ "default": "120"
+ }
+ }
+ ]
+ },
+ {
+ "label": "SSL/TLS",
+ "properties": [
+ {
+ "widget-type": "toggle",
+ "label": "Verify HTTPS Trust Certificates",
+ "name": "verifyHttps",
+ "widget-attributes": {
+ "default": "true",
+ "on": {
+ "label": "True",
+ "value": "true"
+ },
+ "off": {
+ "label": "False",
+ "value": "false"
+ }
+ }
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Keystore File",
+ "name": "keystoreFile"
+ },
+ {
+ "widget-type": "select",
+ "label": "Keystore Type",
+ "name": "keystoreType",
+ "widget-attributes": {
+ "default": "Java KeyStore (JKS)",
+ "values": [
+ "Java KeyStore (JKS)",
+ "Java Cryptography Extension KeyStore (JCEKS)",
+ "PKCS #12"
+ ]
+ }
+ },
+ {
+ "widget-type": "password",
+ "label": "Keystore Password",
+ "name": "keystorePassword"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Keystore Key Algorithm",
+ "name": "keystoreKeyAlgorithm",
+ "widget-attributes": {
+ "default": "SunX509"
+ }
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Keystore Cert Alias",
+ "name": "keystoreCertAlias"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "TrustStore File",
+ "name": "trustStoreFile"
+ },
+ {
+ "widget-type": "select",
+ "label": "TrustStore Type",
+ "name": "trustStoreType",
+ "widget-attributes": {
+ "default": "Java KeyStore (JKS)",
+ "values": [
+ "Java KeyStore (JKS)",
+ "Java Cryptography Extension KeyStore (JCEKS)",
+ "PKCS #12"
+ ]
+ }
+ },
+ {
+ "widget-type": "password",
+ "label": "TrustStore Password",
+ "name": "trustStorePassword"
+ },
+ {
+ "widget-type": "textbox",
+ "label": "TrustStore Key Algorithm",
+ "name": "trustStoreKeyAlgorithm",
+ "widget-attributes": {
+ "default": "SunX509"
+ }
+ },
+ {
+ "widget-type": "csv",
+ "label": "Transport Protocols",
+ "name": "transportProtocols",
+ "widget-attributes": {
+ "default": "TLSv1.2"
+ }
+ },
+ {
+ "widget-type": "textbox",
+ "label": "Cipher Suites",
+ "name": "cipherSuites"
+ }
+ ]
+ }
+ ],
+ "filters": [
+ {
+ "name": "Proxy authentication",
+ "condition": {
+ "property": "proxyUrl",
+ "operator": "exists"
+ },
+ "show": [
+ {
+ "name": "proxyUsername",
+ "type": "property"
+ },
+ {
+ "name": "proxyPassword",
+ "type": "property"
+ }
+ ]
+ },
+ {
+ "name": "OAuth 2 disabled",
+ "condition": {
+ "property": "oauth2Enabled",
+ "operator": "equal to",
+ "value": "false"
+ },
+ "show": [
+ {
+ "name": "username",
+ "type": "property"
+ },
+ {
+ "name": "password",
+ "type": "property"
+ }
+ ]
+ },
+ {
+ "name": "OAuth 2 enabled",
+ "condition": {
+ "property": "oauth2Enabled",
+ "operator": "equal to",
+ "value": "true"
+ },
+ "show": [
+ {
+ "name": "authUrl",
+ "type": "property"
+ },
+ {
+ "name": "tokenUrl",
+ "type": "property"
+ },
+ {
+ "name": "clientId",
+ "type": "property"
+ },
+ {
+ "name": "clientSecret",
+ "type": "property"
+ },
+ {
+ "name": "scopes",
+ "type": "property"
+ },
+ {
+ "name": "refreshToken",
+ "type": "property"
+ }
+ ]
+ },
+ {
+ "name": "SSL HTTPS Verification",
+ "condition": {
+ "property": "verifyHttps",
+ "operator": "equal to",
+ "value": "true"
+ },
+ "show": [
+ {
+ "name": "trustStoreFile",
+ "type": "property"
+ },
+ {
+ "name": "trustStoreType",
+ "type": "property"
+ },
+ {
+ "name": "trustStorePassword",
+ "type": "property"
+ },
+ {
+ "name": "trustStoreKeyAlgorithm",
+ "type": "property"
+ }
+ ]
+ }
+ ]
+}
diff --git a/widgets/HTTP-batchsource.json b/widgets/HTTP-batchsource.json
index 6c05378..373101e 100644
--- a/widgets/HTTP-batchsource.json
+++ b/widgets/HTTP-batchsource.json
@@ -426,6 +426,11 @@
"default": "SunX509"
}
},
+ {
+ "widget-type": "textbox",
+ "label": "Keystore Cert Alias",
+ "name": "keystoreCertAlias"
+ },
{
"widget-type": "textbox",
"label": "TrustStore File",
@@ -503,7 +508,7 @@
"name": "Proxy authentication",
"condition": {
"property": "proxyUrl",
- "operator": "exists",
+ "operator": "exists"
},
"show": [
{
diff --git a/widgets/HTTP-streamingsource.json b/widgets/HTTP-streamingsource.json
index e7abdb6..798a6fe 100644
--- a/widgets/HTTP-streamingsource.json
+++ b/widgets/HTTP-streamingsource.json
@@ -425,6 +425,11 @@
"default": "SunX509"
}
},
+ {
+ "widget-type": "textbox",
+ "label": "Keystore Cert Alias",
+ "name": "keystoreCertAlias"
+ },
{
"widget-type": "textbox",
"label": "TrustStore File",