diff --git a/docs/HTTP-action.md b/docs/HTTP-action.md new file mode 100644 index 0000000..3073d02 --- /dev/null +++ b/docs/HTTP-action.md @@ -0,0 +1,119 @@ +# HTTP Action + +Description +----------- +This plugin executing HTTP/HTTPS requests + +Properties +---------- + +### General + +**URL:** Url to fetch to the first page. +The url must start with a protocol (e.g. http://). + +**HTTP Method:** HTTP request method. + +**Headers:** Headers to send with each HTTP request. + +**Request body:** Body to send with each HTTP request. + + +### Basic Authentication + +**Username:** Username for basic authentication. + +**Password:** Password for basic authentication. + +### HTTP Proxy + +**Proxy URL:** Proxy URL. Must contain a protocol, address and port. + +**Username:** Proxy username. + +**Password:** Proxy password. + +### Error Handling + +**HTTP Errors Handling:** Defines the error handling strategy to use for certain HTTP response codes. +The left column contains a regular expression for HTTP status code. The right column contains an action which +is done in case of match. If HTTP status code matches multiple regular expressions, the first specified in mapping +is matched. + +Example: + +| HTTP Code Regexp | Error Handling | +| ----------------- |:-----------------------:| +| 2.. | Success | +| 401 | Retry and fail | +| 4.. | Fail | +| 5.. | Retry and send to error | +| .* | Fail | + +Note: pagination types "Link in response header", "Link in response body", "Token in response body" do not support +"Send to error", "Skip", "Retry and send to error", "Retry and skip" options. + +**Non-HTTP Error Handling:** Error handling strategy to use when the HTTP response cannot be transformed to an output record. +Possible values are:
+Stop on error - Fails pipeline due to erroneous record.
+Send to error - Sends erroneous record's text to error port and continues.
+Skip on error - Ignores erroneous records. + +**Connect Timeout:** Maximum time in seconds connection initialization is allowed to take. + +### OAuth2 + +**OAuth2 Enabled:** If true, plugin will perform OAuth2 authentication. + +**Auth URL:** Endpoint for the authorization server used to retrieve the authorization code. + +**Token URL:** Endpoint for the resource server, which exchanges the authorization code for an access token. + +**Client ID:** Client identifier obtained during the Application registration process. + +**Client Secret:** Client secret obtained during the Application registration process. + +**Scopes:** Scope of the access request, which might have multiple space-separated values. + +**Refresh Token:** Token used to receive accessToken, which is end product of OAuth2. + +### SSL/TLS + +**Verify HTTPS Trust Certificates:** If false, untrusted trust certificates (e.g. self signed), will not lead to an +error. Do not disable this in production environment on a network you do not entirely trust. Especially public internet. + +**Keystore File:** A path to a file which contains keystore. + +**Keystore Type:** Format of a keystore. + +**Keystore Password:** Password for a keystore. If a keystore is not password protected leave it empty. + +**Keystore Key Algorithm:** An algorithm used for keystore. + +**Keystore Cert Alias** + + Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores. + +Below is an example how the store need to be prepared: + ``` + cat client.crt client.key > client-bundle.pem + + openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS} + + keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \ + -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS} + ``` + +**TrustStore File:** A path to a file which contains truststore. + +**TrustStore Type:** Format of a truststore. + +**TrustStore Password:** Password for a truststore. If a truststore is not password protected leave it empty. + +**TrustStore Key Algorithm:** An algorithm used for truststore. + +**Transport Protocols:** Transport protocols which are allowed for connection. + +**Cipher Suites:** Cipher suites which are allowed for connection. +Colons, commas or spaces are also acceptable separators. + diff --git a/docs/HTTP-batchsource.md b/docs/HTTP-batchsource.md index 37ed1e2..1e22786 100644 --- a/docs/HTTP-batchsource.md +++ b/docs/HTTP-batchsource.md @@ -403,6 +403,20 @@ error. Do not disable this in production environment on a network you do not ent **Keystore Key Algorithm:** An algorithm used for keystore. +**Keystore Cert Alias** + + Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores. + +Below is an example how the store need to be prepared: + ``` + cat client.crt client.key > client-bundle.pem + + openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS} + + keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \ + -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS} + ``` + **TrustStore File:** A path to a file which contains truststore. **TrustStore Type:** Format of a truststore. diff --git a/docs/HTTP-streamingsource.md b/docs/HTTP-streamingsource.md index 4cad538..eafe35f 100644 --- a/docs/HTTP-streamingsource.md +++ b/docs/HTTP-streamingsource.md @@ -410,6 +410,20 @@ error. Do not disable this in production environment on a network you do not ent **Keystore Key Algorithm:** An algorithm used for keystore. +**Keystore Cert Alias** + +Alias of the key in the keystore to be used for communication. This options is supported only by X.509 keys or keystores. + +Below is an example how the store need to be prepared: + ``` + cat client.crt client.key > client-bundle.pem + + openssl pkcs12 -export -in client-bundle.pem -out full-chain.keycert.p12 -name ${CERT_ALIAS} + + keytool -importkeystore -srckeystore full-chain.keycert.p12 -srcstoretype pkcs12 -srcalias ${CERT_ALIAS} \ + -destkeystore identity.jks -deststoretype jks -destalias ${CERT_ALIAS} + ``` + **TrustStore File:** A path to a file which contains truststore. **TrustStore Type:** Format of a truststore. diff --git a/icons/HTTP-action.png b/icons/HTTP-action.png new file mode 100644 index 0000000..933bf62 Binary files /dev/null and b/icons/HTTP-action.png differ diff --git a/pom.xml b/pom.xml index d49ef2e..87d37be 100644 --- a/pom.xml +++ b/pom.xml @@ -21,7 +21,7 @@ HTTP Plugins io.cdap http-plugins - 1.4.0-SNAPSHOT + 1.4.2-SNAPSHOT @@ -79,10 +79,12 @@ 6.1.1 3.9 1.12 + 1.2 + 1.2.17 2.8.5 2.3.0 4.5.9 - 2.4.0-SNAPSHOT + 2.4.0 2.9.9 4.11 2.7.1 @@ -93,6 +95,20 @@ + + + commons-logging + commons-logging + ${common.logging.version} + compile + + + log4j + log4j + ${log4j.version} + compile + + io.cdap.cdap cdap-api diff --git a/src/main/java/io/cdap/plugin/http/action/HttpAction.java b/src/main/java/io/cdap/plugin/http/action/HttpAction.java new file mode 100644 index 0000000..e611809 --- /dev/null +++ b/src/main/java/io/cdap/plugin/http/action/HttpAction.java @@ -0,0 +1,101 @@ +/* + * Copyright © 2021 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package io.cdap.plugin.http.action; + +import io.cdap.cdap.api.annotation.Description; +import io.cdap.cdap.api.annotation.Name; +import io.cdap.cdap.api.annotation.Plugin; +import io.cdap.cdap.etl.api.PipelineConfigurer; +import io.cdap.cdap.etl.api.action.Action; +import io.cdap.cdap.etl.api.action.ActionContext; +import io.cdap.plugin.http.common.RetryPolicy; +import io.cdap.plugin.http.common.error.HttpErrorHandler; +import io.cdap.plugin.http.common.error.RetryableErrorHandling; +import io.cdap.plugin.http.common.http.HttpClient; +import io.cdap.plugin.http.common.http.HttpConstants; +import io.cdap.plugin.http.common.http.HttpResponse; +import org.awaitility.Awaitility; +import org.awaitility.core.ConditionTimeoutException; +import org.awaitility.pollinterval.FixedPollInterval; +import org.awaitility.pollinterval.IterativePollInterval; +import org.awaitility.pollinterval.PollInterval; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Objects; +import java.util.concurrent.TimeUnit; + +/** + * HTTP Action Plugin + */ +@Plugin(type = Action.PLUGIN_TYPE) +@Name(HttpConstants.HTTP_PLUGIN_NAME) +@Description("Action that runs a HTTP command") +public class HttpAction extends Action { + + private static final Logger LOG = LoggerFactory.getLogger(HttpAction.class); + private final HttpActionConfig config; + + public HttpAction(HttpActionConfig config) { + this.config = config; + } + + private boolean makeRequest(HttpClient httpClient, String uri, HttpErrorHandler errorHandler) throws IOException { + try (HttpResponse response = new HttpResponse(httpClient.executeHTTP(uri))) { + RetryableErrorHandling errorHandlingStrategy = errorHandler.getErrorHandlingStrategy(response.getStatusCode()); + + if (errorHandlingStrategy == RetryableErrorHandling.FAIL) { + String body = "null"; + try { + body = response.getBody(); + } catch (Exception ignore) { + } + LOG.warn( + String.format("Request to the url '%s' failed with error code %s and message: %s", + uri, + response.getStatusCode(), + body + )); + } + return !errorHandlingStrategy.shouldRetry(); + } + } + + @Override + public void run(ActionContext actionContext) throws Exception { + try (HttpClient httpClient = new HttpClient(config)) { + HttpErrorHandler httpErrorHandler = new HttpErrorHandler(config); + PollInterval pollInterval = (config.getRetryPolicy().equals(RetryPolicy.LINEAR)) + ? FixedPollInterval.fixed(Objects.requireNonNull(config.getLinearRetryInterval()), TimeUnit.SECONDS) + : IterativePollInterval.iterative(duration -> duration.multiply(2)); + + Awaitility + .await().with() + .pollInterval(pollInterval) + .timeout(config.getMaxRetryDuration(), TimeUnit.SECONDS) + .until(() -> makeRequest(httpClient, config.getUrl(), httpErrorHandler)); + } catch (ConditionTimeoutException e) { + LOG.warn(String.format("Request to the url '%s' failed due to timeout", config.getUrl())); + } + } + + @Override + public void configurePipeline(PipelineConfigurer pipelineConfigurer) { + super.configurePipeline(pipelineConfigurer); + config.validate(); + } +} diff --git a/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java b/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java new file mode 100644 index 0000000..101b8c9 --- /dev/null +++ b/src/main/java/io/cdap/plugin/http/action/HttpActionConfig.java @@ -0,0 +1,495 @@ +/* + * Copyright © 2021 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package io.cdap.plugin.http.action; + +import io.cdap.cdap.api.annotation.Description; +import io.cdap.cdap.api.annotation.Macro; +import io.cdap.cdap.api.annotation.Name; +import io.cdap.cdap.api.plugin.PluginConfig; +import io.cdap.cdap.etl.api.validation.InvalidConfigPropertyException; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.RetryPolicy; +import io.cdap.plugin.http.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity; +import io.cdap.plugin.http.common.error.RetryableErrorHandling; +import io.cdap.plugin.http.common.http.HttpConstants; +import io.cdap.plugin.http.common.http.IHttpConfig; +import io.cdap.plugin.http.common.http.KeyStoreType; + +import java.net.URI; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; +import java.util.regex.PatternSyntaxException; + +import javax.annotation.Nullable; + +/** + * Http Action plugin Config + */ +public class HttpActionConfig extends PluginConfig implements IHttpConfig { + + @Name(HttpConstants.PROPERTY_URL) + @Description("Url to fetch to the first page. The url must start with a protocol (e.g. http://).") + @Macro + protected String url; + + @Name(HttpConstants.PROPERTY_HTTP_METHOD) + @Description("HTTP request method.") + @Macro + protected String httpMethod; + + @Name(HttpConstants.PROPERTY_HEADERS) + @Nullable + @Description("Headers to send with each HTTP request.") + @Macro + protected String headers; + + @Nullable + @Name(HttpConstants.PROPERTY_REQUEST_BODY) + @Description("Body to send with each HTTP request.") + @Macro + protected String requestBody; + + @Nullable + @Name(HttpConstants.PROPERTY_USERNAME) + @Description("Username for basic authentication.") + @Macro + protected String username; + + @Nullable + @Name(HttpConstants.PROPERTY_PASSWORD) + @Description("Password for basic authentication.") + @Macro + protected String password; + + @Nullable + @Name(HttpConstants.PROPERTY_PROXY_URL) + @Description("Proxy URL. Must contain a protocol, address and port.") + @Macro + protected String proxyUrl; + + @Nullable + @Name(HttpConstants.PROPERTY_PROXY_USERNAME) + @Description("Proxy username.") + @Macro + protected String proxyUsername; + + @Nullable + @Name(HttpConstants.PROPERTY_PROXY_PASSWORD) + @Description("Proxy password.") + @Macro + protected String proxyPassword; + + @Nullable + @Name(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING) + @Description("Defines the error handling strategy to use for certain HTTP response codes." + + "The left column contains a regular expression for HTTP status code. The right column contains an action which" + + "is done in case of match. If HTTP status code matches multiple regular expressions, " + + "the first specified in mapping is matched.") + protected String httpErrorsHandling; + + @Name(HttpConstants.PROPERTY_ERROR_HANDLING) + @Description("Error handling strategy to use when the HTTP response cannot be transformed to an output record.") + protected String errorHandling; + + + @Name(HttpConstants.PROPERTY_CONNECT_TIMEOUT) + @Description("Maximum time in seconds connection initialization is allowed to take.") + @Macro + protected Integer connectTimeout; + + @Name(HttpConstants.PROPERTY_READ_TIMEOUT) + @Description("Maximum time in seconds fetching data from the server is allowed to take.") + @Macro + protected Integer readTimeout; + + @Name(HttpConstants.PROPERTY_OAUTH2_ENABLED) + @Description("If true, plugin will perform OAuth2 authentication.") + protected String oauth2Enabled; + + @Nullable + @Name(HttpConstants.PROPERTY_AUTH_URL) + @Description("Endpoint for the authorization server used to retrieve the authorization code.") + @Macro + protected String authUrl; + + @Nullable + @Name(HttpConstants.PROPERTY_TOKEN_URL) + @Description("Endpoint for the resource server, which exchanges the authorization code for an access token.") + @Macro + protected String tokenUrl; + + @Nullable + @Name(HttpConstants.PROPERTY_CLIENT_ID) + @Description("Client identifier obtained during the Application registration process.") + @Macro + protected String clientId; + + @Nullable + @Name(HttpConstants.PROPERTY_CLIENT_SECRET) + @Description("Client secret obtained during the Application registration process.") + @Macro + protected String clientSecret; + + @Nullable + @Name(HttpConstants.PROPERTY_SCOPES) + @Description("Scope of the access request, which might have multiple space-separated values.") + @Macro + protected String scopes; + + @Nullable + @Name(HttpConstants.PROPERTY_REFRESH_TOKEN) + @Description("Token used to receive accessToken, which is end product of OAuth2.") + @Macro + protected String refreshToken; + + @Name(HttpConstants.PROPERTY_VERIFY_HTTPS) + @Description("If false, untrusted trust certificates (e.g. self signed), will not lead to an" + + "error. Do not disable this in production environment on a network you do not entirely trust. " + + "Especially public internet.") + @Macro + protected String verifyHttps; + + @Nullable + @Name(HttpConstants.PROPERTY_KEYSTORE_FILE) + @Description("A path to a file which contains keystore.") + @Macro + protected String keystoreFile; + + @Nullable + @Name(HttpConstants.PROPERTY_KEYSTORE_TYPE) + @Description("Format of a keystore.") + @Macro + protected String keystoreType; + + @Nullable + @Name(HttpConstants.PROPERTY_KEYSTORE_PASSWORD) + @Description("Password for a keystore. If a keystore is not password protected leave it empty.") + @Macro + protected String keystorePassword; + + @Nullable + @Name(HttpConstants.PROPERTY_KEYSTORE_KEY_ALGORITHM) + @Description("An algorithm used for keystore.") + @Macro + protected String keystoreKeyAlgorithm; + + @Nullable + @Name(HttpConstants.PROPERTY_TRUSTSTORE_FILE) + @Description("A path to a file which contains truststore.") + @Macro + protected String trustStoreFile; + + @Nullable + @Name(HttpConstants.PROPERTY_TRUSTSTORE_TYPE) + @Description("Format of a truststore.") + @Macro + protected String trustStoreType; + + @Nullable + @Name(HttpConstants.PROPERTY_TRUSTSTORE_PASSWORD) + @Description("Password for a truststore. If a truststore is not password protected leave it empty.") + @Macro + protected String trustStorePassword; + + @Nullable + @Name(HttpConstants.PROPERTY_TRUSTSTORE_KEY_ALGORITHM) + @Description("An algorithm used for truststore.") + @Macro + protected String trustStoreKeyAlgorithm; + + @Nullable + @Name(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS) + @Description("Transport protocols which are allowed for connection.") + @Macro + protected String transportProtocols; + + @Nullable + @Name(HttpConstants.PROPERTY_CIPHER_SUITES) + @Description("Cipher suites which are allowed for connection. " + + "Colons, commas or spaces are also acceptable separators.") + @Macro + protected String cipherSuites; + + @Name(HttpConstants.PROPERTY_KEYSTORE_CERT_ALIAS) + @Macro + @Nullable + @Description("Alias of the key in the keystore to be used for communication") + protected String keystoreCertAliasName; + + @Override + public String getUrl() { + return url; + } + + @Override + public String getHttpMethod() { + return httpMethod; + } + + @Override + @Nullable + public String getHeaders() { + return headers; + } + + @Override + @Nullable + public String getRequestBody() { + return requestBody; + } + + @Override + @Nullable + public String getUsername() { + return username; + } + + @Override + @Nullable + public String getPassword() { + return password; + } + + @Override + @Nullable + public String getProxyUrl() { + return proxyUrl; + } + + @Override + @Nullable + public String getProxyUsername() { + return proxyUsername; + } + + @Override + @Nullable + public String getProxyPassword() { + return proxyPassword; + } + + @Override + public ErrorHandling getErrorHandling() { + return BaseHttpSourceConfig.getEnumValueByString( + ErrorHandling.class, + errorHandling, + HttpConstants.PROPERTY_ERROR_HANDLING + ); + } + + @Override + public List getHttpErrorHandlingEntries() { + Map httpErrorsHandlingMap = BaseHttpSourceConfig.getMapFromKeyValueString(httpErrorsHandling); + List results = new ArrayList<>(httpErrorsHandlingMap.size()); + + for (Map.Entry entry : httpErrorsHandlingMap.entrySet()) { + String regex = entry.getKey(); + try { + results.add(new HttpErrorHandlerEntity(Pattern.compile(regex), + BaseHttpSourceConfig.getEnumValueByString(RetryableErrorHandling.class, + entry.getValue(), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING))); + } catch (PatternSyntaxException e) { + // We embed causing exception message into this one. Since this message is shown on UI when validation fails. + throw new InvalidConfigPropertyException( + String.format("Error handling regex '%s' is not valid. %s", regex, e.getMessage()), + HttpConstants.PROPERTY_HTTP_ERROR_HANDLING + ); + } + } + return results; + } + + @Override + public Long getLinearRetryInterval() { + return 0L; + } + + @Override + public Long getMaxRetryDuration() { + return 1L; + } + + @Override + public RetryPolicy getRetryPolicy() { + return RetryPolicy.LINEAR; + } + + @Override + public Integer getConnectTimeout() { + return connectTimeout; + } + + @Override + public Integer getReadTimeout() { + return 30; + } + + @Override + public Boolean getOauth2Enabled() { + return Boolean.parseBoolean(oauth2Enabled); + } + + @Override + @Nullable + public String getAuthUrl() { + return authUrl; + } + + @Override + @Nullable + public String getTokenUrl() { + return tokenUrl; + } + + @Override + @Nullable + public String getClientId() { + return clientId; + } + + @Override + @Nullable + public String getClientSecret() { + return clientSecret; + } + + @Override + @Nullable + public String getRefreshToken() { + return refreshToken; + } + + @Override + public Boolean getVerifyHttps() { + return Boolean.parseBoolean(verifyHttps); + } + + @Override + @Nullable + public String getKeystoreFile() { + return keystoreFile; + } + + @Override + @Nullable + public KeyStoreType getKeystoreType() { + return BaseHttpSourceConfig.getEnumValueByString( + KeyStoreType.class, + keystoreType, + HttpConstants.PROPERTY_KEYSTORE_TYPE + ); + } + + @Override + @Nullable + public String getKeystorePassword() { + return keystorePassword; + } + + @Override + @Nullable + public String getKeystoreKeyAlgorithm() { + return keystoreKeyAlgorithm; + } + + @Override + @Nullable + public String getTrustStoreFile() { + return trustStoreFile; + } + + @Override + @Nullable + public KeyStoreType getTrustStoreType() { + return BaseHttpSourceConfig.getEnumValueByString( + KeyStoreType.class, + trustStoreType, + HttpConstants.PROPERTY_TRUSTSTORE_TYPE + ); + } + + @Override + @Nullable + public String getTrustStorePassword() { + return trustStorePassword; + } + + @Override + @Nullable + public String getTrustStoreKeyAlgorithm() { + return trustStoreKeyAlgorithm; + } + + @Override + @Nullable + public String getCipherSuites() { + return cipherSuites; + } + + @Override + @Nullable + public String getKeystoreCertAliasName() { + return keystoreCertAliasName; + } + + @Override + @Nullable + public Map getHeadersMap() { + return BaseHttpSourceConfig.getMapFromKeyValueString(headers); + } + + @Override + public List getTransportProtocolsList() { + return BaseHttpSourceConfig.getListFromString(transportProtocols); + } + + @Override + public void validate() { + // Validate URL + if (!containsMacro(HttpConstants.PROPERTY_URL)) { + try { + // replace with placeholder with anything just during pagination + new URI(getUrl()); + } catch (URISyntaxException e) { + throw new InvalidConfigPropertyException( + String.format("URL value is not valid: '%s'", getUrl()), e, HttpConstants.PROPERTY_URL); + } + } + + // Validate OAuth2 properties + if (!containsMacro(HttpConstants.PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) { + String reasonOauth2 = "OAuth2 is enabled"; + BaseHttpSourceConfig.assertIsSet(getAuthUrl(), HttpConstants.PROPERTY_AUTH_URL, reasonOauth2); + BaseHttpSourceConfig.assertIsSet(getTokenUrl(), HttpConstants.PROPERTY_TOKEN_URL, reasonOauth2); + BaseHttpSourceConfig.assertIsSet(getClientId(), HttpConstants.PROPERTY_CLIENT_ID, reasonOauth2); + BaseHttpSourceConfig.assertIsSet(getClientSecret(), HttpConstants.PROPERTY_CLIENT_SECRET, reasonOauth2); + BaseHttpSourceConfig.assertIsSet(getRefreshToken(), HttpConstants.PROPERTY_REFRESH_TOKEN, reasonOauth2); + } + + if (!containsMacro(HttpConstants.PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) { + BaseHttpSourceConfig.assertIsNotSet( + getTrustStoreFile(), + HttpConstants.PROPERTY_TRUSTSTORE_FILE, + String.format("trustore settings are ignored due to disabled %s", HttpConstants.PROPERTY_VERIFY_HTTPS) + ); + } + } + +} diff --git a/src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java b/src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java similarity index 82% rename from src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java rename to src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java index a554dd6..6933ce9 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/BaseHttpSourceConfig.java +++ b/src/main/java/io/cdap/plugin/http/common/BaseHttpSourceConfig.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common; +package io.cdap.plugin.http.common; import com.google.common.base.Strings; import io.cdap.cdap.api.annotation.Description; @@ -23,13 +23,15 @@ import io.cdap.cdap.etl.api.validation.InvalidConfigPropertyException; import io.cdap.cdap.etl.api.validation.InvalidStageException; import io.cdap.plugin.common.ReferencePluginConfig; -import io.cdap.plugin.http.source.common.error.ErrorHandling; -import io.cdap.plugin.http.source.common.error.HttpErrorHandlerEntity; -import io.cdap.plugin.http.source.common.error.RetryableErrorHandling; -import io.cdap.plugin.http.source.common.http.KeyStoreType; -import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory; -import io.cdap.plugin.http.source.common.pagination.PaginationType; -import io.cdap.plugin.http.source.common.pagination.page.PageFormat; +import io.cdap.plugin.http.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity; +import io.cdap.plugin.http.common.error.RetryableErrorHandling; +import io.cdap.plugin.http.common.http.HttpConstants; +import io.cdap.plugin.http.common.http.IHttpConfig; +import io.cdap.plugin.http.common.http.KeyStoreType; +import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory; +import io.cdap.plugin.http.common.pagination.PaginationType; +import io.cdap.plugin.http.common.pagination.page.PageFormat; import java.io.IOException; import java.net.URI; @@ -49,28 +51,13 @@ /** * Base configuration for HTTP Streaming and Batch plugins. */ -public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { - public static final String PROPERTY_REFERENCE_NAME = "referenceName"; - public static final String PROPERTY_URL = "url"; - public static final String PROPERTY_HTTP_METHOD = "httpMethod"; - public static final String PROPERTY_HEADERS = "headers"; - public static final String PROPERTY_REQUEST_BODY = "requestBody"; +public abstract class BaseHttpSourceConfig extends ReferencePluginConfig implements IHttpConfig { public static final String PROPERTY_FORMAT = "format"; public static final String PROPERTY_RESULT_PATH = "resultPath"; public static final String PROPERTY_FIELDS_MAPPING = "fieldsMapping"; public static final String PROPERTY_CSV_SKIP_FIRST_ROW = "csvSkipFirstRow"; - public static final String PROPERTY_USERNAME = "username"; - public static final String PROPERTY_PASSWORD = "password"; - public static final String PROPERTY_PROXY_URL = "proxyUrl"; - public static final String PROPERTY_PROXY_USERNAME = "proxyUsername"; - public static final String PROPERTY_PROXY_PASSWORD = "proxyPassword"; - public static final String PROPERTY_HTTP_ERROR_HANDLING = "httpErrorsHandling"; - public static final String PROPERTY_ERROR_HANDLING = "errorHandling"; - public static final String PROPERTY_RETRY_POLICY = "retryPolicy"; public static final String PROPERTY_LINEAR_RETRY_INTERVAL = "linearRetryInterval"; public static final String PROPERTY_MAX_RETRY_DURATION = "maxRetryDuration"; - public static final String PROPERTY_CONNECT_TIMEOUT = "connectTimeout"; - public static final String PROPERTY_READ_TIMEOUT = "readTimeout"; public static final String PROPERTY_PAGINATION_TYPE = "paginationType"; public static final String PROPERTY_START_INDEX = "startIndex"; public static final String PROPERTY_MAX_INDEX = "maxIndex"; @@ -80,47 +67,27 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { public static final String PROPERTY_NEXT_PAGE_URL_PARAMETER = "nextPageUrlParameter"; public static final String PROPERTY_CUSTOM_PAGINATION_CODE = "customPaginationCode"; public static final String PROPERTY_WAIT_TIME_BETWEEN_PAGES = "waitTimeBetweenPages"; - public static final String PROPERTY_OAUTH2_ENABLED = "oauth2Enabled"; - public static final String PROPERTY_AUTH_URL = "authUrl"; - public static final String PROPERTY_TOKEN_URL = "tokenUrl"; - public static final String PROPERTY_CLIENT_ID = "clientId"; - public static final String PROPERTY_CLIENT_SECRET = "clientSecret"; - public static final String PROPERTY_SCOPES = "scopes"; - public static final String PROPERTY_REFRESH_TOKEN = "refreshToken"; - public static final String PROPERTY_VERIFY_HTTPS = "verifyHttps"; - public static final String PROPERTY_KEYSTORE_FILE = "keystoreFile"; - public static final String PROPERTY_KEYSTORE_TYPE = "keystoreType"; - public static final String PROPERTY_KEYSTORE_PASSWORD = "keystorePassword"; - public static final String PROPERTY_KEYSTORE_KEY_ALGORITHM = "keystoreKeyAlgorithm"; - public static final String PROPERTY_TRUSTSTORE_FILE = "trustStoreFile"; - public static final String PROPERTY_TRUSTSTORE_TYPE = "trustStoreType"; - public static final String PROPERTY_TRUSTSTORE_PASSWORD = "trustStorePassword"; - public static final String PROPERTY_TRUSTSTORE_KEY_ALGORITHM = "trustStoreKeyAlgorithm"; - public static final String PROPERTY_TRANSPORT_PROTOCOLS = "transportProtocols"; - public static final String PROPERTY_CIPHER_SUITES = "cipherSuites"; - public static final String PROPERTY_SCHEMA = "schema"; - public static final String PAGINATION_INDEX_PLACEHOLDER_REGEX = "\\{pagination.index\\}"; public static final String PAGINATION_INDEX_PLACEHOLDER = "{pagination.index}"; - @Name(PROPERTY_URL) + @Name(HttpConstants.PROPERTY_URL) @Description("Url to fetch to the first page. The url must start with a protocol (e.g. http://).") @Macro protected String url; - @Name(PROPERTY_HTTP_METHOD) + @Name(HttpConstants.PROPERTY_HTTP_METHOD) @Description("HTTP request method.") @Macro protected String httpMethod; - @Name(PROPERTY_HEADERS) + @Name(HttpConstants.PROPERTY_HEADERS) @Nullable @Description("Headers to send with each HTTP request.") @Macro protected String headers; @Nullable - @Name(PROPERTY_REQUEST_BODY) + @Name(HttpConstants.PROPERTY_REQUEST_BODY) @Description("Body to send with each HTTP request.") @Macro protected String requestBody; @@ -153,48 +120,48 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { protected String csvSkipFirstRow; @Nullable - @Name(PROPERTY_USERNAME) + @Name(HttpConstants.PROPERTY_USERNAME) @Description("Username for basic authentication.") @Macro protected String username; @Nullable - @Name(PROPERTY_PASSWORD) + @Name(HttpConstants.PROPERTY_PASSWORD) @Description("Password for basic authentication.") @Macro protected String password; @Nullable - @Name(PROPERTY_PROXY_URL) + @Name(HttpConstants.PROPERTY_PROXY_URL) @Description("Proxy URL. Must contain a protocol, address and port.") @Macro protected String proxyUrl; @Nullable - @Name(PROPERTY_PROXY_USERNAME) + @Name(HttpConstants.PROPERTY_PROXY_USERNAME) @Description("Proxy username.") @Macro protected String proxyUsername; @Nullable - @Name(PROPERTY_PROXY_PASSWORD) + @Name(HttpConstants.PROPERTY_PROXY_PASSWORD) @Description("Proxy password.") @Macro protected String proxyPassword; @Nullable - @Name(PROPERTY_HTTP_ERROR_HANDLING) + @Name(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING) @Description("Defines the error handling strategy to use for certain HTTP response codes." + "The left column contains a regular expression for HTTP status code. The right column contains an action which" + "is done in case of match. If HTTP status code matches multiple regular expressions, " + "the first specified in mapping is matched.") protected String httpErrorsHandling; - @Name(PROPERTY_ERROR_HANDLING) + @Name(HttpConstants.PROPERTY_ERROR_HANDLING) @Description("Error handling strategy to use when the HTTP response cannot be transformed to an output record.") protected String errorHandling; - @Name(PROPERTY_RETRY_POLICY) + @Name(HttpConstants.PROPERTY_RETRY_POLICY) @Description("Policy used to calculate delay between retries.") protected String retryPolicy; @@ -209,12 +176,12 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { @Macro protected Long maxRetryDuration; - @Name(PROPERTY_CONNECT_TIMEOUT) + @Name(HttpConstants.PROPERTY_CONNECT_TIMEOUT) @Description("Maximum time in seconds connection initialization is allowed to take.") @Macro protected Integer connectTimeout; - @Name(PROPERTY_READ_TIMEOUT) + @Name(HttpConstants.PROPERTY_READ_TIMEOUT) @Description("Maximum time in seconds fetching data from the server is allowed to take.") @Macro protected Integer readTimeout; @@ -276,47 +243,47 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { @Macro protected Long waitTimeBetweenPages; - @Name(PROPERTY_OAUTH2_ENABLED) + @Name(HttpConstants.PROPERTY_OAUTH2_ENABLED) @Description("If true, plugin will perform OAuth2 authentication.") protected String oauth2Enabled; @Nullable - @Name(PROPERTY_AUTH_URL) + @Name(HttpConstants.PROPERTY_AUTH_URL) @Description("Endpoint for the authorization server used to retrieve the authorization code.") @Macro protected String authUrl; @Nullable - @Name(PROPERTY_TOKEN_URL) + @Name(HttpConstants.PROPERTY_TOKEN_URL) @Description("Endpoint for the resource server, which exchanges the authorization code for an access token.") @Macro protected String tokenUrl; @Nullable - @Name(PROPERTY_CLIENT_ID) + @Name(HttpConstants.PROPERTY_CLIENT_ID) @Description("Client identifier obtained during the Application registration process.") @Macro protected String clientId; @Nullable - @Name(PROPERTY_CLIENT_SECRET) + @Name(HttpConstants.PROPERTY_CLIENT_SECRET) @Description("Client secret obtained during the Application registration process.") @Macro protected String clientSecret; @Nullable - @Name(PROPERTY_SCOPES) + @Name(HttpConstants.PROPERTY_SCOPES) @Description("Scope of the access request, which might have multiple space-separated values.") @Macro protected String scopes; @Nullable - @Name(PROPERTY_REFRESH_TOKEN) + @Name(HttpConstants.PROPERTY_REFRESH_TOKEN) @Description("Token used to receive accessToken, which is end product of OAuth2.") @Macro protected String refreshToken; - @Name(PROPERTY_VERIFY_HTTPS) + @Name(HttpConstants.PROPERTY_VERIFY_HTTPS) @Description("If false, untrusted trust certificates (e.g. self signed), will not lead to an" + "error. Do not disable this in production environment on a network you do not entirely trust. " + "Especially public internet.") @@ -324,72 +291,78 @@ public abstract class BaseHttpSourceConfig extends ReferencePluginConfig { protected String verifyHttps; @Nullable - @Name(PROPERTY_KEYSTORE_FILE) + @Name(HttpConstants.PROPERTY_KEYSTORE_FILE) @Description("A path to a file which contains keystore.") @Macro protected String keystoreFile; @Nullable - @Name(PROPERTY_KEYSTORE_TYPE) + @Name(HttpConstants.PROPERTY_KEYSTORE_TYPE) @Description("Format of a keystore.") @Macro protected String keystoreType; @Nullable - @Name(PROPERTY_KEYSTORE_PASSWORD) + @Name(HttpConstants.PROPERTY_KEYSTORE_PASSWORD) @Description("Password for a keystore. If a keystore is not password protected leave it empty.") @Macro protected String keystorePassword; @Nullable - @Name(PROPERTY_KEYSTORE_KEY_ALGORITHM) + @Name(HttpConstants.PROPERTY_KEYSTORE_KEY_ALGORITHM) @Description("An algorithm used for keystore.") @Macro protected String keystoreKeyAlgorithm; @Nullable - @Name(PROPERTY_TRUSTSTORE_FILE) + @Name(HttpConstants.PROPERTY_TRUSTSTORE_FILE) @Description("A path to a file which contains truststore.") @Macro protected String trustStoreFile; @Nullable - @Name(PROPERTY_TRUSTSTORE_TYPE) + @Name(HttpConstants.PROPERTY_TRUSTSTORE_TYPE) @Description("Format of a truststore.") @Macro protected String trustStoreType; @Nullable - @Name(PROPERTY_TRUSTSTORE_PASSWORD) + @Name(HttpConstants.PROPERTY_TRUSTSTORE_PASSWORD) @Description("Password for a truststore. If a truststore is not password protected leave it empty.") @Macro protected String trustStorePassword; @Nullable - @Name(PROPERTY_TRUSTSTORE_KEY_ALGORITHM) + @Name(HttpConstants.PROPERTY_TRUSTSTORE_KEY_ALGORITHM) @Description("An algorithm used for truststore.") @Macro protected String trustStoreKeyAlgorithm; @Nullable - @Name(PROPERTY_TRANSPORT_PROTOCOLS) + @Name(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS) @Description("Transport protocols which are allowed for connection.") @Macro protected String transportProtocols; @Nullable - @Name(PROPERTY_CIPHER_SUITES) + @Name(HttpConstants.PROPERTY_CIPHER_SUITES) @Description("Cipher suites which are allowed for connection. " + "Colons, commas or spaces are also acceptable separators.") @Macro protected String cipherSuites; - @Name(PROPERTY_SCHEMA) + @Name(HttpConstants.PROPERTY_SCHEMA) @Macro @Nullable @Description("Output schema. Is required to be set.") protected String schema; + @Name(HttpConstants.PROPERTY_KEYSTORE_CERT_ALIAS) + @Macro + @Nullable + @Description("Alias of the key in the keystore to be used for communication") + protected String keystoreCertAliasName; + protected BaseHttpSourceConfig(String referenceName) { super(referenceName); } @@ -461,11 +434,11 @@ public String getHttpErrorsHandling() { } public ErrorHandling getErrorHandling() { - return getEnumValueByString(ErrorHandling.class, errorHandling, PROPERTY_ERROR_HANDLING); + return getEnumValueByString(ErrorHandling.class, errorHandling, HttpConstants.PROPERTY_ERROR_HANDLING); } public RetryPolicy getRetryPolicy() { - return getEnumValueByString(RetryPolicy.class, retryPolicy, PROPERTY_RETRY_POLICY); + return getEnumValueByString(RetryPolicy.class, retryPolicy, HttpConstants.PROPERTY_RETRY_POLICY); } @Nullable @@ -574,7 +547,7 @@ public String getKeystoreFile() { @Nullable public KeyStoreType getKeystoreType() { - return getEnumValueByString(KeyStoreType.class, keystoreType, PROPERTY_KEYSTORE_TYPE); + return getEnumValueByString(KeyStoreType.class, keystoreType, HttpConstants.PROPERTY_KEYSTORE_TYPE); } @Nullable @@ -594,7 +567,7 @@ public String getTrustStoreFile() { @Nullable public KeyStoreType getTrustStoreType() { - return getEnumValueByString(KeyStoreType.class, trustStoreType, PROPERTY_TRUSTSTORE_TYPE); + return getEnumValueByString(KeyStoreType.class, trustStoreType, HttpConstants.PROPERTY_TRUSTSTORE_TYPE); } @Nullable @@ -623,10 +596,15 @@ public Schema getSchema() { return Strings.isNullOrEmpty(schema) ? null : Schema.parseJson(schema); } catch (IOException e) { throw new InvalidConfigPropertyException("Unable to parse output schema: " + - schema, e, PROPERTY_SCHEMA); + schema, e, HttpConstants.PROPERTY_SCHEMA); } } + @Nullable + public String getKeystoreCertAliasName() { + return keystoreCertAliasName; + } + @Nullable public Map getHeadersMap() { return getMapFromKeyValueString(headers); @@ -641,12 +619,13 @@ public List getHttpErrorHandlingEntries() { try { results.add(new HttpErrorHandlerEntity(Pattern.compile(regex), getEnumValueByString(RetryableErrorHandling.class, - entry.getValue(), PROPERTY_HTTP_ERROR_HANDLING))); + entry.getValue(), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING))); } catch (PatternSyntaxException e) { // We embed causing exception message into this one. Since this message is shown on UI when validation fails. throw new InvalidConfigPropertyException( - String.format( - "Error handling regex '%s' is not valid. %s", regex, e.getMessage()), PROPERTY_HTTP_ERROR_HANDLING); + String.format("Error handling regex '%s' is not valid. %s", regex, e.getMessage()), + HttpConstants.PROPERTY_HTTP_ERROR_HANDLING + ); } } return results; @@ -672,18 +651,18 @@ public List getTransportProtocolsList() { public void validate() { // Validate URL - if (!containsMacro(PROPERTY_URL)) { + if (!containsMacro(HttpConstants.PROPERTY_URL)) { try { // replace with placeholder with anything just during pagination new URI(getUrl().replaceAll(PAGINATION_INDEX_PLACEHOLDER_REGEX, "0")); } catch (URISyntaxException e) { throw new InvalidConfigPropertyException( - String.format("URL value is not valid: '%s'", getUrl()), e, PROPERTY_URL); + String.format("URL value is not valid: '%s'", getUrl()), e, HttpConstants.PROPERTY_URL); } } // Validate HTTP Error Handling Map - if (!containsMacro(PROPERTY_HTTP_ERROR_HANDLING)) { + if (!containsMacro(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING)) { List httpErrorsHandlingEntries = getHttpErrorHandlingEntries(); boolean supportsSkippingPages = PaginationIteratorFactory .createInstance(this, null).supportsSkippingPages(); @@ -695,7 +674,8 @@ public void validate() { postRetryStrategy.equals(ErrorHandling.SKIP)) { throw new InvalidConfigPropertyException( String.format("Error handling strategy '%s' is not support in combination with pagination type", - httpErrorsHandlingEntry.getStrategy(), getPaginationType()), PROPERTY_HTTP_ERROR_HANDLING); + httpErrorsHandlingEntry.getStrategy(), getPaginationType() + ), HttpConstants.PROPERTY_HTTP_ERROR_HANDLING); } } } @@ -703,7 +683,7 @@ public void validate() { // Validate Linear Retry Interval - if (!containsMacro(PROPERTY_RETRY_POLICY) && getRetryPolicy() == RetryPolicy.LINEAR) { + if (!containsMacro(HttpConstants.PROPERTY_RETRY_POLICY) && getRetryPolicy() == RetryPolicy.LINEAR) { assertIsSet(getLinearRetryInterval(), PROPERTY_LINEAR_RETRY_INTERVAL, "retry policy is linear"); } @@ -745,7 +725,7 @@ public void validate() { throw new InvalidConfigPropertyException( String.format("Url '%s' must contain '%s' placeholder when pagination type is '%s'", getUrl(), PAGINATION_INDEX_PLACEHOLDER, getPaginationType()), - PROPERTY_URL); + HttpConstants.PROPERTY_URL); } break; case CUSTOM: @@ -785,18 +765,21 @@ PAGINATION_INDEX_PLACEHOLDER, getPaginationType()), } // Validate OAuth2 properties - if (!containsMacro(PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) { + if (!containsMacro(HttpConstants.PROPERTY_OAUTH2_ENABLED) && this.getOauth2Enabled()) { String reasonOauth2 = "OAuth2 is enabled"; - assertIsSet(getAuthUrl(), PROPERTY_AUTH_URL, reasonOauth2); - assertIsSet(getTokenUrl(), PROPERTY_TOKEN_URL, reasonOauth2); - assertIsSet(getClientId(), PROPERTY_CLIENT_ID, reasonOauth2); - assertIsSet(getClientSecret(), PROPERTY_CLIENT_SECRET, reasonOauth2); - assertIsSet(getRefreshToken(), PROPERTY_REFRESH_TOKEN, reasonOauth2); + assertIsSet(getAuthUrl(), HttpConstants.PROPERTY_AUTH_URL, reasonOauth2); + assertIsSet(getTokenUrl(), HttpConstants.PROPERTY_TOKEN_URL, reasonOauth2); + assertIsSet(getClientId(), HttpConstants.PROPERTY_CLIENT_ID, reasonOauth2); + assertIsSet(getClientSecret(), HttpConstants.PROPERTY_CLIENT_SECRET, reasonOauth2); + assertIsSet(getRefreshToken(), HttpConstants.PROPERTY_REFRESH_TOKEN, reasonOauth2); } - if (!containsMacro(PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) { - assertIsNotSet(getTrustStoreFile(), PROPERTY_TRUSTSTORE_FILE, - String.format("trustore settings are ignored due to disabled %s", PROPERTY_VERIFY_HTTPS)); + if (!containsMacro(HttpConstants.PROPERTY_VERIFY_HTTPS) && !getVerifyHttps()) { + assertIsNotSet( + getTrustStoreFile(), + HttpConstants.PROPERTY_TRUSTSTORE_FILE, + String.format("trustore settings are ignored due to disabled %s", HttpConstants.PROPERTY_VERIFY_HTTPS) + ); } } diff --git a/src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java b/src/main/java/io/cdap/plugin/http/common/EnumWithValue.java similarity index 94% rename from src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java rename to src/main/java/io/cdap/plugin/http/common/EnumWithValue.java index 653a12d..562caaf 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/EnumWithValue.java +++ b/src/main/java/io/cdap/plugin/http/common/EnumWithValue.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common; +package io.cdap.plugin.http.common; /** * Enum which has string values as a part of it. diff --git a/src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java b/src/main/java/io/cdap/plugin/http/common/RetryPolicy.java similarity index 95% rename from src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java rename to src/main/java/io/cdap/plugin/http/common/RetryPolicy.java index 2d0157d..c2cb8cb 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/RetryPolicy.java +++ b/src/main/java/io/cdap/plugin/http/common/RetryPolicy.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common; +package io.cdap.plugin.http.common; /** * An enum which represents a retry policy. diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java b/src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java similarity index 91% rename from src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java rename to src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java index 47f06b5..7bea805 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/error/ErrorHandling.java +++ b/src/main/java/io/cdap/plugin/http/common/error/ErrorHandling.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.error; +package io.cdap.plugin.http.common.error; -import io.cdap.plugin.http.source.common.EnumWithValue; +import io.cdap.plugin.http.common.EnumWithValue; /** * Indicates error handling strategy which will be used during reading HTTP records. diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java similarity index 68% rename from src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java rename to src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java index c6a802f..172d947 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandler.java +++ b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandler.java @@ -13,9 +13,10 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.error; +package io.cdap.plugin.http.common.error; + +import io.cdap.plugin.http.common.http.IHttpConfig; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -29,23 +30,31 @@ public class HttpErrorHandler { private static final Logger LOG = LoggerFactory.getLogger(HttpErrorHandler.class); private List httpErrorsHandlingEntries; + private boolean showWarnings = true; - public HttpErrorHandler(BaseHttpSourceConfig config) { + public HttpErrorHandler(IHttpConfig config) { this.httpErrorsHandlingEntries = config.getHttpErrorHandlingEntries(); } + public HttpErrorHandler(IHttpConfig config, boolean showWarnings) { + this.httpErrorsHandlingEntries = config.getHttpErrorHandlingEntries(); + this.showWarnings = showWarnings; + } + public RetryableErrorHandling getErrorHandlingStrategy(int httpCode) { String httpCodeString = Integer.toString(httpCode); - for (HttpErrorHandlerEntity httpErrorsHandlingEntry : httpErrorsHandlingEntries) { + for (HttpErrorHandlerEntity httpErrorsHandlingEntry: httpErrorsHandlingEntries) { Matcher matcher = httpErrorsHandlingEntry.getPattern().matcher(httpCodeString); if (matcher.matches()) { return httpErrorsHandlingEntry.getStrategy(); } } - LOG.warn(String.format("No error handling strategy defined for HTTP status code '%d'. " + - "Please correct httpErrorsHandling.", httpCode)); + if (showWarnings) { + LOG.warn(String.format("No error handling strategy defined for HTTP status code '%d'. " + + "Please correct httpErrorsHandling.", httpCode)); + } return RetryableErrorHandling.FAIL; } } diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java similarity index 96% rename from src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java rename to src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java index 09a803a..8e4e417 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/error/HttpErrorHandlerEntity.java +++ b/src/main/java/io/cdap/plugin/http/common/error/HttpErrorHandlerEntity.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.error; +package io.cdap.plugin.http.common.error; import java.util.Objects; import java.util.regex.Pattern; diff --git a/src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java b/src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java similarity index 94% rename from src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java rename to src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java index 0efa839..e910ca3 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/error/RetryableErrorHandling.java +++ b/src/main/java/io/cdap/plugin/http/common/error/RetryableErrorHandling.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.error; +package io.cdap.plugin.http.common.error; -import io.cdap.plugin.http.source.common.EnumWithValue; +import io.cdap.plugin.http.common.EnumWithValue; /** * Indicates error handling strategy which will be used to handle unexpected http status codes. diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java b/src/main/java/io/cdap/plugin/http/common/http/HttpClient.java similarity index 84% rename from src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java rename to src/main/java/io/cdap/plugin/http/common/http/HttpClient.java index bb41293..aa6b151 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/HttpClient.java +++ b/src/main/java/io/cdap/plugin/http/common/http/HttpClient.java @@ -13,11 +13,11 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; import com.google.common.base.Charsets; import com.google.common.base.Strings; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; import org.apache.http.Header; import org.apache.http.HttpHost; import org.apache.http.auth.AuthScope; @@ -45,11 +45,11 @@ */ public class HttpClient implements Closeable { private final Map headers; - private final BaseHttpSourceConfig config; + private final IHttpConfig config; private final StringEntity requestBody; private CloseableHttpClient httpClient; - public HttpClient(BaseHttpSourceConfig config) { + public HttpClient(IHttpConfig config) { this.config = config; this.headers = config.getHeadersMap(); @@ -96,29 +96,31 @@ private CloseableHttpClient createHttpClient() throws IOException { httpClientBuilder.setSSLSocketFactory(new SSLConnectionSocketFactoryCreator(config).create()); // set timeouts - Long connectTimeoutMillis = TimeUnit.SECONDS.toMillis(config.getConnectTimeout()); - Long readTimeoutMillis = TimeUnit.SECONDS.toMillis(config.getReadTimeout()); + int connectTimeoutMillis = (int) TimeUnit.SECONDS.toMillis(config.getConnectTimeout()); + int readTimeoutMillis = (int) TimeUnit.SECONDS.toMillis(config.getReadTimeout()); RequestConfig.Builder requestBuilder = RequestConfig.custom(); - requestBuilder.setSocketTimeout(readTimeoutMillis.intValue()); - requestBuilder.setConnectTimeout(connectTimeoutMillis.intValue()); - requestBuilder.setConnectionRequestTimeout(connectTimeoutMillis.intValue()); + requestBuilder.setSocketTimeout(readTimeoutMillis); + requestBuilder.setConnectTimeout(connectTimeoutMillis); + requestBuilder.setConnectionRequestTimeout(connectTimeoutMillis); httpClientBuilder.setDefaultRequestConfig(requestBuilder.build()); // basic auth CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); if (!Strings.isNullOrEmpty(config.getUsername()) && !Strings.isNullOrEmpty(config.getPassword())) { - AuthScope authScope = new AuthScope(HttpHost.create(config.getUrl())); - credentialsProvider.setCredentials(authScope, - new UsernamePasswordCredentials(config.getUsername(), config.getPassword())); + credentialsProvider.setCredentials( + new AuthScope(HttpHost.create(config.getUrl())), + new UsernamePasswordCredentials(config.getUsername(), config.getPassword()) + ); } // proxy and proxy auth if (!Strings.isNullOrEmpty(config.getProxyUrl())) { HttpHost proxyHost = HttpHost.create(config.getProxyUrl()); if (!Strings.isNullOrEmpty(config.getProxyUsername()) && !Strings.isNullOrEmpty(config.getProxyPassword())) { - credentialsProvider.setCredentials(new AuthScope(proxyHost), - new UsernamePasswordCredentials( - config.getProxyUsername(), config.getProxyPassword())); + credentialsProvider.setCredentials( + new AuthScope(proxyHost), + new UsernamePasswordCredentials(config.getProxyUsername(), config.getProxyPassword()) + ); } httpClientBuilder.setProxy(proxyHost); } diff --git a/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java b/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java new file mode 100644 index 0000000..0e10acb --- /dev/null +++ b/src/main/java/io/cdap/plugin/http/common/http/HttpConstants.java @@ -0,0 +1,63 @@ +/* + * Copyright © 2017 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package io.cdap.plugin.http.common.http; + +/** + * HTTP Plugin Constants + */ +public final class HttpConstants { + private HttpConstants() { + + } + + public static final String HTTP_PLUGIN_NAME = "HTTP"; + + // Field Name Constants + public static final String PROPERTY_URL = "url"; + public static final String PROPERTY_HTTP_METHOD = "httpMethod"; + public static final String PROPERTY_HEADERS = "headers"; + public static final String PROPERTY_REQUEST_BODY = "requestBody"; + public static final String PROPERTY_USERNAME = "username"; + public static final String PROPERTY_PASSWORD = "password"; + public static final String PROPERTY_PROXY_URL = "proxyUrl"; + public static final String PROPERTY_PROXY_USERNAME = "proxyUsername"; + public static final String PROPERTY_PROXY_PASSWORD = "proxyPassword"; + public static final String PROPERTY_HTTP_ERROR_HANDLING = "httpErrorsHandling"; + public static final String PROPERTY_ERROR_HANDLING = "errorHandling"; + public static final String PROPERTY_RETRY_POLICY = "retryPolicy"; + public static final String PROPERTY_CONNECT_TIMEOUT = "connectTimeout"; + public static final String PROPERTY_READ_TIMEOUT = "readTimeout"; + public static final String PROPERTY_OAUTH2_ENABLED = "oauth2Enabled"; + public static final String PROPERTY_AUTH_URL = "authUrl"; + public static final String PROPERTY_TOKEN_URL = "tokenUrl"; + public static final String PROPERTY_CLIENT_ID = "clientId"; + public static final String PROPERTY_CLIENT_SECRET = "clientSecret"; + public static final String PROPERTY_SCOPES = "scopes"; + public static final String PROPERTY_REFRESH_TOKEN = "refreshToken"; + public static final String PROPERTY_VERIFY_HTTPS = "verifyHttps"; + public static final String PROPERTY_KEYSTORE_FILE = "keystoreFile"; + public static final String PROPERTY_KEYSTORE_TYPE = "keystoreType"; + public static final String PROPERTY_KEYSTORE_PASSWORD = "keystorePassword"; + public static final String PROPERTY_KEYSTORE_KEY_ALGORITHM = "keystoreKeyAlgorithm"; + public static final String PROPERTY_TRUSTSTORE_FILE = "trustStoreFile"; + public static final String PROPERTY_TRUSTSTORE_TYPE = "trustStoreType"; + public static final String PROPERTY_TRUSTSTORE_PASSWORD = "trustStorePassword"; + public static final String PROPERTY_TRUSTSTORE_KEY_ALGORITHM = "trustStoreKeyAlgorithm"; + public static final String PROPERTY_TRANSPORT_PROTOCOLS = "transportProtocols"; + public static final String PROPERTY_CIPHER_SUITES = "cipherSuites"; + public static final String PROPERTY_SCHEMA = "schema"; + public static final String PROPERTY_KEYSTORE_CERT_ALIAS = "keystoreCertAlias"; +} diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java b/src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java similarity index 98% rename from src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java rename to src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java index f8d5254..21c2cc9 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/HttpResponse.java +++ b/src/main/java/io/cdap/plugin/http/common/http/HttpResponse.java @@ -14,7 +14,7 @@ * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; import org.apache.http.Header; import org.apache.http.HttpEntity; diff --git a/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java b/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java new file mode 100644 index 0000000..09d9375 --- /dev/null +++ b/src/main/java/io/cdap/plugin/http/common/http/IHttpConfig.java @@ -0,0 +1,124 @@ +/* + * Copyright © 2019 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package io.cdap.plugin.http.common.http; + +import io.cdap.plugin.http.common.RetryPolicy; +import io.cdap.plugin.http.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.HttpErrorHandlerEntity; +import java.util.List; +import java.util.Map; +import javax.annotation.Nullable; + +/** + * Interface for the HTTP Plugins Config + */ +public interface IHttpConfig { + + String getUrl(); + + String getHttpMethod(); + + @Nullable + String getHeaders(); + + @Nullable + String getRequestBody(); + + @Nullable + String getUsername(); + + @Nullable + String getPassword(); + + @Nullable + String getProxyUrl(); + + @Nullable + String getProxyUsername(); + + @Nullable + String getProxyPassword(); + + ErrorHandling getErrorHandling(); + + List getHttpErrorHandlingEntries(); + + Long getLinearRetryInterval(); + + Long getMaxRetryDuration(); + + RetryPolicy getRetryPolicy(); + + Integer getConnectTimeout(); + + Integer getReadTimeout(); + + Boolean getOauth2Enabled(); + + @Nullable + String getAuthUrl(); + + @Nullable + String getTokenUrl(); + + @Nullable + String getClientId(); + + @Nullable + String getClientSecret(); + + @Nullable + String getRefreshToken(); + + Boolean getVerifyHttps(); + + @Nullable + String getKeystoreFile(); + + @Nullable + KeyStoreType getKeystoreType(); + + @Nullable + String getKeystorePassword(); + + @Nullable + String getKeystoreKeyAlgorithm(); + + @Nullable + String getTrustStoreFile(); + + @Nullable + KeyStoreType getTrustStoreType(); + + @Nullable + String getTrustStorePassword(); + + @Nullable + String getTrustStoreKeyAlgorithm(); + + @Nullable + String getCipherSuites(); + + @Nullable + String getKeystoreCertAliasName(); + + @Nullable + Map getHeadersMap(); + + List getTransportProtocolsList(); + + void validate(); +} diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java b/src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java similarity index 91% rename from src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java rename to src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java index 6acbf12..5d607af 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/KeyStoreType.java +++ b/src/main/java/io/cdap/plugin/http/common/http/KeyStoreType.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; -import io.cdap.plugin.http.source.common.EnumWithValue; +import io.cdap.plugin.http.common.EnumWithValue; /** * An enum which represent a type of keystore or truststore. diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java b/src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java similarity index 94% rename from src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java rename to src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java index 27ac280..1928683 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/OAuthUtil.java +++ b/src/main/java/io/cdap/plugin/http/common/http/OAuthUtil.java @@ -13,10 +13,10 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; import com.google.gson.JsonElement; -import io.cdap.plugin.http.source.common.pagination.page.JSONUtil; +import io.cdap.plugin.http.common.pagination.page.JSONUtil; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.utils.URIBuilder; diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java b/src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java similarity index 72% rename from src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java rename to src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java index 1883f9b..729c852 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/SSLConnectionSocketFactoryCreator.java +++ b/src/main/java/io/cdap/plugin/http/common/http/SSLConnectionSocketFactoryCreator.java @@ -14,14 +14,10 @@ * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; import com.google.common.base.Strings; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - import java.io.IOException; import java.io.InputStream; import java.nio.file.Files; @@ -42,11 +38,9 @@ * Class which creates an SSLConnectionSocketFactory. */ public class SSLConnectionSocketFactoryCreator { - private static final Logger LOG = LoggerFactory.getLogger(SSLConnectionSocketFactoryCreator.class); - - private final BaseHttpSourceConfig config; + private final IHttpConfig config; - public SSLConnectionSocketFactoryCreator(BaseHttpSourceConfig config) { + public SSLConnectionSocketFactoryCreator(IHttpConfig config) { this.config = config; } @@ -59,6 +53,7 @@ public SSLConnectionSocketFactory create() { SSLContext sslContext = SSLContext.getInstance("TLS"); // "TLS" means rely system properties sslContext.init(getKeyManagers(), getTrustManagers(), null); + return new SSLConnectionSocketFactory(sslContext, config.getTransportProtocolsList().toArray(new String[0]), cipherSuites, SSLConnectionSocketFactory.getDefaultHostnameVerifier()); } catch (KeyManagementException | CertificateException | NoSuchAlgorithmException | KeyStoreException @@ -70,27 +65,30 @@ public SSLConnectionSocketFactory create() { private KeyManager[] getKeyManagers() throws CertificateException, NoSuchAlgorithmException, KeyStoreException, IOException, UnrecoverableKeyException { - KeyStore keystore = loadKeystore(config.getKeystoreFile(), config.getKeystoreType().name(), - config.getKeystorePassword()); - String keyStorePassword = config.getKeystorePassword(); + KeyStore keystore = loadKeystore(config.getKeystoreFile(), config.getKeystoreType().name(), keyStorePassword); // we have to manually fall back to default keystore. SSLContext won't provide such a functionality. if (keystore == null) { String keyStore = System.getProperty("javax.net.ssl.keyStore"); String keyStoreType = System.getProperty("javax.net.ssl.keyStoreType", KeyStore.getDefaultType()); keyStorePassword = System.getProperty("javax.net.ssl.keyStorePassword", ""); - keystore = loadKeystore(keyStore, keyStoreType, keyStorePassword); } - String keystoreAlgorithm = - (Strings.isNullOrEmpty(config.getKeystoreKeyAlgorithm())) ? KeyManagerFactory.getDefaultAlgorithm() + String keystoreAlgorithm = (Strings.isNullOrEmpty(config.getKeystoreKeyAlgorithm())) + ? KeyManagerFactory.getDefaultAlgorithm() : config.getKeystoreKeyAlgorithm(); + KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(keystoreAlgorithm); - char[] passwordArr = (keyStorePassword == null) ? null : keyStorePassword.toCharArray(); - keyManagerFactory.init(keystore, passwordArr); - return keyManagerFactory.getKeyManagers(); + keyManagerFactory.init( + keystore, + (keyStorePassword == null) ? null : keyStorePassword.toCharArray() + ); + + return (Strings.isNullOrEmpty(config.getKeystoreCertAliasName())) + ? keyManagerFactory.getKeyManagers() + : X509KeyManagerAliasWrapper.getKeyManagers(keyManagerFactory, config.getKeystoreCertAliasName()); } private TrustManager[] getTrustManagers() @@ -100,13 +98,17 @@ private TrustManager[] getTrustManagers() return new TrustManager[] { new TrustAllTrustManager() }; } - KeyStore trustStore = loadKeystore(config.getTrustStoreFile(), config.getTrustStoreType().name(), - config.getTrustStorePassword()); + KeyStore trustStore = loadKeystore( + config.getTrustStoreFile(), + config.getTrustStoreType().name(), + config.getTrustStorePassword() + ); + TrustManager[] trustManagers = null; if (trustStore != null) { - String trustStoreAlgorithm = - (Strings.isNullOrEmpty(config.getTrustStoreKeyAlgorithm())) ? TrustManagerFactory.getDefaultAlgorithm() - : config.getTrustStoreKeyAlgorithm(); + String trustStoreAlgorithm = (Strings.isNullOrEmpty(config.getTrustStoreKeyAlgorithm())) + ? TrustManagerFactory.getDefaultAlgorithm() + : config.getTrustStoreKeyAlgorithm(); TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(trustStoreAlgorithm); trustManagerFactory.init(trustStore); trustManagers = trustManagerFactory.getTrustManagers(); @@ -117,13 +119,15 @@ private TrustManager[] getTrustManagers() private static KeyStore loadKeystore(String keystoreFile, String type, String password) throws IOException, CertificateException, NoSuchAlgorithmException, KeyStoreException { - KeyStore keystore = null; - if (keystoreFile != null) { - keystore = KeyStore.getInstance(type); - char[] passwordArr = (password == null) ? null : password.toCharArray(); - try (InputStream is = Files.newInputStream(Paths.get(keystoreFile))) { - keystore.load(is, passwordArr); - } + if (keystoreFile == null) { + return null; + } + + KeyStore keystore = KeyStore.getInstance(type); + char[] passwordArr = (password == null) ? null : password.toCharArray(); + + try (InputStream is = Files.newInputStream(Paths.get(keystoreFile))) { + keystore.load(is, passwordArr); } return keystore; } diff --git a/src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java b/src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java similarity index 95% rename from src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java rename to src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java index c90743e..86f0b93 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/http/TrustAllTrustManager.java +++ b/src/main/java/io/cdap/plugin/http/common/http/TrustAllTrustManager.java @@ -14,7 +14,7 @@ * the License. */ -package io.cdap.plugin.http.source.common.http; +package io.cdap.plugin.http.common.http; import java.security.cert.X509Certificate; import javax.net.ssl.X509TrustManager; diff --git a/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java b/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java new file mode 100644 index 0000000..c1a2f8a --- /dev/null +++ b/src/main/java/io/cdap/plugin/http/common/http/X509KeyManagerAliasWrapper.java @@ -0,0 +1,87 @@ +/* + * Copyright ┬⌐ 2021 Cask Data, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); you may not + * use this file except in compliance with the License. You may obtain a copy of + * the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package io.cdap.plugin.http.common.http; + +import java.net.Socket; +import java.security.Principal; +import java.security.PrivateKey; +import java.security.cert.X509Certificate; + +import javax.net.ssl.KeyManager; +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.X509KeyManager; + + +/** + * This is just wrapper over SunX509KeyManagerImpl with possibility to provide specific alias + * + * Usage example: + * X509KeyManagerAliasWrapper.getKeyManagers(keyManagerFactory, CERT_ALIAS); + */ +public class X509KeyManagerAliasWrapper implements X509KeyManager { + + private final X509KeyManager originalKeyManager; + private final String certAlias; + + public X509KeyManagerAliasWrapper(X509KeyManager originalKeyManager, String certAlias) { + this.originalKeyManager = originalKeyManager; + this.certAlias = certAlias; + } + + public static KeyManager[] getKeyManagers(KeyManagerFactory keyManagerFactory, String certAlias) { + KeyManager[] keyManagers = keyManagerFactory.getKeyManagers(); + + // Current implementation only support X509 Certificates + if (keyManagers.length != 1) { + return keyManagers; + } + if (!(keyManagers[0] instanceof X509KeyManager)) { + return keyManagers; + } + + return new KeyManager[]{ new X509KeyManagerAliasWrapper((X509KeyManager) keyManagers[0], certAlias) }; + }; + + @Override + public String chooseClientAlias(String[] strings, Principal[] principals, Socket socket) { + return certAlias; + } + + @Override + public String[] getClientAliases(String s, Principal[] principals) { + return originalKeyManager.getClientAliases(s, principals); + } + + @Override + public String[] getServerAliases(String s, Principal[] principals) { + return originalKeyManager.getServerAliases(s, principals); + } + + @Override + public String chooseServerAlias(String s, Principal[] principals, Socket socket) { + return originalKeyManager.chooseServerAlias(s, principals, socket); + } + + @Override + public X509Certificate[] getCertificateChain(String s) { + return originalKeyManager.getCertificateChain(s); + } + + @Override + public PrivateKey getPrivateKey(String s) { + return originalKeyManager.getPrivateKey(s); + } +} diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java similarity index 89% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java index ece3486..b05cd89 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/BaseHttpPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/BaseHttpPaginationIterator.java @@ -13,19 +13,19 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; - -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.RetryPolicy; -import io.cdap.plugin.http.source.common.error.ErrorHandling; -import io.cdap.plugin.http.source.common.error.HttpErrorHandler; -import io.cdap.plugin.http.source.common.error.RetryableErrorHandling; -import io.cdap.plugin.http.source.common.http.HttpClient; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.page.PageFactory; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; -import io.cdap.plugin.http.source.common.pagination.state.UrlPaginationIteratorState; +package io.cdap.plugin.http.common.pagination; + +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.RetryPolicy; +import io.cdap.plugin.http.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.HttpErrorHandler; +import io.cdap.plugin.http.common.error.RetryableErrorHandling; +import io.cdap.plugin.http.common.http.HttpClient; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.page.PageFactory; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.pagination.state.UrlPaginationIteratorState; import org.awaitility.Awaitility; import org.awaitility.core.ConditionTimeoutException; import org.awaitility.pollinterval.FixedPollInterval; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java similarity index 85% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java index ad1e832..f7a9dea 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/CustomPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/CustomPaginationIterator.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.apache.http.Header; import java.io.IOException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java similarity index 84% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java index 69084fa..4dd41f1 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/IncrementAnIndexPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/IncrementAnIndexPaginationIterator.java @@ -13,13 +13,13 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.IndexPaginationIteratorState; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.IndexPaginationIteratorState; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java b/src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java similarity index 98% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java rename to src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java index 077ce11..7d0157d 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/JythonPythonExecutor.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/JythonPythonExecutor.java @@ -14,7 +14,7 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; import com.google.common.base.Strings; import org.python.core.PyException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java similarity index 86% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java index cdc815d..296cb9d 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseBodyPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseBodyPaginationIterator.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java similarity index 85% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java index 5e59b35..78e8985 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/LinkInResponseHeaderPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/LinkInResponseHeaderPaginationIterator.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.apache.http.Header; import org.apache.http.HeaderElement; import org.slf4j.Logger; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java similarity index 78% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java index 80f48e3..386048f 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/NonePaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/NonePaginationIterator.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java similarity index 89% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java rename to src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java index c2cad0d..8eec5c7 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorFactory.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationIteratorFactory.java @@ -13,10 +13,10 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; /** * A factory which creates instance of {@BaseHttpPaginationIterator} in accordance to pagination type configured in diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java similarity index 91% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java rename to src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java index d567f58..984c4c8 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/PaginationType.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/PaginationType.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.EnumWithValue; +import io.cdap.plugin.http.common.EnumWithValue; /** * An enum which represent a type of pagination. diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java b/src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java similarity index 85% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java index d3f8804..5788553 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/TokenPaginationIterator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/TokenPaginationIterator.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.apache.http.client.utils.URIBuilder; import org.slf4j.Logger; import org.slf4j.LoggerFactory; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java similarity index 90% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java index f6958e0..9d3ab6b 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BasePage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/BasePage.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.Closeable; import java.util.Arrays; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java similarity index 91% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java index 5905196..17fef12 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/BlobPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/BlobPage.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.api.data.schema.Schema; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.IOException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java similarity index 90% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java index 393e675..3c7ed25 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/DelimitedPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/DelimitedPage.java @@ -14,12 +14,12 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.format.StructuredRecordStringConverter; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.IOException; import java.util.NoSuchElementException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java similarity index 88% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java index 657dee3..3590b3f 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/HttpErrorPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/HttpErrorPage.java @@ -13,14 +13,14 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.etl.api.InvalidEntry; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.error.ErrorHandling; -import io.cdap.plugin.http.source.common.error.HttpErrorHandler; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.HttpErrorHandler; +import io.cdap.plugin.http.common.http.HttpResponse; /** * Represents a page with an error diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java similarity index 97% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java index f700aa4..8354ec2 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/InvalidEntryCreator.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/InvalidEntryCreator.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.api.data.schema.Schema; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java similarity index 99% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java index a0dc95e..b48b8e0 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtil.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/JSONUtil.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import com.google.common.annotations.VisibleForTesting; import com.google.gson.JsonArray; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java similarity index 97% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java index b89d5af..b50fff7 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/JsonPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/JsonPage.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import com.google.gson.JsonElement; import com.google.gson.JsonObject; @@ -21,8 +21,8 @@ import io.cdap.cdap.api.data.schema.Schema; import io.cdap.cdap.etl.api.InvalidEntry; import io.cdap.cdap.format.StructuredRecordStringConverter; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import org.apache.commons.lang3.StringUtils; import java.util.ArrayList; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java similarity index 92% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java index f686f98..bc01e37 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageEntry.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageEntry.java @@ -13,11 +13,11 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.etl.api.InvalidEntry; -import io.cdap.plugin.http.source.common.error.ErrorHandling; +import io.cdap.plugin.http.common.error.ErrorHandling; /** * Represents a single entry found on page. The entry can either be an {@link InvalidEntry} or {@link StructuredRecord}. diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java similarity index 87% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java index 60b7e9c..a1690a2 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFactory.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFactory.java @@ -13,11 +13,11 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.error.HttpErrorHandler; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.error.HttpErrorHandler; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.IOException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java similarity index 89% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java index 527cf3f..c1f0296 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/PageFormat.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/PageFormat.java @@ -13,9 +13,9 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; -import io.cdap.plugin.http.source.common.EnumWithValue; +import io.cdap.plugin.http.common.EnumWithValue; /** * An enum which represent format of the page. diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java similarity index 93% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java index a3f18b6..50be7ab 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/RecordPerLinePage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/RecordPerLinePage.java @@ -13,12 +13,12 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.api.data.schema.Schema; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.BufferedReader; import java.io.IOException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java similarity index 87% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java index b95564a..d5d7839 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/TextPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/TextPage.java @@ -13,11 +13,11 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import io.cdap.cdap.api.data.format.StructuredRecord; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import java.io.IOException; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java similarity index 95% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java index c50c990..52cab5d 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlPage.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlPage.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import com.fasterxml.jackson.databind.node.ArrayNode; import com.google.gson.JsonArray; @@ -25,8 +25,8 @@ import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.api.data.schema.Schema; import io.cdap.cdap.format.StructuredRecordStringConverter; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpResponse; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpResponse; import org.w3c.dom.Document; import java.util.Iterator; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java similarity index 98% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java rename to src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java index b39a89b..71ef81c 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/page/XmlUtil.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/page/XmlUtil.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.node.TextNode; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java similarity index 89% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java rename to src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java index 1ad9f9f..563f430 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/IndexPaginationIteratorState.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/IndexPaginationIteratorState.java @@ -14,9 +14,9 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination.state; +package io.cdap.plugin.http.common.pagination.state; -import io.cdap.plugin.http.source.common.pagination.IncrementAnIndexPaginationIterator; +import io.cdap.plugin.http.common.pagination.IncrementAnIndexPaginationIterator; import java.util.Objects; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java similarity index 92% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java rename to src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java index 4ae7093..ece1900 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/PaginationIteratorState.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/PaginationIteratorState.java @@ -14,7 +14,7 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination.state; +package io.cdap.plugin.http.common.pagination.state; import java.io.Serializable; diff --git a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java b/src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java similarity index 95% rename from src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java rename to src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java index b549be5..4b3804f 100644 --- a/src/main/java/io/cdap/plugin/http/source/common/pagination/state/UrlPaginationIteratorState.java +++ b/src/main/java/io/cdap/plugin/http/common/pagination/state/UrlPaginationIteratorState.java @@ -14,7 +14,7 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination.state; +package io.cdap.plugin.http.common.pagination.state; import java.util.Objects; diff --git a/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java b/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java index 4f15644..27db419 100644 --- a/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java +++ b/src/main/java/io/cdap/plugin/http/sink/batch/HTTPSink.java @@ -30,6 +30,8 @@ import io.cdap.cdap.etl.api.batch.BatchSink; import io.cdap.cdap.etl.api.batch.BatchSinkContext; import io.cdap.cdap.format.StructuredRecordStringConverter; +import io.cdap.plugin.http.common.http.HttpConstants; + import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -60,7 +62,7 @@ * Sink plugin to send the messages from the pipeline to an external http endpoint. */ @Plugin(type = BatchSink.PLUGIN_TYPE) -@Name("HTTP") +@Name(HttpConstants.HTTP_PLUGIN_NAME) @Description("Sink plugin to send the messages from the pipeline to an external http endpoint.") public class HTTPSink extends BatchSink { diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java index dc3be1b..72df8ca 100644 --- a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java +++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSource.java @@ -31,8 +31,9 @@ import io.cdap.cdap.etl.api.batch.BatchSource; import io.cdap.cdap.etl.api.batch.BatchSourceContext; import io.cdap.plugin.common.LineageRecorder; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.page.PageEntry; +import io.cdap.plugin.http.common.http.HttpConstants; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.page.PageEntry; import org.apache.hadoop.io.NullWritable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -43,11 +44,9 @@ * Plugin returns records from HTTP source specified by link. Pagination via APIs is supported. */ @Plugin(type = BatchSource.PLUGIN_TYPE) -@Name(HttpBatchSource.NAME) +@Name(HttpConstants.HTTP_PLUGIN_NAME) @Description("Read data from HTTP endpoint.") public class HttpBatchSource extends BatchSource { - static final String NAME = "HTTP"; - private static final Logger LOG = LoggerFactory.getLogger(HttpBatchSource.class); private final HttpBatchSourceConfig config; diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java index 1b02eec..74d3c81 100644 --- a/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java +++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpBatchSourceConfig.java @@ -15,7 +15,7 @@ */ package io.cdap.plugin.http.source.batch; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; /** * Provides all the configurations required for configuring the {@link HttpBatchSource} plugin. diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java index fa9b5e1..c0d82d8 100644 --- a/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java +++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpInputFormat.java @@ -15,7 +15,7 @@ */ package io.cdap.plugin.http.source.batch; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.page.BasePage; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.mapreduce.InputFormat; import org.apache.hadoop.mapreduce.InputSplit; diff --git a/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java b/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java index 993c20f..1f172d3 100644 --- a/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java +++ b/src/main/java/io/cdap/plugin/http/source/batch/HttpRecordReader.java @@ -17,9 +17,9 @@ import com.google.gson.Gson; import com.google.gson.GsonBuilder; -import io.cdap.plugin.http.source.common.pagination.BaseHttpPaginationIterator; -import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.BaseHttpPaginationIterator; +import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory; +import io.cdap.plugin.http.common.pagination.page.BasePage; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.mapreduce.InputSplit; diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java index 2585818..a424cb2 100644 --- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java +++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpInputDStream.java @@ -18,10 +18,10 @@ import com.google.common.base.Throwables; import io.cdap.cdap.api.data.format.StructuredRecord; -import io.cdap.plugin.http.source.common.pagination.BaseHttpPaginationIterator; -import io.cdap.plugin.http.source.common.pagination.PaginationIteratorFactory; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.state.PaginationIteratorState; +import io.cdap.plugin.http.common.pagination.BaseHttpPaginationIterator; +import io.cdap.plugin.http.common.pagination.PaginationIteratorFactory; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.state.PaginationIteratorState; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.rdd.RDD; import org.apache.spark.streaming.StreamingContext; diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java index 4d9367d..7e3d5d8 100644 --- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java +++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSource.java @@ -26,7 +26,9 @@ import io.cdap.cdap.etl.api.streaming.StreamingSource; import io.cdap.plugin.common.Constants; import io.cdap.plugin.common.IdUtils; +import io.cdap.plugin.http.common.http.HttpConstants; import org.apache.spark.streaming.api.java.JavaDStream; + import scala.reflect.ClassTag; import scala.reflect.ClassTag$; @@ -35,10 +37,9 @@ * For paginated APIs once the last page is reached it waits for the next pages. */ @Plugin(type = StreamingSource.PLUGIN_TYPE) -@Name(HttpStreamingSource.NAME) +@Name(HttpConstants.HTTP_PLUGIN_NAME) @Description(HttpStreamingSource.DESCRIPTION) public class HttpStreamingSource extends StreamingSource { - static final String NAME = "HTTP"; static final String DESCRIPTION = "Read data from HTTP endpoint periodically waiting for updates"; private HttpStreamingSourceConfig config; diff --git a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java index 333afb8..9d3be85 100644 --- a/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java +++ b/src/main/java/io/cdap/plugin/http/source/streaming/HttpStreamingSourceConfig.java @@ -19,7 +19,7 @@ import io.cdap.cdap.api.annotation.Description; import io.cdap.cdap.api.annotation.Macro; import io.cdap.cdap.api.annotation.Name; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; import javax.annotation.Nullable; diff --git a/src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java b/src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java similarity index 96% rename from src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java rename to src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java index 16d22b5..ae67e14 100644 --- a/src/test/java/io/cdap/plugin/http/source/common/pagination/PaginationIteratorTest.java +++ b/src/test/java/io/cdap/plugin/http/common/pagination/PaginationIteratorTest.java @@ -14,17 +14,17 @@ * the License. */ -package io.cdap.plugin.http.source.common.pagination; +package io.cdap.plugin.http.common.pagination; import io.cdap.cdap.api.data.format.StructuredRecord; import io.cdap.cdap.api.data.schema.Schema; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpClient; +import io.cdap.plugin.http.common.pagination.page.BasePage; +import io.cdap.plugin.http.common.pagination.page.JSONUtil; +import io.cdap.plugin.http.common.pagination.page.PageEntry; +import io.cdap.plugin.http.common.pagination.page.PageFormat; import io.cdap.plugin.http.source.batch.HttpBatchSourceConfig; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; -import io.cdap.plugin.http.source.common.http.HttpClient; -import io.cdap.plugin.http.source.common.pagination.page.BasePage; -import io.cdap.plugin.http.source.common.pagination.page.JSONUtil; -import io.cdap.plugin.http.source.common.pagination.page.PageEntry; -import io.cdap.plugin.http.source.common.pagination.page.PageFormat; import org.apache.http.Header; import org.apache.http.HttpEntity; import org.apache.http.StatusLine; diff --git a/src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java b/src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java similarity index 98% rename from src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java rename to src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java index 5401db9..c94679e 100644 --- a/src/test/java/io/cdap/plugin/http/source/common/pagination/page/JSONUtilTest.java +++ b/src/test/java/io/cdap/plugin/http/common/pagination/page/JSONUtilTest.java @@ -13,7 +13,7 @@ * License for the specific language governing permissions and limitations under * the License. */ -package io.cdap.plugin.http.source.common.pagination.page; +package io.cdap.plugin.http.common.pagination.page; import com.google.gson.JsonObject; import org.junit.Assert; diff --git a/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java b/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java index c001c64..6c27b09 100644 --- a/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java +++ b/src/test/java/io/cdap/plugin/http/etl/BaseHttpBatchSourceETLTest.java @@ -37,8 +37,9 @@ import io.cdap.cdap.test.DataSetManager; import io.cdap.cdap.test.TestConfiguration; import io.cdap.cdap.test.WorkflowManager; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpConstants; import io.cdap.plugin.http.source.batch.HttpBatchSource; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; import org.junit.BeforeClass; import org.junit.ClassRule; import org.junit.Rule; @@ -85,20 +86,20 @@ public static void setupTestClass() throws Exception { public List getPipelineResults(Map sourceProperties) throws Exception { Map allProperties = new ImmutableMap.Builder() .put("referenceName", name.getMethodName()) - .put(BaseHttpSourceConfig.PROPERTY_HTTP_METHOD, "GET") - .put(BaseHttpSourceConfig.PROPERTY_OAUTH2_ENABLED, "false") - .put(BaseHttpSourceConfig.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail") - .put(BaseHttpSourceConfig.PROPERTY_ERROR_HANDLING, "stopOnError") - .put(BaseHttpSourceConfig.PROPERTY_RETRY_POLICY, "linear") + .put(HttpConstants.PROPERTY_HTTP_METHOD, "GET") + .put(HttpConstants.PROPERTY_OAUTH2_ENABLED, "false") + .put(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail") + .put(HttpConstants.PROPERTY_ERROR_HANDLING, "stopOnError") + .put(HttpConstants.PROPERTY_RETRY_POLICY, "linear") .put(BaseHttpSourceConfig.PROPERTY_MAX_RETRY_DURATION, "10") .put(BaseHttpSourceConfig.PROPERTY_LINEAR_RETRY_INTERVAL, "1") .put(BaseHttpSourceConfig.PROPERTY_WAIT_TIME_BETWEEN_PAGES, "0") - .put(BaseHttpSourceConfig.PROPERTY_CONNECT_TIMEOUT, "60") - .put(BaseHttpSourceConfig.PROPERTY_READ_TIMEOUT, "120") - .put(BaseHttpSourceConfig.PROPERTY_VERIFY_HTTPS, "true") - .put(BaseHttpSourceConfig.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)") - .put(BaseHttpSourceConfig.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)") - .put(BaseHttpSourceConfig.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2") + .put(HttpConstants.PROPERTY_CONNECT_TIMEOUT, "60") + .put(HttpConstants.PROPERTY_READ_TIMEOUT, "120") + .put(HttpConstants.PROPERTY_VERIFY_HTTPS, "true") + .put(HttpConstants.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)") + .put(HttpConstants.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)") + .put(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2") .putAll(sourceProperties) .build(); diff --git a/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java b/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java index 07a594e..5f52575 100644 --- a/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java +++ b/src/test/java/io/cdap/plugin/http/etl/HttpSourceETLTest.java @@ -22,7 +22,8 @@ import io.cdap.cdap.api.data.schema.Schema; import io.cdap.cdap.etl.mock.test.HydratorTestBase; import io.cdap.cdap.test.TestConfiguration; -import io.cdap.plugin.http.source.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.BaseHttpSourceConfig; +import io.cdap.plugin.http.common.http.HttpConstants; import org.junit.Assert; import org.junit.ClassRule; import org.junit.Rule; @@ -70,7 +71,7 @@ public void testIncrementAnIndex() throws Exception { ); Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/rest/api/2/search?maxResults=2&startAt={pagination.index}") //"https://issues.cask.co/rest/api/2/search?maxResults=2&startAt={pagination.index}") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json") @@ -79,7 +80,7 @@ public void testIncrementAnIndex() throws Exception { "type:/fields/issuetype/name,description:/fields/description," + "projectCategory:/fields/project/projectCategory/name,isSubtask:/fields/issuetype/subtask," + "fixVersions:/fields/fixVersions") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Increment an index") .put(BaseHttpSourceConfig.PROPERTY_START_INDEX, "0") .put(BaseHttpSourceConfig.PROPERTY_INDEX_INCREMENT, "2") @@ -124,14 +125,14 @@ public void testIncrementAnIndexXml() throws Exception { // https://services.odata.org // /V3/Northwind/Northwind.svc/Customers?$inlinecount=allpages&$top=5&$skip={pagination.index} Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/V3/Northwind/Northwind.svc" + + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/V3/Northwind/Northwind.svc" + "/Customers?$inlinecount=allpages&$top=5&$skip={pagination.index}") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "xml") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/feed/entry") .put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "companyName:content/properties/CompanyName,postalCode:content/properties/PostalCode," + "country:content/properties/Country,phone:content/properties/Phone,fax:content/properties/Fax/text()") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Increment an index") .put(BaseHttpSourceConfig.PROPERTY_START_INDEX, "0") .put(BaseHttpSourceConfig.PROPERTY_INDEX_INCREMENT, "20") @@ -177,12 +178,12 @@ public void testLinkInResponseBody() throws Exception { Schema.of(Schema.Type.STRING)) ); Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, + .put(HttpConstants.PROPERTY_URL, // http://confluence.atlassian.com/rest/api/space/ADMINJIRASERVER0710/content/page?limit=100&start=0 getServerAddress() + "/rest/api/space/ADMINJIRASERVER0710/content/page?limit=2&start=0") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Link in response body") .put(BaseHttpSourceConfig.PROPERTY_NEXT_PAGE_FIELD_PATH, "/_links/next") .build(); @@ -222,13 +223,13 @@ public void testLinkInResponseHeader() throws Exception { )) ); Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, + .put(HttpConstants.PROPERTY_URL, //"https://api.github.com/search/code?q=Salesforce+user:data-integrations") getServerAddress() + "/search/code?q=Salesforce+user:data-integrations&per_page=2") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/items") .put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "path:/path,score:/score,repository:/repository") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Link in response header") .build(); @@ -282,11 +283,11 @@ public void testNonePagination() throws Exception { Map properties = new ImmutableMap.Builder() // https://searchcode.com/api/codesearch_I/?q=cdap - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results") .put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "repo:repo,language:/language,file:/filename,url:/url") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None") .build(); @@ -307,9 +308,9 @@ public void testNonePaginationBlob() throws Exception { ); Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/blob") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/blob") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "blob") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None") .build(); @@ -347,9 +348,9 @@ public void testNonePaginationCSV() throws Exception { Map properties = new ImmutableMap.Builder() // https://s3.theeventscalendar.com/uploads/2014/09/test-data-venues11.csv - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/uploads/2014/09/test-data-venues11.csv") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/uploads/2014/09/test-data-venues11.csv") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "csv") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None") .build(); @@ -370,9 +371,9 @@ public void testNonePaginationText() throws Exception { ); //"http://www.columbia.edu/~fdc/sample.html") Map properties = new ImmutableMap.Builder() - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/sample.html") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/sample.html") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "text") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None") .build(); @@ -395,11 +396,11 @@ public void testNonePaginationXml() throws Exception { Map properties = new ImmutableMap.Builder() // https://www.w3.org/2003/05/soap-envelope/ - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/2003/05/soap-envelope/") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/2003/05/soap-envelope/") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "xml") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/schema/complexType[@name='Fault']/sequence/element") .put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "name:@name,type:@type") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "None") .build(); @@ -449,11 +450,11 @@ public void testPaginationCustom() throws Exception { Map properties = new ImmutableMap.Builder() // https://searchcode.com/api/codesearch_I/?q=cdap&per_page=2 - .put(BaseHttpSourceConfig.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap&per_page=2") + .put(HttpConstants.PROPERTY_URL, getServerAddress() + "/api/codesearch_I/?q=cdap&per_page=2") .put(BaseHttpSourceConfig.PROPERTY_FORMAT, "json") .put(BaseHttpSourceConfig.PROPERTY_RESULT_PATH, "/results") .put(BaseHttpSourceConfig.PROPERTY_FIELDS_MAPPING, "repo:repo,language:/language,file:/filename,url:/url") - .put(BaseHttpSourceConfig.PROPERTY_SCHEMA, schema.toString()) + .put(HttpConstants.PROPERTY_SCHEMA, schema.toString()) .put(BaseHttpSourceConfig.PROPERTY_PAGINATION_TYPE, "Custom") .put(BaseHttpSourceConfig.PROPERTY_CUSTOM_PAGINATION_CODE, paginationCode) .build(); @@ -485,20 +486,20 @@ public void testPaginationCustom() throws Exception { protected Map getProperties(Map sourceProperties) { return new ImmutableMap.Builder() .put("referenceName", testName.getMethodName()) - .put(BaseHttpSourceConfig.PROPERTY_HTTP_METHOD, "GET") - .put(BaseHttpSourceConfig.PROPERTY_OAUTH2_ENABLED, "false") - .put(BaseHttpSourceConfig.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail") - .put(BaseHttpSourceConfig.PROPERTY_ERROR_HANDLING, "stopOnError") - .put(BaseHttpSourceConfig.PROPERTY_RETRY_POLICY, "linear") + .put(HttpConstants.PROPERTY_HTTP_METHOD, "GET") + .put(HttpConstants.PROPERTY_OAUTH2_ENABLED, "false") + .put(HttpConstants.PROPERTY_HTTP_ERROR_HANDLING, "2..:Success,.*:Fail") + .put(HttpConstants.PROPERTY_ERROR_HANDLING, "stopOnError") + .put(HttpConstants.PROPERTY_RETRY_POLICY, "linear") .put(BaseHttpSourceConfig.PROPERTY_MAX_RETRY_DURATION, "10") .put(BaseHttpSourceConfig.PROPERTY_LINEAR_RETRY_INTERVAL, "1") .put(BaseHttpSourceConfig.PROPERTY_WAIT_TIME_BETWEEN_PAGES, "0") - .put(BaseHttpSourceConfig.PROPERTY_CONNECT_TIMEOUT, "60") - .put(BaseHttpSourceConfig.PROPERTY_READ_TIMEOUT, "120") - .put(BaseHttpSourceConfig.PROPERTY_VERIFY_HTTPS, "true") - .put(BaseHttpSourceConfig.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)") - .put(BaseHttpSourceConfig.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)") - .put(BaseHttpSourceConfig.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2") + .put(HttpConstants.PROPERTY_CONNECT_TIMEOUT, "60") + .put(HttpConstants.PROPERTY_READ_TIMEOUT, "120") + .put(HttpConstants.PROPERTY_VERIFY_HTTPS, "true") + .put(HttpConstants.PROPERTY_KEYSTORE_TYPE, "Java KeyStore (JKS)") + .put(HttpConstants.PROPERTY_TRUSTSTORE_TYPE, "Java KeyStore (JKS)") + .put(HttpConstants.PROPERTY_TRANSPORT_PROTOCOLS, "TLSv1.2") .putAll(sourceProperties) .build(); } diff --git a/widgets/HTTP-action.json b/widgets/HTTP-action.json new file mode 100644 index 0000000..4f2ab93 --- /dev/null +++ b/widgets/HTTP-action.json @@ -0,0 +1,384 @@ +{ + "metadata": { + "spec-version": "1.5" + }, + "configuration-groups": [ + { + "label": "General", + "properties": [ + { + "widget-type": "textbox", + "label": "URL", + "name": "url" + }, + { + "widget-type": "select", + "label": "HTTP Method", + "name": "httpMethod", + "widget-attributes": { + "values": [ + "GET", + "POST", + "PUT", + "DELETE", + "HEAD" + ], + "default": "GET" + } + }, + { + "widget-type": "keyvalue", + "label": "Headers", + "name": "headers", + "widget-attributes": { + "showDelimiter": "false" + } + }, + { + "widget-type": "textarea", + "name": "requestBody", + "label": "Request Body", + "widget-attributes": { + "rows": "5" + } + } + ] + }, + { + "label": "OAuth2", + "properties": [ + { + "widget-type": "toggle", + "label": "OAuth2 Enabled", + "name": "oauth2Enabled", + "widget-attributes": { + "default": "false", + "on": { + "label": "True", + "value": "true" + }, + "off": { + "label": "False", + "value": "false" + } + } + }, + { + "widget-type": "textbox", + "label": "Auth URL", + "name": "authUrl" + }, + { + "widget-type": "textbox", + "label": "Token URL", + "name": "tokenUrl" + }, + { + "widget-type": "textbox", + "label": "Client ID", + "name": "clientId" + }, + { + "widget-type": "password", + "label": "Client Secret", + "name": "clientSecret" + }, + { + "widget-type": "textbox", + "label": "Scopes", + "name": "scopes" + }, + { + "widget-type": "textbox", + "label": "Refresh Token", + "name": "refreshToken" + } + ] + }, + { + "label": "Basic Authentication", + "properties": [ + { + "widget-type": "textbox", + "label": "Username", + "name": "username" + }, + { + "widget-type": "password", + "label": "Password", + "name": "password" + } + ] + }, + { + "label": "HTTP Proxy", + "properties": [ + { + "widget-type": "textbox", + "label": "Proxy URL", + "name": "proxyUrl" + }, + { + "widget-type": "textbox", + "label": "Username", + "name": "proxyUsername" + }, + { + "widget-type": "password", + "label": "Password", + "name": "proxyPassword" + } + ] + }, + { + "label": "Error Handling", + "properties": [ + { + "widget-type": "keyvalue-dropdown", + "label": "HTTP Errors Handling", + "name": "httpErrorsHandling", + "widget-attributes": { + "default": "2..:Success,.*:Fail", + "showDelimiter": "false", + "dropdownOptions": [ + "Success", + "Fail" + ], + "key-placeholder": "HTTP Status Code Regex" + } + }, + { + "widget-type": "radio-group", + "label": "Non-HTTP Error Handling", + "name": "errorHandling", + "widget-attributes": { + "layout": "inline", + "default": "stopOnError", + "options": [ + { + "id": "stopOnError", + "label": "Stop on error" + } + ] + } + }, + { + "widget-type": "number", + "label": "Connect Timeout", + "name": "connectTimeout", + "widget-attributes": { + "min": "0", + "default": "120" + } + }, + { + "widget-type": "number", + "label": "Read Timeout", + "name": "readTimeout", + "widget-attributes": { + "min": "0", + "default": "120" + } + } + ] + }, + { + "label": "SSL/TLS", + "properties": [ + { + "widget-type": "toggle", + "label": "Verify HTTPS Trust Certificates", + "name": "verifyHttps", + "widget-attributes": { + "default": "true", + "on": { + "label": "True", + "value": "true" + }, + "off": { + "label": "False", + "value": "false" + } + } + }, + { + "widget-type": "textbox", + "label": "Keystore File", + "name": "keystoreFile" + }, + { + "widget-type": "select", + "label": "Keystore Type", + "name": "keystoreType", + "widget-attributes": { + "default": "Java KeyStore (JKS)", + "values": [ + "Java KeyStore (JKS)", + "Java Cryptography Extension KeyStore (JCEKS)", + "PKCS #12" + ] + } + }, + { + "widget-type": "password", + "label": "Keystore Password", + "name": "keystorePassword" + }, + { + "widget-type": "textbox", + "label": "Keystore Key Algorithm", + "name": "keystoreKeyAlgorithm", + "widget-attributes": { + "default": "SunX509" + } + }, + { + "widget-type": "textbox", + "label": "Keystore Cert Alias", + "name": "keystoreCertAlias" + }, + { + "widget-type": "textbox", + "label": "TrustStore File", + "name": "trustStoreFile" + }, + { + "widget-type": "select", + "label": "TrustStore Type", + "name": "trustStoreType", + "widget-attributes": { + "default": "Java KeyStore (JKS)", + "values": [ + "Java KeyStore (JKS)", + "Java Cryptography Extension KeyStore (JCEKS)", + "PKCS #12" + ] + } + }, + { + "widget-type": "password", + "label": "TrustStore Password", + "name": "trustStorePassword" + }, + { + "widget-type": "textbox", + "label": "TrustStore Key Algorithm", + "name": "trustStoreKeyAlgorithm", + "widget-attributes": { + "default": "SunX509" + } + }, + { + "widget-type": "csv", + "label": "Transport Protocols", + "name": "transportProtocols", + "widget-attributes": { + "default": "TLSv1.2" + } + }, + { + "widget-type": "textbox", + "label": "Cipher Suites", + "name": "cipherSuites" + } + ] + } + ], + "filters": [ + { + "name": "Proxy authentication", + "condition": { + "property": "proxyUrl", + "operator": "exists" + }, + "show": [ + { + "name": "proxyUsername", + "type": "property" + }, + { + "name": "proxyPassword", + "type": "property" + } + ] + }, + { + "name": "OAuth 2 disabled", + "condition": { + "property": "oauth2Enabled", + "operator": "equal to", + "value": "false" + }, + "show": [ + { + "name": "username", + "type": "property" + }, + { + "name": "password", + "type": "property" + } + ] + }, + { + "name": "OAuth 2 enabled", + "condition": { + "property": "oauth2Enabled", + "operator": "equal to", + "value": "true" + }, + "show": [ + { + "name": "authUrl", + "type": "property" + }, + { + "name": "tokenUrl", + "type": "property" + }, + { + "name": "clientId", + "type": "property" + }, + { + "name": "clientSecret", + "type": "property" + }, + { + "name": "scopes", + "type": "property" + }, + { + "name": "refreshToken", + "type": "property" + } + ] + }, + { + "name": "SSL HTTPS Verification", + "condition": { + "property": "verifyHttps", + "operator": "equal to", + "value": "true" + }, + "show": [ + { + "name": "trustStoreFile", + "type": "property" + }, + { + "name": "trustStoreType", + "type": "property" + }, + { + "name": "trustStorePassword", + "type": "property" + }, + { + "name": "trustStoreKeyAlgorithm", + "type": "property" + } + ] + } + ] +} diff --git a/widgets/HTTP-batchsource.json b/widgets/HTTP-batchsource.json index 6c05378..373101e 100644 --- a/widgets/HTTP-batchsource.json +++ b/widgets/HTTP-batchsource.json @@ -426,6 +426,11 @@ "default": "SunX509" } }, + { + "widget-type": "textbox", + "label": "Keystore Cert Alias", + "name": "keystoreCertAlias" + }, { "widget-type": "textbox", "label": "TrustStore File", @@ -503,7 +508,7 @@ "name": "Proxy authentication", "condition": { "property": "proxyUrl", - "operator": "exists", + "operator": "exists" }, "show": [ { diff --git a/widgets/HTTP-streamingsource.json b/widgets/HTTP-streamingsource.json index e7abdb6..798a6fe 100644 --- a/widgets/HTTP-streamingsource.json +++ b/widgets/HTTP-streamingsource.json @@ -425,6 +425,11 @@ "default": "SunX509" } }, + { + "widget-type": "textbox", + "label": "Keystore Cert Alias", + "name": "keystoreCertAlias" + }, { "widget-type": "textbox", "label": "TrustStore File",