Difficulties to access my Opensearch server using the plugin

[remiborredon]

Hello,

I'm discorvering Opensearch and Fluentd and I'm trying to use your plugin to send my logs to Opensearch on a k8s cluster.

For deploying opensearch, I'm using the latest version of the helm chart available at: https://github.com/opensearch-project/helm-charts withtout any modification.

For deploying fluentd, I'm using the helm chart available here : https://github.com/fluent/helm-charts?msclkid=f09cc1a8d05d11ec8c465e18fe6e4345

For installing the plugin, I added the line " - fluent-plugin-opensearch" in the plugins sections. I also modified the configuration to send the data to OpenSearch as follow:

[Logs](https://10.56.131.69/dashboard/#/log/default/fluentd-nxj5b/pod?namespace=default&container=fluentd)
Journaux de
fluentd
dans
fluentd-nxj5b
Successfully installed faraday-httpclient-1.0.1
Successfully installed faraday-multipart-1.0.3
Successfully installed faraday-patron-1.0.0
Successfully installed faraday-rack-1.0.0
Successfully installed faraday-retry-1.0.3
Successfully installed faraday-1.10.0
Successfully installed opensearch-transport-1.0.0
Successfully installed opensearch-api-1.0.0
Successfully installed opensearch-ruby-1.0.0
Successfully installed jmespath-1.6.1
Successfully installed aws-partitions-1.587.0
Successfully installed aws-eventstream-1.2.0
Successfully installed aws-sigv4-1.5.0
Successfully installed aws-sdk-core-3.130.2
Successfully installed faraday_middleware-aws-sigv4-0.6.1
Successfully installed fluent-plugin-opensearch-1.0.4
16 gems installed
2022-05-10 12:39:38 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.6.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.4'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.5'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-05-10 12:39:39 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2022-05-10 12:39:39 +0000 [info]: gem 'fluentd' version '1.12.4'
2022-05-10 12:39:39 +0000 [info]: using configuration file: <ROOT>
<label @FLUENT_LOG>
<match **>
@type null
@id ignore_fluent_logs
</match>
</label>
<source>
@type syslog
label @proftpd-sftp
port 5140
bind "0.0.0.0"
tag "proftpd-sftp.*"
</source>
<label @proftpd-sftp>
<match **>
@type opensearch
host "opensearch-cluster-master.default.svc"
port 9200
index_name "fluentd"
scheme https
ssl_verify false
</match>
</label>
<source>
@type prometheus
@id in_prometheus
bind "0.0.0.0"
port 24231
metrics_path "/metrics"
</source>
<source>
@type prometheus_monitor
@id in_prometheus_monitor
</source>
<source>
@type prometheus_output_monitor
@id in_prometheus_output_monitor
</source>
</ROOT>
2022-05-10 12:39:39 +0000 [info]: starting fluentd-1.12.4 pid=1 ruby="2.6.7"
2022-05-10 12:39:39 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/../../../etc/fluent/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-elasticsearch-5.0.3/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2022-05-10 12:39:41 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="null"
2022-05-10 12:39:41 +0000 [info]: adding match in @proftpd-sftp pattern="**" type="opensearch"
The client is unable to verify distribution due to security privileges on the server side. Some functionality may not be compatible if the server is running an unsupported product.
2022-05-10 12:39:43 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [401] Unauthorized
2022-05-10 12:39:43 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2022-05-10 12:39:47 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [401] Unauthorized
2022-05-10 12:39:47 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
2022-05-10 12:39:55 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [401] Unauthorized
2022-05-10 12:39:55 +0000 [warn]: #0 Remaining retry: 12. Retry to communicate after 8 second(s).
2022-05-10 12:40:01 +0000 [info]: Received graceful stop
2022-05-10 12:40:11 +0000 [warn]: #0 Could not communicate to OpenSearch, resetting connection and trying again. [401] Unauthorized
2022-05-10 12:40:11 +0000 [warn]: #0 Remaining retry: 11. Retry to communicate after 16 second(s).

It really looks that this is only a credential issue (I was not able to find what are the default credentials in OpenSearch) but on the other hand, when I look into the OpenSearch log, I can see this:

> [2022-05-10T12:13:02,389][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-cluster-master-0] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=null}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465722e64656661756c742e7376633a393230300d0a557365722d4167656e743a206f70656e7365617263682d727562792f312e302e302028525542595f56455253494f4e3a20322e362e373b206c696e7578207838365f36343b20466172616461792076312e31302e30290d0a4163636570743a202a2f2a0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206f70656e7365617263682d636c75737465722d6d61737465722e64656661756c742e7376633a393230300d0a557365722d4167656e743a206f70656e7365617263682d727562792f312e302e302028525542595f56455253494f4e3a20322e362e373b206c696e7578207838365f36343b20466172616461792076312e31302e30290d0a4163636570743a202a2f2a0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1213) ~[netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) ~[netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.73.Final.jar:4.1.73.Final]
... 16 more

This looks to be something completely different and I don't know in which direction I shall go.

Could someone help me on that?

Thanks in advance!