Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Logstash JDBC input (v5.4.1) crash pipeline when configured server is unreachable for some reason #127

Open
vintur opened this issue Apr 26, 2023 · 1 comment
Open

Logstash JDBC input (v5.4.1) crash pipeline when configured server is unreachable for some reason #127

vintur opened this issue Apr 26, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@vintur
Copy link

vintur commented Apr 26, 2023 •
edited
Loading

Logstash information:

Please include the following information:

  1. Logstash version:
    logstash 8.6.2
  2. Logstash installation source:
    yum localinstall logstash-8.6.2-x86_64.rpm
  3. How is Logstash being run:
    as a service
  4. How was the Logstash Plugin installed:
    logstash-integration-jdbc (5.4.1) was installed by default after upgrade.

JVM (e.g. java -version):
openjdk 17.0.6 2023-01-17
OpenJDK Runtime Environment Temurin-17.0.6+10 (build 17.0.6+10)
OpenJDK 64-Bit Server VM Temurin-17.0.6+10 (build 17.0.6+10, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
Linux Servername 3.10.0-1160.88.1.el7.x86_64 #1 SMP Sat Feb 18 13:27:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.9 (Maipo)

Description of the problem including expected versus actual behavior:
The pipeline with configured jdbc inputs crashes at startup when the configured database server is unreachable for some reason, for example not reachable via DNS or if the datebase is not listening on the configured port.

Expected behavior would be to just ignore the non reachable database and try it again next scheduled run.

Steps to reproduce:
Input configuration:

jdbc {
        jdbc_driver_library => "/etc/logstash/ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        jdbc_connection_string => "${CONNECTION_STRING}"
        jdbc_user => "${USER}"
        jdbc_password => "${PWD}"
        statement_filepath => "/etc/logstash/statement/statement.txt"
        schedule => "${SCHEDULE}"
        record_last_run => true
        tracking_column_type => "timestamp"
        tracking_column => "completion_time"
        use_column_value => true
        last_run_metadata_path => "/var/lib/logstash/.logstash_last_run"
        tags => ["jdbc_input"]
}

Provide logs (if relevant):

#################################### log 1 #########################################
[2023-04-21T08:15:44,715][INFO ][logstash.javapipeline    ][nameofpipeline] Starting pipeline {:pipeline_id=>"nameofpipeline", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x74cc1016 run>"}
[2023-04-21T08:15:44,995][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline Java execution initialization time {"seconds"=>0.28}
[2023-04-21T08:15:46,390][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T08:15:46,433][ERROR][logstash.inputs.jdbc     ][nameofpipeline] 
java.sql.SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known
	at oracle.jdbc.driver.T4CConnection.handleLogonNetException(oracle/jdbc/driver/T4CConnection.java:1099) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:879) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:1132) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:104) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:837) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:664) ~[ojdbc11.jar:23.2.0.0.0]
.... 400 more stack trace lines
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) ~[?:?]
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
	at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]
Caused by: oracle.net.ns.NetException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known
	at oracle.net.resolver.NavAddress.parseAddressParamsAndNavigate(NavAddress.java:208) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavAddress.navigate(NavAddress.java:83) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavDescription.navigate(NavDescription.java:214) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.NavServiceAlias.navigate(NavServiceAlias.java:61) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.resolveAddrTree(AddrResolution.java:1106) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.resolveTNSAddress(AddrResolution.java:918) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.initConnStrategy(AddrResolution.java:613) ~[ojdbc11.jar:23.2.0.0.0]
	at oracle.net.resolver.AddrResolution.<init>(AddrResolution.java:430) ~[ojdbc11.jar:23.2.0.0.0]
.... 400 more stack trace lines
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) [?:?]
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141) [jruby.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64) [jruby.jar:?]
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby.jar:?]
	at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:309) [jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) [jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-04-21T08:15:46,456][ERROR][logstash.inputs.jdbc     ][nameofpipeline] Unable to connect to database. Tried 1 times {:message=>"Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known", :exception=>Sequel::DatabaseConnectionError, :cause=>#<Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known>, :backtrace=>["oracle.jdbc.driver.T4CConnection.handleLogonNetException(oracle/jdbc/driver/T4CConnection.java:1099)", "oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:879)", "oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:1132)", "oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:104)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:837)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:664)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188)", "org.jruby.RubyClass.new(org/jruby/RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)", "java.lang.Thread.run(java/lang/Thread.java:833)"]}
[2023-04-21T08:15:46,469][WARN ][logstash.inputs.jdbc     ][nameofpipeline] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known", :cause=>"#<Java::JavaSql::SQLException: ORA-17868: Unknown host specified.: dbserver.domain.example.com: Name or service not known>"}
[2023-04-21T08:15:46,680][ERROR][logstash.javapipeline    ][nameofpipeline] Pipeline error {:pipeline_id=>"nameofpipeline", :exception=>#<LogStash::ConfigurationError: Can't create a connection pool to the database>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:318:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x74cc1016 run>"}
[2023-04-21T08:15:46,684][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline terminated {"pipeline.id"=>"nameofpipeline"}


#################################### log 2 #########################################


9:28,891][INFO ][logstash.javapipeline    ][nameofpipeline] Starting pipeline {:pipeline_id=>"nameofpipeline", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>375, "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x37d669c2 run>"}
[2023-04-21T15:39:29,200][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline Java execution initialization time {"seconds"=>0.31}
[2023-04-21T15:39:30,453][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:31,040][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:31,565][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:32,105][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:39:32,761][INFO ][logstash.inputs.jdbc     ][nameofpipeline] ECS compatibility is enabled but `target` option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the `target` option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2023-04-21T15:51:02,833][ERROR][logstash.inputs.jdbc     ][nameofpipeline] 
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
	at oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:774) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:688) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:39) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:691) ~[ojdbc8.jar:12.2.0.1.0]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
	at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472) ~[jruby.jar:?]
	at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330) ~[jruby.jar:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118) ~[?:?]
	at org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507) ~[jruby.jar:?]
	at org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen) ~[jruby.jar:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309) ~[?:?]
	at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234) ~[?:?]
	at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191) ~[?:?]
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143) ~[?:?]
	at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
	at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
	at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521) ~[ojdbc8.jar:12.2.0.1.0]
	at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at org.jruby.runtime.Block.call(Block.java:143) [jruby.jar:?]
	at org.jruby.RubyProc.call(RubyProc.java:309) [jruby.jar:?]
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107) [jruby.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.IOException: Operation interrupted, socket connect lapse 15000 ms. dbserver.domain.example.com/10.0.0.1 1521 0 1 true
	at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.InterruptedIOException: Operation interrupted
	at oracle.net.nt.TimeoutSocketChannel.handleInterrupt(TimeoutSocketChannel.java:311) ~[ojdbc8.jar:12.2.0.1.0]
.... 400 more stack trace lines
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-04-21T15:51:02,844][ERROR][logstash.inputs.jdbc     ][nameofpipeline] Unable to connect to database. Tried 1 times {:message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :exception=>Sequel::DatabaseConnectionError, :cause=>#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>, :backtrace=>["oracle.jdbc.driver.T4CConnection.logon(oracle/jdbc/driver/T4CConnection.java:774)", "oracle.jdbc.driver.PhysicalConnection.connect(oracle/jdbc/driver/PhysicalConnection.java:688)", "oracle.jdbc.driver.T4CDriverExtension.getConnection(oracle/jdbc/driver/T4CDriverExtension.java:39)", "oracle.jdbc.driver.OracleDriver.connect(oracle/jdbc/driver/OracleDriver.java:691)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:472)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:330)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.adapters.jdbc.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/adapters/jdbc.rb:237)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.new_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:248)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.make_new(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.assign_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.acquire(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:139)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.connection_pool.threaded.hold(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/connection_pool/threaded.rb:91)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.synchronize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:293)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.test_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:302)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.misc.initialize(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/misc.rb:188)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.database.connecting.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/database/connecting.rb:57)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.sequel_minus_5_dot_62_dot_0.lib.sequel.core.connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/sequel-5.62.0/lib/sequel/core.rb:124)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:118)", "org.jruby.RubyKernel.loop(org/jruby/RubyKernel.java:1507)", "org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(org/jruby/RubyKernel$INVOKER$s$0$0$loop.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.jdbc_connect(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:115)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.plugin_mixins.jdbc.jdbc.open_jdbc_connection(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/plugin_mixins/jdbc/jdbc.rb:158)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_4_dot_1.lib.logstash.inputs.jdbc.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:309)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_mixin_minus_ecs_compatibility_support_minus_1_dot_3_dot_0_minus_java.lib.logstash.plugin_mixins.ecs_compatibility_support.target_check.register(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:309)", "java.lang.Thread.run(java/lang/Thread.java:833)"]}
[2023-04-21T15:51:02,847][WARN ][logstash.inputs.jdbc     ][nameofpipeline] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :cause=>"#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>"}
[2023-04-21T15:51:03,544][ERROR][logstash.javapipeline    ][nameofpipeline] Pipeline error {:pipeline_id=>"nameofpipeline", :exception=>#<LogStash::ConfigurationError: Can't create a connection pool to the database>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-jdbc-5.4.1/lib/logstash/inputs/jdbc.rb:318:in `register'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:392:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:317:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/01-input.conf", "/etc/logstash/conf.d/11-filter-preprocess-lumberjack.conf", "/etc/logstash/conf.d/90-output.conf"], :thread=>"#<Thread:0x37d669c2 run>"}
[2023-04-21T15:51:03,547][INFO ][logstash.javapipeline    ][nameofpipeline] Pipeline terminated {"pipeline.id"=>"nameofpipeline"}
[2023-04-21T15:51:03,551][ERROR][logstash.agent           ] Failed to execute action {:id=>:nameofpipeline, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<nameofpipeline>, action_result: false", :backtrace=>nil}

Something similar to this:
logstash-plugins/logstash-input-jdbc#144

My thread in discussion:
https://discuss.elastic.co/t/logstash-jdbc-input-v5-4-1-crash-pipeline-when-configured-server-is-unreachable-or-if-the-server-is-not-listening-on-db-port/330530
@vintur vintur added the bug Something isn't working label Apr 26, 2023
@vintur vintur changed the title Logstash JDBC inputcrash pipeline when configured server is unreachable for some reason Logstash JDBC input crash pipeline when configured server is unreachable for some reason Apr 26, 2023
@vintur
Copy link
Author

vintur commented May 5, 2023 •
edited
Loading

My workaround for now is to install an older version:
bin/logstash-plugin install --version '5.2.4' logstash-integration-jdbc

Getting similar errors but the pipeline does not crash:
[2023-05-05T15:11:31,051][WARN ][logstash.inputs.jdbc ][pipelinename [6faf299d28a81ec8695e11ca859aaf26d751639446c01b99ea916ea01c0c5cc5] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection", :cause=>"#<Java::JavaSql::SQLRecoverableException: IO Error: The Network Adapter could not establish the connection>"}

@vintur vintur changed the title Logstash JDBC input crash pipeline when configured server is unreachable for some reason Logstash JDBC input (v5.4.1) crash pipeline when configured server is unreachable for some reason May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vintur