Error reported when using dubbo3.0.11:java.lang.IllegalStateException: Serialized class co.elastic.apm.agent.impl.transaction.Span must implement java.io.Serializable

[org-kepe]

Dubbo server encountered an error when calling other Dubbo client

io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: Serialized class co.elastic.apm.agent.impl.transaction.Span must implement java.io.Serializable
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:125)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:307)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:891)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868)
at io.netty.channel.ChannelDuplexHandler.write(ChannelDuplexHandler.java:115)
at org.apache.dubbo.remoting.transport.netty4.NettyClientHandler.write(NettyClientHandler.java:88)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:891)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:956)
at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1263)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Serialized class co.elastic.apm.agent.impl.transaction.Span must implement java.io.Serializable
at com.alibaba.com.caucho.hessian.io.SerializerFactory.getDefaultSerializer(SerializerFactory.java:417)
at com.alibaba.com.caucho.hessian.io.SerializerFactory.getSerializer(SerializerFactory.java:391)
at com.alibaba.com.caucho.hessian.io.Hessian2Output.writeObject(Hessian2Output.java:411)
at com.alibaba.com.caucho.hessian.io.MapSerializer.writeObject(MapSerializer.java:98)
at com.alibaba.com.caucho.hessian.io.Hessian2Output.writeObject(Hessian2Output.java:412)
at org.apache.dubbo.common.serialize.hessian2.Hessian2ObjectOutput.writeObject(Hessian2ObjectOutput.java:92)
at org.apache.dubbo.common.serialize.ObjectOutput.writeAttachments(ObjectOutput.java:57)
at org.apache.dubbo.rpc.protocol.dubbo.DubboCodec.encodeRequestData(DubboCodec.java:210)
at org.apache.dubbo.remoting.exchange.codec.ExchangeCodec.encodeRequest(ExchangeCodec.java:257)
at org.apache.dubbo.remoting.exchange.codec.ExchangeCodec.encode(ExchangeCodec.java:71)
at org.apache.dubbo.rpc.protocol.dubbo.DubboCountCodec.encode(DubboCountCodec.java:47)
at org.apache.dubbo.remoting.transport.netty4.NettyCodecAdapter$InternalEncoder.encode(NettyCodecAdapter.java:69)
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
... 22 more

jdk:1.8.215
dubbo:3.0.11

[SylvainJuge]

Hi,

Thanks for reporting this @org-kepe , could you provide some extra context on when this happens ?

For example:

• is that something that is happening once or does it happens on every request ?
• can you describe a bite more the scenario here ? I assume an Dubbo client calling a Dubbo server, but are both sides instrumented with the Elastic APM agent ?

[org-kepe]

The calling relationship is roughly as follows:
client A->Server B->Server C
An error occurred when B called C, and only B had APM.

Call using the method of 'GenericService.$invoke', sometimes calling 'EchoService.$echo' to test connectivity.

This error doesn't happen every time. I tested it for a day and found two or three times, but I haven't found any patterns yet. But regardless of whether there are errors or not, Dubbo's transaction and span will not be reported

[org-kepe]

Hi,

Thanks for reporting this @org-kepe , could you provide some extra context on when this happens ?

For example:

• is that something that is happening once or does it happens on every request ?
• can you describe a bite more the scenario here ? I assume an Dubbo client calling a Dubbo server, but are both sides instrumented with the Elastic APM agent ?

I know how this bug was caused.
apm code:

dubbo 3.0 code:

The span will always return null.

Perhaps it can be solved through other methods:
RpcContext.getContext().get().get(DubboTraceHelper.SPAN_KEY);

[JonasKunz]

Hi @org-kepe , unfortunately we currently only support dubbo 2.x versions, as those are the ones we currently test with.
I'll clarify that on our supported technologies page.

I've also opened #3546 and will close this issue in favor.
Unfortunately adding Dubbo 3+ support is currently not our top priority, therefore we won't be able to get to it soon. Community contributions are welcome here!