Slow reports after upgrading to v2.13.2

[cdmwebs]

Describe the bug

We're still working to narrow down the actual issue, but after upgrading our app from v2.13.0 to v2.13.2, one of our async tasks that previously took ~3 seconds ballooned to ~3 minutes. The SQL body of the insert statements are fairly large.

We tested our docker image, application server, database connection, VPC timing, and finally disabled Appsignal to try to figure out where the issue was. We noticed an immediate improvement with Appsignal disabled.

We upgraded to 2.13.2 on 11/25, which is roughly the same time the issue appears in our samples. We then downgraded our staging and production servers to 2.13.0 and (so far) performance has returned to prior levels.

Still trying to see if we can provide additional info but wanted to open this to get the ball rolling. It could possibly be related to #971.

To Reproduce

Steps to reproduce the behavior:

TBD

[unflxw]

Hi @cdmwebs, thank you for reporting this!

If you think it could be related to these large SQL INSERT statements, the first thing I'd suggest is to try setting the instrument_ecto config option to false. This should disable AppSignal's instrumentation of SQL queries. If doing so on version 2.13.2 mitigates the issue you're encountering, it would hint towards the SQL query being the issue.

In between the two versions of our Elixir integration that you upgraded from, we changed how INSERT INTO ... VALUES queries are sanitized by our agent. This might be the underlying cause of the issue. If possible, I would ask you to provide a sample of one of those queries, as unaltered as possible, over at [email protected] -- please include a link to this GitHub issue in the email.

I see that you also mention that you think it might be related to #971. Do you have a theory as to why? Does your application use check-ins, or fetch data using Hackney?

[cdmwebs]

That does seem relevant, @unflxw. I forgot about the NIF. We'll get a query over to you soon. Thanks for the quick response.

[codingbull]

If doing so on version 2.13.2 mitigates the issue you're encountering, it would hint towards the SQL query being the issue.

Yes, setting instrument_ecto: false does seem to mitigate the slowness on appsignal: 2.13.2 and appsignal_phoenix: 2.6.0

[shairyar]

https://app.intercom.com/a/inbox/yzor8gyw/inbox/shared/unassigned/conversation/16410700388990?view=List (private link)

[unflxw]

Hi @cdmwebs @codingbull,

AppSignal for Elixir 2.13.3 has been released, with a fix to the issue above. 🎉 Please let us know if you run into any problems with it.

[codingbull]

Can confirm that 2.13.3 has addressed this slowness. Thanks!