[Bug]: Account Does Not Sync to SimpleFIN

[weswitt]

Verified issue does not already exist?

• I have searched and found no existing issue
• I will be providing steps how to reproduce the bug (in most cases this will also mean uploading a demo budget file)

What happened?

After adding a new account that connects to SimpleFIN I do not see any transactions or a correct balance. I can see that SimpleFIN is returning the correct data as I can run CURL to download the accounts from SimpleFIN and all the data are returned. When doing the "Bank Sync" I see the error "504 Gateway Time-out". I'm syncing to 7 different banks with many accounts so maybe AB is not waiting long enough since SimpleFIN returns all transactions for all accounts in one response.

Where are you hosting Actual?

Docker

What browsers are you seeing the problem on?

Chrome

Operating System

Windows 10

[weswitt]

OK, I enabled debug logging and I'm seeing this:

Error parsing JSON response: error code: 524
SyntaxError: Unexpected token e in JSON at position 0
at JSON.parse ()
at IncomingMessage. (file:///app/src/app-simplefin/app-simplefin.js:332:26)
at IncomingMessage.emit (node:events:529:35)
at endReadableNT (node:internal/streams/readable:1400:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
2024-10-10T20:54:38.138Z info: POST 200 /simplefin/transactions

[youngcw]

Can you ping simplefin from your server running Actual?

[weswitt]

Yes. And as I said in the bug report I can download from SimpleFIN using a python script

[weswitt]

OK I debugged this further. I found that AB is issuing a request to SimpleFIN with the arguments "/accounts?start-date=1720828800&end-date=1730419200&pending=1", which a 4 month date range. This is likely a lot of data. So I used these same parameters in a python script and I found that what comes back is a response code of 524, timeout. The payload returned is HTML rendered by Cloudflare. This seems to be a problem with either Cloudflare or SimpleFN. I also observed that the timeout happened after 1m40.458s, so even if proper JSON data was eventually returned AB's 60 second timeout would have terminated the call.

@youngcw what do you think about this??

[youngcw]

The sync will always request 90 days of data unless you have less than that time in your account history. That 1m40s sounds like a long time to get data back. It should process on a few seconds.

[weswitt]

This is definitely a Cloudflare timeout, see https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/. Cloudflare will return a 524 after 100 seconds if the server does not respond. I am seeing the failure after exactly 100 seconds. This is absolutely caused by the fact that SimpleFIN is collecting a lot data by AB's request.

Actual Budget could fix this by making multiple calls for shorter date ranges and then combine the data.

[weswitt]

If I reduce the date range to one month it works fine.

[weswitt]

I modified getAccounts() in app-simplefin.js to never request more than 1 month and I now have things syncing again. This is not ideal because for new accounts it missed historical transactions.

I sent a support email to SimpleFIN to see if they can resolve this on their end.

[youngcw]

Did you ever hear back about this?

[d4vastu]

My issue is pretty odd. If I click bank sync some transactions do come in. However, if I want receive all new so actual matches my bank, I have to log in to simple fin, click adjust, wait for it to say my bank is already connected, then select bank sync again. Then everything works until the next day.

[youngcw]

My issue is pretty odd. If I click bank sync some transactions do come in. However, if I want receive all new so actual matches my bank, I have to log in to simple fin, click adjust, wait for it to say my bank is already connected, then select bank sync again. Then everything works until the next day.

@d4vastu Are you waiting until after simplefin has synced with your banks? Simplefin only syncs once a day so it can be up to 2 days before your transactions will show in Actual depending on the timing of things

[weswitt]

I did hear back from SimpleFIN. Their initial suggestion was the same as mine -- break up the sync calls into one month chunks. They sent another email back saying: "I've made a note to look into this, and I'll reach out to the Actual Budget folks, too. It looks like the timeout is happening on Cloudflare's side (and is by default set to 100 seconds). I'll see if we can extend that.
Thanks again for bringing this up!".

Ultimately you could say this is a SimpleFIN bug, but I really think that AB needs to change their logic to make smaller requests so their code is defensive against these issues.

[youngcw]

Somewhat soon we will update the simplefin calls to only request a single account if you only request one. That probably will help your issue.

[weswitt]

That is a great solution and will definitely fix the problem. Thanks a lot!!

[youngcw]

That option wasn't available until a few days ago unfortunately, but its been added on simplefin's side so we can start using it now.

[weswitt]

Yea, I was going to ask about that as their documented API only returned data for all accounts. Great timing for this.

[weswitt]

I picked up your changes to app-simplefin.js. Seems to fix the problem, thanks.

Where did you find docs for the new SimpleFIN API to query for a single account?? I looked on the site and cannot find the docs.

[youngcw]

You may have to look at their github to see them as they are so new