feat(tests): initial performance test #2828

njlie · 2024-07-30T23:39:33Z

Changes proposed in this pull request

Adds a script that can be run against the local environment.
- Script may be run inside a Docker container as well.
Basic shell script to verify that the local playground is live before running the script.

Context

The performance test script exercises the createReceiver, createQuote, and createOutgoingPayment GraphQL endpoints.

Checklist

netlify · 2024-07-30T23:39:49Z

✅ Deploy Preview for brilliant-pasca-3e80ec canceled.

Name	Link
🔨 Latest commit	`ce5da86`
🔍 Latest deploy log	https://app.netlify.com/sites/brilliant-pasca-3e80ec/deploys/66c36890114453000818364d

BlairCurrey · 2024-08-02T17:50:33Z

test/performance/scripts/run-docker-tests.sh

Seems like this is unused... Remove?

Both pnpm --filter performance run-tests and pnpm --filter performance run-tests-docker use run-test.sh

Good catch, thanks. Meant to remove it

BlairCurrey · 2024-08-05T18:51:44Z

test/performance/scripts/create-outgoing-payments.js

+      headers
+    }
+  )
+  const receiver = JSON.parse(createReceiverResponse.body).data.createReceiver


Im seeing some errors like this when spinning up the localenv and running pnpm --filter performance run-tests-docker:

time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createReceiver' of undefined\n\tat file:///scripts/create-outgoing-payments.js:125:64(84)\n" executor=constant-vus scenario=default source=stacktrace time="2024-08-05T18:40:26Z" level=error msg="TypeError: Cannot read property 'createQuote' of undefined\n\tat file:///scripts/create-outgoing-payments.js:159:58(128)\n" executor=constant-vus scenario=default source=stacktrace

9 TypeErrors for createReceiver and then 1 createQuote. Looks like its indicating data in JSON.parse(createReceiverResponse.body).data is undefined.

Im not sure what to make of it though because then the VUs seem to run fine. I see some quote times in http://localhost:4500/ but not sure there is enough info there to make much sense of the errors.

If I run pnpm --filter performance run-tests-docker again without restarting the localenv I dont see these TypeErrors.

As a side note, I was surprised to see no outgoing payment based metrics but this is actually consistent with the bruno peer-to-peer example (guess it happens outside of the outgoing payment lifecycle where these are recorded).

view of dashboard after running:

I think these errors are performance issues being exposed by the test. It seems to work more consistently after bringing the number of virtual users down to one, so I suspect that these are symptoms of the service being overwhelmed.

makes sense... I actually dont see it anymore

BlairCurrey · 2024-08-05T18:58:37Z

test/performance/scripts/create-outgoing-payments.js

+  // A number specifying the number of VUs to run concurrently.
+  vus: 10,
+  // A string specifying the total duration of the test run.
+  duration: '30s'


Should we bump this up? Im seeing this:

time="2024-08-05T18:51:25Z" level=warning msg="No script iterations fully finished, consider making the test duration longer"

Looks like I need to bump it up to match the issue description anyways:

For now, we can set the scenario to be quite small, a single virtual user (VU) across 10 minutes. We should be able to see the local playground Grafana dashboard properly populate with metrics (and traces).

BlairCurrey · 2024-08-06T03:35:05Z

test/performance/scripts/run-tests.sh

+fi
+
+# Verify that cloud nine wallet address is live
+if curl -s --head --request GET "$c9_wallet_address" | grep "HTTP/1.[01]" > /dev/null; then


So this is kinda funny....

The outgoing payment metrics should show after all. I was wrong here and the peer-to-peer bruno tests DO populate the outgoing payment based metrics (I was just impatient - underestimated the delay). So the performance tests should populate them too.

You can confirm this beyond the metrics if you do docker logs rafiki-cloud-nine-mock-ase-1 after running the tests and you should see many of these:

Error: No account found for wallet address at handleOutgoingPaymentCreated (/home/rafiki/localenv/mock-account-servicing-entity/build/index.js:2200:11) ...

This is showing the mock ase cant find the wallet address in the handle outgoing payment create webhook handler and then it doesnt deposit liquidity and the payment doesnt get funded. I looked into the DB and saw that we had 2 Grace Franklin wallet addresses. https://cloud-nine-wallet-backend/accounts/gfranklin and https://localhost:3000/accounts/gfranklin. Looks like the localhost:3000 one is getting created by the mock ase's wallet address not found handler when we ping here. In other words, you can change gfranklin to anything in c9_wallet_address and it works because it gets created if it doesnt exit. Then when the mock ASE looks up the wallet address, it expects this new one while the one specified in the performance test script is the original so it never gets funded.

Perhaps we can ping something else? Or nothing? http://cloud-nine-wallet-backend:3000/accounts/gfranklin connects but creates a new record as well, and we cant reach by https://cloud-nine-wallet-backend/accounts/gfranklin. Originally I noticed I could change the script to do const CLOUD_NINE_WALLET_ADDRESS = 'https://localhost:3000/accounts/gfranklin' but this is just kinda hacking around the unintended wallet address creation.

metrics showing in the dashboard setting constCLOUD_NINE_WALLET_ADDRESS = 'https://localhost:3000/accounts/gfranklin' in the script

I investigated this a little bit and I think it's a separate issue from there being different wallet addresses with the same name. I think when the webhook is generated it is populated with the database id of the wallet address, but the mock ASE passes that id into an instance of account-provider from mock-account-service-lib, which is seeded with different ids from what gets set in the database.

I get this same issue with the webhook handler not finding the wallet address coming up when I perform the Bruno peer-to-peer example, not sure how it got through when you tried it...

koekiebox

LGtM 🚀

mkurapov · 2024-08-09T17:34:39Z

test/performance/README.md

+
+- [Running local playground for Rafiki](../../localenv/README.md)
+
+If the local environment isn't running it may be started


I think you meant to finish this sentence. We should also mention how to start it with telemetry enabled

mkurapov · 2024-08-09T17:41:40Z

test/performance/scripts/create-outgoing-payments.js

+  const c9WalletAddresses = JSON.parse(c9WalletAddressesRes.body).data
+    .walletAddresses.edges
+  const c9WalletAddress = c9WalletAddresses.find(
+    (edge) => edge.node.url === CLOUD_NINE_WALLET_ADDRESS
+  ).node


I think we could move this into the setup function of the script, such that we don't need to fetch them every iteration. We should give a good error if the request to find the specific wallet address fails

mkurapov · 2024-08-19T11:41:46Z

test/performance/scripts/create-outgoing-payments.js

+  // The following section contains configuration options for execution of this
+  // test script in Grafana Cloud.
+  //
+  // See https://grafana.com/docs/grafana-cloud/k6/get-started/run-cloud-tests-from-the-cli/
+  // to learn about authoring and running k6 test scripts in Grafana k6 Cloud.
+  //
+  // cloud: {
+  //   // The ID of the project to which the test is assigned in the k6 Cloud UI.
+  //   // By default tests are executed in default project.
+  //   projectID: "",
+  //   // The name of the test in the k6 Cloud UI.
+  //   // Test runs with the same name will be grouped.
+  //   name: "script.js"
+  // },
+
+  // Uncomment this section to enable the use of Browser API in your tests.
+  //
+  // See https://grafana.com/docs/k6/latest/using-k6-browser/running-browser-tests/ to learn more
+  // about using Browser API in your test scripts.
+  //
+  // scenarios: {
+  //   // The scenario name appears in the result summary, tags, and so on.
+  //   // You can give the scenario any name, as long as each name in the script is unique.
+  //   ui: {
+  //     // Executor is a mandatory parameter for browser-based tests.
+  //     // Shared iterations in this case tells k6 to reuse VUs to execute iterations.
+  //     //
+  //     // See https://grafana.com/docs/k6/latest/using-k6/scenarios/executors/ for other executor types.
+  //     executor: 'shared-iterations',
+  //     options: {
+  //       browser: {
+  //         // This is a mandatory parameter that instructs k6 to launch and
+  //         // connect to a chromium-based browser, and use it to run UI-based
+  //         // tests.
+  //         type: 'chromium',
+  //       },
+  //     },
+  //   },
+  // }


I think we can remove this comment, if more scenarios are needed we can access the docs then

mkurapov · 2024-08-19T11:42:42Z

test/performance/scripts/create-outgoing-payments.js

+          metadata: {
+            description: null,
+            externalRef: null
+          },


Suggested change

metadata: {

description: null,

externalRef: null

},

There doesn't seem to be a change here?

I was just commenting we can remove metadata if it doesn't contain any info

mkurapov · 2024-08-19T12:39:00Z

test/performance/README.md

+The test can also be run inside of a Docker container on the same Docker network as the Local Playground:
+
+```
+pnpm --filter performance run-tests-docker


this script is useful such that we don't need to set the host names?

I took this from the description of the issue:

Since the local playground is within docker, we will need to run this script in docker as well, e.g.
docker run --rm -i grafana/k6 run - script.js

(Though not setting the host names is a benefit).

mkurapov

🚀

github-actions bot added the type: tests Testing related label Jul 30, 2024

njlie force-pushed the nl/2810/initial-load-test-2 branch from a835667 to 89e7e19 Compare August 1, 2024 21:06

njlie requested review from koekiebox, BlairCurrey, mkurapov and sabineschaller August 1, 2024 22:53

njlie marked this pull request as ready for review August 1, 2024 22:53

BlairCurrey reviewed Aug 2, 2024

View reviewed changes

BlairCurrey reviewed Aug 5, 2024

View reviewed changes

BlairCurrey reviewed Aug 6, 2024

View reviewed changes

BlairCurrey previously approved these changes Aug 7, 2024

View reviewed changes

koekiebox previously approved these changes Aug 7, 2024

View reviewed changes

mkurapov reviewed Aug 10, 2024

View reviewed changes

njlie dismissed stale reviews from koekiebox and BlairCurrey via 79c43f3 August 13, 2024 11:11

njlie requested review from mkurapov, koekiebox and BlairCurrey August 13, 2024 11:18

njlie added 10 commits August 13, 2024 16:17

feat(tests): initial performance test

f903591

feat: add test shell scripts, run inside docker network

22791bd

feat: run script in docker container

dbd8e4d

chore: formatting

9e3bc2d

chore: remove unused script

6c04e11

fix: update test script parameters

a44b0bf

fix: ping wallet address hosts instead of wallet addresses

169026d

fix README

96c3d73

feat: move wallet address retrieval into setup function

1507336

feat: add another error condition

c01d543

formatting

14ea6e5

njlie force-pushed the nl/2810/initial-load-test-2 branch from f914afd to 14ea6e5 Compare August 13, 2024 14:18

BlairCurrey previously approved these changes Aug 13, 2024

View reviewed changes

fix: typechecks

78fa93c

njlie dismissed BlairCurrey’s stale review via 78fa93c August 16, 2024 12:57

njlie requested a review from BlairCurrey August 19, 2024 10:16

mkurapov previously approved these changes Aug 19, 2024

View reviewed changes

chore: remove comments

854f854

njlie dismissed mkurapov’s stale review via 854f854 August 19, 2024 13:37

chore: remove empty metadata object

ce5da86

mkurapov approved these changes Aug 19, 2024

View reviewed changes

njlie requested a review from mkurapov August 20, 2024 07:54

mkurapov approved these changes Aug 20, 2024

View reviewed changes

njlie merged commit 5496612 into main Aug 20, 2024
42 checks passed

njlie deleted the nl/2810/initial-load-test-2 branch August 20, 2024 08:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tests): initial performance test #2828

feat(tests): initial performance test #2828

njlie commented Jul 30, 2024 •

edited

Loading

netlify bot commented Jul 30, 2024 •

edited

Loading

BlairCurrey Aug 2, 2024

njlie Aug 5, 2024

BlairCurrey Aug 5, 2024

BlairCurrey Aug 5, 2024

njlie Aug 5, 2024 •

edited

Loading

BlairCurrey Aug 6, 2024

BlairCurrey Aug 5, 2024

njlie Aug 5, 2024 •

edited

Loading

BlairCurrey Aug 6, 2024 •

edited

Loading

BlairCurrey Aug 6, 2024

njlie Aug 6, 2024

koekiebox left a comment

mkurapov Aug 9, 2024

mkurapov Aug 9, 2024

mkurapov Aug 19, 2024

mkurapov Aug 19, 2024

njlie Aug 19, 2024

mkurapov Aug 19, 2024

mkurapov Aug 19, 2024

njlie Aug 19, 2024

mkurapov left a comment


		- [Running local playground for Rafiki](../../localenv/README.md)

		If the local environment isn't running it may be started

feat(tests): initial performance test #2828

feat(tests): initial performance test #2828

Conversation

njlie commented Jul 30, 2024 • edited Loading

Changes proposed in this pull request

Context

Checklist

netlify bot commented Jul 30, 2024 • edited Loading

✅ Deploy Preview for brilliant-pasca-3e80ec canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

njlie Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

njlie Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

BlairCurrey Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koekiebox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkurapov left a comment

Choose a reason for hiding this comment

njlie commented Jul 30, 2024 •

edited

Loading

netlify bot commented Jul 30, 2024 •

edited

Loading

njlie Aug 5, 2024 •

edited

Loading

njlie Aug 5, 2024 •

edited

Loading

BlairCurrey Aug 6, 2024 •

edited

Loading