Transition audio test to mocha testing framework #2264

nainkunal933 · 2022-06-01T01:20:46Z

Issue #:

Description of changes:
JS SDK currently uses KITE as its test engine for integration / end to end tests. KITE includes a lot of overhead to tests and a lot of KITE features are not currently used. Unit tests use mocha as their testing framework and it is much easier to use.

JS SDK uses a very old version of KITE and finding documentation on KITE issues has been a pain. This PR will start the transition of integration tests from KITE to mocha. A new folder named mochaTests will house these tests. This PR will

transition audio integration test to mocha,
add support for DeviceFarm as a grid provider along with SauceLabs, and
setup the infrastructure for future test transition.

Testing:

Can these tested using a demo application? Please provide reproducible step-by-step instructions.

No, these are integration test changes

Checklist:

Have you successfully run npm run build:release locally?
Yes
Do you add, modify, or delete public API definitions? If yes, has that been reviewed and approved?
NA
Do you change the wire protocol, e.g. the request method? If yes, has that been reviewed and approved?
NA

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

integration/mochaTests/configs/BaseConfig.js

simmkyu

Nice detailed doc about mocha tests! I added comments mostly about syntaxes and layout.

integration/mochaTests/README.md

ltrung · 2022-06-06T23:23:38Z

integration/mocha-tests/README.md

@@ -0,0 +1,111 @@
+# Mocha Tests
+
+The Amazon Chime SDK team is transitioning integration tests from KITE to Mocha. Starting with audio tests, we will transition all the integration tests. The `integration/mocha-tests` directory contains the mocha version of integration tests.


This seems like internal stuff. Not sure if we need to put it here.

ltrung · 2022-06-06T23:24:04Z

integration/mocha-tests/README.md

+The Amazon Chime SDK team is transitioning integration tests from KITE to Mocha. Starting with audio tests, we will transition all the integration tests. The `integration/mocha-tests` directory contains the mocha version of integration tests.
+
+## Test Types
+There are two types of Mocha tests: integration and browser compatibility.


Not correct. browser compatibility is integration tests.

ltrung · 2022-06-06T23:24:04Z

integration/mocha-tests/README.md

+The Amazon Chime SDK team is transitioning integration tests from KITE to Mocha. Starting with audio tests, we will transition all the integration tests. The `integration/mocha-tests` directory contains the mocha version of integration tests.
+
+## Test Types
+There are two types of Mocha tests: integration and browser compatibility.


Not correct. browser compatibility is integration tests.

ltrung · 2022-06-06T23:30:17Z

integration/mocha-tests/tsconfig.json

@@ -0,0 +1,22 @@
+{


can we reuse the same ts config from the main library https://github.com/aws/amazon-chime-sdk-js/tree/main/config

Actually there is no ts file then why do we need it ?

ltrung · 2022-06-06T23:32:31Z

integration/mocha-tests/package.json

+    "lint": "eslint --config eslintrc.json ./ --ext .ts,.tsx,.js --fix"
+  },
+  "author": "",
+  "license": "ISC",


Probably want to be Apache.

ltrung · 2022-06-06T23:32:31Z

integration/mocha-tests/package.json

+    "lint": "eslint --config eslintrc.json ./ --ext .ts,.tsx,.js --fix"
+  },
+  "author": "",
+  "license": "ISC",


Probably want to be Apache.

ltrung · 2022-06-06T23:32:46Z

integration/mocha-tests/package.json

+  "version": "1.0.0",
+  "description": "",
+  "scripts": {
+    "clean": "rm -rf temp && rm -rf ../kite-allure-reports && rm -rf logs",


why do we have kite-allure-reports here?

ltrung · 2022-06-06T23:32:46Z

integration/mocha-tests/package.json

+  "version": "1.0.0",
+  "description": "",
+  "scripts": {
+    "clean": "rm -rf temp && rm -rf ../kite-allure-reports && rm -rf logs",


why do we have kite-allure-reports here?

ltrung · 2022-06-06T23:37:23Z

integration/mocha-tests/README.md

+- `logger.log` outputs messages to the console in different colors based on the log level.
+
+Integration tests use `mocha` as their test framework. Mocha prints the test results and some limited logs to the console automatically. Any console logs inside `it` will print before the mocha logs.
+This makes debugging hard as the logs seem out of sync and random. To get around this issue, buffered logging was added. Inside of `it` blocks, `logger.pushLogs` can be used to add logs to the buffer and a call to `logger.printLogs` inside of the `afterEach` hook will print the logs in the desired order.


Then should this be the default then? Why do we want dev to make decision if one is clearly better to avoid issue?

ltrung · 2022-06-06T23:37:23Z

integration/mocha-tests/README.md

+- `logger.log` outputs messages to the console in different colors based on the log level.
+
+Integration tests use `mocha` as their test framework. Mocha prints the test results and some limited logs to the console automatically. Any console logs inside `it` will print before the mocha logs.
+This makes debugging hard as the logs seem out of sync and random. To get around this issue, buffered logging was added. Inside of `it` blocks, `logger.pushLogs` can be used to add logs to the buffer and a call to `logger.printLogs` inside of the `afterEach` hook will print the logs in the desired order.


Then should this be the default then? Why do we want dev to make decision if one is clearly better to avoid issue?

ltrung · 2022-06-06T23:50:24Z

integration/mocha-tests/utils/HelperFunctions.js

+const { spawnSync, spawn } = require('child_process');
+
+// Run the command asynchronously without blocking the Node.js event loop.
+function runAsync(command, args, options) {


Feel like this should not be in this test framework as it is just a script helper.

ltrung · 2022-06-06T23:51:17Z

integration/mocha-tests/tests/AudioTest.js

+ * 6. Same attendee mutes the audio
+ * 7. Checks if the other participant is not able to hear the audio
+ * */
+describe('AudioTest', async function () {


Does that mean we run all tests here single vs multiple sessions every time?

ltrung · 2022-06-06T23:51:17Z

integration/mocha-tests/tests/AudioTest.js

+ * 6. Same attendee mutes the audio
+ * 7. Checks if the other participant is not able to hear the audio
+ * */
+describe('AudioTest', async function () {


Does that mean we run all tests here single vs multiple sessions every time?

ltrung · 2022-06-07T00:19:19Z

integration/mocha-tests/tests/AudioTest.js

+      });
+
+      describe('setup', async function () {
+        it('should open the meeting demo in two tabs', async function () {


Should not have to do this and just have one single code flow for single vs multiple sessions. Maybe we can create a helper fucntion to map the window and page etc.

ltrung · 2022-06-07T00:19:19Z

integration/mocha-tests/tests/AudioTest.js

+      });
+
+      describe('setup', async function () {
+        it('should open the meeting demo in two tabs', async function () {


Should not have to do this and just have one single code flow for single vs multiple sessions. Maybe we can create a helper fucntion to map the window and page etc.

ltrung · 2022-06-07T00:19:43Z

integration/mocha-tests/README.md

+There are two types of Mocha tests: integration and browser compatibility.
+
+### Integration Tests
+Integration tests are minimal tests that run on the latest Chrome on macOS. These tests are used to test the basic functionality across the more popular browsers.


Need more details on the test infra and how to write a test.

ltrung · 2022-06-07T00:19:43Z

integration/mocha-tests/README.md

+There are two types of Mocha tests: integration and browser compatibility.
+
+### Integration Tests
+Integration tests are minimal tests that run on the latest Chrome on macOS. These tests are used to test the basic functionality across the more popular browsers.


Need more details on the test infra and how to write a test.

nainkunal933 requested a review from a team as a code owner June 1, 2022 01:20

nainkunal933 force-pushed the mocha-transition branch from 8eb9faf to 7847b51 Compare June 1, 2022 01:24

nainkunal933 self-assigned this Jun 1, 2022

LeviSklaroff reviewed Jun 1, 2022

View reviewed changes

integration/mochaTests/configs/BaseConfig.js Outdated Show resolved Hide resolved

simmkyu requested changes Jun 1, 2022

View reviewed changes

nainkunal933 force-pushed the mocha-transition branch 3 times, most recently from 67c2df8 to cfe6aca Compare June 2, 2022 20:31

Transition audio test to mocha testing framework

87caf78

nainkunal933 force-pushed the mocha-transition branch from cfe6aca to 87caf78 Compare June 2, 2022 20:33

ltrung reviewed Jun 6, 2022

View reviewed changes

ltrung reviewed Jun 7, 2022

View reviewed changes

Merge branch 'main' into mocha-transition

f7aede3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition audio test to mocha testing framework #2264

Transition audio test to mocha testing framework #2264

nainkunal933 commented Jun 1, 2022

simmkyu left a comment

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 6, 2022

ltrung Jun 7, 2022

ltrung Jun 7, 2022

ltrung Jun 7, 2022

ltrung Jun 7, 2022

		@@ -0,0 +1,111 @@
		# Mocha Tests

		The Amazon Chime SDK team is transitioning integration tests from KITE to Mocha. Starting with audio tests, we will transition all the integration tests. The `integration/mocha-tests` directory contains the mocha version of integration tests.

Transition audio test to mocha testing framework #2264

Are you sure you want to change the base?

Transition audio test to mocha testing framework #2264

Conversation

nainkunal933 commented Jun 1, 2022

simmkyu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment