Add test workflow #60

mob-sakai · 2021-01-18T12:44:35Z

Changes

Add test workflow on the latest patch versions for each minor version.
- Run unity-changeset list --versions --latest-patch --json --min 2018.3 to get the latest patch versions for each minor version.
  - The minimum version to test is controlled by --min <version> option.

2018.3 -> 2018.3.1.14f1
2018.4 -> 2018.4.30f1
...
2020.1 -> 2020.1.17f1
2020.2 -> 2020.2.1f1

You can add version-module combinations to be excluded.
- Example: 7e8a801#diff-faff1af3d8ff408964a57b2e475f69a6b7c7b71c9978cccc8f471798caac2c88R83-R84

# Exclude android for Unity 2019.2 or earlier
EXCLUDES="$EXCLUDES `echo \"$VERSIONS\" | jq -c '[ .[] | select(test(\"2018|2019.1|2019.2\")) | { version: ., module: \"android\"} ]'`"

# [{"version":"2019.2.21f1","module":"android"}, ..., {"version":"2018.3.14f1","module":"android"}]

Test using the GitHub Actions game-ci/unity-builder@main and game-ci/unity-test-runner@main.
- Build and test a (almost default) Unity project.
- The Unity project support Unity 2018.3 or later.
If push or PR changes any Dockerfile (or reference-project-test), it will automatically test.
- Example: https://github.com/mob-sakai/docker/actions/runs/493271756
  - Tests fail with 2019.1.x, webgl, linux-il2cpp as reported in other issues
- Alternatively, you can run it manually. (workflow_dispatch)
- This result shows the version-module combinations that should be fixed (and those that are successful).

Checklist

Read the contribution guide and accept the code of conduct
Readme (updated or not needed)

NOTE

This workflow requires a secret UNITY_LICENSE that contains of an Unity license file (*.ulf).
This pull request does not provide a "complete" test, but a "simple" test. You can add tests to the test depending on the version.

webbertakken · 2021-01-19T00:24:27Z

This is a great piece of work!

The workflow looks a lot better and more nuanced than the initial one that I created and I think it can act as a great replacement!
I will need a bit more time to review it.

Please note that we're also in the process of moving reference projects to the unity-actions repository. This might mean that we will be using dispatch methods to run tests between repositories.

I also wanted to say great job on https://github.com/mob-sakai/unity-changeset

Get back to you soon.

webbertakken · 2021-01-19T00:28:48Z

.github/workflows/test.yml

+on:
+  workflow_dispatch:
+  push:
+    paths:
+      - "**/Dockerfile"
+      - .github/workflows/test.yml
+      - reference-project-test/*
+  pull_request:
+    types:
+      - "opened"
+      - "synchronize"
+    paths:
+      - "**/Dockerfile"
+      - .github/workflows/test.yml
+      - reference-project-test/*


What's the rationale about becoming so specific about the triggers?

For example; why not simply trigger on all default PR events?

on: pull_request: {}

What's the rationale about becoming so specific about the triggers?

This is to prevent unnecessary testing due to changes in files that are not related to building the image (e.g. changing only Readme.md).
The test (48 jobs) takes about 1 h.
No problem?

I'd say lets keep the paths, but remove the specific types. What do you think?

workflow_dispatch and push may be redundant.
(They support the following)

Trigger from other workflows or manually

When a change is pushed directly

I'd say lets keep the paths, but remove the specific types. What do you think?

OK, I'll remove it.

I think we should keep push and workflow_dispatch.
These will work in the forked repository, so we can test before creating a PR.

webbertakken · 2021-01-19T00:45:16Z

.github/workflows/test.yml

+          # For 'linux-il2cpp' module, switch the script backend to 'IL2CPP'
+          if [ "${{ matrix.module }}" = "linux-il2cpp" ] ; then
+            mv -f reference-project-test/ProjectSettings/ProjectSettings_il2cpp.asset reference-project-test/ProjectSettings/ProjectSettings.asset
+          fi


Note that altering the project will make the branch "dirty", meaning that builder will reject it unless "allowDirty" is passed. Maybe for testing purposes that would be ok.

Compared the changes between the il2cpp vs default project settings - note to self:

@GabLeRoux this might be a nice way to reduce the amount of test projects; simply swap some settings only.

@mob-sakai do you think unity-builder should differentiate between these target platforms? (And would it remove the need for this custom setup part?

- module: base platform: StandaloneLinux64 - module: linux-il2cpp platform: StandaloneLinux64

@webbertakken
In unity-builder and unity-test-runner, should we distinguish between the "StandaloneLinux64-Mono" platform and the "StandaloneLinux64-IL2CPP" platform?
Or do you mean to support the base and linux-il2cpp platforms in unity-builder and unity-test-runner?

No, the way we implemented it now is that StandaloneLinux64 (which doesn't have il2cpp until 2019.3) defaults to base and uses il2cpp from 2019.3 upwards.

Base is just a fallback because il2cpp doesn't exist, and we assumed that il2cpp would be the default/desired target platform for linux (and if not, that non-il2cpp would also be possible to build with il2cpp image).

I'm not sure what's best here. Suggestions?

Ah, I understand what you mean.

My Suggestion:

- run: | # For 2019.3 or later, non-il2cpp would also be possible to build with il2cpp image. if [ `echo "${{ matrix.version }}" | grep -v '$2018|2019.1|2019.2$'` ] && [ "${{ matrix.module }}" = 'base' ] ; then echo "MODULE=linux-il2cpp" >> $GITHUB_ENV else echo "MODULE=${{ matrix.module }}" >> $GITHUB_ENV fi ########################### # Build editor # ########################### - name: Build # if: steps.build-1.outcome == 'failure' uses: docker/build-push-action@v2 id: build-1 continue-on-error: true timeout-minutes: 40 with: context: . file: editor/Dockerfile build-args: | baseImage=base:dev hubImage=hub:dev version=${{ matrix.version }} changeSet=${{ env.CHANGESET }} module=${{ env.MODULE }} tags: editor:dev

BTW, unity-test-runner seems to use unityci/editor:<unity-version>-base-0 for testing.
https://github.com/game-ci/unity-test-runner/blob/main/src/model/image-tag.js

Yea I like your suggestion, lets go for that!

BTW, unity-test-runner seems to use unityci/editor:-base-0 for testing.

Yes, this is simply a result of our iterative approach. It should be able to test for all platforms to test platform-specific code paths. There's this issue that talks about custom image, but ideally we should also be able to use unity-test-runner in the same way as unity-builder, and specify targetPlatform.

… with il2cpp image

webbertakken · 2021-01-20T13:57:04Z

Great work!

mob-sakai · 2021-01-20T14:07:06Z

@webbertakken
Thanks!
Please add a secret UNITY_LICENSE that contains of an Unity license file (*.ulf).

webbertakken · 2021-01-20T17:23:06Z

Added and reran the workflow, seems like there's still a discrepancy in activation for some versions, but the majority seems to work perfectly! Thanks a lot for your contribution.

mob-sakai · 2021-01-20T18:02:59Z

@webbertakken
Thanks!
It seems that the machine IDs in the license file do not match.
You will probably need to generate a new alf file and activate it again.
(The same *.ulf is available for all Unity versions)

###########################
#    Building platform    #
###########################

Initiating legacy licensing module

LICENSE SYSTEM [2021120 14:33:52] Next license update check is after 2020-12-05T13:51:26


LICENSE SYSTEM [2021120 14:33:52] 576562626572264761624c65526f7578 != 00330-80181-13778-AA212


LICENSE SYSTEM [2021120 14:33:52] 576562626572264761624c65526f7578 != 00523_58_119258_05.0

DisplayProgressbar: Unity license
[UnityConnectServicesConfig] config is NOT valid, switching to default
[0120/143353:ERROR:browser_main_loop.cc(161)] Running without the SUID sandbox! See https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment for more information on developing with the sandbox on.
[0120/143353:ERROR:sandbox_linux.cc(308)] InitializeSandbox() called with multiple threads in process gpu-process
Cancelling DisplayDialog: Failed to activate/update license Missing or bad username or password. Please try again using valid credentials or contact [email protected]
This should not be called in batch mode.

mob-sakai · 2021-01-20T18:19:45Z

The test appears to be successful, but the license activation fails.
As you mention in #64, it is not treated as a failure.

webbertakken · 2021-01-20T19:43:14Z

Right my bad, I thought that was the correct license I had lying around in my downloads folder.

Updated and running again.

mob-sakai added 2 commits January 18, 2021 16:09

test: add test project (for Unity 2018.x or later)

1b6f5c1

test: add test on the latest Unity patch versions for push and PR

b63e969

This was referenced Jan 18, 2021

fix: add extra dependencies for webgl and il2cpp #61

Merged

fix: add extra dependencies for 2019.1 #59

Merged

webbertakken self-requested a review January 19, 2021 00:26

webbertakken reviewed Jan 19, 2021

View reviewed changes

This was referenced Jan 19, 2021

Could not find valid clang executable at clang (2019.3.x-linux-il2cpp) #62

Closed

fix: add IL2CPP_ADDITIONAL_ARGS for 2019.3.x-linux-il2cpp #63

Merged

mob-sakai added 2 commits January 20, 2021 10:40

test: remove the specific types for PR

ed9b10f

test: for 2019.3 or later, non-il2cpp would also be possible to build…

33f7a9f

… with il2cpp image

webbertakken merged commit 387eb7c into game-ci:main Jan 20, 2021

mob-sakai deleted the test/workflow branch January 20, 2021 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test workflow #60

Add test workflow #60

mob-sakai commented Jan 18, 2021 •

edited

Loading

webbertakken commented Jan 19, 2021

webbertakken Jan 19, 2021

mob-sakai Jan 19, 2021

webbertakken Jan 19, 2021

mob-sakai Jan 19, 2021

mob-sakai Jan 19, 2021 •

edited

Loading

mob-sakai Jan 20, 2021

mob-sakai Jan 20, 2021 •

edited

Loading

webbertakken Jan 19, 2021 •

edited

Loading

mob-sakai Jan 19, 2021

webbertakken Jan 19, 2021

mob-sakai Jan 20, 2021

mob-sakai Jan 20, 2021

mob-sakai Jan 20, 2021

webbertakken Jan 20, 2021

mob-sakai Jan 20, 2021

webbertakken commented Jan 20, 2021

mob-sakai commented Jan 20, 2021

webbertakken commented Jan 20, 2021 •

edited

Loading

mob-sakai commented Jan 20, 2021

mob-sakai commented Jan 20, 2021

webbertakken commented Jan 20, 2021

Add test workflow #60

Add test workflow #60

Conversation

mob-sakai commented Jan 18, 2021 • edited Loading

Changes

Checklist

NOTE

webbertakken commented Jan 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mob-sakai Jan 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mob-sakai Jan 20, 2021 • edited Loading

Choose a reason for hiding this comment

webbertakken Jan 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webbertakken commented Jan 20, 2021

mob-sakai commented Jan 20, 2021

webbertakken commented Jan 20, 2021 • edited Loading

mob-sakai commented Jan 20, 2021

mob-sakai commented Jan 20, 2021

webbertakken commented Jan 20, 2021

mob-sakai commented Jan 18, 2021 •

edited

Loading

mob-sakai Jan 19, 2021 •

edited

Loading

mob-sakai Jan 20, 2021 •

edited

Loading

webbertakken Jan 19, 2021 •

edited

Loading

webbertakken commented Jan 20, 2021 •

edited

Loading