Add ability to run tests in sequential order

[billychow68]

While running the tests in parallel speeds up the execution overall, there are times when executing tests in sequential order is needed. Currently, I'm managing this in a shell script calling "Commader YAML TEST" individually.

I could be wrong but I believe I read somewhere that the --dir will run the test files (YAML) in sequential order? This request is specifically for running the tests sequentially within the YAML.

[dylanhitt]

Hey @billychow68 you are correct in that --dir does execute test against files in sequential order. Would you be looking for something similar to what is described in issue #96?

[billychow68]

Hi @dylanhitt. I believe a new depends property would do the job as long as I could chain together 2 or more tests. Would that be possible?

[SimonBaeumer]

Hey @dylanhitt and @billychow68 :)

At the moment the tests are executed sequentially.
The problem is that the golang yaml parser does not keep the order in which the tests were defined inside the yaml file.

The depends feature is a "little" bit tricky in some cases:
circular: testA depends on testB depends on testA
double: testC depends on testB and testC depends on testA
It is an interesting challenge to implement but would explode in complexity inside a simple tool like commander.

A simpler solution would be to sort the tests alphabetically, then you can define tests with numbers as prefixes like 001 - test.
Further an option --exit-on-error or -e could stop the test execution if an error occurs like it would in a depends property.

Any thoughts on this?

[dylanhitt]

@SimonBaeumer I agree that depends on isn't the best implementation as we are depending on the user a little too much. I do like the idea of prefixing with numbers. Two questions:

1. Should we allow duplicate sequential values? I don't see a reason not too, it appears it wouldn't increase complexity.
2. What are your thought on a separate field for order or some similar name. Example:

it should print hello world:
  order: 1
  command: echo hello world
  stdout: hello world
  exit-code: 0

it should validate exit code:
  order: 2
  command: exit 1
  exit-code: 1

another test:
  order: 2
  command: exit 1
  exit-code: 1

This is all mainly cosmetic and I'm not attached to either.

[billychow68]

A simpler solution would be to sort the tests alphabetically, then you can define tests with numbers as prefixes like 001 - test.

I like this approach as it's natural to include the prefix (like a test case ID) along with the title of the test case.

Also would --concurrency affect how tests will run sequentially or would it need to be set to say a value of 1? Or no affect whatsoever?

[SimonBaeumer]

--concurrency configured the amount of tests being executed concurrently, I removed it in newer versions because it caused too much problems.
The execution order can't be guaranteed if it is concurrent.

@dylanhitt @billychow68
Do you want it to implement? Otherwise I would go for it.
It should just be an alphabetically ordering after parsing the test suite.

We can discuss a concurrent execution in another ticket, created an issue where it can be discussed #130

[dylanhitt]

@SimonBaeumer you got it. I believe I have enough on my plate.

[SimonBaeumer]

Tests are now executed in alphabetical order, will be released in v2.3.0. Fixed in #142