Workflow for Clean Test Vectors Database

The test vectors can be accessed here.

Currently, three set of test vectors are hosted on the database:

• Positive vectors: the test vectors in this category pass the conformance test
• Negative vectors: the test vectors in this category fail the conformance test. Please note that currently, database does not include the reason why the test vector fails. This should be included.
• Vectors that have issues: the test vectors in this category either fails the conformance test when it is supposed to pass (positive) or passes the conformance test when it is supposed to fail (negative).

Visibility of test vectors:

In data base:

• Only the first two set of vectors should be visible.
• The third category should be “hidden/flagged”. There should be an open GitHub issue here. Owner should assigned to the vector (preferably by github/e-mail account) that is supposed to resolve issue.

To upload a new positive test vector:

• NOTE that this is not in place yet, but under development.
• The test vector is checked by latest conformance software.
  • If passes, it is a candidate to be published. If not, it is “rejected”.
• Manual checking if it fulfills other requirements is done.
  • If it does, it is added to the database and published
• One or several contact persons is added to the data base.

After updates of conformance software:

• Regression test on the test vectors before conformance software is merged into main branch
  • NOTE: (Under development) Conformance software can work with authentication to login to the test vectors database, does regression test on these test vectors and provide "PASS" or "FAIL" status.
  • If the result does not change compared to the previous version of the conformance software, no action is needed.
  • If the result changes, conformance software developer is informed. In this case:
    • The developer checks if there is a problem in conformance software. If yes, the problem gets fixed and the regression testing is done from start. If not, the test vector is taken down and owner is informed to resolve.

Assign features to test vectors to support dash.js testing:

• NOTE that this is not in place yet.
• NOTE the links for dash.js repository and dash.js players.
• NOTE that dash.js has a list of small test vectors to be run every time. This subset should be small, but can be extended over time.
• Feature description should link to IOP.
  • NOTE: Needs revisiting for v5.
• There should be a feature/flag in data base for a smoke test.
• It would be good if the test vector is also including a flag if tested and runs on dash.js (and potentially a browser).