Add regenerate data files tests #7866

yuhaoth · 2023-07-03T09:16:04Z

Description

This PR try to guarantee tests/data_files are generated and there is no missed or un-tracked files in final output.

At this moment, there are still some wrong commands or no generate commands available. Those files are list with uncategorized.lst. And this PR will ignore those files.

Some commands are fixed in this file due to it can not be covered by uncategorized.lst.

#7649 and #7662 fix some wrong commands.

find_missed.py is for finding missed commands
test_new_data_files.sh is for local test.

gilles-peskine-arm · 2023-07-06T08:46:07Z

Context: the directory tests/data_files contains many keys, certificates and related files (CSR, CRL, …) that are used by tests. In principle, tests/data_files/Makefile contains instructions to re-create these files. We keep these files in the repository for several reasons:

Historically, Makefile didn't exist, we just had the files themselves.
Many files contain embedded randomness or depend on the generation time, so we don't get identical files if you re-generate them.
The generation depends on several tools and on specific versions. For example I'm not sure if re-generating all the files works with a single openssl version.

The goal of this pull request is to validate that the instructions in the Makefile are correct. At the moment, nothing in the CI uses the Makefile.

tests/scripts/all.sh

gilles-peskine-arm · 2023-07-10T18:19:10Z

tests/scripts/regenerate_data_files_tests.sh

+component_test_regenerate_data_files_default () {
+    helper_regenerate_data_files neat
+    scripts/config.py full
+    helper_datafile_run_tests "default configuration with refreshed data files"


I don't think this is a good approach for checking that the new files are good. We're checking that the tests pass, but that doesn't mean, but that doesn't mean that the tests do the same thing.

I would prefer a more reliable approach: compare the new files with the old files, using a comparison method that ignores some parts of the files (content of keys, signatures and hashes), comparing only the ASN.1 structure, the metadata that isn't derived from random data (names, serial numbers, etc.), and whether signatures are correct. It will take us some time to do it, but I think the benefits are worth it.

Concretely, here's the general structure that I'm thinking of for the regeneration test.

Understood. I will remove them later.

And I'd like reserve helper_datafile_run_tests as local test utility. I will move it to another scripts.

For non-ASN.1 files and malformed files, how to compare them ? Do you have any idea?

For files without a valid ASN.1 structure, I don't have a plan to compare. Hopefully we won't need to regenerate those often.

I post those tests in this PR and reduce the scope of this PR.

About the compare script, I'd like write it in next PR. I think the script should only compare modified files. I have add TODO tag in regenerate_data_files_tests.sh . Please check it.

I limit openssl version to 3.0.2. I think it can generate all files. And I will fix miss or wrong commands in #7888.

tests/data_files/README.md

Still reviewing, accidentally pushed the button

tgonzalezorlandoarm · 2023-08-11T13:17:42Z

I would need some more explanation:

If "The goal of this pull request is to validate that the instructions in the Makefile are correct", how are we doing this on this PR?

yuhaoth · 2023-08-21T09:28:28Z

rebase to resolve conflicts

The components will remove all files, regenerate and check if there are files that is missed or not committed. In future, we should check modified files are expected. Signed-off-by: Jerry Yu <[email protected]>

And set openssl to 3.0.2 Signed-off-by: Jerry Yu <[email protected]>

- change neat target also. Signed-off-by: Jerry Yu <[email protected]>

Signed-off-by: Jerry Yu <[email protected]>

This is for regenerate all parse input files Signed-off-by: Jerry Yu <[email protected]>

Signed-off-by: Jerry Yu <[email protected]>

Those files are not commited and list in `all_intermediate` and remove invalid final file Signed-off-by: Jerry Yu <[email protected]>

Signed-off-by: Jerry Yu <[email protected]>

The file is used by x509_cert_ino and verified with expected output Signed-off-by: Jerry Yu <[email protected]>

- reserved list includes files that used as command input - uncategoried includes files that commands fail or not covered and should be resolved in future Signed-off-by: Jerry Yu <[email protected]>

Signed-off-by: Jerry Yu <[email protected]>

yuhaoth linked an issue Jul 3, 2023 that may be closed by this pull request

Add faketime test script #7729

Open

yuhaoth added the needs-preceding-pr Requires another PR to be merged first label Jul 3, 2023

yuhaoth marked this pull request as draft July 3, 2023 09:45

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch 5 times, most recently from d745ebe to b16e5ba Compare July 6, 2023 08:39

yuhaoth added needs-work and removed needs-preceding-pr Requires another PR to be merged first labels Jul 6, 2023

tgonzalezorlandoarm self-assigned this Jul 6, 2023

yuhaoth mentioned this pull request Jul 6, 2023

Add more missing commands for data files #7888

Draft

3 tasks

tgonzalezorlandoarm self-requested a review July 6, 2023 13:29

tgonzalezorlandoarm removed their assignment Jul 6, 2023

tgonzalezorlandoarm requested changes Jul 7, 2023

View reviewed changes

tests/scripts/all.sh Outdated Show resolved Hide resolved

yuhaoth mentioned this pull request Jul 10, 2023

Add xxd for data file regenerating tests Mbed-TLS/mbedtls-test#112

Merged

gilles-peskine-arm reviewed Jul 10, 2023

View reviewed changes

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch 3 times, most recently from 7893044 to c613aa3 Compare July 12, 2023 03:21

yuhaoth mentioned this pull request Jul 12, 2023

Add hexdump command Mbed-TLS/mbedtls-test#113

Merged

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch 4 times, most recently from f790c75 to c4d8823 Compare July 13, 2023 07:43

yuhaoth marked this pull request as ready for review July 13, 2023 10:02

yuhaoth added component-test Test framework and CI scripts needs-review Every commit must be reviewed by at least two team members, and removed needs-work labels Jul 13, 2023

tgonzalezorlandoarm reviewed Aug 11, 2023

View reviewed changes

tests/data_files/README.md Outdated Show resolved Hide resolved

tgonzalezorlandoarm previously approved these changes Aug 11, 2023

View reviewed changes

tgonzalezorlandoarm previously approved these changes Aug 16, 2023

View reviewed changes

yuhaoth dismissed tgonzalezorlandoarm’s stale review via 2cc372f August 21, 2023 09:28

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch from 46fdd7b to 2cc372f Compare August 21, 2023 09:28

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch from 2cc372f to ea0827a Compare August 22, 2023 02:11

yuhaoth added 20 commits November 16, 2023 18:14

Add regenerate data files test.

8061add

The components will remove all files, regenerate and check if there are files that is missed or not committed. In future, we should check modified files are expected. Signed-off-by: Jerry Yu <[email protected]>

Add tool chain check

7d2fcdf

And set openssl to 3.0.2 Signed-off-by: Jerry Yu <[email protected]>

add tidy target to remove all files except reserverd files

7e718d6

- change neat target also. Signed-off-by: Jerry Yu <[email protected]>

remove intermediate files from git repo

3bc7c04

Signed-off-by: Jerry Yu <[email protected]>

add parse input variable and target

d3b39e1

This is for regenerate all parse input files Signed-off-by: Jerry Yu <[email protected]>

Add parse input files into all_parse_input

6b86534

Signed-off-by: Jerry Yu <[email protected]>

move files to intermediate

0161da6

Those files are not commited and list in `all_intermediate` and remove invalid final file Signed-off-by: Jerry Yu <[email protected]>

disable parallel build for data files

8daf6b4

Signed-off-by: Jerry Yu <[email protected]>

Keep consistent with other command

42b01b2

Signed-off-by: Jerry Yu <[email protected]>

copy server5-tricky-ip-san.crt.der

d030aee

The file is used by x509_cert_ino and verified with expected output Signed-off-by: Jerry Yu <[email protected]>

Update reserved/uncategoried list

ffa675d

- reserved list includes files that used as command input - uncategoried includes files that commands fail or not covered and should be resolved in future Signed-off-by: Jerry Yu <[email protected]>

Add readme

4e82220

Signed-off-by: Jerry Yu <[email protected]>

fix typo and grammar issues

04c759e

Signed-off-by: Jerry Yu <[email protected]>

improve command document

31916d7

Signed-off-by: Jerry Yu <[email protected]>

fix typo issues

d2edf49

Signed-off-by: Jerry Yu <[email protected]>

adjust position of variable assign

a7d7af0

Signed-off-by: Jerry Yu <[email protected]>

Add more uncategorized files

2836c1a

Signed-off-by: Jerry Yu <[email protected]>

Describe how we checked the output files

d9bd547

Signed-off-by: Jerry Yu <[email protected]>

resolve conflict with base branch

005f346

Signed-off-by: Jerry Yu <[email protected]>

reserver server1 config file

52ff195

Signed-off-by: Jerry Yu <[email protected]>

yuhaoth force-pushed the pr/add-regenerate-data-files-tests branch from a8feae9 to 52ff195 Compare November 16, 2023 10:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add regenerate data files tests #7866

Add regenerate data files tests #7866

yuhaoth commented Jul 3, 2023 •

edited

Loading

gilles-peskine-arm commented Jul 6, 2023

gilles-peskine-arm Jul 10, 2023

gilles-peskine-arm Jul 10, 2023

yuhaoth Jul 11, 2023

gilles-peskine-arm Jul 11, 2023

yuhaoth Jul 13, 2023

tgonzalezorlandoarm commented Aug 11, 2023

yuhaoth commented Aug 21, 2023

Add regenerate data files tests #7866

Are you sure you want to change the base?

Add regenerate data files tests #7866

Conversation

yuhaoth commented Jul 3, 2023 • edited Loading

Description

gilles-peskine-arm commented Jul 6, 2023

gilles-peskine-arm Jul 10, 2023

Choose a reason for hiding this comment

gilles-peskine-arm Jul 10, 2023

Choose a reason for hiding this comment

yuhaoth Jul 11, 2023

Choose a reason for hiding this comment

gilles-peskine-arm Jul 11, 2023

Choose a reason for hiding this comment

yuhaoth Jul 13, 2023

Choose a reason for hiding this comment

tgonzalezorlandoarm commented Aug 11, 2023

yuhaoth commented Aug 21, 2023

yuhaoth commented Jul 3, 2023 •

edited

Loading