Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit Tests compareResults numeric ID based ground truth values are fragile #328

Open
blipper opened this issue Apr 18, 2021 · 2 comments
Open
Assignees
Labels
extensive Bigger piece of work

Comments

@blipper
Copy link
Contributor

blipper commented Apr 18, 2021

Is your feature request related to a problem? Please describe.

A typical test pattern in unittets is
TEST_F(IFDSTaintAnalysisTest, TaintTest_04) {
initialize({PathToLlFiles + "dummy_source_sink/taint_04_cpp_dbg.ll"});
IFDSSolver_P TaintSolver(*TaintProblem);
TaintSolver.solve();
map<int, set> GroundTruth;
GroundTruth[19] = set{"18"};
GroundTruth[24] = set{"23"};
compareResults(GroundTruth);
}

Specifying individual IDs is very fragile since it depends on how clang compiles and a single change can break everything.

Describe the solution you'd like
A more flexible/less fragile pattern

e.g. specify the instruction pattern (with filters/qulifiers)
%kIVSize = alloca i32, align 4, !psr.id !35 | ID: 7

Something like
IDLookup.Builder.IRpattern("%kIVSize = alloca i32, align 4").inFunction("main").afterIRpattern("%p = alloca %struct.Params").build().getID();

@MMory MMory added the extensive Bigger piece of work label Apr 26, 2021
@MMory
Copy link
Member

MMory commented Apr 26, 2021

I agree that this is fragile. So far we have not had problems with that, which I don't mean as an excuse not to do it right.

@blipper
Copy link
Contributor Author

blipper commented Jul 26, 2021

One example is the differences than result between stdlibc++ and libcxx

@pdschubert pdschubert assigned meret-boe and unassigned pdschubert Oct 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extensive Bigger piece of work
Projects
None yet
Development

No branches or pull requests

4 participants