Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use 1.0 for good AtomMapping scores, 0.0 for bad AtomMapping scores #529

Merged
merged 5 commits into from
Aug 28, 2023

Conversation

richardjgowers
Copy link
Contributor

@richardjgowers richardjgowers commented Aug 23, 2023

Previously 0.0 was used for "good" scores and 1.0 for "bad" scores, so networks could be created according to considering the total sum of all weights across a network as the cost function. This created friction with existing and ongoing efforts, where 1.0 is considered perfect "similarity" and 0.0 is bad/no similarity.

This PR flips scores around to align with community practice.

Developers certificate of origin

@richardjgowers richardjgowers changed the title The great score flip Use 1.0 for good AtomMapping scores, 0.0 for bad AtomMapping scores Aug 23, 2023
@codecov
Copy link

codecov bot commented Aug 24, 2023

Codecov Report

Patch coverage: 95.78% and project coverage change: -0.10% ⚠️

Comparison is base (e797d52) 92.07% compared to head (0fbf018) 91.97%.
Report is 23 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #529      +/-   ##
==========================================
- Coverage   92.07%   91.97%   -0.10%     
==========================================
  Files         113      116       +3     
  Lines        6938     7054     +116     
==========================================
+ Hits         6388     6488     +100     
- Misses        550      566      +16     
Files Changed Coverage Δ
openfe/protocols/openmm_rfe/_rfe_utils/relative.py 82.84% <ø> (-0.12%) ⬇️
openfe/protocols/openmm_rfe/equil_rfe_methods.py 94.02% <ø> (ø)
openfecli/commands/gather.py 85.25% <90.19%> (-6.79%) ⬇️
openfe/protocols/openmm_rfe/equil_rfe_settings.py 100.00% <100.00%> (ø)
openfe/setup/atom_mapping/lomap_scorers.py 100.00% <100.00%> (ø)
openfe/setup/ligand_network_planning.py 100.00% <100.00%> (ø)
openfe/tests/protocols/test_rfe_tokenization.py 97.14% <100.00%> (ø)
...nfe/tests/setup/atom_mapping/test_lomap_scorers.py 100.00% <100.00%> (ø)
openfe/tests/setup/test_network_planning.py 100.00% <100.00%> (ø)
openfe/tests/utils/test_system_probe.py 100.00% <100.00%> (ø)
... and 6 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@hannahbaumann
Copy link
Contributor

This looks good to me! Should we check whether networks generated with the flipped score match networks generated using the original score?

Copy link
Contributor

@Yoshanuikabundi Yoshanuikabundi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few nits with docstring style and phrasing.

docs/cookbook/creating_atom_mappings.rst Outdated Show resolved Hide resolved
"""Maximum command substructure rule

This rule was originally defined as::
This rule is defined as::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

openfe/setup/atom_mapping/lomap_scorers.py Outdated Show resolved Hide resolved
openfe/setup/atom_mapping/lomap_scorers.py Outdated Show resolved Hide resolved
openfe/setup/ligand_network_planning.py Outdated Show resolved Hide resolved
openfe/setup/atom_mapping/lomap_scorers.py Outdated Show resolved Hide resolved
Copy link
Member

@IAlibay IAlibay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we check whether networks generated with the flipped score match networks generated using the original score?

Is test_minimal_spanning_network_regression sufficient here? Ottherwise I agree, we probably should consider a more in-depth check.

@richardjgowers
Copy link
Contributor Author

I'll check with Ben once he's back, but I think the perses scorer already was in this direction, so that doesn't require flipping

@richardjgowers
Copy link
Contributor Author

@IAlibay this is the regression test here: https://github.com/OpenFreeEnergy/openfe/blob/main/openfe/tests/setup/test_network_planning.py#L198

It looks like a non-trivial (it's not just radial, it's got plenty of edges) network, so I doubt we'd be getting it through luck if we hadn't correctly done this change

@IAlibay
Copy link
Member

IAlibay commented Aug 28, 2023

@richardjgowers am I good to merge here? Would be good to unblock this for @hannahbaumann to test out the newly generated MSTs.

@richardjgowers richardjgowers merged commit d2129d5 into main Aug 28, 2023
6 of 7 checks passed
@richardjgowers richardjgowers deleted the the_great_score_flip branch August 28, 2023 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants