-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add check to run_mtl_loop that archived directories match the tiles-specstatus file #808
Conversation
Pinging @araichoor as a reminder. This still probably still isn't urgent yet, but it would be nice to merge this this week. |
sorry that fell off my radar. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, so here are minor comments/suggestions, which you can ignore if you think it s better to do so.
…e have to be main/DARK or main/BRIGHT/
Thanks for your careful review @araichoor. I addressed all of your comments. I went in the direction you suggested of splitting out the archiving check by I've left one request for an opinion from @schlafly as part of my response to the final comment of your review. Otherwise I think this PR should be ready for merging. I rechecked the code using the same test as detailed at the top of this thread. |
thanks for implementing my suggestions! I just thought of a possible minor drawback of making the check on {survey,obscon} only, but I think there s nothing to change in the code. I thought about the following:
(and we could put that instruction in the dailyops wiki). |
I think "dark" and "bright" being "out-of-sync" is fine, though, right? As far as I know we treat "bright" and "dark" as completely separate surveys? So, although we've never just run Sometimes we run an MTL update where there are, e.g., some new dark tiles but no new bright tiles and the Another way of looking at this (which is why I liked your suggestion) is that we might want to proceed with an MTL update for dark tiles even if there's an archiving issue for bright tiles. Am I missing something that intertwines the bright and dark parts of the survey? |
'I think "dark" and "bright" being "out-of-sync" is fine, though, right?' => yes, sure, no big deal actually 'Am I missing something that intertwines the bright and dark parts of the survey?' => no you re right! I was just slightly worried that we end up in a case where sorry for the noise! |
This PR adds a
check_archiving()
function that is used inrun_mtl_loop
to test whether archived tile directories match theARCHIVEDATE
column in thetiles-spectstatus.ecsv
file. The check is restricted to bright and dark tiles from the Main Survey.See #807 and desihub/desisurveyops#141 for more context.
run_mtl_loop
now also includes a--noarchivecheck
flag that can be used to bypass the archiving check (e.g., to speed up simulations or the alt-MTL process).I tested this pretty extensively by:
archive
directories in/pscratch/sd/a/adamyers/arxivsandbox/tiles/archive/
.tiles-specstatus.ecsv
file to/pscratch/sd/a/adamyers/arxivsandbox/ops/
.20994
and25442
to/pscratch/sd/a/adamyers/arxivsandbox/tiles/original/
.3333
from/pscratch/sd/a/adamyers/arxivsandbox/ops/tiles-specstatus.ecsv
(the original file is retained at/pscratch/sd/a/adamyers/arxivsandbox/ops/tiles-specstatus.ecsv.orig
).run_mtl_loop DARK --zcatdir /pscratch/sd/a/adamyers/arxivsandbox/ --mtldir /pscratch/sd/a/adamyers/arxivsandbox/mtl
The resulting error message was:
@araichoor: I'd appreciate a second pair of eyes on this PR before I tag
desitarget
and we make the check a standard part of daily ops.I don't think this is critical to resolve on the timescale of the long weekend, though!