DM-43077: Convert all tasks to use CalibrateImageTask outputs #157
Conversation
parejkoj
force-pushed
the
tickets/DM-43077
branch
2 times, most recently
from
544a9fa to
cbc6df9
Compare
parejkoj
force-pushed
the
tickets/DM-43077
branch
from
cbc6df9 to
dc10c06
Compare
| - transformSourceTable | ||
| - consolidateSourceTable | ||
| - updateVisitSummary | ||
| description: | | ||
| Visit-level tasks. | ||
| Allowed data query constraints: visit | ||
|
|
||
| writeRecalibratedSourceTable, transformSourceTable run per-detector | ||
| transformSourceTable run per-detector |
There was a problem hiding this comment.
| transformSourceTable run per-detector | |
| transformSourceTable runs per-detector |
| description: | | ||
| Visit-level tasks. | ||
| Allowed data query constraints: visit | ||
|
|
||
| writeRecalibratedSourceTable, transformSourceTable run per-detector | ||
| transformSourceTable run per-detector |
There was a problem hiding this comment.
| transformSourceTable run per-detector | |
| transformSourceTable runs per-detector |
There was a problem hiding this comment.
or include reprocessVisitImage.
| description: | | ||
| Visit-level tasks. | ||
| Allowed data query constraints: visit | ||
|
|
||
| writeRecalibratedSourceTable, transformSourceTable run per-detector | ||
| transformSourceTable runs per-detector |
There was a problem hiding this comment.
Is there a reason why reprocessVisitImage is not included here like many others below?
| @@ -52,15 +50,14 @@ subsets: | |||
| step2d: | |||
| subset: | |||
| - finalizeCharacterization | |||
| - writeRecalibratedSourceTable | |||
There was a problem hiding this comment.
Is reprocessVisitImage not included here on purpose?
| class: lsst.pipe.tasks.characterizeImage.CharacterizeImageTask | ||
| config: | ||
| doApCorr: false | ||
| doDeblend: false |
There was a problem hiding this comment.
Could you comment briefly on Jira about the lack of this option in calibrateImage? deblending seems to be always on but they have been turned off in characterizeImage.
There was a problem hiding this comment.
Deblending in calibrateImage is generally quite fast, because the detection threshold is high, and only happens after psf sources are found, not before. In characterizeImage it wasn't strictly necessary (see DM-36958) since that task was only finding psf sources and objectSizeStarSelector doesn't care about blendedness. There's no "turn off this step" options in calibrateImage, because all of the steps are either necessary for the output, or so fast you don't notice them.
There was a problem hiding this comment.
I was less concerned about the speed but a comment in the code said turning it on messes things up. That comment is 8 years old now, but DM-36958 seem to have gotten stuck in want of an RFC. That's what has left me confused here.
There was a problem hiding this comment.
The important thing here is that we never want to deblend during psf determination, and calibrateImage doesn't.
parejkoj
force-pushed
the
tickets/DM-43077
branch
from
dc10c06 to
6f4a2b0
Compare
TallJimbo
force-pushed
the
tickets/DM-43077
branch
2 times, most recently
from
bf425d1 to
4cc9227
Compare
* characterizeImage, writeSourceTable/writePreSourceTable are no longer necessary, as those steps are done inside calibrateImage. * Remove unnecessary configuration of quickLook as calibrateImage is already quite stripped-down. * Remove unnecessary imSim config that is already in the obs_lsst config override.
This does the work of writeRecalibratedSourceTable, and writes both an afw and Arrow version of the table, plus writes the final PVI. It should be run after all visit-level calibrations have been finished and the final visit summary produced.
calibrateImage detects significantly fewer sources overall, so needs less deblending.
ci_imsim doesn't have much of the visit-level processing (e.g. fgcmcal), but we still have to run reprocessVisitImage to get the full source catalog, since calibrateImage only produces a catalog of stars, with fewer measurements. This then requires treating the stars catalog as a "preSourceTable" and adding some other tasks that would normally only be run in a "full" pipeline.
This should be replaced by analyzeCalibrateImageMetadata when it is available (DM-46684).
parejkoj
force-pushed
the
tickets/DM-43077
branch
from
4cc9227 to
cd196af
Compare
to maintain inheritance of pipelines, and not duplicate definitions of tasks in the pipelines
No description provided.