diff --git a/README.rst b/README.rst index bb772ad..068c1f0 100644 --- a/README.rst +++ b/README.rst @@ -110,7 +110,7 @@ action is to setup a clean python environment into which the installation will occur. This mitigates any possible dependency conflicts with other packages you use. -The recommended method of setting up a new envrionment is with ``conda``: +The recommended method of setting up a new environment is with ``conda``: .. code-block:: console diff --git a/doc/conf.py b/doc/conf.py index 067f08b..ee5dffc 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -47,6 +47,7 @@ "sphinx.ext.mathjax", "sphinx.ext.ifconfig", "sphinx.ext.viewcode", + "sphinx_subfigure", # "sphinx.ext.autosectionlabel", ] diff --git a/doc/figures/pypeit_spec2d_noise_postscrub.png b/doc/figures/pypeit_spec2d_noise_postscrub.png new file mode 100644 index 0000000..98532df Binary files /dev/null and b/doc/figures/pypeit_spec2d_noise_postscrub.png differ diff --git a/doc/figures/pypeit_spec2d_noise_prescrub.png b/doc/figures/pypeit_spec2d_noise_prescrub.png new file mode 100644 index 0000000..04c1e2c Binary files /dev/null and b/doc/figures/pypeit_spec2d_noise_prescrub.png differ diff --git a/doc/figures/scrubber_line_fitting.png b/doc/figures/scrubber_line_fitting.png new file mode 100644 index 0000000..3038ce1 Binary files /dev/null and b/doc/figures/scrubber_line_fitting.png differ diff --git a/doc/include/dependencies_table.rst b/doc/include/dependencies_table.rst index 5ec30a2..79bef31 100644 --- a/doc/include/dependencies_table.rst +++ b/doc/include/dependencies_table.rst @@ -1,6 +1,6 @@ -================================ =========================================================================================================================================================================== -Python Version ``>=3.10,<3.12`` -Required for users ``astropy>=5.1``, ``ccdproc``, ``darkdetect``, ``matplotlib``, ``numpy>=1.22``, ``pysimplegui``, ``requests``, ``scipy>=1.9``, ``setuptools``, ``setuptools_scm``, ``tqdm`` -Optional ``pypeit`` requirements ``pypeit[specutils]>=1.14.0`` -Required for developers ``black``, ``pylint``, ``pypeit[specutils]>=1.14.0``, ``pyyaml``, ``sphinx``, ``sphinx-automodapi``, ``sphinx_rtd_theme==1.2.2``, ``stomp.py``, ``xmltodict`` -================================ =========================================================================================================================================================================== +================================ =================================================================================================================================================================================== +Python Version ``>=3.10,<3.12`` +Required for users ``astropy>=5.1``, ``ccdproc``, ``darkdetect``, ``matplotlib``, ``numpy>=1.22``, ``pysimplegui``, ``requests``, ``scipy>=1.9``, ``setuptools``, ``setuptools_scm``, ``tqdm`` +Optional ``pypeit`` requirements ``pypeit[specutils]>=1.14.0`` +Required for developers ``black``, ``pylint``, ``pypeit[specutils]>=1.14.0``, ``pyyaml``, ``sphinx``, ``sphinx-automodapi``, ``sphinx-subfigure``, ``sphinx_rtd_theme==1.2.2``, ``stomp.py``, ``xmltodict`` +================================ =================================================================================================================================================================================== diff --git a/doc/scrub_deveny_pickup.rst b/doc/scrub_deveny_pickup.rst index ad52696..022e264 100644 --- a/doc/scrub_deveny_pickup.rst +++ b/doc/scrub_deveny_pickup.rst @@ -60,6 +60,9 @@ signal. This patten is then subtracted from the original image and saved to a FITS file for usual data reduction processing (preferably with PypeIt) to yield an extracted 1D spectrum for analysis. +Example Pre- and Post-Scrubbed Data Products +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + To illustrate the need for and utility of this tool, :numref:`spec1d_comps` shows a comparison of the extracted 1D spectra for two different object types from two different programs (and nights) from both the original and scrubbed @@ -81,6 +84,36 @@ from the spectra. estimates a mean :math:`SNR = 4.8` for the scrubbed galaxy spectrum and a mean :math:`SNR = 7.2` for the scrubbed white dwarf spectrum. +Of equivalent interest to the quality of the extracted spectra is the noise +remaining after extraction of sky and objects. Shown in :numref:`prepost_noise` +are the noise analysis plots from PyepIt for the pre- and post-scrubbed +versions of the 2D spectral image shown in :numref:`raw_frame`. The images and +pixel histograms are of the residual noise image, which is the science image +minus the object and sky models, and then divided by the uncertainty image. +The ideal pixel histogram would be a gaussian with width :math:`\sigma=1`. + +.. _prepost_noise: +.. subfigure:: A|B + :gap: 2px + :class-grid: outline + + .. image:: figures/pypeit_spec2d_noise_prescrub.png + :alt: Pre-scrubbed noise analysis plot + + .. image:: figures/pypeit_spec2d_noise_postscrub.png + :alt: Post-scrubbed noise analysis plot + + -- Noise analysis of the pre- (top) and post-scrubbed (bottom) versions of + the frame shown in :numref:`raw_frame`. Image values are residual image + divided by uncertainty, so the ideal pixel histogram would be a gaussian + with width :math:`\sigma=1`. Note the improvement in both the visual + quality of the scrubbed frame and the width and shape of the pixel histogram + compared to the pre-scrubbed frame. + + +Outline +^^^^^^^ + This document begins with a description of how to use this tool to clean the EMI pickup noise from your data, and moves on to lay out the details of what the tool does to your data and other points to consider. @@ -493,11 +526,11 @@ than that predicted from the FFT (green dashed line in the second panel of Pickup Noise Pattern Construction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Finally, we use the sinusoid fits to produce a pattern image. This is the +The final result of sinusoid fits is a constructed pattern image. This is the zero-mean sinusoid (sans quadratic polynomial) that *should* represent only the -EMI pickup noise (as an additive AC-only signal). Subtract that and make -additional QA plots to show how totally awesome this is!!! - +EMI pickup noise (as an additive AC-only signal). :numref:`image_comparisons` +shows the process of pattern construction and its effect on the processed +science image. .. _image_comparisons: .. figure:: figures/scrubber_image_comparisons.* @@ -629,3 +662,14 @@ panel in :numref:`image_comparisons`). The result is shown in discussed above) from artifacts and ringing in the FFT. The peak at 169.1 pixels, indicating the frequency with the most power in the flattened array: most likely the period of the AC EMI pickup noise. + +Also, say something about the actual line-by-line fits in terms of how good a +sinusoid fits each one. Show a figure like :numref:`line_by_line`. + +.. _line_by_line: +.. figure:: figures/scrubber_line_fitting.png + :class: with-shadow + :alt: Line-by-line fitting examples + + -- Examples of individual line fits for 4 randomly selected lines from the + image shown in :numref:`raw_frame`. diff --git a/obstools/scrub_deveny_pickup.py b/obstools/scrub_deveny_pickup.py index 4555e25..c943254 100644 --- a/obstools/scrub_deveny_pickup.py +++ b/obstools/scrub_deveny_pickup.py @@ -131,7 +131,7 @@ def iterative_pypeit_clean( try: # Look for the spec2d file spec2d_file = [ - next(d.joinpath("Science").glob(f"spec2d_{filename.stem}-*")) + next(d.joinpath("Science").glob(f"spec2d_{filename.stem}-*.fits")) for d in pyp_dir ][0] except (StopIteration, IndexError): diff --git a/setup.cfg b/setup.cfg index f7d60d4..5088119 100644 --- a/setup.cfg +++ b/setup.cfg @@ -48,6 +48,7 @@ install_requires = docs = sphinx sphinx-automodapi + sphinx-subfigure sphinx_rtd_theme==1.2.2 broker = @@ -61,6 +62,7 @@ pypeit = dev = sphinx sphinx-automodapi + sphinx-subfigure sphinx_rtd_theme==1.2.2 pylint black