Demonstration prepared for the MWGaia-DN Introductory School, Coimbra, 2024. This demo showcases working with HEALPix-partitioned survey catalogs via LSDB, and time domain analysis with nested-dask.
- Slide deck
- LSDB (on GitHub) (on ReadTheDocs)
- HiPSCat (on GitHub) (on ReadTheDocs)
- nested-dask (on GitHub) (on ReadTheDocs)
- nested-pandas (on GitHub) (on ReadTheDocs)
You can follow along with this demo by creating your own local environment, or accessing the LINCC-hub (a shared cloud-hosted JupyterHub). We recommend using LINCC-hub because of the close proximity to our servers hosting the survey data. Also, the code should run much quicker and smoother because the instances have powerful resources available.
- You'll need an account on LINCC-hub. You can sign up by completing this form, and following the steps. Please complete this prior to attending the demo. You will receive an invite to the GitHub group at the email that is registered with your GitHub acccount. You may need to accept the membership invite.
- BEFORE STARTING YOUR SERVER, note that you should not use the default size! On the "Server Options" page select "Need more CPU or memory...?" and choose a "Large" server.
- To get started, log into https://lsst.dirac.dev/ with your GitHub sign-on. If this fails, please reach out to [email protected].
- After your server has started up (it will take 3 minutes at least!), clone this repo in LINCC-hub ("New" > "Terminal")
git clone https://github.com/lincc-frameworks/MWGaia-DN_Introductory_School
- Open the notebooks with the "MWGaia-DN School" kernel. If next to the kernel name there is a label saying "Not Trusted", click it and "Trust" the notebook. Running, especially the first cells, should take a minute. Feel free to run it before the demo starts if you want to follow along.
- Work through the notebooks and have fun.
- Shutdown each notebook after you're done to use less memory.
If installing in your own hardware, create a virtual environment and install the relevant packages:
>> conda create --name lincc python=3.11
>> conda activate lincc
>> pip install lsdb pyvo ipyaladin cesium scikit-learn aiohttp
In this notebook we will learn how to:
- Load object and source catalogs (lazily)
- Perform crossmatching with existing
LSDBcatalogs - Save the results of a science workflow to disk
In this notebook we will learn how to:
- Query and filter catalog data
- Compute time-series features for LSDB catalogs using
nested-dask - Plot light curves and periodograms
In this notebook we will learn how to:
- Use VizieR TAP query to access tables and store/handle them in
LSDB - Use those catalogs to perform crossmatching with existing
LSDBcatalogs - Perform time-series analysis and exploration with
nested-dask
Watch the following LINCC Tech Talk to learn more about LSDB. Other relevant talks can be found in the LSST Discovery Alliance website.
This project is supported by Schmidt Sciences.
This project is based upon work supported by the National Science Foundation under Grant No. AST-2003196.
This project acknowledges support from the DIRAC Institute in the Department of Astronomy at the University of Washington. The DIRAC Institute is supported through generous gifts from the Charles and Lisa Simonyi Fund for Arts and Sciences, and the Washington Research Foundation.