Create new fork option. This will bring you to the "Create new fork" screen:
And select the Create new fork option. This will bring you to the "Create new fork" screen:
!!! note Uncheck the "Copy the main branch only" option
Uncheck the \"Copy the main branch only\" option
Failure to uncheck this will prevent you from pulling in updates from the development branch easily.
As you make changes and want to commit them to your github repos, you will be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a git status in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the git status command. All other git commands will be relative to your current working directory as well.
To make runs in either CAM or CAM-SIMA the commands are identical:
./create_newcase\u2026 #(the commands will differ and are specified later in this document)\ncd YourCaseDirectory\n./xmlchange\u2026 #(specific commands are specified later in this document)\n./case.setup\n./case.build\n./case.submit\nDepending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes.
For more detailed information on case creation and building, see https://ncar.github.io/CAM/doc/build/html/users_guide/building-and-running-cam.html
"},{"location":"conversion/ccpp-conversion-guide/#prep-work","title":"Prep Work","text":""},{"location":"conversion/ccpp-conversion-guide/#conversion-spreadsheet","title":"Conversion Spreadsheet","text":"Put the parameterization that you are going to convert into the conversion spreadsheet https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0
"},{"location":"conversion/ccpp-conversion-guide/#create-github-issues","title":"Create Github Issues","text":"Create a Github Issue in the ESCOMP/CAM repo that states which physics parameterization you are planning to convert to the CCPP framework. Then create another issue in the ESCOMP atmospheric physics repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in ESCOMP atmospheric physics and be removed from ESCOMP/CAM.
"},{"location":"conversion/ccpp-conversion-guide/#setting-up-your-sandbox","title":"Setting up your sandbox","text":"Make sure you have github forks for both ESCOMP/CAM-SIMA and ESCOMP/atmospheric_physics. If needed see https://github.com/ESCOMP/Cam/wiki/CAM-Development-Workflow-in-GitHub#how-to-makestore-revisions-to-your-personal-cam-repository-github-fork
To begin, fork ESCOMP/CAM-SIMA:
And select the Create new fork option. This will bring you to the \"Create new fork\" screen:
!!! note Uncheck the \"Copy the main branch only\" option
Failure to uncheck this will prevent you from pulling in updates from the development branch easily.
As you make changes and want to commit them to your github repos, you will be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a git status in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the git status command. All other git commands will be relative to your current working directory as well.
In order to describe the build process, we need to define several source and build directories:
<srcroot> : The CAM-SIMA sandbox being built. From a case directory, this can be found with ./xmlquery SRCROOT.<caseroot>: The location of the case being built.<src_mods>: The location of the CAM-SIMA source mods, located at <caseroot>/SourceMods/src.cam.<bldroot> : The location of CAM-SIMA generated code and compiled code (.o and .mod). See atm/obj under ./xmlquery OBJROOT.Given the context above, the following is the CAM-SIMA build sequence:
ConfigCAM object, used to store build configuration information. Code can be found in <srcroot>/cime_config/cam_config.py<case>: A CIME case object<case_log>: A python logger, which is usually created by CIME itselfconfig_dict: A dictionary of configure options, with the dictionary keys being the name of the config variable/option, and the dictionary values being special config variable objects that contain the following properties:name: The name of the config option/variable. This is what is used as the key in the config_dict dictionarydesc: A text description of what this particular config option is, and what sort of values the config option should have.value: The value of that config option/variable. Currently can only be an integer, string, or list of integers or strings.valid_vals: Optional property that contains either a list or range of valid values for this particular config entryvalid_type: Optional property for a list-type config option that states what data type the list elements can have.is_nml_attr: Logical that states whether this config option can be used as an XML attribute in a namelist definition file.ConfigCAM object also has various build-in methods that are used to call other steps in the build workflow, change the value of a given config variable, or print extra config info to the provided python logger.ConfigCAM object currently has no uses outside buildnml and buildlib, and so is not saved after those steps are complete.BuildCacheCAM object (<srcroot>/cime_config/cam_build_cache.py).<build_cache>: An optional cache file (created by previous build). This file is created in the case build directory (bld/atm/obj).The diagram above displays everything that occurs when CAM-SIMA's buildnml is called by CIME, which occurs after the user calls preview_namelists, case.build, or case.submit.
All blue boxes represent source code, and are listed as \"source file: function/object used\", and all objects or functions that have a \"+\" are automatically tested whenever a Pull Request is opened, updated, or merged.
All orange boxes are input files that are used by the code, while all green boxes are output files that are generated. It is important to note that additional files are generated as well, specifically a build cache and a CCPP datatable, but those files are used internally in the build scripts shown here and not by the final model executable, and thus aren't listed here.
Finally, the arrows show the order of operations, starting with buildnml, with the top two source code boxes representing python classes that are used by the functions/objects directly below them.
*Static images can be found at the bottom of this page
"},{"location":"design/cam-run-process/#cam-sima-api","title":"CAM-SIMA API","text":"Upon running ./case.submit the core CAM-SIMA driver code* is in $CAM-SIMA/src/control/cam_comp.F90. This section lays out each of the subroutines within cam_comp.F90. The subroutines in cam_comp.F90 are set up to mirror the phases of the Common Community Physics Package (CCPP).
* cam_comp.F90 subroutines are called by the NUOPC cap: $CAM-SIMA/src/cpl/nuopc/atm_comp_nuopc.F90
cam_init sets up the metadata and configuration objects/modules for the run. It is called once at startup. Below are the variables passed in:
eccen in cam_control_mod obliqr (in) Earth's obliquity in radians used to set module-level obliqr in cam_control_mod lambm0 (in) Mean longitude of perihelion at the vernal equinox (radians) used to set module-level lambm0 in cam_control_mod mvelpp (in) Earth's moving vernal equinox longitude of perihelion plus pi (radians) used to set module-level mvelpp in cam_control_mod perpetual_run (in) flag to determine if perpetual mode is enabled passed to time manager init perpetual_ymd (in) perpetual year, month, day (YYYYMMDD) used to determine the sun position and interpolate boundary data sets passed to time manager init dtime (in) model timestep size in seconds passed to time manager init start_ymd (in) start date (YYYYMMDD) passed to time manager init start_tod (in) start time of day (sec) passed to time manager init ref_ymd (in) reference date (YYYYMMDD) (defaults to start_ymd) passed to time manager init ref_tod (in) reference time of day (sec) (defaults to start_tod) passed to time manager init stop_ymd (in) stop date (YYYYMMDD) passed to time manager init stop_tod (in) stop time of day (sec) passed to time manager init curr_ymd (in) current date (YYYYMMDD) (same as start date) passed to time manager init curr_tod (in) current time of day (sec) (same as start tod) passed to time manager init cam_in (inout) surface exchange object - coupler to CAM-SIMA allocated if this is an initial run cam_out (inout) surface exchange object - CAM-SIMA to coupler allocated if this is an initial run ** For additional information on run types, see the CESM Tutorial
cam_init calls the following key subroutines (locations) in this order:
cam_ctrl_init (src/control/cam_control_mod.F90): Sets the module-level run configuration variables; logs configurations to the atm logcam_ctrl_set_orbit (src/control/cam_control_mod.F90): Sets the module-level orbital variablestimemgr_init (src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm logread_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)cam_ctrl_set_physics_type (src/control/cam_control_mod.F90): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm logcam_initfiles_open (src/control/cam_initfiles.F90): Opens initial or restart file, and topography file if specifiedcam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)air_composition_init (src/data/air_composition.F90): Initializes air-composition-dependent model constantsmodel_grid_init (src/dynamics/<dycore>/dyn_grid.F90): Initializes model grids and decompositionscam_ccpp_initialize_constituents ($CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90): initializes the constituent data array; after this point, we cannot add new constituentsdyn_init (src/dynamics/<dycore>/dyn_comp.F90): Initializes the dynamical coreatm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocates and sets up surface exchange dataphys_init (src/physics/utils/phys_comp.F90): Initializes physics (includes call to CCPP cap to run init phases of schemes in the Suite Definition File (SDF)stepon_init (src/dynamics/<dycore>/stepon.F90): Initializes dynamics <--> physics couplingcam_timestep_init is called at the start of each timestep. It has no input/output/inout variables.
The routine calls the following subroutines (locations) in this order:
stepon_timestep_init (src/dynamics/<dycore>/stepon.F90): First phase of dynamics (couple from dynamics to physics); also returns timestep for physicsphys_timestep_init (src/physics/utils/phys_comp.F90):src/data/registry.xml) from the ncdata filecam_run1 is the first \"run\" phase called in the physics loop. It is called every timestep BEFORE the mediator/surface coupler and calls the following subroutine (location):
phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_before_coupler\" group in the SDFcam_run2 is the second \"run\" phase called in the physics loop. It is called every timestep AFTER the mediator/coupler. Input/output variables:
cam_run2 calls these subroutines (locations):
phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_after_coupler\" group in the SDFstepon_run2 (src/dynamics/<dycore>/stepon.F90): The second phase of dynamics (couple from physics to dynamics)cam_run3 is the third \"run\" phase called in the physics loop. It is called every timestep AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:
cam_run3 calls the following subroutine (location):
stepon_run3 (src/dynamics/<dycore>/stepon.F90): Calls dyn_run, which runs the dycorecam_run4 currently does nothing! (but it is called every timestep)
cam_timestep_final runs at the end of each timestep. In/out variables:
cam_timestep_final calls the following subroutines (locations):
history_write_files (src/history/cam_history.F90): Writes fields to user-configured history files (if applicable)history_wrap_up (src/history/cam_history.F90): Closes files and zeros buffers as necessaryphys_timestep_final (src/physics/utils/phys_comp.F90):ncdata_check is set in user_nl_cam, calls physics_check_data ($CASE/bld/atm/obj/phys_init/physics_inputs.F90) to perform snapshot checkingcam_final is called once at the end of the model execution. In/out variables:
cam_final calls the following subroutines (locations):
phys_final (src/physics/utils/phys_comp.F90): calls \"final\" phase of all schemes in the SDFstepon_final (src/dynamics/<dycore>/stepon.F90): finalizes dycore (doesn't currently do anything)atm2hub_deallocate and hub2atm_deallocate (src/control/camsrfexch.F90): deallocate cam_in/cam_out objectsThe core Common Community Physics Package (CCPP) documentation can be found here. This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:
CCPP-compliant physics schemes must adhere to the following criteria:
dependencies, but we're not getting into that now):Metadata example (snippet taken from kessler.meta)
[ precl ]\n standard_name = total_precipitation_rate_at_surface\n long_name = Total precipitation rate at surface\n units = m s-1\n dimensions = (horizontal_loop_extent)\n type = real | kind = kind_phys\n intent = out\n[ relhum ]\n standard_name = relative_humidity\n long_name = Relative humidity\n units = percent\n dimensions = (horizontal_loop_extent, vertical_layer_dimension)\n type = real | kind = kind_phys\n intent = out\n[ scheme_name ]\n standard_name = scheme_name\n units = none\n type = character | kind = len=64\n dimensions = ()\n intent = out\n\nCCPP-compliant physics schemes are organized into suite definition files (SDFs). An SDF tells the framework which schemes will be run in what order. Separating schemes into \"groups\" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<suite name=\"kessler\" version=\"1.0\">\n <group name=\"physics_before_coupler\">\n <scheme>calc_exner</scheme>\n <scheme>temp_to_potential_temp</scheme>\n <scheme>calc_dry_air_ideal_gas_density</scheme>\n <scheme>wet_to_dry_water_vapor</scheme>\n <scheme>wet_to_dry_cloud_liquid_water</scheme>\n <scheme>wet_to_dry_rain</scheme>\n <scheme>kessler</scheme>\n <scheme>potential_temp_to_temp</scheme>\n <scheme>dry_to_wet_water_vapor</scheme>\n <scheme>dry_to_wet_cloud_liquid_water</scheme>\n <scheme>dry_to_wet_rain</scheme>\n <scheme>kessler_update</scheme>\n <scheme>qneg</scheme>\n <scheme>geopotential_temp</scheme>\n <scheme>cam_state_diagnostics</scheme>\n <scheme>kessler_diagnostics</scheme>\n </group>\n <group name=\"physics_after_coupler\">\n <scheme>cam_tend_diagnostics</scheme>\n </group>\n</suite>\nThe framework code is primarily python code that generates Fortran caps. The class structure looks like:
Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:
<host>_ccpp_cap.F90: contains the interface layer between the host and the suite(s)ccpp_<suite>_cap.F90: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in orderccpp_datatable.xml: consolidates metadata into an XML file to be used as desired by the host modelHow CAM-SIMA and the CCPP come together:
"},{"location":"design/ccpp-in-cam-sima/#host-model","title":"Host Model","text":"The core host model code is what is held in the CAM-SIMA github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (src/data/registry.xml).
The CCPP physics scheme code exists in the atmospheric_physics repository, which exists as a submodule of CAM-SIMA in the following location: $CAM-SIMA/src/physics/ncar_ccpp
SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.
The diagnostics directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).
The to_be_ccppized directory contains physics schemes and utilties that have not been CCPP-ized, but were needed by an CCPP-ized scheme.
The utilities directory contains schemes that are used regularly, such as tendency applicators and state converters. See below for more.
The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location: $CASE/bld/atm/obj/ccpp/
All CCPP phases are called from the physics driver (src/physics/utils/phys_comp.F90). You can see the order of these calls more thoroughly in the documented run sequence.
For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:
src/data/physconst.F90) so that a scheme can access an existing host model variablesrc/data/registry.xml)The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.
"},{"location":"design/ccpp-in-cam-sima/#state-and-tendency-variables","title":"State and tendency variables","text":"Two of the most commonly used and referred-to objects in CAM-SIMA are:
The Fortran for both objects is auto-generated by CAM-SIMA based on the registry ($CAM-SIMA/src/data/registry.xml). The generated code can be found here: $CASE/bld/atm/obj/cam_registry/physics_types.F90. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.
The physics_state object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.
Some examples of these core \"state\" variables include temperature (T), eastward wind (U), and northward wind (V)
As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called time splitting, which means that all physics since the last state update get the same input state. The alternative, process splitting, means that the output state of one scheme serves as the input state of the next.
NOTE: Although constituents are handled independently of the physics_state object (they are handled by the CCPP framework), they ARE considered state variables.
The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being \"updated\" by the scheme. Some of these tendency variables are held within the physics_tend object, but others are internal to the physics.
The module $CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90 includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: state_var = state_var + tend*dt where dt is the timestep size.
Some definitions to start (as written by a non-scientist, so there is more nuance than this!):
src/data/air_composition.F90qneg will set constituent values that are less than the minimum to the minimumThere are three ways to identify a quantity as a constituent in CAM-SIMA:
cam_register_constituents (in src/control/cam_comp.F90). Currently, we are always adding water vapor as a host constituent because it is expected by the SE dycore.advected = Truesrc/physics/ncar_ccpp/musica/micm/micm.F90The registration and initializaiton of the constituent data array and the constituent properties object are done through calls to the generated CCPP cap.
ccpp_model_constituents_t objectccpp_model_constituents_t objectConstituent values and properties can be accessed from the host side and from the physics in the following ways:
cam_constituents.F90 module, which is an interface to the CCPP cap, which is in turn an interface to the constituents object[ q ]\n standard_name = ccpp_constituents\n units = none\n type = real | kind = kind_phys\n dimensions = (horizontal_loop_extent,vertical_layer_dimension,number_of_ccpp_constituents)\n intent = inout\n[ const_props ]\n standard_name = ccpp_constituent_properties\n units = None\n type = ccpp_constituent_prop_ptr_t\n dimensions = (number_of_ccpp_constituents)\n intent = in\nThis section can be removed when constituents are documented in the CCPP Framework documentation.
"},{"location":"design/constituents/#constituent-object-fortran","title":"Constituent object (Fortran)","text":"The constituent object (found in $CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is ccpp_model_constituents_t.
This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):
const_metadata (ccpp_constituent_properties)num_layer_vars (number_of_ccpp_constituents)num_advected_vars (number_of_ccpp_advected_constituents)vars_layer (ccpp_constituents)The const_metadata property is of type ccpp_constituent_prop_ptr_t, which contains a pointer to a ccpp_constituent_properties_t object, as depicted above. This object contains all of the constituent properties for the constituents in the constituents array, with the same indices as the constituents array.
The ccpp_model_constituents_t type also contains a hash table of constituent properties for more efficient searching, as well as several methods used by the generated cap code. Some methods are highlighted below:
The constituents-related code generation routines provide an interface to the constituents object. These routines can be found in $CAM-SIMA/ccpp_framework/scripts/constituents.py, primarily within the \u201cwrite_host_routines\u201d function. The (most often used) generated routines related to constituents are:
The routines above are generated during ./preview_namelists or ./case.build and can be found here: $CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90
CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.
"},{"location":"design/history/#history-initialization","title":"History Initialization","text":""},{"location":"design/history/#reading-and-processing-the-history-configuration","title":"Reading and processing the history configuration","text":"cime_config/hist_config.py in _HIST_CONFIG_ENTRY_TYPES.user_nl_cam)<keyword>;<volume>: <value> (see examples below)atm_in equivalent indicates how hist_config.py parses these into a namelist to be read by SIMA):hist_config.py also contains the HistoryVolConfig class (all the info pertaining to a single history file), the HistoryConfig class (all the history configuration information including a dict of HistoryVolConfig objects), and helper classes.HistoryConfig object is created in buildnml out of entries in user_nl_cam and written to run/atm_in.atm_in, the HistoryConfig object must contain all the logic in setting default values.src/history/cam_hist_file.F90cam_hist_file.F90 is hist_read_namelist_config, which is called by history_readnl in src/history/cam_history.F90. This function reads in the hist_config_arrays_nl namelist group, allocates the history field arrays, and then uses those arrays to read in the hist_file_config_nl namelist group (via a call to read_namelist_entry).cam_history.F90\u2019s module-level hist_configs array of hist_file_t objects (the size of this array is the number of user-configured volumes).hist_file_t object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (config_set_up_fields), set up the metadata of the file (config_define_file), and write history fields to the file (config_write_time_dependent_variables).hist_file_t object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the hist_field_info_t (in src/history/buffers/src/hist_field.F90), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are hist_buffer_t) that will and do hold the actual data.The possible fields to be output by the history infrastructure are tracked in cam_history.F90 via the possible_field_list hash table. It is populated during init time by calls to the subroutine history_add_field (found in src/history/cam_history.F90). \u201cInit time,\u201d means that all calls to history_add_field must occur during the execution of cam_init (found in src/control/cam_comp.F90).
dyn_init or stepon_initsrc/physics/ncar_ccpp/diagnostics/cam_diagnostics.F90physics_before_coupler group in the suite definition file (SDF)cam_tend_diagnostics scheme in src/physics/ncar_ccpp/diagnotics/cam_diagnostics.F90physics_after_coupler group in the SDFsrc/physics/ncar_ccpp/diagnostics Each call to history_add_field adds a new field to the end of the possible_field_list_head linked list. At the end of cam_init, the possible field list linked list is used to print the list (to the atm.log* file) and then is converted to the possible_field_list hash table. A sample of the history field list is seen below.
***************** HISTORY FIELD LIST ******************\n T K avg air_temperature\n ZM m avg geopotential_height_wrt_surface\n PHIS m2 s-2 ins surface_geopotential\n PMID Pa avg air_pressure\n PDELDRY Pa avg air_pressure_thickness_of_dry_air\n Q kg kg-1 avg water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water\n CLDLIQ kg kg-1 avg cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water \n RAINQM kg kg-1 avg rain_mixing_ratio_wrt_moist_air_and_condensed_water\n TTEND K s-1 avg tendency_of_air_temperature_due_to_model_physics\n *************** END HISTORY FIELD LIST ****************\nOutside of CAM-SIMA init and finalize time, history buffers can be populated with data via a call to history_out_field (found in src/history/cam_history.F90)
The subroutine history_out_field iterates over the hist_configs array and populates the buffer(s) of the hist_field_info_t object of the hist_file_t object if the field name in the call is active on that file (e.g. the field was configured via the namelist to be output for that volume).
history_out_field can exist anywhere except _init and _finalhistory_out_field can exist anywhere except in dyn_init, stepon_init, and stepon_finalhistory_out_field calls are included in the run phase of the same schemes described in the section above.The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:
The bolded step #2 above is what determines if we need to define a new history file. The situations where we would need to define a new file are:
hist_max_frames) and closed itWe determine if it's time for a new file with the following line of code:
mod(num_samples, hist_configs(file_idx)%max_frame()) == 0\nIf it is indeed time to define a new file, we call the config_define_file subroutine (which is found in src/history/cam_hist_file.F90) for the volume:
call hist_configs(file_idx)%define_file(restart, logname, host, model_doi_url)\nThe cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:
The bolded step #3 above occurs any time the criteria for #1 is satisfied. At this point, the following call is made to write the history fields (whose data has been stored in their buffers via calls to history_out_field):
call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)\nIt is during this call that we increment the number of samples written for this volume and actually write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).
"},{"location":"design/history/#defining-history-restart-files","title":"Defining history restart files","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/history/#writing-a-history-restart-file","title":"Writing a history restart file","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/history/#reading-a-history-restart-file","title":"Reading a history restart file","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/sima-design-goals/","title":"Design goals & features","text":"Motivated by the Singletrack project (a precursor to SIMA), the CAM-SIMA project was created to build a new CAM infrastructure to meet the SIMA science and computational needs. This page documents those needs and some of the features that implement them.
"},{"location":"design/sima-design-goals/#cam-needs-to-be-more-run-time-configurable","title":"CAM needs to be more run-time configurable","text":"The chunk structure in CAM physics served its purpose when threading was the only way to accelerate code. However, to make the best use of both threading and modern accelerators, a flexible chunking mechanism is required. The new infrastructure enables this by using flat arrays for all fields.
In order to continue to allow CAM to grow without an ever increasing cost of bringing in new features, CAM must be more modular. A classic example is chemistry which ACOM would like to make modular but which is currently entwined with CAM in many areas (e.g., code in CAM repository, extensive configuration code dedicated to chemistry, extensive namelist building code and data dedicated to chemistry, large number of use cases containing chemistry configuration data). The new CAM infrastructure contains features to increase modularity.
Modularity will allow CAM to move components to external repositories. This process cuts development costs for both CAM and for the component (e.g., as has happened with PUMAS). Some ways this is accomplished are listed here:
Use of the CCPP to build physics suites also makes CAM more modular because the CCPP treats physics schemes as modular which allows flexibility in building physics suites. The CCPP takes care of making sure variables are available before they are used and also builds the code currently handled via hand-written code in the various versions of physpkg.F90.
"},{"location":"design/sima-design-goals/#run-time-data-checking","title":"Run-time data checking","text":"CAM needs data to run but the data needs vary with the simulation. The new infrastructure facilitates this.
CAM currently has a few ways to run offline testing (e.g., SCAM, PORT). The new infrastructure builds these capabilities in for more efficient and flexible use.
To enable efficient quality control, the new infrastructure implements a number of continuous integration (CI) techniques.
The standards in this document are largely prescriptive, i.e., they should be followed.
While some legacy code will not follow these rules, efforts SHOULD be made to improve the code whenever you are working on it (e.g., bug fixes, enhancements).
"},{"location":"development/cam-coding-standards/#general-coding-standards","title":"General coding standards","text":""},{"location":"development/cam-coding-standards/#must","title":"MUST","text":"See tips for configuring editors
"},{"location":"development/cam-coding-standards/#should","title":"SHOULD","text":"We expect all python code to pass with a perfect score (10) using pylint with this version of pylintrc. However, external repos may have their own pylintrc version depending on their needs.
We also expect all python files to follow the black code style and format.
Finally, one can also follow the Google Python Style Guide.
"},{"location":"development/cam-coding-standards/#fortran-coding-standards","title":"Fortran coding standards","text":"The standards described in this section represent the CAM Fortran standards. Other Fortran standards:
use statementsif statements (i.e., all if statements should have a then if the statement is on more than one line)-HUGE(1), real: NaN, character: 'UNSET').pure keyword.intent for dummy arguments except for pointers.1.5_r8, not 1.5 or 1.5D0. Literals must also include the decimal point.character(len=*) or character(len=CL)).implicit none statement in the preamble (after the use statements). Module routines then do not need this statement.call subroutine(x, optional_y=y) instead of call subroutine(x, y) for the optional variable optional_y).SAVE attribute and is not thread safe.nullify statement.pcols, even if only a subset of the variable is used in the subroutine or function. #if, #ifdef). Always try for runtime variable logic instead.pure keyword if a subroutine has no side effects.==, /=, <, >=) not old character versions (e.g., .eq.).private. Public interfaces are declared after the private declaration.private module interfaces (i.e., subroutines and functions) should be declared private in the module header..F90).pure attribute. If they cannot, the reason should be included as a comment in the function's preamble.return at the end of a subroutine).use statements should be brought in at the smallest scope possible (e.g. inside individual subroutines instead of at the module level).module, subroutine, if, do), indent that scope relative to the scoping statement (recommended 3 spaces but each module should at least be self consistent).if, else, end, do, and while.::only, i.e., only:, not only :..emacs file)","text":"(add-hook 'before-save-hook 'delete-trailing-whitespace)\n(setq-default indent-tabs-mode nil)\n(setq tab-width 4)\n.vimrc file)","text":" autocmd BufWritePre * :%s/\\s\\+$//e\nThis page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.
"},{"location":"development/cam-testing/#python-unit-testing","title":"Python unit testing","text":"CAM-SIMA supports two kinds of python unit tests, doctest and unittest tests, both of which are part of the standard python library.
All unittest tests should be in:
CAM-SIMA/test/unit
while all files used by the tests should be in:
CAM-SIMA/test/unit/sample_files
All unittest tests are automatically run via Github Actions whenever a Pull Request (PR) is opened, modified, or merged.
All doctest tests are also run automatically as long as the scripts they are located in are under CAM-SIMA/cime_config or CAM-SIMA/src/data.
To manually run all of the unit tests at any time, simply run the following shell script:
CAM-SIMA/test/run_tests.sh
Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables. If so, then it would likely be best as a doctest. Otherwise it should be a unittest test. Failure to follow this rule of thumb could result in test failures in the Github Actions workflow. Also remember to add your new tests to the run_tests.sh script so future users can easily run the tests manually.
Any python script which is added or modified via a PR will automatically be analyzed using pylint, and must have a score of 9.5 or greater in order to not be marked as a test failure. The hidden pylintrc file used for the analysis can be found here:
CAM-SIMA/test/.pylintrc
Users can also manually run pylint against the core python build scripts by running the following shell script:
CAM-SIMA/test/pylint_test.sh
Please note that pylint is not part of the standard python library, and so it may need to be installed before being able to run the shell script.
NOTE: Regression testing on Derecho should be done for every PR before merging!
Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the test_driver.sh script under $CAM-SIMA/test/system. To run the tests associated with a particular compiler option one can do the following commands:
For running GNU tests*:
env CAM_FC=gnu ./test_driver.sh -f\nFor running Intel tests*:
env CAM_FC=intel ./test_driver.sh -f\nRunning the script will produce a directory in your scratch space labeled aux_sima_<CAM_FC>_<timestamp>, where <CAM_FC> is the compiler you chose, and <timestamp> is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.
Inside the directory you should see an executable labeled cs.status.*. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled PASS except the SUBMIT step, which should be labeled PEND. Any other label indicates that a test may have failed, and should be investigated.
Finally, the tests themselves are listed in <CAM-SIMA>/cime_config/testdefs/testlist_cam.xml. Any files that need to be included in order for the tests to run properly are located in <CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX, where XXX is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found online here.
*Note: you may also have to include the environment variable CAM_ACCOUNT on derecho, which points to your account key
The test list can be found here: $CAM-SIMA/cime_config/testdefs/testlist_cam.xml
<machine> XML entry<test> XML entry with the following structure:<test compset=\"<COMPSET_NAME>\" grid=\"<GRID_ALIAS>\" name=\"<TEST_TYPE>_<TEST_MOD>\" testmods=\"<RELPATH_TO_TESTMODS_DIR>\">\n <machines>\n <machine name=\"<MACH_NAME>\" compiler=\"<COMPILER>\" category=\"<TEST_CATEGORY>\"/>\n </machines>\n <options>\n <option name=\"comment\">COMMENT HERE</option>\n <option name=\"wallclock\">WALLCLOCK_TIME</option>\n </options>\n</test>\n<COMPSET_NAME>: component set alias (or long name) - you can see more about compsets here<GRID_ALIAS>: model grid/resolution you'd like to run on - you can see more about grids here<TEST_TYPE>: type of test to be run. You can find the testing types here.<TEST_MOD>: test modifier that changes the default behavior of the test type. More here<RELPATH_TO_TESTMODS_DIR>: relative path to the testmods directory for this run; usually looks something like \"cam/some_directory_name/\"user_nl_cam and/or shell_commands)$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/<MACH_NAME>: machine name - will almost definitely be either derecho or izumi<COMPILER>: compiler to be used (options: gnu, nag, intel, nvhpc)<TEST_CATEGORY>: group of tests that this test belongs to - the default run by test_driver.sh is aux_sima (which is run for each PR to CAM-SIMA)WALLCLOCK_TIME: maximum amount of time that the job will be allowed to runHere is an example test entry for a 2-timestep smoke test of kessler physics on the MPAS grid, run with both intel and gnu
<test compset=\"FKESSLER\" grid=\"mpasa480_mpasa480\" name=\"SMS_Ln2\" testmods=\"cam/outfrq_kessler_mpas_derecho_nooutput/\">\n <machines>\n <machine name=\"derecho\" compiler=\"intel\" category=\"aux_sima\"/>\n <machine name=\"derecho\" compiler=\"gnu\" category=\"aux_sima\"/>\n </machines>\n <options>\n <option name=\"wallclock\">00:10:00</option>\n <option name=\"comment\">GNU build test for MPAS dycore (with Kessler physics)</option>\n </options>\n </test>\nStart with the CAM wiki.
"},{"location":"development/debugging/#cam-sima-specific-debugging","title":"CAM-SIMA-specific debugging","text":""},{"location":"development/debugging/#build-errors","title":"Build errors","text":"Debugging tips if you get build errors:
bld/atm.bldlog.*PETXXX - if there was an issue on the ESMF side of things, it will show up in one of these (there will be one PET file per processor)./xmlchange NTASKS=1), do a clean rebuild (as instructed), and run again; maybe running in serial will identify the error***************** HISTORY FIELD LIST ****************** in the atm.log* file; if it's not there, the error occurred at init timeCAM-SIMA time step advanced message in the atm.log* file), it's likely that one or more variable that you introduced or modified has gone off the rails (value has become very large or very small or zero)development and:development) to identify where the numbers are going awry OR$CAM-SIMA/tools/find_max_nonzero_index.F90)$CAM-SIMA/tools/find_max_nonzero_index.F90(COURTNEY - ASK CHERYL TO HELP WITH THIS)
./case.build)bash to change to bash (if not already)np=1\nnthreads=1\n\nsource .env_mach_specific.sh\n\nRUNDIR=`./xmlquery RUNDIR -value`\nEXEROOT=`./xmlquery EXEROOT -value`\nLID=`date '+%y%m%d-%H%M%S'`\n\ncd $RUNDIR\nmkdir timing\nmkdir timing/checkpoints\necho `pwd`\nexport OMP_NUM_THREADS=$nthreads\ntotalview ${EXEROOT}/cesm.exe\nexit to exit the totalview window and give up the node Git also offers extensive built-in (i.e., command line) help, although it can be hard to understand until you are familiar with basic git concepts:
git help
git help COMMAND
man gittutorial
man giteveryday
"},{"location":"development/git-basics/#how-to-set-up-your-git-development-environment","title":"How to set up your git development environment","text":"There are several stages in setting up your git and GitHub development environment. Below, they are broken into three sections which represent the different stages:
Create a new personal CAM-SIMA fork of the ESCOMP/CAM-SIMA repo:
Set up SSH keys on GitHub (optional):
If you will be pushing changes to GitHub (either to your own fork or to shared forks), or if you will be pulling changes from private repositories on GitHub, then it's worth the time to set up ssh keys for each machine you'll be using. By doing so, you won't have to enter your password when pushing to or pulling from GitHub.
See here for instructions. After doing this, you can use the ssh form of GitHub URLs (e.g., git@github.com:ESCOMP/cam-sima.git) in place of the https form.
Configure GitHub notifications (optional): It is important to keep track of activity on GitHub but there are ways to manage how you are notified.
git has global settings which apply to all repository clones on your machine. These settings reside in a file called .gitconfig in your home directory. Below are some required and some optional but recommended global git settings to apply to any new machine where you will do CAM-SIMA development (e.g., Derecho, Izumi, personal laptop). Apply the settings below or simply copy a .gitconfig file from a machine that is already configured.
git config --global user.name \"Your Name\"\ngit config --global user.email <GitHub email address>\nWe recommend that you use your UCAR email address as your GitHub email address but if you use another address, you can add your UCAR email address by clicking on your profile picture in the upper-right of any GitHub page, then clicking on \"Settings\", and then on \"Emails\".
"},{"location":"development/git-basics/#recommended-git-global-configuration-settings","title":"Recommended git global configuration settings","text":"You can set which editor to use for log messages, etc., with:
git config --global core.editor <editor of your choice: emacs, vi, vim, etc>\n(See http://swcarpentry.github.io/git-novice/02-setup for specific settings to use for many common editor choices.)
The following setting generates better patches than the default:
git config --global diff.algorithm histogram\nThe following setting makes it easier to resolve conflicts if you're doing conflict resolution by hand as opposed to with a dedicated conflict resolution tool. Without this setting, you'll just see your version and the version you're merging in delimited by conflict markers. With this setting, you'll also see the common ancestor of the two sides of the merge, which can make it much easier to figure out how to resolve the conflict:
git config --global merge.conflictstyle diff3\nAlternatively, look into using a graphical conflict-resolution tool such as kdiff3 or the Emacs built-in M-x vc-resolve-coflicts.
We recommend that you set git to not push anything by default:
git config --global push.default nothing\nThis can help prevent you from accidentally pushing work to the wrong repository.
"},{"location":"development/git-basics/#configuring-git-on-shared-machines","title":"Configuring git on shared machines","text":"If using git on shared resources, such as on the login nodes for CISL machines, then one may find their git commands being killed by sys admins due to git spawning too many threads and thus blocking (or at least slowing down) other users. To avoid this situation, you can limit the number of threads git spawns for various activities by setting the following git config variables:
git config --global --add index.threads 8\ngit config --global --add grep.threads 8\ngit config --global --add pack.threads 8\nPlease note that a limit of 8 threads was chosen specifically for CISL machines. If you are using a separate shared system you may find it beneficial to choose a different thread limit.
"},{"location":"development/git-basics/#git-tools-for-bash","title":"git tools for Bash","text":"There are two helpful things you can do to streamline your use of git if you use the bash shell. These are completely optional, but improve your git experience. These are documented in the appendix of the excellent Pro Git book.
"},{"location":"development/git-basics/#updating-your-login-to-a-github-token","title":"Updating your login to a GitHub token","text":"If you have been denied access to push to a GitHub repo, note that the old password system has been deprecated and no longer works. If you do not make changes, you will no longer be able to work with GitHub on personal forks and private repos.
If you only ever git clone from public repositories like ESCOMP and ESMCI, you may ignore the rest of this wiki page.
If you've already created and are using a GitHub token, you may ignore the rest of this wiki page.
GitHub will soon be requiring that you use a GitHub generated token (basically a long password that they autogenerate for you).
Store your credentials on every machine you use GitHub on.
On a private / secure Linux system:
git config --global credential.helper store
Note that the store option to git's credential helper stores your GitHub token in plain text. If you are on a public machine, you should store your GitHub token in a password manager and use the credential cache mechanism to open command-line access to your GitHub repositories:
git config --global credential.helper cache --timeout <seconds>
where <seconds> is the number of seconds git 'remembers' your token. For example, to only have to enter your token once per day:
git config --global credential.helper cache --timeout 86400
On Windows:
git config --global credential.helper wincred - On Macs (make sure you select the Mac tab in the middle of the window): see this documentation
On each machine, do a git clone of your personal repo. It will ask for your username and password. The password needs to be your saved token. You should only need to do this once.
git clone of your personal repo. If it clones without asking for your username/password you are done.Here are some commands for creating and working with clones:
"},{"location":"development/git-basics/#create-a-new-clone","title":"Create a new clone","text":"git clone https://github.com/<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\nor
git clone git@github.com:<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\nwhere <GitHub userid> is your GitHub account login ID. Some useful options to the clone command are:
--origin <origin name>: A clone knows where it came from and by default, calls that location, \"origin\". You can change that name with the --origin option. This can come in handy when dealing with multiple upstream repositories. Change the clone directory name: By default, the clone is created in a directory with the same name as the repository. You can change this by adding a directory name to the clone command. Use the appropriate example below:git clone https://github.com/<GitHub userid>/CAM-SIMA <clone_dir_name>\ncd <clone_dir_name>\nor
git clone git@github.com:<GitHub userid>/CAM-SIMA <clone_dir_name>\ncd <clone_dir_name>\ngit checkout <tag>\nnote that <tag> can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., origin/branch_name), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see Working with branches
When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits).
git branch <new branch name> <tag or branch name>\nfor example
git branch new_feature cam6_2_024\ngit checkout <new branch name>\ngit-fleximod (e.g., CAM-SIMA, CESM), always run that tool after checking out a new branch or tag:bin/git-fleximod update\nWhile working with clones created using the methods above will be sufficient for most if not all of your development needs, there may be times when you will want to access or compare your code with code from a different repository. git has no problem storing revisions from multiple repositories in a single clone!
To begin, your clone probably has a single remote (also known as an upstream repository). To see the current status of which upstream repositories are configured for your clone, use the git remote command:
git remote\nTo see the location of the remote repositories in your current directory:
git remote -v\nYou should see something like:
origin https://github.com/gituser/CAM-SIMA (fetch)\norigin https://github.com/gituser/CAM-SIMA (push)\nThis tells you the \"upstream\" location from where new code is downloaded (when you run git fetch origin) or where code is uploaded (when you run git push origin <branch>). Note that most git commands are purely local, using only information in the .git directory of your clone.
You can rename an existing remote:
git remote rename origin ESCOMP\nYou can set the remote name as part of a clone command (the default is 'origin'):
git clone -o ESCOMP https://github.com/ESCOMP/cam-sima\nTo add a new upstream repository, use the remote add command. For example:
git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA\ngit fetch --tags ESCOMP\nYou should see messages much like a new clone when you execute the git fetch command. Note that you can call the new remote anything, in this example we are calling it ESCOMP.
Note that while this section explains how to update your local branch to the ESCOMP/CAM-SIMA/development branch, the instructions can easily be generalized for any branch from any upstream remote.
Before starting, you should have either:
git commit and that git status shows no modified files).Add the upstream remote, if you have not already done so (see Adding remotes).
Merge the specific remote/branch into your branch. In this example, it is ESCOMP/development
git fetch ESCOMP\ngit merge ESCOMP/development\nIf you have a git clone, you can view differences between commits or tags. As far as git diff is concerned, a commit hash is the same as a tag so in the examples below will use <tag>.
git diff <tag1> <tag2>\n git diff <tag>\n git diff --name-only <tag> [ <tag2> ]\n git diff <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\ngit has a command, difftool, that can run a graphical tool on each file that is different between two commits.
opendiff as the graphical difference tool: git config --global diff.tool opendiff\n git difftool --tool-help\ndifftool on <file1> and <file2> git difftool <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\ndifftool on all files in a changeset (answer 'y' or 'n' for each file): git difftool <tag> [ <tag2> ]\ndifftool on all files in a changeset (i.e., same as 'y' for every file): yes | git difftool <tag> [ <tag2> ]\ndiffmerge","text":"This section contains the complete modifications needed for using the graphical tool diffmerge for \"git difftool\" and \"git mergetool\"
There is a tool called diffmerge which enables both side-by-side comparison of edited files in git as well as providing a three way-pane for editing merge conflicts. This tool is available for download at: https://sourcegear.com/diffmerge/. It has been installed on izumi and derecho by the system administrators in public areas, so you don't need to download it for those machines.
To use the differencing tool type: git difftool
If after a git merge the git command says there are conflicts, then you may type git mergetool to allow you to resolve the conflicts and complete the merge. The mergetool with diffmerge properly installed will display three panes. From left to right these panes are:
Code being merged in Merged code Your code\nThe panes are opened in a visual editor, and any changes you make in the middle pane, may be saved by selecting the save icon at the top of the screen and then exiting the window. This will finalize the merge for that particular file.
For a presentation which can be used as a tutorial, you may refer to: Presentation on diffmerge tool for git
The following modifications may be copy/pasted into the user's .gitconfig file which resides in the home directory. Since the potential for making an editing mistake is possible, it is recommended that a copy be made of the .gitconfig file prior to these edits in case an error is discovered.
If you have problems, check out the tips at the bottom of this page
[diff]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 algorithm = histogram\n[difftooldiffmerge]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool]\n\u00a0 \u00a0 \u00a0 \u00a0 prompt = false\n[push]\n\u00a0 \u00a0 \u00a0 \u00a0 default = simple\n[merge]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 ff = false\n[mergetool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge --merge --result=$MERGED $REMOTE $BASE $LOCAL\n[mergetool]\n\u00a0 \u00a0 \u00a0 \u00a0 keepBackup = false\nUseful tips
ncarenv/23.09 or newer (to check say \"module list\")A: git clone <path_name> [<local_dir>] where <path_name> is the location of the source git clone and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).
A: git clone ssh://<user>@<machine>:<path_name> [<local_dir>] where <user> is your user name on the remote machine, <machine> is the machine name (e.g., derecho.hpc.ucar.edu), <path_name> is the location of the source git clone on the remote machine, and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).
A: There a a few ways to do this:
git diff or git difftool with your existing branches or development)A first step is to find the link to the fork's branch. Just below the PR is a line that starts with a colored oval (e.g., \"Open\" in green) and looks something like:
Octocat wants to merge 123 commits into ESCOMP:development from Octocat:amazing-new-feature\nClicking on the last part (Octocat:amazing-new-feature) will take you to the correct branch where you can browse the code (the first method above). If you want to download that code, click the green \"Code\" button and then click the clipboard icon. Be sure to take note of the branch name.
git remote add octocat https://github.com/octocat/CAM-SIMA.git\n git fetch --no-tags octocat\n git checkout octocat/amazing-new-feature\nInstead of the checkout you can also do diffs:
git difftool origin/development octocat/amazing-new-feature\n git clone -b amazing-new-feature octocat https://github.com/octocat/CAM-SIMA.git octocat_cam\n cd octocat_cam-sima\nA: A code review must fulfill three purposes:
The first two steps are usually completed by a single SE although SEs engaged in a final review will often find missed errors. This is similar to peer reviewers finding problems with a paper even after reviews done by colleagues.
"},{"location":"development/git-faq/#q-how-do-i-update-my-branch-to-the-current-cam_development","title":"Q: How do I update my branch to the current cam_development?","text":"A: see this section
"},{"location":"development/git-fleximod/","title":"git-fleximod","text":""},{"location":"development/git-fleximod/#populate-andor-update-your-externals","title":"Populate and/or update your externals","text":"Run bin/git-fleximod update
The process will either:
bin/git-fleximod update -f to force overwrite all externalsTo add or update an external repo to CAM-SIMA, the following steps must be done:
.gitmodules file at the head of CAM-SIMA to add or update the external. Explanations for what each entry in the .gitmodules file is can be found on Github here.gitmodules file has been updated, go to the head of CAM-SIMA and run bin/git-fleximod update. This will bring in the new external code into your local repo (but will not commit them)..gitmodules file, and the updated submodule itself. An easy way to make sure you have commited everything is to run git status and make sure there are no files or directories that have been modified but are still un-staged.Once all of that is done then congrats! A new external has been successfully added/updated.
"},{"location":"development/tool-recommendations/","title":"Tool recommendations","text":"This page lists recommendations for various tools one might use during CAM-SIMA development.
Please note that these standards are currently in development, and thus any of them could change at any time.
"},{"location":"development/tool-recommendations/#version-control","title":"Version Control","text":"The recommended version control tool is git.
"},{"location":"development/tool-recommendations/#repository-hosting","title":"Repository Hosting","text":"We recommend Github for hosting software repositories
"},{"location":"development/tool-recommendations/#cicd","title":"CI/CD","text":"When possible, we recommend running any CI/CD workflows using Github Actions.
"},{"location":"development/tool-recommendations/#container-software","title":"Container Software","text":"We recommend using Docker for building and running containers.
More to Come!
"},{"location":"usage/creating-a-case/","title":"Creating, configuring, and running a case","text":"Because CAM-SIMA uses the CIME build system, creating and configuration a case is mostly the same as it was in CAM7 (and in CESM). But! We're going to assume you know nothing.
"},{"location":"usage/creating-a-case/#1-run-create_newcase","title":"1. Runcreate_newcase","text":"Before you create a case:
bin/git-fleximod update to update and/or populate your externals$CAM-SIMA/cime/scriptsAssuming you have completed the above steps and have a nice checkout of a development branch of CAM-SIMA, let's go!
Here is the (basic) structure of the command for create_newcase:
./create_newcase --case <CASEDIR> --compset <COMPSET_NAME> --res <RESOLUTION> --compiler <COMPILER> --project <PROJECT KEY> --run-unsupported\nTo break that down piece by piece:
--case <CASEDIR> (optional): The case argument specifies the full path to the directory you would like to be created, in which your case will reside.run and bld directories will be automatically placed within the case directory--case /glade/derecho/scratch/<username>/<case_name>--case /scratch/cluster/<username>/<case_name>$CAM-SIMA/cime/scripts directory (and, eventually, your run and bld directories will default to a new directory in your scratch space)--compset <COMPSET_NAME> (required): the compset (or \"component set\") tells CIME which base-level configurations you want to run with (including model versions and input data sets)create_newcase command can be either:$CAM-SIMA/cime_config/config_compsets.xml)<initialization time>_<atmosphere model>_<land model>_<sea-ice model>_<ocean model>_<river runoff model>_<land ice model>_<wave model>--res <RESOLUTION> (required): the resolution specifies both the dycore being run and the grid.<atmosphere>_<ocean>_<mask>$CAM-SIMA/ccs_config/modelgrid_aliases_nuopc.xmlne3pg3_ne3pg3_mg37 or ne5_ne5_mg37--compiler <COMPILER> (optional): the compiler you wish to use--project <PROJECT KEY>: a project key to charge to--run-unsupported: this flag is mostly in place to indicate to scientists that the configuration they're choosing to run is not scientifically supported.Given all that, let's say you run the following command on izumi (assuming you're this mysterious person \"courtneyp\"):
./create_newcase --case /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 --compset FKESSLER --res ne5_ne5_mg37 --compiler gnu --run-unsupported\nWhat will happen is that a new case directory will be created here: /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 that will be configured to run the FKESSLER compset (long name: 2000_CAM%KESSLER_SLND_SICE_SOCN_SROF_ SGLC_SWAV) with the SE dycore and the ne5 grid.
Navigate into your shiny new case directory and run ./case.setup
From here, there are A LOT of configuration options. We'll highlight a few here that you'll use often.
"},{"location":"usage/creating-a-case/#xml-configurations","title":"XML configurations","text":"Many configurable settings can be found within the env_*.xml files in your case directory. In order to protect again typos, it's not advised to edit those directly. Instead, you will run ./xmlchange commands to alter those settings (and ./xmlquery can give you the current setting). You can find additional information on the configurable xml variables here. A few to highlight:
CAM_CONFIG_OPTS: This is where we tell CAM-SIMA what physics scheme(s) we wish to run, as well as if we wish to run the null dycore.--physics-suites kessler --analytic_ic, which means we're running the suite_kessler.xml SDF with the SE dycore (with analytic initial conditions - no need to supply an ncdata initial conditions file)./xmlchange CAM_CONFIG_OPTS = \"--physics-suites kessler --dyn none\"CAM_LINKED_LIBS: Hopefully, you won't have to change this often; however, if you are getting linking errors during your build, try turning off the linked libraries (./xmlchange CAM_LINKED_LIBS='')DOUT_S: This is the flag to archive log files of successful runs. During development, we advise turning this off so your log files don't disappear on you (./xmlchange DOUT_S=False)STOP_OPTION & STOP_N: How long you are going to run the model. If STOP_N is 8 and STOP_OPTION is \"ndays\", you are setting the model up to run for 8 days.STOP_OPTION are: 'nsteps', 'nseconds', 'nminutes', 'nhours', 'ndays', 'nmonths', 'nyears'STOP_N is an integerDEBUG: A flag to turn on debug mode for the run (runs without compiler optimizations) - this is very useful during development! It defaults to \"False\", so you can turn it on with ./xmlchange DEBUG=TrueRUNDIR: this is the path to the run directory for your case. The bld directory will exist one level up.If you run ./xmlchange, the output will tell you if you need to re-set up and/or do a clean build of your case.
There are countless namelist configuration options. These can be modified by updating the user_nl_cam file in your case directory. Namelist options can be updated without rebuilding the case.
Run ./case.build
The console output will tell you the progress of the build. If you get to MODEL BUILD HAS FINISHED SUCCESSFULLY, hooray!
Debugging tips
"},{"location":"usage/creating-a-case/#4-run-the-case","title":"4. Run the case","text":"Run ./case.submit
The job will be submitted to the system's queue. You can see the status of your job with the qstat command. Once it is finished, the log files will be in the run directory (unless it ran successfully AND your archiver is on)
Debugging tips
"},{"location":"usage/history/","title":"History & model output","text":""},{"location":"usage/history/#configuring-the-namelist","title":"Configuring the namelist","text":"The syntax for updating a the configuration for a history file is:
<namelist_option>;<volume>: <setting>\nPossible namelist options for your user_nl_cam:
hist_add_avg_fields: add average fields to a specified history filehist_add_inst_fields: add instantaneous fields to a specified history filehist_add_min_fields: add minimum fields to a specified history filehist_add_max_fields: add maximum fields to a specified history filehist_remove_fields: remove a given field from a specified history filehist_file_type: type of file (options are \"history\", \"satellite\", and \"initial_value\") - defaults to \"history\"hist_max_frames: maximum number of samples written to a specified history file (after which a new one will be created)hist_output_frequency: frequency with which to write samples to a specified history file (syntax is <integer>*<multiplier> like 3*nhours)hist_precision: precision of the specified history file (options are \"REAL32\" and \"REAL64\") - defaults to \"REAL32\"hist_write_nstep0: logical for whether or not to write the nstep = 0 sample (defaults to .false.)hist_filename_template: template filename for the specified history fileExample Take the following sample user_nl_cam:
hist_output_frequency;h1: 5*ndays\nhist_max_frames;h1: 3\nhist_add_inst_fields;h1: U\nhist_add_inst_fields;h1: V, Q\nhist_precision;h1: REAL64\nhist_filename_spec;h1: my-history-file%m-%d\nhist_write_nstep0;h1: .false.\nIt will be parsed by hist_config.py and this will be the relevant section of atm_in:
&hist_config_arrays_nl\n hist_num_inst_fields = 3\n hist_num_avg_fields = 2\n hist_num_min_fields = 0\n hist_num_max_fields = 0\n hist_num_var_fields = 0\n/\n\n&hist_file_config_nl\n hist_volume = 'h0'\n hist_avg_fields = 'T', 'Q'\n hist_max_frames = 1\n hist_output_frequency = '1*month'\n hist_precision = 'REAL32'\n hist_file_type = 'history'\n hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'\n hist_write_nstep0 = .false.\n/\n\n&hist_file_config_nl\n hist_volume = 'h1'\n hist_inst_fields = 'U', \u2018V\u2019, \u2018Q\u2019\n hist_max_frames = 3\n hist_output_frequency = '5*ndays'\n hist_precision = 'REAL64'\n hist_file_type = 'history'\n hist_filename_spec = 'my-history-file%m-%d'\n hist_write_nstep0 = .false.\n/\nIn plain English, a one-month run with these history configuration will result in a total of three files that will look something like these:
During init time, fields can be added to the possible field list with a call to history_add_field:
history_add_field(diagnostic_name, standard_name, vdim_name, avgflag, units, gridname, flag_xyfill, mixing_ratio)
Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will be the name in netcdf output file standard_name No string CCPP standard name for the variable vdim_name No string vertical dimension: 'horiz_only' for no vertical dimension; 'lev' for vertical_layer_dimension; 'ilev' for vertical_interface_dimension avgflag No string default average flag; options: 'avg', 'lst' (instantaneous), 'min', 'max', 'var' (standard deviation) units No string variable units gridname Yes string gridname on which the variable's data is mapped (defaults to the physics grid) flag_xyfill Yes string fill value for variable values mixing_ratio Yes string constituent mixing ratio type ('wet' or 'dry'); not set to anything if not passed inExample:
call history_add_field('Q', 'water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water', 'lev', 'avg', 'kg kg-1', mixing_ratio='wet')\nIt's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_add_field call will be in the init phase.
After init time, a variable's current values can be captured for output with a call to history_out_field:
history_out_field(diagnostic_name, field_values)
Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will cause an error if the diagnostic name was not added via history_add_field field_values No real variable values to be stored in the history buffer(s)Example:
call history_out_field('Q', const_array(:,:,const_idx))\nIt's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_out_field call(s) will likely be in the run phase.
The output files can be found in your run directory. They are in netCDF format.
See the ADF for lots that you can do with your results!
A few useful commands for inspecting netCDF data:
ncdump -h file.nc\nncdump -v <varname> file.nc\ncprnc <file1_path> <file2_path>\nncks -F -d time,<start time>,<end time>,<step> <snapshot_file> <output_file_name>\nExample to get time samples 3-5 from a file onto a new file called \"split-file.nc\":
ncks -F -d time,3,5,1 file.nc split-file.nc\nncdiff -v <field_name> -o <output_file> <file1> <file2>\n*cprnc can be found:
/glade/p/cesmdata/cseg/tools/cime/tools/cprnc/cprnc/fs/cgd/csm/tools/cime/tools/cprnc/cprncTo make runs in either CAM or CAM-SIMA the commands are identical:
./create_newcase\u2026 #(the commands will differ and are specified later in this document)\ncd YourCaseDirectory\n./xmlchange\u2026 #(specific commands are specified later in this document)\n./case.setup\n./case.build\n./case.submit\nDepending on which machine you are on, you may prefer to run the ./case.build command on a compute node instead of the login node due to user resource utilization limits on the login nodes.
For more detailed information on case creation and building, see https://ncar.github.io/CAM/doc/build/html/users_guide/building-and-running-cam.html
"},{"location":"conversion/ccpp-conversion-guide/#prep-work","title":"Prep Work","text":""},{"location":"conversion/ccpp-conversion-guide/#conversion-spreadsheet","title":"Conversion Spreadsheet","text":"Put the parameterization that you are going to convert into the conversion spreadsheet https://docs.google.com/spreadsheets/d/1_1TTpnejam5jfrDqAORCCZtfkNhMRcu7cul37YTr_WM/edit#gid=0
"},{"location":"conversion/ccpp-conversion-guide/#create-github-issues","title":"Create Github Issues","text":"Create a Github Issue in the ESCOMP/CAM repo that states which physics parameterization you are planning to convert to the CCPP framework. Then create another issue in the ESCOMP atmospheric physics repo describing the same physics parameterization that you are now planning to add to the collection of NCAR CCPP physics suites. Doing this allows the software engineers to keep track of which physics routines are being worked on, and which still need to be assigned. The goal of converting the physics parameterization is to ultimately have the CCPP-ized physics package reside in ESCOMP atmospheric physics and be removed from ESCOMP/CAM.
"},{"location":"conversion/ccpp-conversion-guide/#setting-up-your-sandbox","title":"Setting up your sandbox","text":"Make sure you have github forks for both ESCOMP/CAM-SIMA and ESCOMP/atmospheric_physics. If needed see https://github.com/ESCOMP/Cam/wiki/CAM-Development-Workflow-in-GitHub#how-to-makestore-revisions-to-your-personal-cam-repository-github-fork
To begin, fork ESCOMP/CAM-SIMA:
And select the Create new fork option. This will bring you to the \"Create new fork\" screen:
Uncheck the \\\"Copy the main branch only\\\" option
Failure to uncheck this will prevent you from pulling in updates from the development branch easily.
As you make changes and want to commit them to your github repos, you will be managing two separate repos. When you issue git commands, be aware of where you are in your code tree. If you want to see changes in CAM-SIMA, you can issue a git status in the main CAM-SIMA directory. If you want to see changes in the atmospheric_physics repo, make sure you are in src/physics/ncar_ccpp before you issue the git status command. All other git commands will be relative to your current working directory as well.
In order to describe the build process, we need to define several source and build directories:
<srcroot> : The CAM-SIMA sandbox being built. From a case directory, this can be found with ./xmlquery SRCROOT.<caseroot>: The location of the case being built.<src_mods>: The location of the CAM-SIMA source mods, located at <caseroot>/SourceMods/src.cam.<bldroot> : The location of CAM-SIMA generated code and compiled code (.o and .mod). See atm/obj under ./xmlquery OBJROOT.Given the context above, the following is the CAM-SIMA build sequence:
ConfigCAM object, used to store build configuration information. Code can be found in <srcroot>/cime_config/cam_config.py<case>: A CIME case object<case_log>: A python logger, which is usually created by CIME itselfconfig_dict: A dictionary of configure options, with the dictionary keys being the name of the config variable/option, and the dictionary values being special config variable objects that contain the following properties:name: The name of the config option/variable. This is what is used as the key in the config_dict dictionarydesc: A text description of what this particular config option is, and what sort of values the config option should have.value: The value of that config option/variable. Currently can only be an integer, string, or list of integers or strings.valid_vals: Optional property that contains either a list or range of valid values for this particular config entryvalid_type: Optional property for a list-type config option that states what data type the list elements can have.is_nml_attr: Logical that states whether this config option can be used as an XML attribute in a namelist definition file.ConfigCAM object also has various build-in methods that are used to call other steps in the build workflow, change the value of a given config variable, or print extra config info to the provided python logger.ConfigCAM object currently has no uses outside buildnml and buildlib, and so is not saved after those steps are complete.BuildCacheCAM object (<srcroot>/cime_config/cam_build_cache.py).<build_cache>: An optional cache file (created by previous build). This file is created in the case build directory (bld/atm/obj).The diagram above displays everything that occurs when CAM-SIMA's buildnml is called by CIME, which occurs after the user calls preview_namelists, case.build, or case.submit.
All blue boxes represent source code, and are listed as \"source file: function/object used\", and all objects or functions that have a \"+\" are automatically tested whenever a Pull Request is opened, updated, or merged.
All orange boxes are input files that are used by the code, while all green boxes are output files that are generated. It is important to note that additional files are generated as well, specifically a build cache and a CCPP datatable, but those files are used internally in the build scripts shown here and not by the final model executable, and thus aren't listed here.
Finally, the arrows show the order of operations, starting with buildnml, with the top two source code boxes representing python classes that are used by the functions/objects directly below them.
*Static images can be found at the bottom of this page
"},{"location":"design/cam-run-process/#cam-sima-api","title":"CAM-SIMA API","text":"Upon running ./case.submit the core CAM-SIMA driver code* is in $CAM-SIMA/src/control/cam_comp.F90. This section lays out each of the subroutines within cam_comp.F90. The subroutines in cam_comp.F90 are set up to mirror the phases of the Common Community Physics Package (CCPP).
* cam_comp.F90 subroutines are called by the NUOPC cap: $CAM-SIMA/src/cpl/nuopc/atm_comp_nuopc.F90
cam_init sets up the metadata and configuration objects/modules for the run. It is called once at startup. Below are the variables passed in:
eccen in cam_control_mod obliqr (in) Earth's obliquity in radians used to set module-level obliqr in cam_control_mod lambm0 (in) Mean longitude of perihelion at the vernal equinox (radians) used to set module-level lambm0 in cam_control_mod mvelpp (in) Earth's moving vernal equinox longitude of perihelion plus pi (radians) used to set module-level mvelpp in cam_control_mod perpetual_run (in) flag to determine if perpetual mode is enabled passed to time manager init perpetual_ymd (in) perpetual year, month, day (YYYYMMDD) used to determine the sun position and interpolate boundary data sets passed to time manager init dtime (in) model timestep size in seconds passed to time manager init start_ymd (in) start date (YYYYMMDD) passed to time manager init start_tod (in) start time of day (sec) passed to time manager init ref_ymd (in) reference date (YYYYMMDD) (defaults to start_ymd) passed to time manager init ref_tod (in) reference time of day (sec) (defaults to start_tod) passed to time manager init stop_ymd (in) stop date (YYYYMMDD) passed to time manager init stop_tod (in) stop time of day (sec) passed to time manager init curr_ymd (in) current date (YYYYMMDD) (same as start date) passed to time manager init curr_tod (in) current time of day (sec) (same as start tod) passed to time manager init cam_in (inout) surface exchange object - coupler to CAM-SIMA allocated if this is an initial run cam_out (inout) surface exchange object - CAM-SIMA to coupler allocated if this is an initial run ** For additional information on run types, see the CESM Tutorial
cam_init calls the following key subroutines (locations) in this order:
cam_ctrl_init (src/control/cam_control_mod.F90): Sets the module-level run configuration variables; logs configurations to the atm logcam_ctrl_set_orbit (src/control/cam_control_mod.F90): Sets the module-level orbital variablestimemgr_init (src/utils/time_manager.F90`): Initializes the time manager; logs configurations to the atm logread_namelist (src/control/runtime_opts.F90): Reads all namelists for the run, including auto-generated scheme namelists (see build process)cam_ctrl_set_physics_type (src/control/cam_control_mod.F90): sets module-level configuration for variables for simple physics and moist physics schemes; logs configurations to atm logcam_initfiles_open (src/control/cam_initfiles.F90): Opens initial or restart file, and topography file if specifiedcam_register_constituents (src/control/cam_comp.F90): Sets the total number and advected number of constituents; currently ALWAYS adds water vapor as constituent (expected by the SE dycore)air_composition_init (src/data/air_composition.F90): Initializes air-composition-dependent model constantsmodel_grid_init (src/dynamics/<dycore>/dyn_grid.F90): Initializes model grids and decompositionscam_ccpp_initialize_constituents ($CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90): initializes the constituent data array; after this point, we cannot add new constituentsdyn_init (src/dynamics/<dycore>/dyn_comp.F90): Initializes the dynamical coreatm2hub_alloc and hub2atm_alloc (src/control/camsrfexch.F90): Allocates and sets up surface exchange dataphys_init (src/physics/utils/phys_comp.F90): Initializes physics (includes call to CCPP cap to run init phases of schemes in the Suite Definition File (SDF)stepon_init (src/dynamics/<dycore>/stepon.F90): Initializes dynamics <--> physics couplingcam_timestep_init is called at the start of each timestep. It has no input/output/inout variables.
The routine calls the following subroutines (locations) in this order:
stepon_timestep_init (src/dynamics/<dycore>/stepon.F90): First phase of dynamics (couple from dynamics to physics); also returns timestep for physicsphys_timestep_init (src/physics/utils/phys_comp.F90):src/data/registry.xml) from the ncdata filecam_run1 is the first \"run\" phase called in the physics loop. It is called every timestep BEFORE the mediator/surface coupler and calls the following subroutine (location):
phys_run1 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_before_coupler\" group in the SDFcam_run2 is the second \"run\" phase called in the physics loop. It is called every timestep AFTER the mediator/coupler. Input/output variables:
cam_run2 calls these subroutines (locations):
phys_run2 (src/physics/utils/phys_comp.F90): Calls the run phase for all physics schemes in the \"physics_after_coupler\" group in the SDFstepon_run2 (src/dynamics/<dycore>/stepon.F90): The second phase of dynamics (couple from physics to dynamics)cam_run3 is the third \"run\" phase called in the physics loop. It is called every timestep AFTER cam_run3 and BEFORE cam_run4 (unsurprisingly). In/out variables:
cam_run3 calls the following subroutine (location):
stepon_run3 (src/dynamics/<dycore>/stepon.F90): Calls dyn_run, which runs the dycorecam_run4 currently does nothing! (but it is called every timestep)
cam_timestep_final runs at the end of each timestep. In/out variables:
cam_timestep_final calls the following subroutines (locations):
history_write_files (src/history/cam_history.F90): Writes fields to user-configured history files (if applicable)history_wrap_up (src/history/cam_history.F90): Closes files and zeros buffers as necessaryphys_timestep_final (src/physics/utils/phys_comp.F90):ncdata_check is set in user_nl_cam, calls physics_check_data ($CASE/bld/atm/obj/phys_init/physics_inputs.F90) to perform snapshot checkingcam_final is called once at the end of the model execution. In/out variables:
cam_final calls the following subroutines (locations):
phys_final (src/physics/utils/phys_comp.F90): calls \"final\" phase of all schemes in the SDFstepon_final (src/dynamics/<dycore>/stepon.F90): finalizes dycore (doesn't currently do anything)atm2hub_deallocate and hub2atm_deallocate (src/control/camsrfexch.F90): deallocate cam_in/cam_out objectsThe core Common Community Physics Package (CCPP) documentation can be found here. This section details the code structure and implementation of the CCPP Framework within CAM-SIMA. That said here's a quick overview of the CCPP:
CCPP-compliant physics schemes must adhere to the following criteria:
dependencies, but we're not getting into that now):Metadata example (snippet taken from kessler.meta)
[ precl ]\n standard_name = total_precipitation_rate_at_surface\n long_name = Total precipitation rate at surface\n units = m s-1\n dimensions = (horizontal_loop_extent)\n type = real | kind = kind_phys\n intent = out\n[ relhum ]\n standard_name = relative_humidity\n long_name = Relative humidity\n units = percent\n dimensions = (horizontal_loop_extent, vertical_layer_dimension)\n type = real | kind = kind_phys\n intent = out\n[ scheme_name ]\n standard_name = scheme_name\n units = none\n type = character | kind = len=64\n dimensions = ()\n intent = out\n\nCCPP-compliant physics schemes are organized into suite definition files (SDFs). An SDF tells the framework which schemes will be run in what order. Separating schemes into \"groups\" also allows the run phases of those groups to be called separately by the host model. Here's an example SDF (from suite_kessler.xml):
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<suite name=\"kessler\" version=\"1.0\">\n <group name=\"physics_before_coupler\">\n <scheme>calc_exner</scheme>\n <scheme>temp_to_potential_temp</scheme>\n <scheme>calc_dry_air_ideal_gas_density</scheme>\n <scheme>wet_to_dry_water_vapor</scheme>\n <scheme>wet_to_dry_cloud_liquid_water</scheme>\n <scheme>wet_to_dry_rain</scheme>\n <scheme>kessler</scheme>\n <scheme>potential_temp_to_temp</scheme>\n <scheme>dry_to_wet_water_vapor</scheme>\n <scheme>dry_to_wet_cloud_liquid_water</scheme>\n <scheme>dry_to_wet_rain</scheme>\n <scheme>kessler_update</scheme>\n <scheme>qneg</scheme>\n <scheme>geopotential_temp</scheme>\n <scheme>cam_state_diagnostics</scheme>\n <scheme>kessler_diagnostics</scheme>\n </group>\n <group name=\"physics_after_coupler\">\n <scheme>cam_tend_diagnostics</scheme>\n </group>\n</suite>\nThe framework code is primarily python code that generates Fortran caps. The class structure looks like:
Given CCPP-compliant physics schemes and one or more SDF, the framework generates caps for the host model to call at the appropriate time. The core files generated by the framework are:
<host>_ccpp_cap.F90: contains the interface layer between the host and the suite(s)ccpp_<suite>_cap.F90: contains one subroutine per phase (including one run phase per group) in which the phases of the schemes within the suite are called in orderccpp_datatable.xml: consolidates metadata into an XML file to be used as desired by the host modelHow CAM-SIMA and the CCPP come together:
"},{"location":"design/ccpp-in-cam-sima/#host-model","title":"Host Model","text":"The core host model code is what is held in the CAM-SIMA github repository, plus code that is generated at build-time (or preview_namelists-time) based on the registry (src/data/registry.xml).
The CCPP physics scheme code exists in the atmospheric_physics repository, which exists as a submodule of CAM-SIMA in the following location: $CAM-SIMA/src/physics/ncar_ccpp
SDFs are located in the root directory of the repository and scheme source code is in the relevant subdirectories.
The diagnostics directory contains all diagnostic schemes (the global ones used for state and tendency output, as well as the scheme-specific diagnostic schemes).
The to_be_ccppized directory contains physics schemes and utilties that have not been CCPP-ized, but were needed by an CCPP-ized scheme.
The utilities directory contains schemes that are used regularly, such as tendency applicators and state converters. See below for more.
The caps generated by the CCPP Framework at model build time (or preview_namelists-time) can be found in the following location: $CASE/bld/atm/obj/ccpp/
All CCPP phases are called from the physics driver (src/physics/utils/phys_comp.F90). You can see the order of these calls more thoroughly in the documented run sequence.
For a CCPP-ized physics scheme to work, the framework needs to be able to find a matching variable on the host side for each input variable for the suite(s) in question. This means that CAM-SIMA needs to allocate, initialize, and provide metadata for these variables. We do this in two ways:
src/data/physconst.F90) so that a scheme can access an existing host model variablesrc/data/registry.xml)The registry-based code generator is run before the CCPP framework does its magic, so, when it's time, the framework can connect the dots between the host model and the physics.
"},{"location":"design/ccpp-in-cam-sima/#state-and-tendency-variables","title":"State and tendency variables","text":"Two of the most commonly used and referred-to objects in CAM-SIMA are:
The Fortran for both objects is auto-generated by CAM-SIMA based on the registry ($CAM-SIMA/src/data/registry.xml). The generated code can be found here: $CASE/bld/atm/obj/cam_registry/physics_types.F90. The objects are used by the host model (CAM-SIMA), while the CCPP physics take the individual component variables as inputs.
The physics_state object in CAM-SIMA contains the current values for a select set of variables that describe the atmosphere, at the resolution specified by the input grid.
Some examples of these core \"state\" variables include temperature (T), eastward wind (U), and northward wind (V)
As a rule, CAM-SIMA physics schemes do not update the state directly and instead return tendencies (see below) which are then applied to the state later in the run phase. This is called time splitting, which means that all physics since the last state update get the same input state. The alternative, process splitting, means that the output state of one scheme serves as the input state of the next.
NOTE: Although constituents are handled independently of the physics_state object (they are handled by the CCPP framework), they ARE considered state variables.
The physics tendencies represent how a given scheme (or schemes) changes the state in a single model timestep. The tendencies are accumulated until it is time to apply them to the state. There is one tendency for each state variable being \"updated\" by the scheme. Some of these tendency variables are held within the physics_tend object, but others are internal to the physics.
The module $CAM-SIMA/src/physics/ncar_ccpp/utilities/physics_tendency_updaters.F90 includes the schemes to apply the tendencies to the relevant state variables. These schemes are added to the SDF whenever the state should be updated. Each calculation looks like: state_var = state_var + tend*dt where dt is the timestep size.
Some definitions to start (as written by a non-scientist, so there is more nuance than this!):
src/data/air_composition.F90qneg will set constituent values that are less than the minimum to the minimumThere are three ways to identify a quantity as a constituent in CAM-SIMA:
cam_register_constituents (in src/control/cam_comp.F90). Currently, we are always adding water vapor as a host constituent because it is expected by the SE dycore.advected = Truesrc/physics/ncar_ccpp/musica/micm/micm.F90The registration and initializaiton of the constituent data array and the constituent properties object are done through calls to the generated CCPP cap.
ccpp_model_constituents_t objectccpp_model_constituents_t objectConstituent values and properties can be accessed from the host side and from the physics in the following ways:
cam_constituents.F90 module, which is an interface to the CCPP cap, which is in turn an interface to the constituents object[ q ]\n standard_name = ccpp_constituents\n units = none\n type = real | kind = kind_phys\n dimensions = (horizontal_loop_extent,vertical_layer_dimension,number_of_ccpp_constituents)\n intent = inout\n[ const_props ]\n standard_name = ccpp_constituent_properties\n units = None\n type = ccpp_constituent_prop_ptr_t\n dimensions = (number_of_ccpp_constituents)\n intent = in\nThis section can be removed when constituents are documented in the CCPP Framework documentation.
"},{"location":"design/constituents/#constituent-object-fortran","title":"Constituent object (Fortran)","text":"The constituent object (found in $CAM-SIMA/ccpp_framework/src/ccpp_constituent_prop_ mod.F90) is a flexible and extendable means of containing necessary constituent data for the framework. The primary object is ccpp_model_constituents_t.
This object, importantly, contains the following properties (for which there is metadata; CCPP standard name in parenthesis):
const_metadata (ccpp_constituent_properties)num_layer_vars (number_of_ccpp_constituents)num_advected_vars (number_of_ccpp_advected_constituents)vars_layer (ccpp_constituents)The const_metadata property is of type ccpp_constituent_prop_ptr_t, which contains a pointer to a ccpp_constituent_properties_t object, as depicted above. This object contains all of the constituent properties for the constituents in the constituents array, with the same indices as the constituents array.
The ccpp_model_constituents_t type also contains a hash table of constituent properties for more efficient searching, as well as several methods used by the generated cap code. Some methods are highlighted below:
The constituents-related code generation routines provide an interface to the constituents object. These routines can be found in $CAM-SIMA/ccpp_framework/scripts/constituents.py, primarily within the \u201cwrite_host_routines\u201d function. The (most often used) generated routines related to constituents are:
The routines above are generated during ./preview_namelists or ./case.build and can be found here: $CASE/bld/atm/obj/ccpp/cam_ccpp_cap.F90
CAM-SIMA history is the mechanism for configuring and generating diagnostic output from a model run. It is also used to generate initial-data files and aids in the model-restart process by saving the state of diagnostic fields whose processing window (e.g., averaging, standard deviation) crosses a restart-write cycle. This page describes the implementation of CAM-SIMA history in CAM-SIMA.
"},{"location":"design/history/#history-initialization","title":"History Initialization","text":""},{"location":"design/history/#reading-and-processing-the-history-configuration","title":"Reading and processing the history configuration","text":"cime_config/hist_config.py in _HIST_CONFIG_ENTRY_TYPES.user_nl_cam)<keyword>;<volume>: <value> (see examples below)atm_in equivalent indicates how hist_config.py parses these into a namelist to be read by SIMA):hist_config.py also contains the HistoryVolConfig class (all the info pertaining to a single history file), the HistoryConfig class (all the history configuration information including a dict of HistoryVolConfig objects), and helper classes.HistoryConfig object is created in buildnml out of entries in user_nl_cam and written to run/atm_in.atm_in, the HistoryConfig object must contain all the logic in setting default values.src/history/cam_hist_file.F90cam_hist_file.F90 is hist_read_namelist_config, which is called by history_readnl in src/history/cam_history.F90. This function reads in the hist_config_arrays_nl namelist group, allocates the history field arrays, and then uses those arrays to read in the hist_file_config_nl namelist group (via a call to read_namelist_entry).cam_history.F90\u2019s module-level hist_configs array of hist_file_t objects (the size of this array is the number of user-configured volumes).hist_file_t object contains information about the configuration options for a given history volume. This includes the maximum number of frames that can be written to the file, the current number of frames on the file, the name of the file, and all of the history fields to be written to the file. It also contains methods to populate the field lists (config_set_up_fields), set up the metadata of the file (config_define_file), and write history fields to the file (config_write_time_dependent_variables).hist_file_t object contains both a hash table and allocatable field list to keep track of the fields written to the file. The core class for each of these is the hist_field_info_t (in src/history/buffers/src/hist_field.F90), which contains information about a history field. This includes the field names, the accumulate flag (average, instantaneous, minimum, etc), units, type, dimensions, and fill value. It also includes the buffer(s) (type is/are hist_buffer_t) that will and do hold the actual data.The possible fields to be output by the history infrastructure are tracked in cam_history.F90 via the possible_field_list hash table. It is populated during init time by calls to the subroutine history_add_field (found in src/history/cam_history.F90). \u201cInit time,\u201d means that all calls to history_add_field must occur during the execution of cam_init (found in src/control/cam_comp.F90).
dyn_init or stepon_initsrc/physics/ncar_ccpp/diagnostics/cam_diagnostics.F90physics_before_coupler group in the suite definition file (SDF)cam_tend_diagnostics scheme in src/physics/ncar_ccpp/diagnotics/cam_diagnostics.F90physics_after_coupler group in the SDFsrc/physics/ncar_ccpp/diagnostics Each call to history_add_field adds a new field to the end of the possible_field_list_head linked list. At the end of cam_init, the possible field list linked list is used to print the list (to the atm.log* file) and then is converted to the possible_field_list hash table. A sample of the history field list is seen below.
***************** HISTORY FIELD LIST ******************\n T K avg air_temperature\n ZM m avg geopotential_height_wrt_surface\n PHIS m2 s-2 ins surface_geopotential\n PMID Pa avg air_pressure\n PDELDRY Pa avg air_pressure_thickness_of_dry_air\n Q kg kg-1 avg water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water\n CLDLIQ kg kg-1 avg cloud_liquid_water_mixing_ratio_wrt_moist_air_and_condensed_water \n RAINQM kg kg-1 avg rain_mixing_ratio_wrt_moist_air_and_condensed_water\n TTEND K s-1 avg tendency_of_air_temperature_due_to_model_physics\n *************** END HISTORY FIELD LIST ****************\nOutside of CAM-SIMA init and finalize time, history buffers can be populated with data via a call to history_out_field (found in src/history/cam_history.F90)
The subroutine history_out_field iterates over the hist_configs array and populates the buffer(s) of the hist_field_info_t object of the hist_file_t object if the field name in the call is active on that file (e.g. the field was configured via the namelist to be output for that volume).
history_out_field can exist anywhere except _init and _finalhistory_out_field can exist anywhere except in dyn_init, stepon_init, and stepon_finalhistory_out_field calls are included in the run phase of the same schemes described in the section above.The cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:
The bolded step #2 above is what determines if we need to define a new history file. The situations where we would need to define a new file are:
hist_max_frames) and closed itWe determine if it's time for a new file with the following line of code:
mod(num_samples, hist_configs(file_idx)%max_frame()) == 0\nIf it is indeed time to define a new file, we call the config_define_file subroutine (which is found in src/history/cam_hist_file.F90) for the volume:
call hist_configs(file_idx)%define_file(restart, logname, host, model_doi_url)\nThe cam_history.F90 subroutine history_write_files (which is called during cam_timestep_final) does three main actions for each of the user-defined history volumes:
The bolded step #3 above occurs any time the criteria for #1 is satisfied. At this point, the following call is made to write the history fields (whose data has been stored in their buffers via calls to history_out_field):
call hist_configs(file_idx)%write_time_dependent_variables(file_idx, restart)\nIt is during this call that we increment the number of samples written for this volume and actually write the data held within the buffer(s) to the netcdf file (as well as the time-dependent metadata for the fields).
"},{"location":"design/history/#defining-history-restart-files","title":"Defining history restart files","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/history/#writing-a-history-restart-file","title":"Writing a history restart file","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/history/#reading-a-history-restart-file","title":"Reading a history restart file","text":"Restarts not yet implemented in CAM-SIMA
"},{"location":"design/sima-design-goals/","title":"Design goals & features","text":"Motivated by the Singletrack project (a precursor to SIMA), the CAM-SIMA project was created to build a new CAM infrastructure to meet the SIMA science and computational needs. This page documents those needs and some of the features that implement them.
"},{"location":"design/sima-design-goals/#cam-needs-to-be-more-run-time-configurable","title":"CAM needs to be more run-time configurable","text":"The chunk structure in CAM physics served its purpose when threading was the only way to accelerate code. However, to make the best use of both threading and modern accelerators, a flexible chunking mechanism is required. The new infrastructure enables this by using flat arrays for all fields.
In order to continue to allow CAM to grow without an ever increasing cost of bringing in new features, CAM must be more modular. A classic example is chemistry which ACOM would like to make modular but which is currently entwined with CAM in many areas (e.g., code in CAM repository, extensive configuration code dedicated to chemistry, extensive namelist building code and data dedicated to chemistry, large number of use cases containing chemistry configuration data). The new CAM infrastructure contains features to increase modularity.
Modularity will allow CAM to move components to external repositories. This process cuts development costs for both CAM and for the component (e.g., as has happened with PUMAS). Some ways this is accomplished are listed here:
Use of the CCPP to build physics suites also makes CAM more modular because the CCPP treats physics schemes as modular which allows flexibility in building physics suites. The CCPP takes care of making sure variables are available before they are used and also builds the code currently handled via hand-written code in the various versions of physpkg.F90.
"},{"location":"design/sima-design-goals/#run-time-data-checking","title":"Run-time data checking","text":"CAM needs data to run but the data needs vary with the simulation. The new infrastructure facilitates this.
CAM currently has a few ways to run offline testing (e.g., SCAM, PORT). The new infrastructure builds these capabilities in for more efficient and flexible use.
To enable efficient quality control, the new infrastructure implements a number of continuous integration (CI) techniques.
The standards in this document are largely prescriptive, i.e., they should be followed.
While some legacy code will not follow these rules, efforts SHOULD be made to improve the code whenever you are working on it (e.g., bug fixes, enhancements).
"},{"location":"development/cam-coding-standards/#general-coding-standards","title":"General coding standards","text":""},{"location":"development/cam-coding-standards/#must","title":"MUST","text":"See tips for configuring editors
"},{"location":"development/cam-coding-standards/#should","title":"SHOULD","text":"We expect all python code to pass with a perfect score (10) using pylint with this version of pylintrc. However, external repos may have their own pylintrc version depending on their needs.
We also expect all python files to follow the black code style and format.
Finally, one can also follow the Google Python Style Guide.
"},{"location":"development/cam-coding-standards/#fortran-coding-standards","title":"Fortran coding standards","text":"The standards described in this section represent the CAM Fortran standards. Other Fortran standards:
use statementsif statements (i.e., all if statements should have a then if the statement is on more than one line)-HUGE(1), real: NaN, character: 'UNSET').pure keyword.intent for dummy arguments except for pointers.1.5_r8, not 1.5 or 1.5D0. Literals must also include the decimal point.character(len=*) or character(len=CL)).implicit none statement in the preamble (after the use statements). Module routines then do not need this statement.call subroutine(x, optional_y=y) instead of call subroutine(x, y) for the optional variable optional_y).SAVE attribute and is not thread safe.nullify statement.pcols, even if only a subset of the variable is used in the subroutine or function. #if, #ifdef). Always try for runtime variable logic instead.pure keyword if a subroutine has no side effects.==, /=, <, >=) not old character versions (e.g., .eq.).private. Public interfaces are declared after the private declaration.private module interfaces (i.e., subroutines and functions) should be declared private in the module header..F90).pure attribute. If they cannot, the reason should be included as a comment in the function's preamble.return at the end of a subroutine).use statements should be brought in at the smallest scope possible (e.g. inside individual subroutines instead of at the module level).module, subroutine, if, do), indent that scope relative to the scoping statement (recommended 3 spaces but each module should at least be self consistent).if, else, end, do, and while.::only, i.e., only:, not only :..emacs file)","text":"(add-hook 'before-save-hook 'delete-trailing-whitespace)\n(setq-default indent-tabs-mode nil)\n(setq tab-width 4)\n.vimrc file)","text":" autocmd BufWritePre * :%s/\\s\\+$//e\nThis page describes the various automated and manual tests that are run for CAM-SIMA whenever the code is modified, as well as instructions for how to add new tests.
"},{"location":"development/cam-testing/#python-unit-testing","title":"Python unit testing","text":"CAM-SIMA supports two kinds of python unit tests, doctest and unittest tests, both of which are part of the standard python library.
All unittest tests should be in:
CAM-SIMA/test/unit
while all files used by the tests should be in:
CAM-SIMA/test/unit/sample_files
All unittest tests are automatically run via Github Actions whenever a Pull Request (PR) is opened, modified, or merged.
All doctest tests are also run automatically as long as the scripts they are located in are under CAM-SIMA/cime_config or CAM-SIMA/src/data.
To manually run all of the unit tests at any time, simply run the following shell script:
CAM-SIMA/test/run_tests.sh
Finally, when adding new tests, determine if the test can be done in only a few lines with minimal interaction with external files or variables. If so, then it would likely be best as a doctest. Otherwise it should be a unittest test. Failure to follow this rule of thumb could result in test failures in the Github Actions workflow. Also remember to add your new tests to the run_tests.sh script so future users can easily run the tests manually.
Any python script which is added or modified via a PR will automatically be analyzed using pylint, and must have a score of 9.5 or greater in order to not be marked as a test failure. The hidden pylintrc file used for the analysis can be found here:
CAM-SIMA/test/.pylintrc
Users can also manually run pylint against the core python build scripts by running the following shell script:
CAM-SIMA/test/pylint_test.sh
Please note that pylint is not part of the standard python library, and so it may need to be installed before being able to run the shell script.
NOTE: Regression testing on Derecho should be done for every PR before merging!
Users can manually run regression tests on Derecho to ensure that the model builds correctly in various configurations. The tests can be run with a local copy of CAM-SIMA by using the test_driver.sh script under $CAM-SIMA/test/system. To run the tests associated with a particular compiler option one can do the following commands:
For running GNU tests*:
env CAM_FC=gnu ./test_driver.sh -f\nFor running Intel tests*:
env CAM_FC=intel ./test_driver.sh -f\nRunning the script will produce a directory in your scratch space labeled aux_sima_<CAM_FC>_<timestamp>, where <CAM_FC> is the compiler you chose, and <timestamp> is the timestamp (starting with the date) of when the tests were started, along with a job submitted to the local batch system.
Inside the directory you should see an executable labeled cs.status.*. Running that command after the submitted job has finished will display the test results. Currently for all tests everything should be labeled PASS except the SUBMIT step, which should be labeled PEND. Any other label indicates that a test may have failed, and should be investigated.
Finally, the tests themselves are listed in <CAM-SIMA>/cime_config/testdefs/testlist_cam.xml. Any files that need to be included in order for the tests to run properly are located in <CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/outfrq_XXX, where XXX is the name of the test. Additional information on the CIME testing system, which is what this testing infrastructure is built on, can be found online here.
*Note: you may also have to include the environment variable CAM_ACCOUNT on derecho, which points to your account key
The test list can be found here: $CAM-SIMA/cime_config/testdefs/testlist_cam.xml
<machine> XML entry<test> XML entry with the following structure:<test compset=\"<COMPSET_NAME>\" grid=\"<GRID_ALIAS>\" name=\"<TEST_TYPE>_<TEST_MOD>\" testmods=\"<RELPATH_TO_TESTMODS_DIR>\">\n <machines>\n <machine name=\"<MACH_NAME>\" compiler=\"<COMPILER>\" category=\"<TEST_CATEGORY>\"/>\n </machines>\n <options>\n <option name=\"comment\">COMMENT HERE</option>\n <option name=\"wallclock\">WALLCLOCK_TIME</option>\n </options>\n</test>\n<COMPSET_NAME>: component set alias (or long name) - you can see more about compsets here<GRID_ALIAS>: model grid/resolution you'd like to run on - you can see more about grids here<TEST_TYPE>: type of test to be run. You can find the testing types here.<TEST_MOD>: test modifier that changes the default behavior of the test type. More here<RELPATH_TO_TESTMODS_DIR>: relative path to the testmods directory for this run; usually looks something like \"cam/some_directory_name/\"user_nl_cam and/or shell_commands)$CAM-SIMA/cime_config/testdefs/testmods_dirs/cam/<MACH_NAME>: machine name - will almost definitely be either derecho or izumi<COMPILER>: compiler to be used (options: gnu, nag, intel, nvhpc)<TEST_CATEGORY>: group of tests that this test belongs to - the default run by test_driver.sh is aux_sima (which is run for each PR to CAM-SIMA)WALLCLOCK_TIME: maximum amount of time that the job will be allowed to runHere is an example test entry for a 2-timestep smoke test of kessler physics on the MPAS grid, run with both intel and gnu
<test compset=\"FKESSLER\" grid=\"mpasa480_mpasa480\" name=\"SMS_Ln2\" testmods=\"cam/outfrq_kessler_mpas_derecho_nooutput/\">\n <machines>\n <machine name=\"derecho\" compiler=\"intel\" category=\"aux_sima\"/>\n <machine name=\"derecho\" compiler=\"gnu\" category=\"aux_sima\"/>\n </machines>\n <options>\n <option name=\"wallclock\">00:10:00</option>\n <option name=\"comment\">GNU build test for MPAS dycore (with Kessler physics)</option>\n </options>\n </test>\nStart with the CAM wiki.
"},{"location":"development/debugging/#cam-sima-specific-debugging","title":"CAM-SIMA-specific debugging","text":""},{"location":"development/debugging/#build-errors","title":"Build errors","text":"Debugging tips if you get build errors:
bld/atm.bldlog.*PETXXX - if there was an issue on the ESMF side of things, it will show up in one of these (there will be one PET file per processor)./xmlchange NTASKS=1), do a clean rebuild (as instructed), and run again; maybe running in serial will identify the error***************** HISTORY FIELD LIST ****************** in the atm.log* file; if it's not there, the error occurred at init timeCAM-SIMA time step advanced message in the atm.log* file), it's likely that one or more variable that you introduced or modified has gone off the rails (value has become very large or very small or zero)development and:development) to identify where the numbers are going awry OR$CAM-SIMA/tools/find_max_nonzero_index.F90)$CAM-SIMA/tools/find_max_nonzero_index.F90(COURTNEY - ASK CHERYL TO HELP WITH THIS)
./case.build)bash to change to bash (if not already)np=1\nnthreads=1\n\nsource .env_mach_specific.sh\n\nRUNDIR=`./xmlquery RUNDIR -value`\nEXEROOT=`./xmlquery EXEROOT -value`\nLID=`date '+%y%m%d-%H%M%S'`\n\ncd $RUNDIR\nmkdir timing\nmkdir timing/checkpoints\necho `pwd`\nexport OMP_NUM_THREADS=$nthreads\ntotalview ${EXEROOT}/cesm.exe\nexit to exit the totalview window and give up the node Git also offers extensive built-in (i.e., command line) help, although it can be hard to understand until you are familiar with basic git concepts:
git help
git help COMMAND
man gittutorial
man giteveryday
"},{"location":"development/git-basics/#how-to-set-up-your-git-development-environment","title":"How to set up your git development environment","text":"There are several stages in setting up your git and GitHub development environment. Below, they are broken into three sections which represent the different stages:
Create a new personal CAM-SIMA fork of the ESCOMP/CAM-SIMA repo:
Set up SSH keys on GitHub (optional):
If you will be pushing changes to GitHub (either to your own fork or to shared forks), or if you will be pulling changes from private repositories on GitHub, then it's worth the time to set up ssh keys for each machine you'll be using. By doing so, you won't have to enter your password when pushing to or pulling from GitHub.
See here for instructions. After doing this, you can use the ssh form of GitHub URLs (e.g., git@github.com:ESCOMP/cam-sima.git) in place of the https form.
Configure GitHub notifications (optional): It is important to keep track of activity on GitHub but there are ways to manage how you are notified.
git has global settings which apply to all repository clones on your machine. These settings reside in a file called .gitconfig in your home directory. Below are some required and some optional but recommended global git settings to apply to any new machine where you will do CAM-SIMA development (e.g., Derecho, Izumi, personal laptop). Apply the settings below or simply copy a .gitconfig file from a machine that is already configured.
git config --global user.name \"Your Name\"\ngit config --global user.email <GitHub email address>\nWe recommend that you use your UCAR email address as your GitHub email address but if you use another address, you can add your UCAR email address by clicking on your profile picture in the upper-right of any GitHub page, then clicking on \"Settings\", and then on \"Emails\".
"},{"location":"development/git-basics/#recommended-git-global-configuration-settings","title":"Recommended git global configuration settings","text":"You can set which editor to use for log messages, etc., with:
git config --global core.editor <editor of your choice: emacs, vi, vim, etc>\n(See http://swcarpentry.github.io/git-novice/02-setup for specific settings to use for many common editor choices.)
The following setting generates better patches than the default:
git config --global diff.algorithm histogram\nThe following setting makes it easier to resolve conflicts if you're doing conflict resolution by hand as opposed to with a dedicated conflict resolution tool. Without this setting, you'll just see your version and the version you're merging in delimited by conflict markers. With this setting, you'll also see the common ancestor of the two sides of the merge, which can make it much easier to figure out how to resolve the conflict:
git config --global merge.conflictstyle diff3\nAlternatively, look into using a graphical conflict-resolution tool such as kdiff3 or the Emacs built-in M-x vc-resolve-coflicts.
We recommend that you set git to not push anything by default:
git config --global push.default nothing\nThis can help prevent you from accidentally pushing work to the wrong repository.
"},{"location":"development/git-basics/#configuring-git-on-shared-machines","title":"Configuring git on shared machines","text":"If using git on shared resources, such as on the login nodes for CISL machines, then one may find their git commands being killed by sys admins due to git spawning too many threads and thus blocking (or at least slowing down) other users. To avoid this situation, you can limit the number of threads git spawns for various activities by setting the following git config variables:
git config --global --add index.threads 8\ngit config --global --add grep.threads 8\ngit config --global --add pack.threads 8\nPlease note that a limit of 8 threads was chosen specifically for CISL machines. If you are using a separate shared system you may find it beneficial to choose a different thread limit.
"},{"location":"development/git-basics/#git-tools-for-bash","title":"git tools for Bash","text":"There are two helpful things you can do to streamline your use of git if you use the bash shell. These are completely optional, but improve your git experience. These are documented in the appendix of the excellent Pro Git book.
"},{"location":"development/git-basics/#updating-your-login-to-a-github-token","title":"Updating your login to a GitHub token","text":"If you have been denied access to push to a GitHub repo, note that the old password system has been deprecated and no longer works. If you do not make changes, you will no longer be able to work with GitHub on personal forks and private repos.
If you only ever git clone from public repositories like ESCOMP and ESMCI, you may ignore the rest of this wiki page.
If you've already created and are using a GitHub token, you may ignore the rest of this wiki page.
GitHub will soon be requiring that you use a GitHub generated token (basically a long password that they autogenerate for you).
Store your credentials on every machine you use GitHub on.
On a private / secure Linux system:
git config --global credential.helper store
Note that the store option to git's credential helper stores your GitHub token in plain text. If you are on a public machine, you should store your GitHub token in a password manager and use the credential cache mechanism to open command-line access to your GitHub repositories:
git config --global credential.helper cache --timeout <seconds>
where <seconds> is the number of seconds git 'remembers' your token. For example, to only have to enter your token once per day:
git config --global credential.helper cache --timeout 86400
On Windows:
git config --global credential.helper wincred - On Macs (make sure you select the Mac tab in the middle of the window): see this documentation
On each machine, do a git clone of your personal repo. It will ask for your username and password. The password needs to be your saved token. You should only need to do this once.
git clone of your personal repo. If it clones without asking for your username/password you are done.Here are some commands for creating and working with clones:
"},{"location":"development/git-basics/#create-a-new-clone","title":"Create a new clone","text":"git clone https://github.com/<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\nor
git clone git@github.com:<GitHub userid>/CAM-SIMA\ncd CAM-SIMA\nwhere <GitHub userid> is your GitHub account login ID. Some useful options to the clone command are:
--origin <origin name>: A clone knows where it came from and by default, calls that location, \"origin\". You can change that name with the --origin option. This can come in handy when dealing with multiple upstream repositories. Change the clone directory name: By default, the clone is created in a directory with the same name as the repository. You can change this by adding a directory name to the clone command. Use the appropriate example below:git clone https://github.com/<GitHub userid>/CAM-SIMA <clone_dir_name>\ncd <clone_dir_name>\nor
git clone git@github.com:<GitHub userid>/CAM-SIMA <clone_dir_name>\ncd <clone_dir_name>\ngit checkout <tag>\nnote that <tag> can also be the name of a branch or a commit hash. If you specify the name of a branch, you will check out the head of the branch. If you name a remote branch (e.g., origin/branch_name), you will create a detached HEAD but you can still use the code. Please note that if you plan on changing the code, first create a branch (see Working with branches
When you create a clone, your clone will contain pointers all the branches that existed at the clone's origin (e.g., the repository at GitHub). While you can check out these branches, however, before attempting to make any changes, you should first create a local version branch (so git can keep track of the local commits).
git branch <new branch name> <tag or branch name>\nfor example
git branch new_feature cam6_2_024\ngit checkout <new branch name>\ngit-fleximod (e.g., CAM-SIMA, CESM), always run that tool after checking out a new branch or tag:bin/git-fleximod update\nWhile working with clones created using the methods above will be sufficient for most if not all of your development needs, there may be times when you will want to access or compare your code with code from a different repository. git has no problem storing revisions from multiple repositories in a single clone!
To begin, your clone probably has a single remote (also known as an upstream repository). To see the current status of which upstream repositories are configured for your clone, use the git remote command:
git remote\nTo see the location of the remote repositories in your current directory:
git remote -v\nYou should see something like:
origin https://github.com/gituser/CAM-SIMA (fetch)\norigin https://github.com/gituser/CAM-SIMA (push)\nThis tells you the \"upstream\" location from where new code is downloaded (when you run git fetch origin) or where code is uploaded (when you run git push origin <branch>). Note that most git commands are purely local, using only information in the .git directory of your clone.
You can rename an existing remote:
git remote rename origin ESCOMP\nYou can set the remote name as part of a clone command (the default is 'origin'):
git clone -o ESCOMP https://github.com/ESCOMP/cam-sima\nTo add a new upstream repository, use the remote add command. For example:
git remote add ESCOMP https://github.com/ESCOMP/CAM-SIMA\ngit fetch --tags ESCOMP\nYou should see messages much like a new clone when you execute the git fetch command. Note that you can call the new remote anything, in this example we are calling it ESCOMP.
Note that while this section explains how to update your local branch to the ESCOMP/CAM-SIMA/development branch, the instructions can easily be generalized for any branch from any upstream remote.
Before starting, you should have either:
git commit and that git status shows no modified files).Add the upstream remote, if you have not already done so (see Adding remotes).
Merge the specific remote/branch into your branch. In this example, it is ESCOMP/development
git fetch ESCOMP\ngit merge ESCOMP/development\nIf you have a git clone, you can view differences between commits or tags. As far as git diff is concerned, a commit hash is the same as a tag so in the examples below will use <tag>.
git diff <tag1> <tag2>\n git diff <tag>\n git diff --name-only <tag> [ <tag2> ]\n git diff <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\ngit has a command, difftool, that can run a graphical tool on each file that is different between two commits.
opendiff as the graphical difference tool: git config --global diff.tool opendiff\n git difftool --tool-help\ndifftool on <file1> and <file2> git difftool <tag> [ <tag2> ] -- <path_to_file1> <path_to_file2>\ndifftool on all files in a changeset (answer 'y' or 'n' for each file): git difftool <tag> [ <tag2> ]\ndifftool on all files in a changeset (i.e., same as 'y' for every file): yes | git difftool <tag> [ <tag2> ]\ndiffmerge","text":"This section contains the complete modifications needed for using the graphical tool diffmerge for \"git difftool\" and \"git mergetool\"
There is a tool called diffmerge which enables both side-by-side comparison of edited files in git as well as providing a three way-pane for editing merge conflicts. This tool is available for download at: https://sourcegear.com/diffmerge/. It has been installed on izumi and derecho by the system administrators in public areas, so you don't need to download it for those machines.
To use the differencing tool type: git difftool
If after a git merge the git command says there are conflicts, then you may type git mergetool to allow you to resolve the conflicts and complete the merge. The mergetool with diffmerge properly installed will display three panes. From left to right these panes are:
Code being merged in Merged code Your code\nThe panes are opened in a visual editor, and any changes you make in the middle pane, may be saved by selecting the save icon at the top of the screen and then exiting the window. This will finalize the merge for that particular file.
For a presentation which can be used as a tutorial, you may refer to: Presentation on diffmerge tool for git
The following modifications may be copy/pasted into the user's .gitconfig file which resides in the home directory. Since the potential for making an editing mistake is possible, it is recommended that a copy be made of the .gitconfig file prior to these edits in case an error is discovered.
If you have problems, check out the tips at the bottom of this page
[diff]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 algorithm = histogram\n[difftooldiffmerge]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n[difftool]\n\u00a0 \u00a0 \u00a0 \u00a0 prompt = false\n[push]\n\u00a0 \u00a0 \u00a0 \u00a0 default = simple\n[merge]\n\u00a0 \u00a0 \u00a0 \u00a0 tool = diffmerge\n\u00a0 \u00a0 \u00a0 \u00a0 ff = false\n[mergetool \"diffmerge\"]\n\u00a0 \u00a0 \u00a0 \u00a0 cmd = diffmerge --merge --result=$MERGED $REMOTE $BASE $LOCAL\n[mergetool]\n\u00a0 \u00a0 \u00a0 \u00a0 keepBackup = false\nUseful tips
ncarenv/23.09 or newer (to check say \"module list\")A: git clone <path_name> [<local_dir>] where <path_name> is the location of the source git clone and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).
A: git clone ssh://<user>@<machine>:<path_name> [<local_dir>] where <user> is your user name on the remote machine, <machine> is the machine name (e.g., derecho.hpc.ucar.edu), <path_name> is the location of the source git clone on the remote machine, and the optional <local_dir> is the name of the clone (default is a local directory of the same name as the original).
A: There a a few ways to do this:
git diff or git difftool with your existing branches or development)A first step is to find the link to the fork's branch. Just below the PR is a line that starts with a colored oval (e.g., \"Open\" in green) and looks something like:
Octocat wants to merge 123 commits into ESCOMP:development from Octocat:amazing-new-feature\nClicking on the last part (Octocat:amazing-new-feature) will take you to the correct branch where you can browse the code (the first method above). If you want to download that code, click the green \"Code\" button and then click the clipboard icon. Be sure to take note of the branch name.
git remote add octocat https://github.com/octocat/CAM-SIMA.git\n git fetch --no-tags octocat\n git checkout octocat/amazing-new-feature\nInstead of the checkout you can also do diffs:
git difftool origin/development octocat/amazing-new-feature\n git clone -b amazing-new-feature octocat https://github.com/octocat/CAM-SIMA.git octocat_cam\n cd octocat_cam-sima\nA: A code review must fulfill three purposes:
The first two steps are usually completed by a single SE although SEs engaged in a final review will often find missed errors. This is similar to peer reviewers finding problems with a paper even after reviews done by colleagues.
"},{"location":"development/git-faq/#q-how-do-i-update-my-branch-to-the-current-cam_development","title":"Q: How do I update my branch to the current cam_development?","text":"A: see this section
"},{"location":"development/git-fleximod/","title":"git-fleximod","text":""},{"location":"development/git-fleximod/#populate-andor-update-your-externals","title":"Populate and/or update your externals","text":"Run bin/git-fleximod update
The process will either:
bin/git-fleximod update -f to force overwrite all externalsTo add or update an external repo to CAM-SIMA, the following steps must be done:
.gitmodules file at the head of CAM-SIMA to add or update the external. Explanations for what each entry in the .gitmodules file is can be found on Github here.gitmodules file has been updated, go to the head of CAM-SIMA and run bin/git-fleximod update. This will bring in the new external code into your local repo (but will not commit them)..gitmodules file, and the updated submodule itself. An easy way to make sure you have commited everything is to run git status and make sure there are no files or directories that have been modified but are still un-staged.Once all of that is done then congrats! A new external has been successfully added/updated.
"},{"location":"development/tool-recommendations/","title":"Tool recommendations","text":"This page lists recommendations for various tools one might use during CAM-SIMA development.
Please note that these standards are currently in development, and thus any of them could change at any time.
"},{"location":"development/tool-recommendations/#version-control","title":"Version Control","text":"The recommended version control tool is git.
"},{"location":"development/tool-recommendations/#repository-hosting","title":"Repository Hosting","text":"We recommend Github for hosting software repositories
"},{"location":"development/tool-recommendations/#cicd","title":"CI/CD","text":"When possible, we recommend running any CI/CD workflows using Github Actions.
"},{"location":"development/tool-recommendations/#container-software","title":"Container Software","text":"We recommend using Docker for building and running containers.
More to Come!
"},{"location":"usage/creating-a-case/","title":"Creating, configuring, and running a case","text":"Because CAM-SIMA uses the CIME build system, creating and configuration a case is mostly the same as it was in CAM7 (and in CESM). But! We're going to assume you know nothing.
"},{"location":"usage/creating-a-case/#1-run-create_newcase","title":"1. Runcreate_newcase","text":"Before you create a case:
bin/git-fleximod update to update and/or populate your externals$CAM-SIMA/cime/scriptsAssuming you have completed the above steps and have a nice checkout of a development branch of CAM-SIMA, let's go!
Here is the (basic) structure of the command for create_newcase:
./create_newcase --case <CASEDIR> --compset <COMPSET_NAME> --res <RESOLUTION> --compiler <COMPILER> --project <PROJECT KEY> --run-unsupported\nTo break that down piece by piece:
--case <CASEDIR> (optional): The case argument specifies the full path to the directory you would like to be created, in which your case will reside.run and bld directories will be automatically placed within the case directory--case /glade/derecho/scratch/<username>/<case_name>--case /scratch/cluster/<username>/<case_name>$CAM-SIMA/cime/scripts directory (and, eventually, your run and bld directories will default to a new directory in your scratch space)--compset <COMPSET_NAME> (required): the compset (or \"component set\") tells CIME which base-level configurations you want to run with (including model versions and input data sets)create_newcase command can be either:$CAM-SIMA/cime_config/config_compsets.xml)<initialization time>_<atmosphere model>_<land model>_<sea-ice model>_<ocean model>_<river runoff model>_<land ice model>_<wave model>--res <RESOLUTION> (required): the resolution specifies both the dycore being run and the grid.<atmosphere>_<ocean>_<mask>$CAM-SIMA/ccs_config/modelgrid_aliases_nuopc.xmlne3pg3_ne3pg3_mg37 or ne5_ne5_mg37--compiler <COMPILER> (optional): the compiler you wish to use--project <PROJECT KEY>: a project key to charge to--run-unsupported: this flag is mostly in place to indicate to scientists that the configuration they're choosing to run is not scientifically supported.Given all that, let's say you run the following command on izumi (assuming you're this mysterious person \"courtneyp\"):
./create_newcase --case /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 --compset FKESSLER --res ne5_ne5_mg37 --compiler gnu --run-unsupported\nWhat will happen is that a new case directory will be created here: /scratch/cluster/courtneyp/kessler-ne5-gnu-0722 that will be configured to run the FKESSLER compset (long name: 2000_CAM%KESSLER_SLND_SICE_SOCN_SROF_ SGLC_SWAV) with the SE dycore and the ne5 grid.
Navigate into your shiny new case directory and run ./case.setup
From here, there are A LOT of configuration options. We'll highlight a few here that you'll use often.
"},{"location":"usage/creating-a-case/#xml-configurations","title":"XML configurations","text":"Many configurable settings can be found within the env_*.xml files in your case directory. In order to protect again typos, it's not advised to edit those directly. Instead, you will run ./xmlchange commands to alter those settings (and ./xmlquery can give you the current setting). You can find additional information on the configurable xml variables here. A few to highlight:
CAM_CONFIG_OPTS: This is where we tell CAM-SIMA what physics scheme(s) we wish to run, as well as if we wish to run the null dycore.--physics-suites kessler --analytic_ic, which means we're running the suite_kessler.xml SDF with the SE dycore (with analytic initial conditions - no need to supply an ncdata initial conditions file)./xmlchange CAM_CONFIG_OPTS = \"--physics-suites kessler --dyn none\"CAM_LINKED_LIBS: Hopefully, you won't have to change this often; however, if you are getting linking errors during your build, try turning off the linked libraries (./xmlchange CAM_LINKED_LIBS='')DOUT_S: This is the flag to archive log files of successful runs. During development, we advise turning this off so your log files don't disappear on you (./xmlchange DOUT_S=False)STOP_OPTION & STOP_N: How long you are going to run the model. If STOP_N is 8 and STOP_OPTION is \"ndays\", you are setting the model up to run for 8 days.STOP_OPTION are: 'nsteps', 'nseconds', 'nminutes', 'nhours', 'ndays', 'nmonths', 'nyears'STOP_N is an integerDEBUG: A flag to turn on debug mode for the run (runs without compiler optimizations) - this is very useful during development! It defaults to \"False\", so you can turn it on with ./xmlchange DEBUG=TrueRUNDIR: this is the path to the run directory for your case. The bld directory will exist one level up.If you run ./xmlchange, the output will tell you if you need to re-set up and/or do a clean build of your case.
There are countless namelist configuration options. These can be modified by updating the user_nl_cam file in your case directory. Namelist options can be updated without rebuilding the case.
Run ./case.build
The console output will tell you the progress of the build. If you get to MODEL BUILD HAS FINISHED SUCCESSFULLY, hooray!
Debugging tips
"},{"location":"usage/creating-a-case/#4-run-the-case","title":"4. Run the case","text":"Run ./case.submit
The job will be submitted to the system's queue. You can see the status of your job with the qstat command. Once it is finished, the log files will be in the run directory (unless it ran successfully AND your archiver is on)
Debugging tips
"},{"location":"usage/history/","title":"History & model output","text":""},{"location":"usage/history/#configuring-the-namelist","title":"Configuring the namelist","text":"The syntax for updating a the configuration for a history file is:
<namelist_option>;<volume>: <setting>\nPossible namelist options for your user_nl_cam:
hist_add_avg_fields: add average fields to a specified history filehist_add_inst_fields: add instantaneous fields to a specified history filehist_add_min_fields: add minimum fields to a specified history filehist_add_max_fields: add maximum fields to a specified history filehist_remove_fields: remove a given field from a specified history filehist_file_type: type of file (options are \"history\", \"satellite\", and \"initial_value\") - defaults to \"history\"hist_max_frames: maximum number of samples written to a specified history file (after which a new one will be created)hist_output_frequency: frequency with which to write samples to a specified history file (syntax is <integer>*<multiplier> like 3*nhours)hist_precision: precision of the specified history file (options are \"REAL32\" and \"REAL64\") - defaults to \"REAL32\"hist_write_nstep0: logical for whether or not to write the nstep = 0 sample (defaults to .false.)hist_filename_template: template filename for the specified history fileExample Take the following sample user_nl_cam:
hist_output_frequency;h1: 5*ndays\nhist_max_frames;h1: 3\nhist_add_inst_fields;h1: U\nhist_add_inst_fields;h1: V, Q\nhist_precision;h1: REAL64\nhist_filename_spec;h1: my-history-file%m-%d\nhist_write_nstep0;h1: .false.\nIt will be parsed by hist_config.py and this will be the relevant section of atm_in:
&hist_config_arrays_nl\n hist_num_inst_fields = 3\n hist_num_avg_fields = 2\n hist_num_min_fields = 0\n hist_num_max_fields = 0\n hist_num_var_fields = 0\n/\n\n&hist_file_config_nl\n hist_volume = 'h0'\n hist_avg_fields = 'T', 'Q'\n hist_max_frames = 1\n hist_output_frequency = '1*month'\n hist_precision = 'REAL32'\n hist_file_type = 'history'\n hist_filename_spec = '%c.cam.%u.%y-%m-%d-%s.nc'\n hist_write_nstep0 = .false.\n/\n\n&hist_file_config_nl\n hist_volume = 'h1'\n hist_inst_fields = 'U', \u2018V\u2019, \u2018Q\u2019\n hist_max_frames = 3\n hist_output_frequency = '5*ndays'\n hist_precision = 'REAL64'\n hist_file_type = 'history'\n hist_filename_spec = 'my-history-file%m-%d'\n hist_write_nstep0 = .false.\n/\nIn plain English, a one-month run with these history configuration will result in a total of three files that will look something like these:
During init time, fields can be added to the possible field list with a call to history_add_field:
history_add_field(diagnostic_name, standard_name, vdim_name, avgflag, units, gridname, flag_xyfill, mixing_ratio)
Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will be the name in netcdf output file standard_name No string CCPP standard name for the variable vdim_name No string vertical dimension: 'horiz_only' for no vertical dimension; 'lev' for vertical_layer_dimension; 'ilev' for vertical_interface_dimension avgflag No string default average flag; options: 'avg', 'lst' (instantaneous), 'min', 'max', 'var' (standard deviation) units No string variable units gridname Yes string gridname on which the variable's data is mapped (defaults to the physics grid) flag_xyfill Yes string fill value for variable values mixing_ratio Yes string constituent mixing ratio type ('wet' or 'dry'); not set to anything if not passed inExample:
call history_add_field('Q', 'water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water', 'lev', 'avg', 'kg kg-1', mixing_ratio='wet')\nIt's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_add_field call will be in the init phase.
After init time, a variable's current values can be captured for output with a call to history_out_field:
history_out_field(diagnostic_name, field_values)
Field Optional? Type Description diagnostic_name No string diagnostic name for the field - will cause an error if the diagnostic name was not added via history_add_field field_values No real variable values to be stored in the history buffer(s)Example:
call history_out_field('Q', const_array(:,:,const_idx))\nIt's important to avoid adding calls to history_add_field to the CCPP-ized physics schemes (to keep them portable). Instead, create a new diagnostics scheme and place that in the diagnostics folder of the atmospheric_physics repository. The history_out_field call(s) will likely be in the run phase.
The output files can be found in your run directory. They are in netCDF format.
See the ADF for lots that you can do with your results!
A few useful commands for inspecting netCDF data:
ncdump -h file.nc\nncdump -v <varname> file.nc\ncprnc <file1_path> <file2_path>\nncks -F -d time,<start time>,<end time>,<step> <snapshot_file> <output_file_name>\nExample to get time samples 3-5 from a file onto a new file called \"split-file.nc\":
ncks -F -d time,3,5,1 file.nc split-file.nc\nncdiff -v <field_name> -o <output_file> <file1> <file2>\n*cprnc can be found:
/glade/p/cesmdata/cseg/tools/cime/tools/cprnc/cprnc/fs/cgd/csm/tools/cime/tools/cprnc/cprnc