Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SABGOM Access issue - new variables in SABGOM model? #5

Open
kwilcox opened this issue Mar 30, 2017 · 19 comments
Open

SABGOM Access issue - new variables in SABGOM model? #5

kwilcox opened this issue Mar 30, 2017 · 19 comments

Comments

@kwilcox
Copy link
Member

kwilcox commented Mar 30, 2017

I think I've figured out the issue... querying by time used to be the way to pull data, but it appears that the aggregation is now dimensioning some variables using time and others using time1. Were new variables added to the aggregation recently?

The time variable is missing values from 180 to 471 while the time1 variable has those values.

@jbzambon
@rsignell-usgs
Copy link
Collaborator

rsignell-usgs commented Mar 30, 2017 •
edited
Loading

Yep! Looking at the DDS reveals:

    Float32 zeta[time2 = 16131][eta_rho = 320][xi_rho = 440];
    Float32 ubar[time2 = 16131][eta_u = 320][xi_u = 439];
    Float32 vbar[time2 = 16131][eta_v = 319][xi_v = 440];
    Float32 u[time2 = 16131][s_rho = 36][eta_u = 320][xi_u = 439];
    Float32 v[time2 = 16131][s_rho = 36][eta_v = 319][xi_v = 440];
    Float32 w[time2 = 16131][s_w = 37][eta_rho = 320][xi_rho = 440];
    Float32 temp[time2 = 16131][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 salt[time2 = 16131][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 NO3[time = 12076][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 NH4[time = 12076][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 chlorophyll[time1 = 12761][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 phytoplankton[time1 = 12761][s_rho = 36][eta_rho = 320][xi_rho = 440];
    Float32 zooplankton[time1 = 12761][s_rho = 36][eta_rho = 320][xi_rho = 440];

@kwilcox
Copy link
Member Author

kwilcox commented Mar 30, 2017

http://oceanus.meas.ncsu.edu apparently has 3 time dimensions and not 2 like http://omgsrv1.meas.ncsu.edu. @jbzambon which should I be using to pull in the data?

@rsignell-usgs
Copy link
Collaborator

The good news is that netcdf subset does the right thing, if you just specify a time value or time range and you don't to worry about the number of different time variables:

http://oceanus.meas.ncsu.edu:8080/thredds/ncss/fmrc/sabgom_roms/SABGOM_ROMS_Forecast_Model_Run_Collection_best.ncd?var=angle&var=f&var=h&var=mask_psi&var=mask_rho&var=mask_u&var=mask_v&var=pm&var=pn&var=NH4&var=NO3&var=chlorophyll&var=phytoplankton&var=zooplankton&var=ubar&var=vbar&var=zeta&var=salt&var=temp&var=u&var=v&var=w&disableLLSubset=on&disableProjSubset=on&horizStride=1&time=2017-04-02T00%3A00%3A00Z&vertCoord=&addLatLon=true&accept=netcdf4

@kwilcox
Copy link
Member Author

kwilcox commented Mar 31, 2017

@rsignell-usgs @jbzambon Can you confirm that you are seeing incorrect (changed) _FillValue attributes when downloaded from NCSS?

THREDDS says the _FillValue is NaN (which is correct, the data contains NaN as needed) but when downloading from NCSS the resulting files contain a _FillValue of 0.f.

@kwilcox
Copy link
Member Author

kwilcox commented Mar 31, 2017 •
edited
Loading

Neverrrmind, the _FillValue attributes are different/incorrect on
http://oceanus.meas.ncsu.edu:8080/thredds/dodsC/fmrc/sabgom_roms/SABGOM_ROMS_Forecast_Model_Run_Collection_best.ncd.html

units: meter second-1
long_name: u-momentum component
time: ocean_time
coordinates: time2_run time2 s_rho lat_u lon_u 
field: u-velocity, scalar, series
_FillValue: 0.0
standard_name: eastward_sea_water_velocity_assuming_no_tide

@jbzambon
Copy link
Contributor

jbzambon commented Mar 31, 2017 via email

@rsignell-usgs
Copy link
Collaborator

@kwilcox , I put the 1 step file I downloaded from oceanus via NCSS on my THREDDS Server:
http://geoport-dev.whoi.edu/thredds/dodsC/sand/usgs/users/rsignell/data/sabgom/SABGOM_ROMS_Forecast_Model_Run_Collection_best.ncd.nc.html

Some vars like temp and salt have _FillValue: 0.0 but others like ubar have _FillValue: 1.0E37.

I don't think there is a bug.

Things like Godiva2 appear to work just fine:
2017-03-31_14-18-37

@kwilcox
Copy link
Member Author

kwilcox commented Mar 31, 2017

See #5 (comment). The _FillValue for a bunch of variables is "0.0", but NaNs are in the file. 🤷‍♂️, I've never seen current vectors or temperature with a fill value of 0.0 so was asking if it was correct!

@rsignell-usgs
Copy link
Collaborator

rsignell-usgs commented Mar 31, 2017 •
edited
Loading

We often set the fill value to 0.0 for temp, salt, u, v, zeta, etc for older ROMS runs before they actually wrote a fill value. Otherwise the land was covered with 0.0 values. Figured values of exactly 0.0 were unlikely in realistic model runs.

I'm sure I gave @jbzambon those sample ROMS catalogs or perhaps even created them. Perhaps they could be changed now if using more modern version of ROMS.

@kwilcox
Copy link
Member Author

kwilcox commented Mar 31, 2017

@kwilcox
Copy link
Member Author

kwilcox commented Apr 5, 2017

The additional time dimensions have dropped off of the DAP aggregation now and its back to only time: http://omgsrv1.meas.ncsu.edu:8080/thredds/dodsC/fmrc/sabgom/SABGOM_Forecast_Model_Run_Collection_best.ncd.html. Does newer data not have any of the variables that were dimensioned by those?

@jbzambon
Copy link
Contributor

jbzambon commented Apr 5, 2017 via email

@rsignell-usgs
Copy link
Collaborator

Hmm, the start/stop times for SABGOM are currently:
2017-Feb-27 15:00
2017-Apr-08 00:00

Is that correct?

@rsignell-usgs
Copy link
Collaborator

rsignell-usgs commented Apr 5, 2017 •
edited
Loading 

import netCDF4

def start_stop(url,tvar):
    nc = netCDF4.Dataset(url)
    ncv = nc.variables
    time_var = ncv[tvar]
    first = netCDF4.num2date(time_var[0],time_var.units)
    last = netCDF4.num2date(time_var[-1],time_var.units)

    print first.strftime('%Y-%b-%d %H:%M')
    print last.strftime('%Y-%b-%d %H:%M')

url='http://omgsrv1.meas.ncsu.edu:8080/thredds/dodsC/fmrc/sabgom/SABGOM_Forecast_Model_Run_Collection_best.ncd'
tvar='time'
start_stop(url,tvar)

yields

2017-Feb-27 15:00
2017-Apr-08 00:00

@jbzambon
Copy link
Contributor

jbzambon commented Apr 5, 2017 via email

@kwilcox
Copy link
Member Author

kwilcox commented Apr 5, 2017

Will the additional variables that were present in the aggregation last week ever come back? They basically broke the ingestion for a bunch of months and now it is working again since the additional time dimensions have gone away.

@jbzambon
Copy link
Contributor

jbzambon commented Apr 5, 2017 via email

@kwilcox
Copy link
Member Author

kwilcox commented Apr 5, 2017

No worries. Should I move to pulling the files down from http://oceanus.meas.ncsu.edu:8080/thredds/catalog/fmrc/sabgom_roms/catalog.html?dataset=fmrc/sabgom_roms/SABGOM_ROMS_Forecast_Model_Run_Collection_best.ncd instead?

@jbzambon
Copy link
Contributor

jbzambon commented Apr 5, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kwilcox @rsignell-usgs @jbzambon