Remove dependency on external prj2epsg service/api

[zhiyuli]

I pasted here the email conversion I had with Shawn on this issue

Drew to Shawn:
I then checked the error log on server and it seems the prj2epsg service is partially down.

I am looking at your code that send prj string to prj2epsg to get a list of matched epsg dict and then pin a epsg url like http://prj2epsg.org/epsg/3003.json to get more info.
But this http://prj2epsg.org/epsg/XXXX.json endpoint it down now.

Could you remind me again why we need to know the epsg code before we upload data to geoserver? (I am sure you told me the reason before but it is not on the top of my head)

Thanks

Shawn Crawley
Dec 19 (2 days ago)

to me
We need the epsg code because there are certain rasters/shapefiles that had a projection that GeoServer did not recognize, even though they are valid (i.e. recognized by ArcMap and other authorities on projections). In some cases, the prj was missing the EPSG code. Another problem is that some projections have multiple epsg codes (a great example is the Google projection. see https://en.wikipedia.org/wiki/Web_Mercator#Identifiers). This prj2epsg service was a work-around to find the more common epsg code, or at least a closest fit. If the closest fit does not match what is already set as the prj, the new epsg code returned from the prj2epsg service is used to "edit" the file with the correct "corrected" epsg using the gdal_edit function (see https://github.com/hydroshare/tethysapp-hydroshare_gis/blob/master/tethysapp/hydroshare_gis/utilities.py#L1510). The gdal_edit function does not reproject or change the actual underlying coordinate data, it just rewrites the project info stored.

I believe there is a better route, but I took the above route initially because I seem to recall that David Tarboton had an issue with my other idea. However, I still think my other idea would be best. I think the best work-around would be to just reproject EVERY dataset into the Web Mercator (3857) projection or the WSG84 (4326) projection. Whichever one openlayers is using to display the data. GeoServer has to reproject the data on the fly anyway if the data is not already in the requested projection. Also, I think it is the on-the-fly reprojecting that causes some weird artifacts in the tiles. I don't remember exactly why Dr. Tarboton rejected it (and maybe he didn't). In the case that it was not, and is not, seen as a bad idea from anyone, that is the change that I suggest. Then we wouldn't need to depend on the prj2epsg service.

Thanks for looking into it, Drew! Let me know if you have anymore questions.

Zhiyu (Drew) Li [email protected]
Dec 19 (2 days ago)

to Shawn
Thanks I see.
I think reprojecting anything into 3857 is a doable approach. But the tool does the reprojection should recognize the original projection first, right? Apparently ArcMap can do it, but can GDAL or any other open source tool recognize all of these uncommon/strange/weird prj string? Also is Geoserver already using GDAL to handle reprojection under the hood?

Thanks

Zhiyu (Drew) Li [email protected]
Dec 19 (2 days ago)

to Shawn
Looks like geoserver doc recommends reprojection if native prj has no matched EPSG code
http://docs.geoserver.org/2.7.1/user/advanced/crshandling/configurecrs.html

Shawn Crawley
Dec 19 (2 days ago)

to me
Cool. Yeah, I guess I'd go with GeoServer on that one, then. And I'm not sure if they are using GDAL under the hood, but they likely are.

Zhiyu (Drew) Li [email protected]
Dec 19 (2 days ago)

to Shawn
OK. I think we are close.
We should let GIS app use GDAL to test each file
Plan A:
If GDAL can recognize prj file and can find a matched EPSG code, we then upload the file to GeoServer directly (assuming GeoServer uses GDAL and can do the same thing)
If GDAL can recognize prj file but cant find a matched EPSG code, we reproject it into 3857
if GDAL cant recognize prj file we throw out a error (ask user to reproject it in ArcMap)

Plan B
we reproject anything into 3857 using GDAL
throw error once GDAL fails to do it (ask user to reproject it in ArcMap)

what do you think?

Thanks

Shawn Crawley
Dec 20 (1 day ago)

to me
Hm... I see benefits to both plans. However, how do you foresee GDAL being used to test each file for a matched EPSG code? Or how would we determine that "GDAL can recognize prj file but cant find a matched EPSG code"? Are you referring to using gdalinfo and parsing the results (like the app already does) to see if it returns an EPSG code in the projection info? I'd almost lean more towards Plan B, just because I think that running the reproject function (which I think is gdalwarp) on every file is the most straight-forward way of checking if a file "passes the projection test" so to speak. And though it would be a little more overhead (loading time) up front, the rendering of the layers would happen quicker since they wouldn't be reprojected on the fly. However, if you do have a good strategy, the will power, and the time to do Plan A, then I'd say that one would be fine.

Let me know.

[zhiyuli]

Updates/Memos:
The latest GDAL 2.2.3 introduced a new function "FindMatches()" that is supposed to be able to replace prj2epsg service.
OGRSpatialReferenceH * OGRSpatialReference::FindMatches()
http://www.gdal.org/classOGRSpatialReference.html#abb83a2ed838ee5e81a2dba8ac640b63c

I will be playing with it and reporting back later.

[zhiyuli]

A initial fix has been pushed to appsdev server, which makes use of a new function newly introduced in GDAL 2.3. Since GDAL 2.3 is still in development phase (latest stable release is 2.2.3), we should evaluate the potential risk of pushing a pre-release GDAL to our production server.

[zhiyuli]

The external service prj2ersi is still down. Since this issue has been blocking GIS app from normal use on apps server for more than 2 weeks over the break.
I just pushed the fix tested on appsdev to apps server.
Be aware this fix depends on a pre-release GDAL 2.3, which may bring in other unwanted issues.

[zhiyuli]

@martinseul
Your bug report on resource https://www.hydroshare.org/resource/684650ad73404d9bb6de3069cfd14938/
brought our attention to the prj2epsg issue. The said fix has replaced prj2epsg with GDAL2.3, but it didnt fix the issue with this particular resource. GIS apps still can't display this resource correctly as this shapefile is properly kind of uncommon or incompatible to GeoServer.

I was able to open this shapefile in ArcMap (it failed to open in GRASS) .
It contains only one feature object (Polygon type, not MultiPlolygon), but this feature is composed of a numbers of small polygons. I guess GeoServer thinks this is a invalid geometry.

[zhiyuli]

A brief summary and take-away for this issue.

GeoServer requires to know EPSG code for all incoming data
http://docs.geoserver.org/stable/en/user/configuration/crshandling/configurecrs.html

1. Only well-known and commonly used prjections have official EPSG codes. Custom/local projections do not have EPSG codes.

2. OGC “Well-known Text Representation of Spatial Reference Systems”
   Also projection string should include EPSG code in it wherever possible

3. Most shapefiles and Tiff files we see or use were exported or edited by ArcGIS. But ArcGIS stores non-OGC projection string in files, and may not include EPSG code in it.

4. Before sending files to GeoServer, GIS app tries to find a matched OGC-WKT projection string (with EPSG code) for every incoming data:
   4-A) if found, replaces original projection string with this OGC projection string (shapefile: *.prj file; Tiff: metadata header);
   4-B) If not found, sends original file to GeoServer anyway (in some cases, rare but possible, GeoServer is still able to recognize it)

5. GIS app was using an external service, prj2espg, for (4), which has been down for weeks. We find GDAL 2.3 (still in development phase as of this writing) introduced a new function "findMatched()" that can do similar things as prj2epsg service. We have updated GIS app to use this development GDAL. (We need to keep an eye on GDAL github and change to use official stable GDAL 2.3 once it is out)

6. To further improve 4-B, we may try manually re-projecting files into EPSG:3857 (Web Mapping Mercator) before sending them to GeoServer. Be aware of potential performance issue on big files!