Data import enhancements #757

gkahiu · 2022-12-22T18:11:30Z

Incorporates enhancements in #712.

On comparing extents, the implementation uses the geometry of the selected region rather than the bounding box. It compares it (the geometry) with either the bounding box (converted into a geometry) of the input raster file or the outline geometry of the vector file. The tolerance is defined in settings to allow for adjustments if needed. I hope this approach meets the expected requirements.

for more information, see https://pre-commit.ci

…trends.earth into data_import_enhancements

azvoleff

This looks good - just a few comments inline. Also, on first running the plugin with this branch I got the below. I haven't seen that message again, so maybe just appears on first run - but any idea what leads to the below?

2022-12-22T13:32:35     WARNING    Traceback (most recent call last):
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 969, in done
              self.validate_input(value)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 1002, in validate_input
              self.ok_clicked()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 1140, in ok_clicked
              remap_ret = self.remap_raster(out_file, self.dlg_agg.nesting.get_list())
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1250, in remap_raster
              warp_ret = self.warp_raster(temp_tif)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1298, in warp_raster
              target_bounds = self.extent_as_list()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1090, in extent_as_list
              bbox = self.region_selector.region_info.geom.boundingBox()
             AttributeError: 'NoneType' object has no attribute 'boundingBox'

azvoleff · 2022-12-22T18:46:23Z

LDMP/jobs/manager.py

+        uris = None
+
+        if job.results.type == ResultType.RASTER_RESULTS:
+            uris = job.results.get_main_uris()


This should likely be using get_all_uris. get_main_uris may be returning vrts linked to other geotiffs (for example if one of the rasters is a TiledRaster

Done, now using get_all_uris.

azvoleff · 2022-12-22T18:55:47Z

LDMP/data_io.py

+        vector_file = self.lineEdit_vector_file.text()
+        if not vector_file:
+            return None
+
        return qgis.core.QgsVectorLayer(
            self.lineEdit_vector_file.text(), "vector file", "ogr"


this could use vector_file

Corrected thanks.

azvoleff · 2022-12-22T19:01:13Z

LDMP/data_io.py

+
+            source_crs = lyr.crs()
+
+            # Combine the geometry for all the features


This is fine for now, but we'll have an issue if countries that span the dateline (like Fiji) try to import land cover data that they have defined in a series of tiles linked with VRTs (as this will merge the polygons for Fiji together, creating a massive polygon from -180 to 180 in WGS84). So we'll in the future need to find a way to use the same login of splitting polygons across the meridian that we do in other tools to allow automatic import of datasets that span the dateline, without creating a dataset that spans the globe.

Alright, makes sense to split.

for more information, see https://pre-commit.ci

gkahiu · 2022-12-22T20:16:53Z

This looks good - just a few comments inline. Also, on first running the plugin with this branch I got the below. I haven't seen that message again, so maybe just appears on first run - but any idea what leads to the below?

2022-12-22T13:32:35     WARNING    Traceback (most recent call last):
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 969, in done
              self.validate_input(value)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 1002, in validate_input
              self.ok_clicked()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\lc_setup.py", line 1140, in ok_clicked
              remap_ret = self.remap_raster(out_file, self.dlg_agg.nesting.get_list())
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1250, in remap_raster
              warp_ret = self.warp_raster(temp_tif)
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1298, in warp_raster
              target_bounds = self.extent_as_list()
              File "C:\Users/azvoleff/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\LDMP\data_io.py", line 1090, in extent_as_list
              bbox = self.region_selector.region_info.geom.boundingBox()
             AttributeError: 'NoneType' object has no attribute 'boundingBox'

Am not really sure but am suspecting its something to do with the datasets - ne_10m_admin_0_countries.shp, ne_10m_admin_1_states_provinces.shp and ne_10m_populated_places.shp. Are these normally available on first time use or have to be downloaded? I have added log messages https://github.com/ConservationInternational/trends.earth/pull/757/files#diff-f5dbeb7765ec8b8d9b12e6b28b59efad8a6b99b2adcc31566eb875508396ce21R149 to help provide some more information. It could be either the datasets are not found, layers are invalid when loaded or the search for matching features did not yield any result.

gkahiu · 2022-12-22T20:17:35Z

I have updated the PR to address some of the comments.

azvoleff · 2022-12-23T18:47:52Z

Thanks @gkahiu. One more issue I've noted on this one: when rasters get expanded to cover the area of interest a user has selected, the no data areas that are added are set to zero. They should be set to -32768 to indicate that they are no data. I would have thought the dstNodata argument to gdal.Warp in RasterImportWorker.work would do this, but apparently not. Any idea what the correct argument is to gdal.Warp to fix this?

for more information, see https://pre-commit.ci

gkahiu · 2022-12-28T11:05:42Z

Thanks @gkahiu. One more issue I've noted on this one: when rasters get expanded to cover the area of interest a user has selected, the no data areas that are added are set to zero. They should be set to -32768 to indicate that they are no data. I would have thought the dstNodata argument to gdal.Warp in RasterImportWorker.work would do this, but apparently not. Any idea what the correct argument is to gdal.Warp to fix this?

Thanks for flagging this - the argument in gdal.warp is fine. The source of the 'zero-based' values is gdal.buildVRT which defaults to zero if no value is specified for no data. I have explicitly included this parameter VRTNodata so that any subsequent operation, including gdal.warp, on the image recognizes -32768 as nodata and cascades it forward using srcNodata and dstNodata parameters respectively. I hope this helps to address the issue, kindly confirm and let me know.

azvoleff · 2023-01-09T16:29:03Z

Thanks @gkahiu. That fixes the nodata issue. I did notice another issue though - I was playing with importing a dataset when choosing a city as the region, with a buffer applied to that city. The extent_as_list function (which ultimately relies on the output of get_admin_bbox in visualization.py, doesn't appear to properly handle cities, or, for that matter, buffered regions. If I use a city as the region, I get an uncropped input file back, even if the city+buffer covers only a small portion of the input file. Similarly if I apply a buffer to the region I have selected (using the buffer option in the settings tool), it doesn't affect the extent of the results.

gkahiu added 8 commits December 13, 2022 09:20

Fix call to geos validity check

032067a

Add RegionSelector widget

faa51e0

Move results to job's root directory.

257acc6

Copy files instead of moving to address temp files.

55b706e

Update tolerance to perform check based on geometry area.

92f098e

Refactor soil organic carbon dialog.

9147b7b

Refactor land productivity import dialog.

749a32b

Enable save/restore LC aggregation.

46fddbd

gkahiu added the Kartoza label Dec 22, 2022

gkahiu requested a review from azvoleff December 22, 2022 18:11

pre-commit-ci bot and others added 3 commits December 22, 2022 18:11

[pre-commit.ci] auto fixes from pre-commit.com hooks

88c6892

for more information, see https://pre-commit.ci

Fix incorrect refactoring.

af6c1dd

Merge branch 'data_import_enhancements' of https://github.com/gkahiu/…

46cb3c9

…trends.earth into data_import_enhancements

azvoleff reviewed Dec 22, 2022

View reviewed changes

gkahiu and others added 5 commits December 22, 2022 22:52

Add log messages in func for getting admin geometry.

13c0d3f

Update uris for raster results in job manager.

c46d569

Use variable containing file location.

145f70e

Comment change.

7e63936

[pre-commit.ci] auto fixes from pre-commit.com hooks

3037b3b

for more information, see https://pre-commit.ci

Fix output of main uri for RasterResults to work for vrt or tiff

745bbc6

gkahiu and others added 4 commits December 27, 2022 01:48

Explicitly set nodata for VRT.

d2e331c

Refactor VRT args.

8f1d6e9

Explicitly set source/destination nodata values.

8c27576

[pre-commit.ci] auto fixes from pre-commit.com hooks

0fd1243

for more information, see https://pre-commit.ci

gkahiu mentioned this pull request Jan 4, 2023

Changing Dataset CRS #758

Open

Fix code to get city from cities.json

056c93b

Add percentage overlap to message bar

c765fd5

azvoleff assigned Samweli Jul 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data import enhancements #757

Data import enhancements #757

gkahiu commented Dec 22, 2022

azvoleff left a comment

azvoleff Dec 22, 2022

gkahiu Dec 22, 2022

azvoleff Dec 22, 2022

gkahiu Dec 22, 2022

azvoleff Dec 22, 2022

gkahiu Dec 22, 2022

gkahiu commented Dec 22, 2022

gkahiu commented Dec 22, 2022

azvoleff commented Dec 23, 2022

gkahiu commented Dec 28, 2022

azvoleff commented Jan 9, 2023 •

edited

Loading


		source_crs = lyr.crs()

		# Combine the geometry for all the features

Data import enhancements #757

Are you sure you want to change the base?

Data import enhancements #757

Conversation

gkahiu commented Dec 22, 2022

azvoleff left a comment

Choose a reason for hiding this comment

azvoleff Dec 22, 2022

Choose a reason for hiding this comment

gkahiu Dec 22, 2022

Choose a reason for hiding this comment

azvoleff Dec 22, 2022

Choose a reason for hiding this comment

gkahiu Dec 22, 2022

Choose a reason for hiding this comment

azvoleff Dec 22, 2022

Choose a reason for hiding this comment

gkahiu Dec 22, 2022

Choose a reason for hiding this comment

gkahiu commented Dec 22, 2022

gkahiu commented Dec 22, 2022

azvoleff commented Dec 23, 2022

gkahiu commented Dec 28, 2022

azvoleff commented Jan 9, 2023 • edited Loading

azvoleff commented Jan 9, 2023 •

edited

Loading