Bug report #13336

Allow raster calculator with huge raster inputs

Added by Simona Z over 1 year ago. Updated about 1 month ago.

Status:Feedback Start Date:09/09/2015
Priority:High Due date:
Assigned to:- % Done:

0%

Category:Raster Calculator
Target version:Version 3.0
Platform: Pull Request or Patch supplied:No
Platform version: Affected version:2.16.0
Status info: Causes crash or corruption:Yes
Resolution: Tag:

Description

Hello,
I am trying to do some raster operation with Qgis' Raster Calculator but I get a minidump error and Qgis crashes (like minidump written to C:\Users\PROPRI~1\AppData\Local\Temp\qgis-20150909-155310-3892-7148-d20c5b7.dmp) every time. I am working with two raster files I created, so I would like to know if it's a problem with my rasters or with QGis

Metadata of raster file (it's a binary file with 255 and 0 values):

Driver
GDAL provider
GTiff
GeoTIFF
Dataset Description
[raster pathname]
AREA_OR_POINT=Area
Band 1
STATISTICS_MAXIMUM=255
STATISTICS_MEAN=34.841265875025
STATISTICS_MINIMUM=0
STATISTICS_STDDEV=87.582012938486
Dimensions
X: 18821 Y: 15837 Bands: 1
Origin
1.71397e+06,5.13445e+06
Pixel Size
2.00004,-2.00001
No Data Value
-99999
Data Type
Float64 - Sixty four bit floating point
Pyramid overviews
Layer Spatial Reference System
+proj=tmerc +lat_0=0 +lon_0=9 +k=0.9996 +x_0=1500000 +y_0=0 +ellps=intl +towgs84=-104.1,-49.1,-9.9,0.971,-2.917,0.714,-11.68 +units=m +no_defs
Layer Extent (layer original source projection)
1713966.4223982943221927,5102780.3500192686915398 : 1751609.1579853519797325,5134454.5267527680844069
Band
Band 1
Band No
1
No Stats
No stats collected yet

qgis-20160120-163811-3596-4868-b403532.7z (2.5 MB) Nicolas Cadieux, 01/20/2016 01:50 pm

simulateRaster.7z (648 Bytes) Nicolas Cadieux, 01/25/2016 09:23 am

backtrace.log (2.7 kB) Johannes Kroeger, 09/03/2016 03:53 am

Associated revisions

Revision a665b7d72e7042aadd4a5cec10e9e9988a46b70f
Added by Nyall Dawson over 1 year ago

Don't crash when using raster calculator with huge raster (refs #13336)

Now we abort and advise the user if insufficient memory is available
to perform the calculation.

At some future stage it would be nice to perform the calculations
in blocks to allow this scenario, but for now it's better not to
crash.

History

Updated by Nyall Dawson over 1 year ago

  • Status changed from Open to Feedback

Can you share your raster file and the expression you are using for the calculation?

Updated by Nicolas Cadieux over 1 year ago

Hi Nyall,

Same problem with 2.12.0 on Windows 7-64
Files are large .tiff(84GB each) but this has worked in the past with 2.6 or 2.8. In my case, both are in the same projection, and I am not changing the output projection. This is simple math. layer1 - layer 2 (creating a normalized LiDAR layer). Same result using virtual raster sets of the same data.

Hope this helps.
Nicolas

Updated by Giovanni Manghi over 1 year ago

  • Category set to Raster Calculator
  • Priority changed from Normal to High

Nicolas Cadieux wrote:

Hi Nyall,

Same problem with 2.12.0 on Windows 7-64 Files are large .tiff(84GB each) but this has worked in the past with 2.6 or 2.8. In my case, both are in the same projection, and I am not changing the output projection. This is simple math. layer1 - layer 2 (creating a normalized LiDAR layer). Same result using virtual raster sets of the same data.

so, to replicate the issue is needed to use such big tiffs or it happens with any size ones? Any shareable dataset?

Updated by Nicolas Cadieux over 1 year ago

Hi,
I just saw the comments. Sorry for not responding. I though I would get an email. I could share a file but it's 80 GB. I bit big but I could send it by ftp server. Or I could wait to find a smaller file that crashes QGIS. I did a test with a 250MB file and it work ok. It crashes with the big stuff. Send me an email to nicolas.cadieux at archeotec.ca if you need files to resolve this issues.

Nicolas

Updated by Giovanni Manghi over 1 year ago

  • Subject changed from minidump error when using Raster calculator to QGIS crashes when using Raster calculator with huge raster inputs
  • Status changed from Feedback to Open

Nicolas Cadieux wrote:

Send me an email to nicolas.cadieux at archeotec.ca if you need files to resolve this issues.

would be happy to try test this huge rasters: giovanni dot manghi at naturalgis dot pt

Updated by Nyall Dawson over 1 year ago

Can you also send me a link to them? nyall dot dawson at gmail dot com

Updated by Nyall Dawson over 1 year ago

Did you manage to send this data? I'd like to address this for 2.14

Updated by Nicolas Cadieux over 1 year ago

Working on that file. For now, I can send you the dmp.

Updated by Nicolas Cadieux over 1 year ago

trying again on the dmp. If it does not work, I will send it by email.

Updated by Nicolas Cadieux over 1 year ago

Hi,

I have a solution. I have uploaded 2 files. A python 2.7 script that will create a simulated raster having the same general properties as my files. That will create a ASC file that has 145,857,241KB. (Will take time...) Then, use the gdal_translate file to save the ASC file as a tif. That file will have 84,013,848KB. With that, you can make a copy of this file (ex rename file1 and file2) and then do some simple raster math. It should crash as expected. (you will need to change the input and output files in these scripts (but as you are programs, you probably know this.)

Contact me if you have problems.
Nicolas

Updated by Nyall Dawson over 1 year ago

  • Tracker changed from Bug report to Feature request
  • Subject changed from QGIS crashes when using Raster calculator with huge raster inputs to Allow raster calculator with huge raster inputs
  • Assigned to deleted (Nyall Dawson)

Updated by Nyall Dawson over 1 year ago

Crash is now avoided, so moving this to a feature request - at some future stage we should perform the calculation in "blocks" to avoid the memory allocation errors.

Updated by Drew Ricketts about 1 year ago

Should this issue be resolved in QGIS 2.14? I'm trying to do a very simple raster calculation on a .tif using a mac, and QGIS crashes without giving a warning about memory.

Updated by Nyall Dawson about 1 year ago

Drew - sounds like a totally different issue.

Updated by Jürgen Fischer 11 months ago

  • Tracker changed from Feature request to Bug report
  • Causes crash or corruption changed from Yes to No

Updated by Jürgen Fischer 11 months ago

  • Tracker changed from Bug report to Feature request

Updated by Giovanni Manghi 10 months ago

  • Tracker changed from Feature request to Bug report
  • Target version set to Version 3.0
  • Affected version changed from 2.10.1 to 2.16.0
  • Causes crash or corruption changed from No to Yes

It still crashes, see #15281 (tested on a clean 2.16 installation on a Linux box with 16GB or ram).

Updated by Johannes Kroeger 8 months ago

I guess I ran into this issue as well with a simple subtraction between two ~40000x40000 rasters.
Attached is a backtrace from gdb.
QGIS eats about 7GB of RAM while trying to do this, there would be 12GB more available for it though so I am not sure if it fills up something.
QGIS version 2.16.1-Nødebo.

It works perfectly fine if I do the same calculation with the gdal_calc.py utility outside QGIS.

If needed I can share the rasters, they are about 700MB each (compressed).

Updated by Giovanni Manghi about 1 month ago

  • Status changed from Open to Feedback

Please try the NEW QGIS raster calculator available in the Processing toolbox of QGIS master (QGIS3).

Updated by Giovanni Manghi about 1 month ago

If needed I can share the rasters, they are about 700MB each (compressed).

I would like to give them a try, thanks!

Also available in: Atom