Bug report #13963

QGIS greadily allocates memory and crashes when editing moderately large shapefiles with the node tool

Added by Karl Kastner over 1 year ago. Updated about 1 month ago.

Status:Closed Start Date:12/08/2015
Priority:High Due date:
Assigned to:- % Done:

0%

Category:Digitising
Target version:-
Platform:Linux / Ubuntu Pull Request or Patch supplied:No
Platform version:14.04 Trusty Affected version:2.8.4
Status info: Causes crash or corruption:Yes
Resolution:fixed/implemented Tag:

Description

When editing moderately large shapefiles (228428 points, 1.1MB << 1% total RAM). QGIS starts to allocate huge amounts of memory and either crashes the system when no ulimit is set or is killed by the kernel. This happens reproducibly by adding the dataset to a new project, going into edit mode, selecting the node tool, selecting more than 1/3 of the points at once and attempting to delete them. This is a downsampled dataset which can be generated and edited with other software on the same machine without hickups. QGIS version is 1:2.8.1-0+20trusty6.

dmesg-2.12.1.txt (247.3 kB) Karl Kastner, 12/09/2015 01:00 am

dmesg-2.8.4.txt (248.2 kB) Karl Kastner, 12/10/2015 03:10 am

kapuas-polygon-merged-100.tgz (1.1 MB) Karl Kastner, 12/10/2015 05:44 am

Shapefile_test.zip (8.8 kB) Hynek Suchy, 12/28/2015 05:30 am

Associated revisions

Revision e503c705a1c4dba07617f6beaf0a58360c857bcf
Added by Jürgen Fischer about 1 year ago

change QgsAbstractGeometryV2::coordinateSequence() to return a
implicitly shared copy of an internal cache instead of recreating the
coordinate sequence again and again.

Improves performance of the nodetool on large features a lot (refs #13963)

Also introduce Qgs(Coordinate|Ring|Point)SequenceV2 typedefs.

Revision 102ba2d3c51b2541f3b56301c58224a517323dd1
Added by Nyall Dawson 10 months ago

Much faster selection of large number of nodes using node tool

Refs #13963

History

Updated by Nyall Dawson over 1 year ago

  • Status changed from Open to Feedback

Please test with 2.8.4 or 2.12.1

Updated by Karl Kastner over 1 year ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Updated by Karl Kastner over 1 year ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Updated by Jürgen Fischer over 1 year ago

Karl Kastner wrote:

Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Because 2.8.4 is the LTR, which is in debian-ltr or ubuntugis-ltr (see download page)

Updated by Karl Kastner over 1 year ago

Thanks for the advice to add the LTR repository. The bug also affects 2.8.4. It does not occur on every deletion though, the steps had to be repeated a couple of times until Qgis crashed. Qgis becomes also relatively slow when activating the node tool, as it automatically starts a background validation. Note that the dataset has some crossing lines due to the downsampling, but this should not make Qgis crash.

dpkg -l qgis
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                     Version                   Architecture              Description
+++-========================================-=========================-=========================-=====================================================================================
ii  qgis                                     1:2.8.4+20trusty-ubuntugi amd64                     Geographic Information System (GIS)

Updated by Jürgen Fischer over 1 year ago

The kernel output is mostly useless. Please attach the shape file.

Updated by Jürgen Fischer over 1 year ago

The archive is missing the dbf.

Updated by Karl Kastner over 1 year ago

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

Updated by Giovanni Manghi over 1 year ago

  • Category changed from Actions to Digitising
  • Status changed from Feedback to Open
  • Priority changed from Low to High

Karl Kastner wrote:

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

weird, but anyway I confirm the issue, at least on 2.8.4 on ubuntu 14.04.

Updated by Giovanni Manghi over 1 year ago

  • Affected version changed from 2.8.1 to 2.8.4

Updated by Hynek Suchy over 1 year ago

Hello!
I would like to confirm this bug.
I am on Windows Vista 32bits Ultimate, QGIS 2.12.1 Lyon, standalone installer.
I am working on a small shapefile with some 300 points.
While editing the shapefile, mainly deleting points or moving them, qgis starts to
slow down and in the end freezes.
I tried to open the shp file outside of current project and I easily recreated the bug.
1. select one point
2. delete it
repeat 11 times and qgis freezes (not responding), but after some time
refreshes and I can delete another point, after 14 repetitions qgis doesn't recover
and I have to kill it from Windows Task Manager.
Enclosing shapefile.

Updated by Giovanni Manghi over 1 year ago

Jurgen, Nyall, shouldn't this be tagged as blocker?

Updated by Gerhard Spieles over 1 year ago

Hi Hynek,
maybe a problem of installed plugins.
See #13869

Updated by Giovanni Manghi over 1 year ago

Gerhard Spieles wrote:

Hi Hynek, maybe a problem of installed plugins. See #13869

usually in my testing boxes I have no 3rd party plugins, but I can test again.

Updated by Hynek Suchy over 1 year ago

Sorry for making a false alarm!
It was CadTools, I deinstalled them (and Digitizing Tools to be sure) and now it works smoothly!
Thank you!

Updated by Karl Kastner over 1 year ago

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.
The bug is also be reproduced with data sets having a valid geometry. (The polygon in the original sample shape file was self intersecting.)
The behaviour depends on the geometry for the same data points:
Polygon: QGIS becomes a memory bomb and crashes.
Line: QGIS takes very long (minutes) to complete the tasks, but does not fill the memory.
Point: QGIS deletes all selected points flawlessly within less than a second.

Updated by Giovanni Manghi over 1 year ago

Karl Kastner wrote:

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.

yes, this is my observation too.

Updated by Jürgen Fischer about 1 year ago

Improvements in PR #2817

Updated by Giovanni Manghi about 1 year ago

  • Status changed from Open to Feedback

Updated by Jürgen Fischer about 1 year ago

50f01a2 also reduces the memory consumption of the geometry classes, which should also help with this issue.

Updated by Giovanni Manghi 11 months ago

  • Status changed from Feedback to Closed
  • Resolution set to fixed/implemented

seems fixed to me at least on 2.14.3, please reopen if necessary

Updated by baditaflorin - 10 months ago

  • Status changed from Closed to Reopened

Tested in 2.14.3 on the Canada provinces and the results are horrible.

If i delete less then 10-20 nodes at a time it takes around 1-2 seconds.

If i select around 100-1000 nodes, it can take up to 10 minutes.

You can download the file from here https://www.dropbox.com/s/j5i1m1tt2cam1qy/canada_states.zip?dl=0

Here it`s a youtube video showing the selection https://youtu.be/tV7VOK4YAZE

Updated by Giovanni Manghi about 1 month ago

  • Status changed from Reopened to Closed

baditaflorin - wrote:

Tested in 2.14.3 on the Canada provinces and the results are horrible.

If i delete less then 10-20 nodes at a time it takes around 1-2 seconds.

If i select around 100-1000 nodes, it can take up to 10 minutes.

You can download the file from here https://www.dropbox.com/s/j5i1m1tt2cam1qy/canada_states.zip?dl=0

Here it`s a youtube video showing the selection https://youtu.be/tV7VOK4YAZE

the original report/issue is solved, there is no more leak if you the node tool.
So in this cases please file a different ticket.
Anyway I tested you dataset on both 2.18.4 and master (qgis3) on both Windows and Linux and is not that slow removing nodes when you selected a lot of them. Is not lighting fast, but it takes a few seconds when you have selected a reasonable large amount. Please try with the latest releases and in case file a new ticket.

Also available in: Atom