Bug report #13963

QGIS greadily allocates memory and crashes when editing moderately large shapefiles with the node tool

Added by Karl Kastner about 1 year ago. Updated 8 months ago.

Status:Reopened Start Date:12/08/2015
Priority:High Due date:
Assigned to:- % Done:

0%

Category:Digitising
Target version:-
Platform:Linux / Ubuntu Pull Request or Patch supplied:No
Platform version:14.04 Trusty Affected version:2.8.4
Status info: Causes crash or corruption:Yes
Resolution:fixed/implemented Tag:

Description

When editing moderately large shapefiles (228428 points, 1.1MB << 1% total RAM). QGIS starts to allocate huge amounts of memory and either crashes the system when no ulimit is set or is killed by the kernel. This happens reproducibly by adding the dataset to a new project, going into edit mode, selecting the node tool, selecting more than 1/3 of the points at once and attempting to delete them. This is a downsampled dataset which can be generated and edited with other software on the same machine without hickups. QGIS version is 1:2.8.1-0+20trusty6.

dmesg-2.12.1.txt (247.3 kB) Karl Kastner, 12/09/2015 01:00 am

dmesg-2.8.4.txt (248.2 kB) Karl Kastner, 12/10/2015 03:10 am

kapuas-polygon-merged-100.tgz (1.1 MB) Karl Kastner, 12/10/2015 05:44 am

Shapefile_test.zip (8.8 kB) Hynek Suchy, 12/28/2015 05:30 am

Associated revisions

Revision e503c705a1c4dba07617f6beaf0a58360c857bcf
Added by Jürgen Fischer about 1 year ago

change QgsAbstractGeometryV2::coordinateSequence() to return a
implicitly shared copy of an internal cache instead of recreating the
coordinate sequence again and again.

Improves performance of the nodetool on large features a lot (refs #13963)

Also introduce Qgs(Coordinate|Ring|Point)SequenceV2 typedefs.

Revision 102ba2d3c51b2541f3b56301c58224a517323dd1
Added by Nyall Dawson 8 months ago

Much faster selection of large number of nodes using node tool

Refs #13963

History

Updated by Nyall Dawson about 1 year ago

  • Status changed from Open to Feedback

Please test with 2.8.4 or 2.12.1

Updated by Karl Kastner about 1 year ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Updated by Karl Kastner about 1 year ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Updated by Jürgen Fischer about 1 year ago

Karl Kastner wrote:

Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Because 2.8.4 is the LTR, which is in debian-ltr or ubuntugis-ltr (see download page)

Updated by Karl Kastner about 1 year ago

Thanks for the advice to add the LTR repository. The bug also affects 2.8.4. It does not occur on every deletion though, the steps had to be repeated a couple of times until Qgis crashed. Qgis becomes also relatively slow when activating the node tool, as it automatically starts a background validation. Note that the dataset has some crossing lines due to the downsampling, but this should not make Qgis crash.

dpkg -l qgis
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                     Version                   Architecture              Description
+++-========================================-=========================-=========================-=====================================================================================
ii  qgis                                     1:2.8.4+20trusty-ubuntugi amd64                     Geographic Information System (GIS)

Updated by Jürgen Fischer about 1 year ago

The kernel output is mostly useless. Please attach the shape file.

Updated by Jürgen Fischer about 1 year ago

The archive is missing the dbf.

Updated by Karl Kastner about 1 year ago

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

Updated by Giovanni Manghi about 1 year ago

  • Category changed from Actions to Digitising
  • Status changed from Feedback to Open
  • Priority changed from Low to High

Karl Kastner wrote:

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

weird, but anyway I confirm the issue, at least on 2.8.4 on ubuntu 14.04.

Updated by Giovanni Manghi about 1 year ago

  • Affected version changed from 2.8.1 to 2.8.4

Updated by Hynek Suchy about 1 year ago

Hello!
I would like to confirm this bug.
I am on Windows Vista 32bits Ultimate, QGIS 2.12.1 Lyon, standalone installer.
I am working on a small shapefile with some 300 points.
While editing the shapefile, mainly deleting points or moving them, qgis starts to
slow down and in the end freezes.
I tried to open the shp file outside of current project and I easily recreated the bug.
1. select one point
2. delete it
repeat 11 times and qgis freezes (not responding), but after some time
refreshes and I can delete another point, after 14 repetitions qgis doesn't recover
and I have to kill it from Windows Task Manager.
Enclosing shapefile.

Updated by Giovanni Manghi about 1 year ago

Jurgen, Nyall, shouldn't this be tagged as blocker?

Updated by Gerhard Spieles about 1 year ago

Hi Hynek,
maybe a problem of installed plugins.
See #13869

Updated by Giovanni Manghi about 1 year ago

Gerhard Spieles wrote:

Hi Hynek, maybe a problem of installed plugins. See #13869

usually in my testing boxes I have no 3rd party plugins, but I can test again.

Updated by Hynek Suchy about 1 year ago

Sorry for making a false alarm!
It was CadTools, I deinstalled them (and Digitizing Tools to be sure) and now it works smoothly!
Thank you!

Updated by Karl Kastner about 1 year ago

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.
The bug is also be reproduced with data sets having a valid geometry. (The polygon in the original sample shape file was self intersecting.)
The behaviour depends on the geometry for the same data points:
Polygon: QGIS becomes a memory bomb and crashes.
Line: QGIS takes very long (minutes) to complete the tasks, but does not fill the memory.
Point: QGIS deletes all selected points flawlessly within less than a second.

Updated by Giovanni Manghi about 1 year ago

Karl Kastner wrote:

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.

yes, this is my observation too.

Updated by Jürgen Fischer about 1 year ago

Improvements in PR #2817

Updated by Giovanni Manghi about 1 year ago

  • Status changed from Open to Feedback

Updated by Jürgen Fischer about 1 year ago

50f01a2 also reduces the memory consumption of the geometry classes, which should also help with this issue.

Updated by Giovanni Manghi 9 months ago

  • Status changed from Feedback to Closed
  • Resolution set to fixed/implemented

seems fixed to me at least on 2.14.3, please reopen if necessary

Updated by baditaflorin - 8 months ago

  • Status changed from Closed to Reopened

Tested in 2.14.3 on the Canada provinces and the results are horrible.

If i delete less then 10-20 nodes at a time it takes around 1-2 seconds.

If i select around 100-1000 nodes, it can take up to 10 minutes.

You can download the file from here https://www.dropbox.com/s/j5i1m1tt2cam1qy/canada_states.zip?dl=0

Here it`s a youtube video showing the selection https://youtu.be/tV7VOK4YAZE

Also available in: Atom