Feature request #8725

Speed improvement in the render of geometries for qgis

Added by Alvaro Huarte over 3 years ago. Updated over 3 years ago.

Status:Closed Start Date:09/30/2013
Priority:Normal Due date:
Assigned to:Alvaro Huarte % Done:

100%

Category:Vectors
Target version:Future Release - Nice to have
Platform: Resolution:fixed/implemented
Platform version: Pull Request or Patch supplied:Yes
Status info:for test Tag:

Description

The render of geometries for vector layers in qgis can be optimized.
There are some tricks that sure make fast the rendering of big shapes:

- Prefilter the points of the geometries to be painted using a map2pixel error tolerance (_getPolygon and _getLineString in rendererv2).
- Prevents divisions in map to devices coordinates.
- Prevents of unnecessary trim of the geometries with the canvas rect.
- Prevents of unnecessary calloc using shared memory buffer (ogr to wkb in ogrFeatureIterator).

I have implements this tricks and qgis has better performance of 300% in some map situations.
I want validate the code and I will try share it.

test_data_qgis_master_RenderingStatsQGIS__Feature-8725.zip (3.9 kB) Alvaro Huarte, 10/05/2013 03:27 am

cfgimage.jpg (43.7 kB) Alvaro Huarte, 10/26/2013 01:30 pm

WarningLabel.jpg (18.6 kB) Alvaro Huarte, 11/20/2013 02:25 am

WarningLabel2.jpg (19.9 kB) Alvaro Huarte, 11/20/2013 02:23 pm

DefaultSimplifyConfig.jpg (77.9 kB) Alvaro Huarte, 11/21/2013 03:30 pm

BadSimplificationWithOffset.jpg (73.8 kB) Alvaro Huarte, 12/13/2013 04:10 pm

History

Updated by Matthias Kuhn over 3 years ago

Hi Alvaro,

Thank you for looking into this issues and contributing.

The best thing to do is to open a pull request on github, so it can be reviewed, commented and eventually merged.

Updated by Alvaro Huarte over 3 years ago

Ok, thank you very much

Updated by Matthias Kuhn over 3 years ago

I forgot to mention that it would be best to make one pull request for each improvement. So in case one of them is not accepted, it will still be possible to merge the good parts.

Updated by Alvaro Huarte over 3 years ago

ok, I will divide this issue in four single childs tasks.
Thank you!

Updated by Giovanni Manghi over 3 years ago

Hi Alvaro, do you have a branch that can be tested? Cheers!

Updated by Alvaro Huarte over 3 years ago

Hi Giovanni, I am testing my changes, if it successly in some days I will try a pull request.
Thank you very much!

Updated by Alvaro Huarte over 3 years ago

I'm making progress, I think I get promising results. Attach a table of results for three tested layers.

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

I'm making progress, I think I get promising results. Attach a table of results for three tested layers.

wow, I can't wait to test your patches! :)

Updated by Alvaro Huarte over 3 years ago

  • % Done changed from 70 to 100
  • Status info set to for test
  • Resolution set to fixed

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

done for test.

github: https://github.com/ahuarte47/QGIS/commit/28691ff96e0ef2a97a8f1d08898b1c7d97941f7c + https://github.com/ahuarte47/QGIS/commit/a8894f8df811c61fe021607d474aa8e117a5f56f

Sorry there are two commits, I forgot to add one edited file

Hi Alvaro, can you please make a Pull Request in the qgis repo? This way other will be able to see it and review and then commit. Thanks!

Updated by Alvaro Huarte over 3 years ago

Done.
I'm new using gitub and I'm learning this tool, Sorry!
Thanks!

Updated by Alvaro Huarte over 3 years ago

  • Resolution changed from fixed to fixed/implemented
  • Pull Request or Patch supplied changed from No to Yes

I have implemented the simplification of geometries in the vector-providers. The general behavior simplifies the geometries post-fetch the feature from the provider, but OGR-provider simplifies the geometries pre-fetch the features having a extra seed improvement.

Te final result gets ~3x painting speed faster that original code

New branch:
https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR

I think ready for test! :-)

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

I have implemented the simplification of geometries in the vector-providers. The general behavior simplifies the geometries post-fetch the feature from the provider, but OGR-provider simplifies the geometries pre-fetch the features having a extra seed improvement.

Te final result gets ~3x painting speed faster that original code

New branch: https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR

I think ready for test! :-)

compiling right now :)

Updated by Giovanni Manghi over 3 years ago

I think ready for test! :-)

got an error during make

Building CXX object src/core/CMakeFiles/qgis_core.dir/qgsmaplayer.cpp.o
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp: In function ‘QgsRectangle calculateBoundingBox(const QVector<QPointF>&)’:
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:102:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:102:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:102:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:103:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:103:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:103:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:104:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:104:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:104:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:105:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:105:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:105:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:109:5: error: ‘x’ was not declared in this scope
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:110:5: error: ‘y’ was not declared in this scope
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp: In function ‘QgsRectangle calculateBoundingBox(QGis::WkbType, unsigned char*, std::size_t)’:
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:125:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:125:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:125:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:126:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:126:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:126:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:127:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:127:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:127:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:128:18: error: ‘numeric_limits’ is not a member of ‘std’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:128:38: error: expected primary-expression before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:128:38: error: expected ‘,’ or ‘;’ before ‘double’
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:135:5: error: ‘x’ was not declared in this scope
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:136:5: error: ‘y’ was not declared in this scope
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp: At global scope:
/home/giovanni/QGIS-Issue_8725-OGR/src/core/qgsmaprequest.cpp:149:20: warning: unused parameter ‘sourceWkb’ [-Wunused-parameter]
make[2]: [ 14%] *** [src/core/CMakeFiles/qgis_core.dir/qgsmaprequest.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
Building CXX object src/core/CMakeFiles/qgis_core.dir/qgsmaplayerregistry.cpp.o
make[1]: *** [src/core/CMakeFiles/qgis_core.dir/all] Error 2
make: *** [all] Error 2

Updated by Alvaro Huarte over 3 years ago

Hi Giovanni, I use Visual Studio 2010 and I don't get these errors.
What compiler do you use ?

I will try fix it!

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

Hi Giovanni, I use Visual Studio 2010 and I don't get these errors. What compiler do you use ?

I will try fix it!

Hi Alvaro, I use Ubuntu Linux, gcc 4.6.3

Updated by Matthias Kuhn over 3 years ago

It's probably a simple #include <limits.h>

Updated by Alvaro Huarte over 3 years ago

Done, thanks!

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

Done, thanks!

Hi Alvaro! I made a few quick tests with shapefiles, measuring just the time to open them (with the "time" command). I have not had the time to try evaluate the improvement when zooming in/out.

The shapefiles (polygons) that I tested are:

http://faunalia.pt/downloads/GTBs_Shapefile.zip
http://faunalia.pt/downloads/MAMMTERR.zip

un-patched QGIS

giovanni@sibirica ~/Desktop $ time qgis MAMMTERR.shp

user 0m10.377s
user 0m10.237s
user 0m10.429s

giovanni@sibirica ~/Desktop $ time qgis GTBs_2011.shp

user 1m2.896s
user 1m5.436s
user 1m4.120s

patched QGIS

giovanni@sibirica ~/Desktop $ time qgis MAMMTERR.shp

user 0m6.352s
user 0m6.244s
user 0m6.820s

giovanni@sibirica ~/Desktop $ time qgis GTBs_2011.shp

user 0m35.278s
user 0m33.286s
user 0m33.602s

So there is really a substantial improvement!

I tested also another shape (big, more that 1.2gb of small polygons) where things get worst(?)

http://www.igeo.pt/scrif/crif/CRIF2011shp.zip

unpatched

giovanni@sibirica ~/Desktop $ time qgis crif2011.shp

user 0m35.186s
user 0m34.870s
user 0m35.214s

patched

giovanni@sibirica ~/Desktop $ time qgis crif2011.shp

user 0m45.571s
user 0m49.311s
user 0m46.279s

Can I test also the vectors you are using to check the rendering times?

Cheers!

Updated by Matthias Kuhn over 3 years ago

Running a rendering performance test once may be strongly biased by other processes running.
For statistically more robust numbers I recommend using qgis_bench

It's really simple to use.

Here some more information:
https://github.com/qgis/QGIS/blob/master/tests/bench/README

Updated by Alvaro Huarte over 3 years ago

Hi, I made tests with 'qgis_bench' to see the speed.

Soft/hard: WindowsXP+SP3, 32bits, 4gb RAM.

My shapefiles tested:

Dataset Value Master Patched
http://faunalia.pt/downloads/GTBs_Shapefile.zip iterations 10 10 100%
total_avg 180.1015625 68.8765625 38%
total_max 185.8125 71.484375 38%
total_maxdev 33.3046875 2.60781249999999 7%
total_min 146.796875 67.265625 45%
total_stdev 11.1268459888756 1.27757571479394 11%
http://faunalia.pt/downloads/MAMMTERR.zip iterations 10 10 100%
total_avg 88.621875 9.49375 10%
total_max 95.09375 10 10%
total_maxdev 53.85625 1.665625 3%
total_min 34.765625 7.828125 22%
total_stdev 17.9542520316393 0.572148158805916 3%
http://www.igeo.pt/scrif/crif/CRIF2011shp.zip iterations 10 10 100%
total_avg 275.1109375 70.275 25%
total_max 286.5625 71.4375 24%
total_maxdev 93.3609375 6.50937500000001 6%
total_min 181.75 63.765625 35%
total_stdev 31.1253721504484 2.18250769614909 7%
http://idena.navarra.es/descargas/CARTO1_Lin_6CNivelD.zip iterations 10 10 100%
total_avg 8.2765625 1.4703125 17%
total_max 8.90625 1.515625 17%
total_maxdev 4.9796875 0.1109375 2%
total_min 3.296875 1.359375 41%
total_stdev 1.66037296379646 0.0578125 3%
http://idena.navarra.es/descargas/GEOLOG_Pol_Litologia.zip iterations 10 10 100%
total_avg 10.371875 1.7953125 17%
total_max 11.03125 1.859375 16%
total_maxdev 5.79375 0.4984375 8%
total_min 4.578125 1.296875 28%
total_stdev 1.93128792438492 0.166455758480294 8%
Urban parcels of cadastre of navarra (Polygon2D of 142680 records, 66mb) iterations 10 10 100%
total_avg 8.6046875 3.3484375 38%
total_max 9.0625 3.375 37%
total_maxdev 2.9953125 0.1296875 4%
total_min 5.609375 3.21875 57%
total_stdev 0.999488394520542 0.0458481273472538 4%
Rustic parcels of cadastre of navarra (Polygon2D of 542658 records, 458mb) iterations 10 10 100%
total_avg 57.8265625 12.6921875 21%
total_max 61.859375 13.03125 21%
total_maxdev 28.4671875 0.957812499999999 3%
total_min 29.359375 11.734375 39%
total_stdev 9.50177422268632 0.32983557101418 3%

Updated by Nathan Woodrow over 3 years ago

Nice!

Updated by Alvaro Huarte over 3 years ago

  • File sketch.png added

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

I thought a bit :-), I think more appropriate to add a new "rendering" tab with this configuration similar to global options form.

I agree

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

Hi, I made tests with 'qgis_bench' to see the speed.

Soft/hard: WindowsXP+SP3, 32bits, 4gb RAM.

very nice results indeed! as soon as I will understand how to use qgis_bench I will make tests too :)

Updated by Alvaro Huarte over 3 years ago

Hi Giovanni, qgis_bench is easy :-)

I execute in a Console Window (CMD)...

{%BIN_PATH%}/qgis_bench -i 10 -p test_project.qgs >> test_project.log

-i: number of desired iterations
-p: project to test

Regards

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

Hi Giovanni, qgis_bench is easy :-)

I execute in a Console Window (CMD)...

{%BIN_PATH%}/qgis_bench -i 10 -p test_project.qgs >> test_project.log

-i: number of desired iterations -p: project to test

Regards

I made also tests with qgis_bench

Dataset Value Master Patched
http://faunalia.pt/downloads/MAMMTERR.zip iterations 10 10 100%
total_avg 14.6881179 3.616626 24%
total_max 15.500969 3.816239 24%
total_maxdev 2.3073439 0.204413 8%
total_min 12.380774 3.412213 27%
total_stdev 0.808622095655375 0.108486504445484 13%
http://faunalia.pt/downloads/GTBs_Shapefile.zip iterations 10 10 100%
total_avg 41.0249639 32.6264391 79%
total_max 42.986687 37.038314 86%
total_maxdev 1.9617231 4.41187489999999 224%
total_min 39.566472 30.029877 75%
total_stdev 1.12526313538864 1.95203781571303 173%
http://www.igeo.pt/scrif/crif/CRIF2011shp.zip iterations 10 10 100%
total_avg 64.6020374 37.1403211 57%
total_max 65.808114 38.874429 59%
total_maxdev 4.3062684 2.76617309999999 64%
total_min 60.295769 34.374148 57%
total_stdev 1.54006099425576 1.13227893163129 73%

Updated by Jürgen Fischer over 3 years ago

  • Subject changed from Speed ​​improvement in the render of geometries for qgis to Speed improvement in the render of geometries for qgis

Updated by Alvaro Huarte over 3 years ago

Hi,

I have implemented a new tab called 'Rendering' in the options form of the layer where the user can configure the drawing simplification.

Regards

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

Hi,

I have implemented a new tab called 'Rendering' in the options form of the layer where the user can configure the drawing simplification.

Regards

Hi Alvaro,
this means that when loading a layer the user will not see any improvement, isn't it? If yes then it is a pity because for large vectors there is also a huge improvement when loading the layer. What about an option in the general qgis options and then the switch in vector properties to allow override the general configuration?

Updated by Alvaro Huarte over 3 years ago

Hi Giovanni,

I propose that by default the simplification is activated (The check-box, by default, will be checked), after the user can disable it for some reason, so I have been asked several people.

About a general option, I agree, but I do not what others think about it.

Giovanni, thank you very much for your tests and support!
Regards

Updated by Jukka Rahkonen over 3 years ago

Hi,

This may be a stupid question, but how does the simplified rendering work when users start to edit vectors? I think that users can not be let to edit those features which are simplified for fast viewing and save edits back to the source shapefile. And what gets saved if used does not edit anything but wants to make a copy by Save as..? Is QGIS saving the simpified layer of can is somehow handle two sort of geometries, the real ones and those temporarily simplified for viewing? Even when other layers are edited the result might be unexpected if user wants to snap vertices of the layer that is edited to vertices of the layer that is rendered as simplified.

Updated by Alvaro Huarte over 3 years ago

Hi @jratike80, this is a good question!, but there is not problem.

The simplification only is applied in the 'getFeatures' request when it becomes from a rendering request (... drawing the layer in the map). Fetch geometries, update geometries... are executed using distinct 'getFeatures' requests where the geometries are fetched without simplification.

Regards

Updated by Jürgen Fischer over 3 years ago

Alvaro Huarte wrote:

The simplification only is applied in the 'getFeatures' request when it becomes from a rendering request (... drawing the layer in the map). Fetch geometries, update geometries... are executed using distinct 'getFeatures' requests where the geometries are fetched without simplification.

You could also combine that with the QgsVectorLayer::isEditable() method - the layer does caching in edit mode. Another potential thing to look at is snapping.

Other things to consider are unaligned accesses. Marco Bernasocchi is currently working in qgis/QGIS-Android on cleaning the code from unaligned accesses (mainly when parsing or construction WKB; ARM doesn't like those), which is basically replacing all occurrences of *foo = bar; with memcpy(foo, &bar, sizeof bar);. I see that you're introducing more of those. Please rework it to use memcpy.

And finally run the stuff through scripts/prepare-commit.sh (or scripts/astyle.sh for already committed stuff).

But enough whining - otherwise it's very nice work - thank you very much :)

Updated by Alvaro Huarte over 3 years ago

Hello Jürgen, thank you very much for your comments!

I have released new commits for the branch https://github.com/ahuarte47/QGIS/compare/Issue_8725-OGR where I replace all '*foo = bar' with 'memcpy'. This has been a lesson for me.

Also I disable the drawing simplification in 'editmode', the geometries are cached for snapping, thanks.

About the scripts, I am going to study

Regards

Updated by Alvaro Huarte over 3 years ago

Hi Jürgen,

After a failed performance test I wanted to do (https://github.com/qgis/QGIS/pull/927#issuecomment-27158451), I retake his message referring to scripts. I do not know very well how execute it. I must use cygwin ?

Thank you very much

Updated by Alvaro Huarte over 3 years ago

To avoid confusion I created a new pull request with the last code.

New pull request:
https://github.com/qgis/QGIS/pull/980

Obsolete pull:
https://github.com/qgis/QGIS/pull/927

Regards

Updated by Nathan Woodrow over 3 years ago

Hey Alvaro,

Really like your work.

I seem to be having a small issue with the build though. When ever I run it in normal mode e.g just running the output qgis.exe file I get a crash as some as I open the file or one feature renders. But - and here is the fun part - attach the debugger and I don't get a crash :)

Running qgis_bench has the same effect, just crashes. Very strange.

Nathan

Updated by Alvaro Huarte over 3 years ago

Hi Nathan, thanks for your comments!

I have test some shapes and all works fine in 'release' build.
Can I test your data ?

Regards

Updated by Alexander Bruy over 3 years ago

Nathan Woodrow wrote:

I seem to be having a small issue with the build though. When ever I run it in normal mode e.g just running the output qgis.exe file I get a crash as some as I open the file or one feature renders. But - and here is the fun part - attach the debugger and I don't get a crash :)

Same here. I can reproduce this with polygon layers. Lines and points opened correctly. Also I found that QGIS compiled from old branch (https://github.com/qgis/QGIS/pull/927) works fine.

Updated by Alexander Bruy over 3 years ago

  • Resolution deleted (fixed/implemented)

Updated by Marco Hugentobler over 3 years ago

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

Updated by Giovanni Manghi over 3 years ago

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

Hi,

speaking as a user here. We all know that rendering performances of large vectors are poor in QGIS when compared to other packages, so I really hope that things can be sorted out. Is for this reason that I asked for a Windows installer that includes the patch (Alex is working on it, thanks!), to allow people test the patch, eventually find the issues and so allow you devs to eventually fix them.

By the way, loading time of large vectors is much better with the patch, this itself is a very welcome improvement.

Updated by Nathan Woodrow over 3 years ago

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Regarding exporting in the composer I would suggest just turning it off at that point and render with the full geometries. Navigating the map is what most people do and printing is normally expected to not be a super quick operation. It should be as fast as possible but if I have to wait 1 minute to get correct results in a print out I will happily, however I'm not going to wait a minute for my screen to render while I am building a map or editing.

IMO this needs to be considered. Not saying it should be merged today, or even next week but I think it's a worthwhile patch. As long as there is a way to switch it off, even as a build time option in case something really bad comes up then I think that would be fine.

Couple this kind of thing with the multithreading and we should see a nice boost in the rendering speed.

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

Hi,

speaking as a user here. We all know that rendering performances of large vectors are poor in QGIS when compared to other packages, so I really hope that things can be sorted out. Is for this reason that I asked for a Windows installer that includes the patch (Alex is working on it, thanks!), to allow people test the patch, eventually find the issues and so allow you devs to eventually fix them.

By the way, loading time of large vectors is much better with the patch, this itself is a very welcome improvement.

Hi, I am working for build a windows installer with this patch.
I build the default installer with the script 'creatensis.pl' but it does not contains my patch. I must study how insert it.

Updated by Matthias Kuhn over 3 years ago

Hi Alvaro,

The effect of disabled anti-aliasing is still perceptible here.
Which portion of the performance is gained by disabling AA actually?

The upper part is rendered without the patch, the lower part with the patch.

Updated by Alvaro Huarte over 3 years ago

Alexander Bruy wrote:

Nathan Woodrow wrote:

I seem to be having a small issue with the build though. When ever I run it in normal mode e.g just running the output qgis.exe file I get a crash as some as I open the file or one feature renders. But - and here is the fun part - attach the debugger and I don't get a crash :)

Same here. I can reproduce this with polygon layers. Lines and points opened correctly. Also I found that QGIS compiled from old branch (https://github.com/qgis/QGIS/pull/927) works fine.

I must study it, the code is the same but in last commit I also did a merge with master branch for make the windows installer.

Updated by Alvaro Huarte over 3 years ago

Nathan Woodrow wrote:

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Regarding exporting in the composer I would suggest just turning it off at that point and render with the full geometries. Navigating the map is what most people do and printing is normally expected to not be a super quick operation. It should be as fast as possible but if I have to wait 1 minute to get correct results in a print out I will happily, however I'm not going to wait a minute for my screen to render while I am building a map or editing.

IMO this needs to be considered. Not saying it should be merged today, or even next week but I think it's a worthwhile patch. As long as there is a way to switch it off, even as a build time option in case something really bad comes up then I think that would be fine.

Couple this kind of thing with the multithreading and we should see a nice boost in the rendering speed.

I agree, this patch improves the 'rendering at large scales', for getting rendering feedback the user can enable this existing option in the global options form of qgis.

To avoid problems in composer I am going disable this patch in 'printing' tasks.

Also, at least windows, there is other important improvement that I want test: Support for 'FileMapping' in the gdal-ogr library (e.g. using boost library) for speed up the access to data. I have experience with it and can be 2x faster.

Updated by Giovanni Manghi over 3 years ago

Nathan Woodrow wrote:

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Regarding exporting in the composer I would suggest just turning it off at that point and render with the full geometries. Navigating the map is what most people do and printing is normally expected to not be a super quick operation. It should be as fast as possible but if I have to wait 1 minute to get correct results in a print out I will happily, however I'm not going to wait a minute for my screen to render while I am building a map or editing.

IMO this needs to be considered. Not saying it should be merged today, or even next week but I think it's a worthwhile patch. As long as there is a way to switch it off, even as a build time option in case something really bad comes up then I think that would be fine.

Couple this kind of thing with the multithreading and we should see a nice boost in the rendering speed.

+1

Updated by Alvaro Huarte over 3 years ago

Matthias Kuhn wrote:

Hi Alvaro,

The effect of disabled anti-aliasing is still perceptible here. Which portion of the performance is gained by disabling AA actually?

The upper part is rendered without the patch, the lower part with the patch.

Hi Matthias, disable antialiasing for far geometries is perceptible, but for large datasets it save rendering time (I will make tests with some shapes and send results). ArcGis, by example, also cheats with that. I think that is more important to get fast framerates that high quality rendering, But, of course, all people has one opinion, I do what seems best to the community.

regards

Updated by Alvaro Huarte over 3 years ago

Hi Matthias, we have another think coming, :-)

A few days ago, you said me... "... another minor thing is, that the simplification code should probably not be in the request, but rather in the place where the simplification is done, so in the iterator or rather in its own simplification class (maybe other parts of the code need it...)".

Still think the same? I have no problem in creating a new utility class that implements this simplification code as static methods.

Thanks for your recommendations.

Updated by Nyall Dawson over 3 years ago

Nathan Woodrow wrote:

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Sorry to nitpick, but just for clarification - are you actually meaning rendering at small scales (further zoomed out) is where the performance gain is? If so, I agree that this is a significant issue which would be great to address.

Updated by Nathan Woodrow over 3 years ago

Nyall Dawson wrote:

Nathan Woodrow wrote:

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Sorry to nitpick, but just for clarification - are you actually meaning rendering at small scales (further zoomed out) is where the performance gain is? If so, I agree that this is a significant issue which would be great to address.

Yeah. Just stuffed up my terms.

Updated by Nyall Dawson over 3 years ago

Ok - in that case I run into this same issue almost daily, and any improvements would be greatly appreciated!

Updated by Alvaro Huarte over 3 years ago

Nyall Dawson wrote:

Ok - in that case I run into this same issue almost daily, and any improvements would be greatly appreciated!

I hope that with this patch, and the improvement of "FileMapping" I want to add to gdal-ogr providers, it result a very good behavior of qgis for render large datasets.

Best regards

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Alexander Bruy wrote:

Nathan Woodrow wrote:

I seem to be having a small issue with the build though. When ever I run it in normal mode e.g just running the output qgis.exe file I get a crash as some as I open the file or one feature renders. But - and here is the fun part - attach the debugger and I don't get a crash :)

Same here. I can reproduce this with polygon layers. Lines and points opened correctly. Also I found that QGIS compiled from old branch (https://github.com/qgis/QGIS/pull/927) works fine.

I must study it, the code is the same but in last commit I also did a merge with master branch for make the windows installer.

I have reverted the last commit with the merge action.
I hope that it fix this build issue.

Updated by Alvaro Huarte over 3 years ago

Matthias Kuhn wrote:

Hi Alvaro,

The effect of disabled anti-aliasing is still perceptible here. Which portion of the performance is gained by disabling AA actually?

The upper part is rendered without the patch, the lower part with the patch.

Hi, I have tests with 'qgis_bench' to see the improvement disabling 'AA' for '1-pixel geometries'.

Soft/hard: WindowsXP+SP3, 32bits, 4gb RAM.

- Patched (1): Disable 'AntiAliasing' for '1-pixel geometries' is not executed.
- Patched (2): Disable 'AntiAliasing' for '1-pixel geometries' is executed.

My shapefiles tested:

Dataset Value Master (0) Patched (1) (1-0) Patched (2) (2-0) (2-1)
http://faunalia.pt/downloads/GTBs_Shapefile.zip iterations 10 10 10
total_avg 211,02344 87,79219 42% 86,05781 41% 2%
total_max 224,60938 89,62500 88,18750
total_maxdev 30,39844 1,83281 2,12969
total_min 180,62500 86,81250 84,03125
total_stdev 11,06205 0,86665 1,35855
http://faunalia.pt/downloads/MAMMTERR.zip iterations 10 10 10
total_avg 84,65156 12,43438 15% 10,98594 13% 12%
total_max 92,84375 15,54688 11,37500
total_maxdev 52,21406 3,11250 1,50156
total_min 32,43750 11,93750 9,48438
total_stdev 17,42508 1,04493 0,51365
http://www.igeo.pt/scrif/crif/CRIF2011shp.zip iterations 10 10 10
total_avg 247,60625 109,75469 44% 77,25156 31% 30%
total_max 257,84375 112,65625 85,67188
total_maxdev 74,05938 16,97344 8,42031
total_min 173,54688 92,78125 74,73438
total_stdev 24,70081 5,67215 3,30560
http://idena.navarra.es/descargas/CARTO1_Lin_6CNivelD.zip iterations 10 10 10
total_avg 7,85781 1,94844 25% 1,58438 20% 19%
total_max 8,40625 2,85938 2,32813
total_maxdev 4,54531 0,91094 0,74375
total_min 3,31250 1,82813 1,45313
total_stdev 1,51537 0,30411 0,24992
http://idena.navarra.es/descargas/GEOLOG_Pol_Litologia.zip iterations 10 10 10
total_avg 9,31406 2,19063 24% 2,03750 22% 7%
total_max 10,06250 3,07813 3,12500
total_maxdev 4,84531 0,88750 1,08750
total_min 4,46875 2,06250 1,89063
total_stdev 1,61862 0,29603 0,36304
Urban parcels of cadastre of navarra (Polygon2D of 142680 records, 66mb) iterations 10 10 10
total_avg 8,20156 5,30000 65% 3,75938 46% 29%
total_max 8,62500 5,54688 3,82813
total_maxdev 3,06094 1,51875 0,15000
total_min 5,14063 3,78125 3,60938
total_stdev 1,02101 0,50745 0,05896
Rustic parcels of cadastre of navarra (Polygon2D of 472658 records, 458mb) iterations 10 10 10
total_avg 43,41719 16,49219 38% 11,77031 27% 29%
total_max 46,25000 17,23438 12,04688
total_maxdev 24,57344 0,74219 0,78594
total_min 18,84375 16,32813 10,98438
total_stdev 8,19141 0,25610 0,27208

I think that disable 'AA' for far geometries is effective, but it is true that, by contrast, a lower visual quality is perceptible especially for maps with geometries isolated.

I think it's worth apply it.
Regards

Updated by Matthias Kuhn over 3 years ago

Thank you for this nice table.
So, disabling AA indeed seems to result in a noticeable difference in some cases.
I would prefer to make it an optional feature the way it looks. I have seen you committed something related to AA, was this a new config option?

Updated by Alvaro Huarte over 3 years ago

Matthias Kuhn wrote:

Thank you for this nice table. So, disabling AA indeed seems to result in a noticeable difference in some cases. I would prefer to make it an optional feature the way it looks. I have seen you committed something related to AA, was this a new config option?

Hi Matthias,

It is very appreciable when the layer is very dense with a lot of small geometries.

Referred to the last commit, I was pending disable 'AA' for '1-pixel geometries' only when the simplification of the layer is activated. Temporal 'QgsSymbolV2RenderContext' objects created in 'QgsSymbolV2' class lose its 'VectorLayer' owner, and I have fixed it.

Referred to append a new extra option for disable the 'AA' of far geometries, I think that if the user already active the 'Fast drawing' implicitly assumes that the drawing quality can be a bit lower. I think that add other option can be strange or too complex to "understand". But I accept that the community send me :-)

Best regards

Updated by Regis Haubourg over 3 years ago

Referred to append a new extra option for disable the 'AA' of far geometries, I think that if the user already active the 'Fast drawing' implicitly assumes that the drawing quality can be a bit lower. I think that add other option can be strange or too complex to "understand". But I accept that the community send me :-)

Hi Alvaro,
In fact, if AA is enabled, symboly will be altered since you can't use symbols smaller than 1 pixel. So user will have to recalibrate symbology with your option for existing projects. We should avoid that. One big point with QGIS is precisely AA rendering, and we must keep antialising enabled. If user wants a Mapinfo-Arcgis style, faster, AA option is already there in general options, and user should gain your speed improvements + AA-OFF improvement.
My two cents, Régis

Updated by Alvaro Huarte over 3 years ago

regis Haubourg wrote:

Referred to append a new extra option for disable the 'AA' of far geometries, I think that if the user already active the 'Fast drawing' implicitly assumes that the drawing quality can be a bit lower. I think that add other option can be strange or too complex to "understand". But I accept that the community send me :-)

Hi Alexandro, In fact, if AA is enabled, symboly will be altered since you can't use symbols smaller than 1 pixel. So user will have to recalibrate symbology with your option for existing projects. We should avoid that. One big point with QGIS is precisely AA rendering, and we must keep antialising enabled. If user wants a Mapinfo-Arcgis style, faster, AA option is already there in general options, and user should gain your speed improvements + AA-OFF improvement. My two cents, Régis

Hi, I don't understand your reply very well, :-)

I may not have described right this behavior:

The 'AA' is only individually disabled for geometries of a layer so far that it fill a '1-pixel' of the map, the rest are painted using the 'AA' option as heretofore.

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Nathan Woodrow wrote:

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

The rendering at large scales is the main issue most people have with large datasets so this is where you need the most performance gain which this patch seems to achieve. When you are at smaller scales you don't have as many geometries so normally QGIS handles things well. Some of my clients have pretty large datasets and this is a major issue for them.

Regarding exporting in the composer I would suggest just turning it off at that point and render with the full geometries. Navigating the map is what most people do and printing is normally expected to not be a super quick operation. It should be as fast as possible but if I have to wait 1 minute to get correct results in a print out I will happily, however I'm not going to wait a minute for my screen to render while I am building a map or editing.

IMO this needs to be considered. Not saying it should be merged today, or even next week but I think it's a worthwhile patch. As long as there is a way to switch it off, even as a build time option in case something really bad comes up then I think that would be fine.

Couple this kind of thing with the multithreading and we should see a nice boost in the rendering speed.

I agree, this patch improves the 'rendering at large scales', for getting rendering feedback the user can enable this existing option in the global options form of qgis.

To avoid problems in composer I am going disable this patch in 'printing' tasks.

Also, at least windows, there is other important improvement that I want test: Support for 'FileMapping' in the gdal-ogr library (e.g. using boost library) for speed up the access to data. I have experience with it and can be 2x faster.

Hi, the last commit disable the 'drawing simplification' for print compositions.
Best regards

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

Hi,

speaking as a user here. We all know that rendering performances of large vectors are poor in QGIS when compared to other packages, so I really hope that things can be sorted out. Is for this reason that I asked for a Windows installer that includes the patch (Alex is working on it, thanks!), to allow people test the patch, eventually find the issues and so allow you devs to eventually fix them.

By the way, loading time of large vectors is much better with the patch, this itself is a very welcome improvement.

Hi Giovanni, now I am going to make a Windows Installer with the current patch state.
Telling you as I have!

Updated by Matthias Kuhn over 3 years ago

Alvaro,

I can imagine to enable this functionality for screen rendering by default sooner or later, but only if all optimization are non-visible. (Sub-pixel simplification / no AA)

On the other hand it seems, that there is the possibility to improve performance even further but at the expense of visible graphics quality degradation. This can be ok for some situations / systems, but needs to be opt-in.

So I would welcome to have the two nicely separated already. Do you think this is a road we can take?

Updated by Alvaro Huarte over 3 years ago

Hi Matthias, I can add other option to the new 'fast drawing' box in the 'rendering' options panel for the 'AA-disabling' advising that it is possible a graphics quality degradation.

If you like well I want to try before prepare the windows installer with this patch Giovanni asked for days.

Updated by Giovanni Manghi over 3 years ago

Hi Giovanni, now I am going to make a Windows Installer with the current patch state. Telling you as I have!

thanks!

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Giovanni Manghi wrote:

Marco Hugentobler wrote:

I've done some tests with the patch, and I have mixed feelings towards applying. The optimisation has most effect in zoomed-out situations where the data is too dense for the scale. But then, the user will still perceive it as slow performance (e.g. rendering 10 or 20 seconds without feedback is still slow). In situations where the data is appropriate for the scale, the differences are very small. On the other side, the patch adds a lot of code and still has potential to cause bugs (e.g. exporting in the composer with low resolution, say 50 dpi, shows gaps in a polygon mosaic).

Hi,

speaking as a user here. We all know that rendering performances of large vectors are poor in QGIS when compared to other packages, so I really hope that things can be sorted out. Is for this reason that I asked for a Windows installer that includes the patch (Alex is working on it, thanks!), to allow people test the patch, eventually find the issues and so allow you devs to eventually fix them.

By the way, loading time of large vectors is much better with the patch, this itself is a very welcome improvement.

Hi Giovanni, now I am going to make a Windows Installer with the current patch state. Telling you as I have!

Done:
http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86

It contains the Windows Installer with the qgis master branch + #8725 patch.
Regards

Updated by Bo Victor Thomsen over 3 years ago

Alvaro -

I've tried your patched version of QGIS. Works nicely with shape files and with large speed improvements. However, QGIS crashes when I'm trying to open a Postgis based layer. Mini dump file available at: https://docs.google.com/file/d/0B_386CU-PnW0cndaZXZOT1FWM2c/edit?usp=docslist_api (It's 20MB)

The postgis layer works in "normal" master.
OS Version: Window 8.1

Regards Bo Victor Thomsen

Updated by Giovanni Manghi over 3 years ago

Bo Victor Thomsen wrote:

Alvaro -

I've tried your patched version of QGIS. Works nicely with shape files and with large speed improvements. However, QGIS crashes when I'm trying to open a Postgis based layer. Mini dump file available at: https://docs.google.com/file/d/0B_386CU-PnW0cndaZXZOT1FWM2c/edit?usp=docslist_api (It's 20MB)

The postgis layer works in "normal" master. OS Version: Window 8.1

Regards Bo Victor Thomsen

confirmed here

Updated by Giovanni Manghi over 3 years ago

Done: http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86

It contains the Windows Installer with the qgis master branch + #8725 patch. Regards

Thanks! I will make some tests!

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Bo Victor Thomsen wrote:

Alvaro -

I've tried your patched version of QGIS. Works nicely with shape files and with large speed improvements. However, QGIS crashes when I'm trying to open a Postgis based layer. Mini dump file available at: https://docs.google.com/file/d/0B_386CU-PnW0cndaZXZOT1FWM2c/edit?usp=docslist_api (It's 20MB)

The postgis layer works in "normal" master. OS Version: Window 8.1

Regards Bo Victor Thomsen

confirmed here

Sorry, now I don't have installed the dump debugging tools.
I have tested with some postgis layers and works fine for me.

The error is throw for all layers ?
Are you using the application compiled from my pull request, or from the windows installer ?
If it is throwed by one layer, can you send me as shapefile for test here ?

Anyway, at night at home, I will setup the dump debugging tools for test the error
Thanks for your support!

Note: The improvement is very perceptible?
Best Regards

Updated by Alvaro Huarte over 3 years ago

Matthias Kuhn wrote:

Alvaro,

I can imagine to enable this functionality for screen rendering by default sooner or later, but only if all optimization are non-visible. (Sub-pixel simplification / no AA)

On the other hand it seems, that there is the possibility to improve performance even further but at the expense of visible graphics quality degradation. This can be ok for some situations / systems, but needs to be opt-in.

So I would welcome to have the two nicely separated already. Do you think this is a road we can take?

Hi Matthias, according your advice, the last commit implements 'simplification hints' allowing configure what drawing simplification type can be executed (Points, BBOX or AA, or any combination).

By default, the full simplification (Points+BBOX+AA) will be applied.

Updated by andre mano over 3 years ago

  • Platform set to Windows 7
  • Platform version set to 64bits

I tested with a shapefile containing 75k polygons and although I don't have a quantitative analysis, I can say that the rendering (even without going to quality threshold) it's at least twice as fast as QGIS 2.0.

I repeated the test with another shapefile containing 1000000 + polygons and the loading and pan/zoom operations became fast enough to make this layer usable in QGIS without loading it first into a spatial DB, specially if you give a high quality threshhold value.

A great add to QGIS!

Updated by Giovanni Manghi over 3 years ago

Note: The improvement is very perceptible?

yes :)

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Note: The improvement is very perceptible?

yes :)

I am delighted to read that.
I hope soon to add "FileMapping" support to gdal-ogr and when approved, another speed up for qgis!

Updated by Alvaro Huarte over 3 years ago

  • Platform deleted (Windows 7)
  • Platform version deleted (64bits)

Alvaro Huarte wrote:

Giovanni Manghi wrote:

Note: The improvement is very perceptible?

yes :)

I am delighted to read that. I hope soon to add "FileMapping" support to gdal-ogr and when approved, another speed up for qgis!

Updated by Alvaro Huarte over 3 years ago

Bo Victor Thomsen wrote:

Alvaro -

I've tried your patched version of QGIS. Works nicely with shape files and with large speed improvements. However, QGIS crashes when I'm trying to open a Postgis based layer. Mini dump file available at: https://docs.google.com/file/d/0B_386CU-PnW0cndaZXZOT1FWM2c/edit?usp=docslist_api (It's 20MB)

The postgis layer works in "normal" master. OS Version: Window 8.1

Regards Bo Victor Thomsen

Hello Bo Victor, you are right, qgis from my windows installer crashes when the user load a postgis layer.

But the error is not related with my patch code, the exception is thrown before load the layer in 'void QgsApplication::preNotify()' method when the 'add' button of the configuration form is clicked.

I think this error is described in http://hub.qgis.org/issues/5597#note-27.

In my compiled version I do not get that error so it has to be what files I have replaced in the master version of qgis for make the windows installer.

I must study it for rebuild the windows installer for test this patch.
Regards

Updated by Bo Victor Thomsen over 3 years ago

Alvaro -
Sorry for being late to the party, I was at a mapping conference. Just to answer your questions even though you seem to have found the error: QGIS crashes just after I've selected one postgis table and clicked the "Add" button in the "PostGres add layer dialog".

Question no 2: The shape file (which works fine) has a speed improvement of 50%. The layer contains 1.300.000 polygon shapes and goes from 40 sec. load to 20 sec. load on my Icore7 PC with SSD.
Regards

Updated by Alvaro Huarte over 3 years ago

Bo Victor Thomsen wrote:

Alvaro - Sorry for being late to the party, I was at a mapping conference. Just to answer your questions even though you seem to have found the error: QGIS crashes just after I've selected one postgis table and clicked the "Add" button in the "PostGres add layer dialog".

Question no 2: The shape file (which works fine) has a speed improvement of 50%. The layer contains 1.300.000 polygon shapes and goes from 40 sec. load to 20 sec. load on my Icore7 PC with SSD. Regards

Hi,
This error is strange for me, and seems like an old error is back, Should I add a issue to the list of QGIS?

Updated by Bo Victor Thomsen over 3 years ago

Alvaro -
I've just tried the Postgis layer on the latest 32 bit QGIS Master and Current(2.0.1-3) installed with the Osgeo4w installer. They both worked without a hitch. So the problem is isolated to the special install build of QGIS with the speed improvements. I don't think you should report it as a general QGIS issue. Is it possible for you to zip the qgis.exe or maybe the "bin" directory of your working copy of the special QGIS and upload it somewhere I could get it?
Regards

Updated by Alvaro Huarte over 3 years ago

Bo Victor Thomsen wrote:

Alvaro - I've just tried the Postgis layer on the latest 32 bit QGIS Master and Current(2.0.1-3) installed with the Osgeo4w installer. They both worked without a hitch. So the problem is isolated to the special install build of QGIS with the speed improvements. I don't think you should report it as a general QGIS issue. Is it possible for you to zip the qgis.exe or maybe the "bin" directory of your working copy of the special QGIS and upload it somewhere I could get it? Regards

Hi, thanks for your reply!

I found my error. Sorry, rookie mistake, I built the windows installer using the QGIS desktop distribution from the OSGeo4W setup, it is 2.0.X... all wrong.

I remade the installer from my master package and gives me no errors in the postgis loading form.

The new installer is (195mb):
http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v20
Only bin folder (8mb):
http://www.filedropper.com/qgis-osgeo4w-210-f8725-bin-x86v20

Best Regards.

Updated by Alvaro Huarte over 3 years ago

Hi,

There is a bug in QgsOgrFeatureIterator::ensureRelevantFields(), the feature cursor always fetchs all attributes ignoring the relevant fields configuration of the feature request.

I am trying to improve the OGR-GDAL provider and Even Rouault (gdal-dev) help me for find this bug (http://hub.qgis.org/issues/9062).

af73f30c fixes this in master!

In many cases the layers are required to be painted without any field, or one or two fields, therefore, other improvement for render vector layers!

Thanks for all!
Regards

Updated by Giovanni Manghi over 3 years ago

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Hi Giovanni, I prepare in brief.
Thanks for you support!

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Hi,

The new installer (195mb):
http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v30

It also contains a proposed improvement in GDAL ( https://github.com/OSGeo/gdal/pull/26 ).
I hope to be accepted :-)

Note:
This installer does not contain any plugin of python, it is a subset of the QGIS 2.1 release for testing proposes.

Regards

Updated by Giovanni Manghi over 3 years ago

Hi Alvaro

with the new installer QGIS crashes when starting.

Cheers!

Alvaro Huarte wrote:

Giovanni Manghi wrote:

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Hi,

The new installer (195mb): http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v30

It also contains a proposed improvement in GDAL ( https://github.com/OSGeo/gdal/pull/26 ). I hope to be accepted :-)

Note: This installer does not contain any plugin of python, it is a subset of the QGIS 2.1 release for testing proposes.

Regards

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Hi Alvaro

with the new installer QGIS crashes when starting.

Cheers!

Alvaro Huarte wrote:

Giovanni Manghi wrote:

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Hi,

The new installer (195mb): http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v30

It also contains a proposed improvement in GDAL ( https://github.com/OSGeo/gdal/pull/26 ). I hope to be accepted :-)

Note: This installer does not contain any plugin of python, it is a subset of the QGIS 2.1 release for testing proposes.

Regards

Sorry, I can not rebuild well the installer, but it is independent of the changes I'm making.
I should download again all OSGeo4W packages.

I try shortly.

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Giovanni Manghi wrote:

Hi Alvaro

with the new installer QGIS crashes when starting.

Cheers!

Alvaro Huarte wrote:

Giovanni Manghi wrote:

therefore, other improvement for render vector layers!

very good news!

any chance for an updated windows installer? I don't need it as I can compile on Linux, it is just to allow the many Windows user to keep testing your (updated) work.

Hi,

The new installer (195mb): http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v30

It also contains a proposed improvement in GDAL ( https://github.com/OSGeo/gdal/pull/26 ). I hope to be accepted :-)

Note: This installer does not contain any plugin of python, it is a subset of the QGIS 2.1 release for testing proposes.

Regards

Sorry, I can not rebuild well the installer, but it is independent of the changes I'm making. I should download again all OSGeo4W packages.

I try shortly.

I have updated my OSGeo4W packages and I hope that new windows installer is fine.
http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x8640

Best Regards

Updated by Bo Victor Thomsen over 3 years ago

Hi Alvaro -

I just tried your new version of QGIS on my Postgis-based dataset (1.3 million buildings, i.e simple polygons with 4-10 nodes). This dataset crashed the first version of "your" QGIS. Now it works without a hitch. Drawing time is around 10 seconds versus 20 seconds using the ordinary version of QGIS. This is nice ;-) !!

Regards
Bo Victor Thomsen
Aests-GIS
Denmark

Updated by Alvaro Huarte over 3 years ago

Bo Victor Thomsen wrote:

Hi Alvaro -

I just tried your new version of QGIS on my Postgis-based dataset (1.3 million buildings, i.e simple polygons with 4-10 nodes). This dataset crashed the first version of "your" QGIS. Now it works without a hitch. Drawing time is around 10 seconds versus 20 seconds using the ordinary version of QGIS. This is nice ;-) !!

Regards Bo Victor Thomsen Aests-GIS Denmark

You have used the last window installer ( http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x8640 ) ?
if so I'm glad I seem to have fixed my problems by making the installer.

For shapefiles, although it depends on the data (#records, size of geometries...), the improvement is still greater.

Thank you very much!

Updated by Bo Victor Thomsen over 3 years ago

Yes, I did use your latest version of QGIS. The drawing sped of QGIS with the postgis dataprovider is now compareable with MapInfo using it's native file format (Same dataset 1.3 mill. polygons, MapInfo w. tabfile: 8 seconds, QGIS w. Postgis: 10 seconds)
Regards
Bo Victor Thomsen

Updated by Alvaro Huarte over 3 years ago

This up guys! :-)

My proposed pull request (http://trac.osgeo.org/gdal/ticket/5272#comment:7) in GDAL-OGR library for optimize the OGR SHP-provider was accepted and Even Rouault has commited the modification with extra contributions.

The last windows installer already contains this improvement, really it contains my porposed pull, but they are equivalents.
Thanks Even!

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

This up guys! :-)

My proposed pull request (http://trac.osgeo.org/gdal/ticket/5272#comment:7) in GDAL-OGR library for optimize the OGR SHP-provider was accepted and Even Rouault has commited the modification with extra contributions.

very nice done Alvaro!

until now the feedback I received is very good, with no noticeable major issue caused by your patch. We will keep testing.

Updated by João Gaspar over 3 years ago

My feedback:

OS: Windows 7 64-bits
Processor: Intel(R) Core(TM)2 Duo CPU T8300 @ 2.4GHz 2.4GHz
RAM: 4.00Gb

QGIS Version: QGIS standalone Testing of the Alvaro(32-bits)

Results:

I test two shapefiles:

-Contours of Portugal (30 meters distance) size 300mb of capacity (with a lot of table features)

Results:
With patch: 4 seconds to load / Without patch: 25 seconds to load

- Landuse Level 2 of Portugal size 25mb of capacity (also with a lot of table features)

Results:
With patch: 11 seconds to load / Without patch: more then a 1 minute to load

I think if you create a spatial index in shapefile will help this option of rendering and save a cople of seconds to load.

With simplify rendering on, is necessary notice the users that this option improve rendering but create some gaps in the visualization process that can be fix after the load of the shapefile and zooming to the desarible scale.

hope this feedback helps.

Great job Alvaro.

Regards

Updated by Alvaro Huarte over 3 years ago

João Gaspar wrote:

My feedback:

*OS: Windows 7 64-bits Processor: Intel(R) Core(TM)2 Duo CPU T8300 @ 2.4GHz 2.4GHz RAM: 4.00Gb*

QGIS Version: QGIS standalone Testing of the Alvaro(32-bits)

Results:

I test two shapefiles:

-Contours of Portugal (30 meters distance) size 300mb of capacity (with a lot of table features)

Results: With patch: 4 seconds to load / Without patch: 25 seconds to load

- Landuse Level 2 of Portugal size 25mb of capacity (also with a lot of table features)

Results: With patch: 11 seconds to load / Without patch: more then a 1 minute to load

I think if you create a spatial index in shapefile will help this option of rendering and save a cople of seconds to load.

With simplify rendering on, is necessary notice the users that this option improve rendering but create some gaps in the visualization process that can be fix after the load of the shapefile and zooming to the desarible scale.

hope this feedback helps.

Great job Alvaro.

Regards

Thank your very much to all!
I am developing other improvement in GDAL related with this patch. Therefore I will release other windows installer and I will append a "warning" label indicating that increase factor may lead to the emergence of gaps.

But these gaps should not appear in the scroll at least, João, it is right?
Thanks!

Updated by João Gaspar over 3 years ago

Sorry my English sometimes tricks me. :p

I don't understand what you want to say with "not appear in the scroll at least".

Updated by Alvaro Huarte over 3 years ago

João Gaspar wrote:

Sorry my English sometimes tricks me. :p

I don't understand what you want to say with "not appear in the scroll at least".

No, my English is bad :-P, I trying to say: if you view gaps when the value of simplification factor configured (slider in the fast-rendering form of the layer) is the mininum.

Updated by João Gaspar over 3 years ago

Ahaha. ;)

In the minimum level don't appear gaps is "clean and smooth". ;)

Only appears when i increase the slider to high values of simplify geometries.

regards

Updated by Alvaro Huarte over 3 years ago

João Gaspar wrote:

Ahaha. ;)

In the minimum level don't appear gaps is "clean and smooth". ;)

Only appears when i increase the slider to high values of simplify geometries.

regards

:-), ok, That's reassuring to me, with the minimum value occurs simplification but should not leave gaps. I'll put a warning for higher values ​​because in those cases it may appear.

Regards

Updated by Regis Haubourg over 3 years ago

Hi Alvaro, I tested your windows package, this is huge! Many thanks...
As a GIS administrator, I would need a general option in QGIS settings to set default behaviour and simplification level for new layers. Is that possible? We will have to deal qgs migration, could we be sure that layers coming from older QGIS will get these default QGIS settings?
Cheers
Régis

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

João Gaspar wrote:

My feedback:

*OS: Windows 7 64-bits Processor: Intel(R) Core(TM)2 Duo CPU T8300 @ 2.4GHz 2.4GHz RAM: 4.00Gb*

QGIS Version: QGIS standalone Testing of the Alvaro(32-bits)

Results:

I test two shapefiles:

-Contours of Portugal (30 meters distance) size 300mb of capacity (with a lot of table features)

Results: With patch: 4 seconds to load / Without patch: 25 seconds to load

- Landuse Level 2 of Portugal size 25mb of capacity (also with a lot of table features)

Results: With patch: 11 seconds to load / Without patch: more then a 1 minute to load

I think if you create a spatial index in shapefile will help this option of rendering and save a cople of seconds to load.

With simplify rendering on, is necessary notice the users that this option improve rendering but create some gaps in the visualization process that can be fix after the load of the shapefile and zooming to the desarible scale.

hope this feedback helps.

Great job Alvaro.

Regards

Thank your very much to all! I am developing other improvement in GDAL related with this patch. Therefore I will release other windows installer and I will append a "warning" label indicating that increase factor may lead to the emergence of gaps.

But these gaps should not appear in the scroll at least, João, it is right? Thanks!

Hi João,
I added a warning label that appears when the user increases the simplification factor above the minimum level.


Regards

Updated by Alvaro Huarte over 3 years ago

regis Haubourg wrote:

Hi Alvaro, I tested your windows package, this is huge! Many thanks... As a GIS administrator, I would need a general option in QGIS settings to set default behaviour and simplification level for new layers. Is that possible? We will have to deal qgs migration, could we be sure that layers coming from older QGIS will get these default QGIS settings? Cheers Régis

Hi Régis, thanks!
I can append a general option, I write it down on my to-do list.
Regards

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Hi João, I added a warning label that appears when the user increases the simplification factor above the minimum level.

To describe it a little better, check the box performs the drawing simplification and accelerates the render of geometries. If the simplification factor is the minimum value, must not appear gaps, the more you increase it, is more likely to appear and so in these cases I present the warning.

As a user I do not need to increase the default value except in extreme cases of very large datasets.

Updated by João Gaspar over 3 years ago

Yeah really good. Describes perfectly the situation. =)

Regards

Updated by Nyall Dawson over 3 years ago

Can I suggest some improvements in the strings?

"Fast drawing" -> "Simplify geometry"

"Note ..." -> "Note: Enabling this option simplifies geometries in this layer to improve rendering speed. The simplification applies only during rendering of the layer and does not modify the layer geometry."

"Simplification factor..." -> "Simplification factor (higher values result in more simplification):"

"Warning..." -> "Warning: Increasing this factor will further speed up rendering, but may result in gaps or topological errors in the layer display."

Updated by Alvaro Huarte over 3 years ago

Nyall Dawson wrote:

Can I suggest some improvements in the strings?

"Fast drawing" -> "Simplify geometry"

"Note ..." -> "Note: Enabling this option simplifies geometries in this layer to improve rendering speed. The simplification applies only during rendering of the layer and does not modify the layer geometry."

"Simplification factor..." -> "Simplification factor (higher values result in more simplification):"

"Warning..." -> "Warning: Increasing this factor will further speed up rendering, but may result in gaps or topological errors in the layer display."

Done, thank you very much Nyall!

Updated by rph - over 3 years ago

Hi, I've tested this on Win7 64 bit with a previous project containing several raster and vector layers. Surprinsingly the whole set got slower: even while having only 2 layers enabled it got really heavy and also affected the scroll of the layers list. For the moment I'm not able to provide the dataset I was working on but I'll try to recreate the problem with a dataset I can supply.
Trying with a clean project with just one (complex) vector layer did speed up things.

Anyone else noticed something similar?

Updated by Alvaro Huarte over 3 years ago

rph - wrote:

Hi, I've tested this on Win7 64 bit with a previous project containing several raster and vector layers. Surprinsingly the whole set got slower: even while having only 2 layers enabled it got really heavy and also affected the scroll of the layers list. For the moment I'm not able to provide the dataset I was working on but I'll try to recreate the problem with a dataset I can supply. Trying with a clean project with just one (complex) vector layer did speed up things.

Anyone else noticed something similar?

Hi, I tried to reproduce this behavior and have not succeeded. I see no connection with this patch, but anyway this QGIS has not linked all data providers, it only contains most popular for test.

Regards

Updated by Giovanni Manghi over 3 years ago

Anyone else noticed something similar?

can you try remove the layers (one by one) from your project and see if you find one that can be the cause?

Anyone else noticed something similar?

I may have seen this (but not sure) with a specific vector layer, made of a lot of polygons that are not possible to simplify, because it is grid polygon vector made of perfect squares.

In this case seem that qgis takes a lot trying figure how simplify, eventually taking an overhead. But as I said is just an impression as I have not tested it seriously. Example

https://www.dropbox.com/s/2leh363cdbupg1w/grid25_clip.tar.gz

Updated by Alvaro Huarte over 3 years ago

Giovanni Manghi wrote:

Anyone else noticed something similar?

can you try remove the layers (one by one) from your project and see if you find one that can be the cause?

Anyone else noticed something similar?

I may have seen this (but not sure) with a specific vector layer, made of a lot of polygons that are not possible to simplify, because it is grid polygon vector made of perfect squares.

In this case seem that qgis takes a lot trying figure how simplify, eventually taking an overhead. But as I said is just an impression as I have not tested it seriously. Example

https://www.dropbox.com/s/2leh363cdbupg1w/grid25_clip.tar.gz

What you're saying makes sense, I may be trying to simplify polygons with <=5 points (grids) which themselves are not generalizable,
I look, thanks Giovanni!!

Updated by Alvaro Huarte over 3 years ago

Alvaro Huarte wrote:

Giovanni Manghi wrote:

Anyone else noticed something similar?

can you try remove the layers (one by one) from your project and see if you find one that can be the cause?

Anyone else noticed something similar?

I may have seen this (but not sure) with a specific vector layer, made of a lot of polygons that are not possible to simplify, because it is grid polygon vector made of perfect squares.

In this case seem that qgis takes a lot trying figure how simplify, eventually taking an overhead. But as I said is just an impression as I have not tested it seriously. Example

https://www.dropbox.com/s/2leh363cdbupg1w/grid25_clip.tar.gz

What you're saying makes sense, I may be trying to simplify polygons with <=5 points (grids) which themselves are not generalizable, I look, thanks Giovanni!!

I answer to myself :-), the patch try not to simplify polygons of 5 points, or linestrings of 2 points, but try to find out if it is possible to disable the 'AntiAliasing' for render the geometry faster. If the end is not possible to disable the AA (The geometry is not so far), an overload occurs in the process.

This can be avoided disabling for this layer the drawing simplification.

Updated by Alvaro Huarte over 3 years ago

regis Haubourg wrote:

Hi Alvaro, I tested your windows package, this is huge! Many thanks... As a GIS administrator, I would need a general option in QGIS settings to set default behaviour and simplification level for new layers. Is that possible? We will have to deal qgs migration, could we be sure that layers coming from older QGIS will get these default QGIS settings? Cheers Régis

Done!

Regards

Updated by Alvaro Huarte over 3 years ago

  • File deleted (sketch.png)

Updated by Alvaro Huarte over 3 years ago

Hi,

I release a new windows installer which contains:

http://www.filedropper.com/qgis-osgeo4w-210-f8725-setup-x86v50

I made new tests with the latest version to see the speed.

Soft/hard: WindowsXP+SP3, 32bits, 4gb RAM.

My shapefiles tested:

Dataset Value Master Patched
http://faunalia.pt/downloads/GTBs_Shapefile.zip iterations 5 5
total_avg 214,14063 84,29375 39%
total_max 222,53125 85,39063
total_maxdev 29,60938 1,21563
total_min 184,53125 83,07813
total_stdev 14,81578 0,93151
http://faunalia.pt/downloads/MAMMTERR.zip iterations 5 5
total_avg 86,60938 10,60625 12%
total_max 98,96875 13,39063
total_maxdev 46,48438 2,78438
total_min 40,12500 9,81250
total_stdev 23,24727 1,39311
http://www.igeo.pt/scrif/crif/CRIF2011shp.zip iterations 5 5
total_avg 287,47500 67,36250 23%
total_max 307,04688 77,32813
total_maxdev 75,53750 9,96563
total_min 211,93750 64,78125
total_stdev 37,77141 4,98346
https://www.dropbox.com/s/2leh363cdbupg1w/grid25_clip.tar.gz iterations 5 5
total_avg 100,86250 54,27500 54%
total_max 101,20313 55,20313
total_maxdev 0,72187 2,93125
total_min 100,14063 51,34375
total_stdev 0,38151 1,46910
http://idena.navarra.es/descargas/CARTO1_Lin_6CNivelD.zip iterations 5 5
total_avg 7,98125 1,30000 16%
total_max 9,35938 2,14063
total_maxdev 4,30938 0,84063
total_min 3,67188 1,06250
total_stdev 2,16033 0,42175
http://idena.navarra.es/descargas/GEOLOG_Pol_Litologia.zip iterations 5 5
total_avg 9,97188 1,79063 18%
total_max 11,26563 2,84375
total_maxdev 4,95625 1,05313
total_min 5,01563 1,50000
total_stdev 2,47837 0,52697
Urban parcels of cadastre of navarra (Polygon2D of 142680 records, 66mb) iterations 5 5
total_avg 50,28438 10,01563 20%
total_max 55,93750 14,60938
total_maxdev 22,23750 4,59375
total_min 28,04688 8,81250
total_stdev 11,11898 2,29725
Rustic parcels of cadastre of navarra (Polygon2D of 542658 records, 458mb) iterations 5 5
total_avg 10,17500 2,65938 26%
total_max 11,04688 2,68750
total_maxdev 3,37813 0,02813
total_min 6,79688 2,64063
total_stdev 1,68913 0,01822

Updated by Nyall Dawson over 3 years ago

Here's some more string updates for this latest addition:

"By default simplify geometries" -> "Simplify geometries by default"

And for the text below the checkbox:

"If checked, new vector layers added to the map will automatically use geometry simplification to speed up rendering. ... (then the rest of what you have)"

Thanks!

Updated by Nyall Dawson over 3 years ago

Alvaro - these improvements look fantastic! Your hard work is much appreciated!

One related idea I was planning on investigating was adding an option to disable rendering of features smaller than a certain size (in mm). You'll see a similar option in the labelling properties for a vector layer under "Rendering" -> "Suppress labelling of features smaller than ". Having the ability to skip rendering of small features could potentially lead to some more performance gains (as well as having cartographic benefits). Obviously, it should be disabled by default for layers as it would only be useful in some circumstances.

There was a bit of discussion on the QGIS dev list about this a month or so ago -- I had a quick initial attempt at implementing it but have got side tracked on composer work (and it's looking like this will tie me up for the whole of the 2.2 dev cycle).

If you wanted to experiment with this, there's code for checking the size of a feature in mm in QgsPalLayerSettings::checkMinimumSizeMM . I had a quick play here https://github.com/nyalldawson/QGIS/commit/27a42e11b31521a521f8435fc4e82d75568c97cb (no gui, hard coded to a minimum size of 2mm) which yielded some benefit. This was done in QgsVectorLayer::drawRendererV2, I think moving it further up the chain could improve the performance some more.

Updated by Alvaro Huarte over 3 years ago

Nyall Dawson wrote:

Here's some more string updates for this latest addition:

"By default simplify geometries" -> "Simplify geometries by default"

And for the text below the checkbox:

"If checked, new vector layers added to the map will automatically use geometry simplification to speed up rendering. ... (then the rest of what you have)"

Thanks!

Done, thanks Nyall!

Updated by Alvaro Huarte over 3 years ago

Nyall Dawson wrote:

Alvaro - these improvements look fantastic! Your hard work is much appreciated!

One related idea I was planning on investigating was adding an option to disable rendering of features smaller than a certain size (in mm). You'll see a similar option in the labelling properties for a vector layer under "Rendering" -> "Suppress labelling of features smaller than ". Having the ability to skip rendering of small features could potentially lead to some more performance gains (as well as having cartographic benefits). Obviously, it should be disabled by default for layers as it would only be useful in some circumstances.

There was a bit of discussion on the QGIS dev list about this a month or so ago -- I had a quick initial attempt at implementing it but have got side tracked on composer work (and it's looking like this will tie me up for the whole of the 2.2 dev cycle).

If you wanted to experiment with this, there's code for checking the size of a feature in mm in QgsPalLayerSettings::checkMinimumSizeMM . I had a quick play here https://github.com/nyalldawson/QGIS/commit/27a42e11b31521a521f8435fc4e82d75568c97cb (no gui, hard coded to a minimum size of 2mm) which yielded some benefit. This was done in QgsVectorLayer::drawRendererV2, I think moving it further up the chain could improve the performance some more.

Hi Nyall, I did not know this discussion!, I have involved in QGIS later. Coincidences of life, this patch proposes an implementation of that issue. Now, I'm a little waiting for comments and suggestions to see if it is interesting to approval.

On the subject of not paint "very small" elements seems like a good idea in case of layers as you mention, in other cases (very dense layers) would appear whole areas unpainted, but I would not mix this with this patch but on a new patch. If you want I can try to implement it.

Regards

Updated by Nyall Dawson over 3 years ago

Alvaro Huarte wrote:

Hi Nyall, I did not know this discussion!, I have involved in QGIS later. Coincidences of life, this patch proposes an implementation of that issue. Now, I'm a little waiting for comments and suggestions to see if it is interesting to approval.

I know - it's great timing! Thanks for all the work you've put in developing these much needed improvements for QGIS.

On the subject of not paint "very small" elements seems like a good idea in case of layers as you mention, in other cases (very dense layers) would appear whole areas unpainted, but I would not mix this with this patch but on a new patch. If you want I can try to implement it.

I don't want to add to your workload, but it'd be great if you could check it out. I've realized I'm not going to get a chance to tackle this for 2.2. Based on your past work you've got a much better understanding of optimising code like this then I do!

Updated by Alvaro Huarte over 3 years ago

Nyall Dawson wrote:

Alvaro Huarte wrote:

Hi Nyall, I did not know this discussion!, I have involved in QGIS later. Coincidences of life, this patch proposes an implementation of that issue. Now, I'm a little waiting for comments and suggestions to see if it is interesting to approval.

I know - it's great timing! Thanks for all the work you've put in developing these much needed improvements for QGIS.

On the subject of not paint "very small" elements seems like a good idea in case of layers as you mention, in other cases (very dense layers) would appear whole areas unpainted, but I would not mix this with this patch but on a new patch. If you want I can try to implement it.

I don't want to add to your workload, but it'd be great if you could check it out. I've realized I'm not going to get a chance to tackle this for 2.2. Based on your past work you've got a much better understanding of optimising code like this then I do!

Thanks Nyall, I also really appreciate the tests, advice and opinions of the community, they serve me to keep learning every day. And your english corrections ;-)

Updated by Jeremy Palmer over 3 years ago

I've just seen this pull request. Dumb question, but why would you implement the user defined
simplification setting? Seems to me it should be sorted by the software automatically, based on extent and the map to pixel setting. Is also overly complicated for the user...

Updated by Alvaro Huarte over 3 years ago

Jeremy Palmer wrote:

I've just seen this pull request. Dumb question, but why would you implement the user defined simplification setting? Seems to me it should be sorted by the software automatically, based on extent and the map to pixel setting. Is also overly complicated for the user...

Hi Jeremy, the software automatically calculates the simplification factor based on geometry and current map2pixel setting (Slider with minimum value).

I added the possibility of increase this factor to get 'extra' simplification to accelerate even more the render of very large layers.
In this case it is possible that appear gaps and one warning message is showed in the configuration panel.

As I said above the user, in most cases, do not worry about modifying this value, but in other cases it may be interesting.
Regards

Updated by Jeremy Palmer over 3 years ago

Hi Alvaro,

Thanks for the reply. In that case it might be worth stating on the slider label that it is a advanced experimental option.

Cheers,
Jeremy

Updated by Alvaro Huarte over 3 years ago

Jeremy Palmer wrote:

Hi Alvaro,

Thanks for the reply. In that case it might be worth stating on the slider label that it is a advanced experimental option.

Cheers, Jeremy

Hi Jeremy, I suppose we should add an entry in the user manual and explain this configuration.
Thanks!

Alvaro

Updated by Alvaro Huarte over 3 years ago

Hi all, There is a modification in behavior of this patch (https://github.com/qgis/QGIS/pull/980).

Now, it contains the advise of @nyalldawson (https://github.com/qgis/QGIS/pull/1020#issuecomment-30263200) to enable AA-disabling only when the simplification slider is greater than minimum value.

Best regards

Updated by Alvaro Huarte over 3 years ago

I am pleased to announce that this patch seems to have been merged!
:-)

Thank you very much for your support and advices!

Updated by Giovanni Manghi over 3 years ago

Alvaro Huarte wrote:

I am pleased to announce that this patch seems to have been merged! :-)

Thank you very much for your support and advices!

Alvaro, many thanks to YOU :)

Updated by Giovanni Manghi over 3 years ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented

Updated by Alvaro Huarte over 3 years ago

What about add documentation to user manual?

Updated by Alvaro Huarte over 3 years ago

Hi, I committed a new version of geometry simplification to speed up the vector drawing.
https://github.com/ahuarte47/QGIS/tree/Issue_8725-revival-optA-to-B

It is inspired in this PR, but it contains several changes based on advice received ( @matthias-kuhn thanks! ) and new features:

  • About code, the simplification is configured in a new QgsSimplifyMethod class which indicates how to simplify the geometries in a feature iterator.
  • Now, the user can define where the simplification executes (There is a new option in settings panel), locally after fetch the geometry from provider, or simplifying it on provider side. e.g. In postgres provider, first option simplifies the geometry already fetched locally, but the second option simplifies the geometry in database using the function ST_Simplify.
  • The settings panel also shows the simplification threshold in pixel units as @timlinux suggested me.

I have not created a new pull request because of I would greatly appreciate if you could test it with your data and SO's. I have tested it in windows I get similar results as old version for shapefiles (OGR-provider). The new simplification on database side for postgres provider improves the old results too. I will create a table of tests to compare data and configurations.

About postgres simplification, the ST_simplify function needs a tolerance parameter, I use map2pixel/5.0 as input value, it is experimental and I must define it better (All ideas are welcome). This simplification can be applicable to other database providers (MySQL, SQL Server, Oracle...)

Thank you very much!

Also available in: Atom