[Qgis-user] QGIS loaded millions of entries

[Alessandro Pasotti]

On Sat, Mar 4, 2017 at 8:26 AM, Tobias Wendorff <
tobias.wendorff at tu-dortmund.de> wrote:

> Am Sa, 4.03.2017, 03:49 schrieb Nyall Dawson:
> >> I've loaded a PostgreSQL table with more than 3 * 10^6 entries.
> >> When accesing the attributes, QGIS downloads all the attributes!
> >> QGIS first gets instable and crashes afterwards.
> >>
> >> Actually, it would be more useful to download only the attributes,
> >> which are in the current view.
> >>
> >> [...]
> >
> > Yep - it's really annoying for large layers. If you go to settings ->
> > options -> data sources you can change "attribute table behaviour" to
> > "show features visible on map". That'll help a bit.
>
> Thanks, I'll try it out soon.
>
> > I've got changes incoming which also improve the performance if you
> > have the "show selected features" option enabled
> > (https://github.com/qgis/QGIS/pull/4164).
>
> Another improvement would be to request the data in stages. Let's
> the attribute view shows 500 entries on each page. For shapefiles,
> sqlite and other database-like formats, it should work with OFFSET
> and LIMIT to minimize the amount of data being loaded:
>
> First page: SELECT * FROM xy OFFSET 0 LIMIT 500
> Second page: SELECT * FROM xy OFFSET 500 LIMIT 500
> etc.
>
> That's what many tools for databank manipulation do.
>
>

Yes, that would be a strategy, but you might run into problem with sorting
and filtering if the data provider does not support for it.

I think that depending on provider's capabilities and number of records in
the data source, different strategies should be implemented: for huge
tables I suppose that a threaded loader or a paged loader are the way to go.

My 2 € cents

-- 
Alessandro Pasotti
w3:   www.itopen.it
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20170304/fe1aacc9/attachment.html>