[Qgis-user] Very Slow Table Operations in QGIS 3.2.3

Tue Oct 9 14:31:54 PDT 2018

My first suggestion would be for, after importing from csv, save it to
geopackages for the rest of the operations. Then, in the end export back to
the necessary file format. I think working directly on text files is not
optimal.

Best regards,

Alexandre Neto

A ter, 9/10/2018, 20:02, Rebecca Bartlett <RebeccaBartlett at cunet.carleton.ca>
escreveu:

> Hello,
>
>
>
> A couple of colleagues and I have been wrestling with a similar issue to
> this and I’ve been hunting around to see if it’s documented, and happily
> saw this email string. I’ve viewed the bug reports (both Kelley’s and the
> related #19771) and while they capture most of what we’ve been running
> into, it’s not an exact match. Here’s our situation and any input would be
> greatly appreciated – including viable workarounds.
>
>
>
> 1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File
> (5721 features; 9 fields; no geometry)
>
> 2.       Using Field Calculator via the attribute table to add a Decimal
> field (length: 10; precision: 2 – the precision is important which is why
> the virtual field workaround without opening the attribute table hasn’t
> worked well with its precision of 3, plus the precision isn’t retained upon
> table export although we may be missing an export option somewhere) and
> input values from an already-existing text field
>
> a.       Copying text field to numeric field for a join
>
> 3.       Field calculation takes less than a second
>
> 4.       Saving the table doesn’t work. I left it running for 90 minutes
> the first time and it has crashed QGIS on colleagues’ computers multiple
> times
>
> a.       Notable that it also doesn’t work on 2.18.24 on Windows machines
> but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we
> had them use 2.18, and only the Windows users had issues)
>
> b.       Still takes a few moments to save on Mac 2.18 – but it works!
>
>
>
> We’ve tried several workarounds and the best one (a.k.a. the one that
> works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via
> the Source Fields tab in Properties, but it still takes >6 minutes to save
> which seems quite long for 5700 features.
>
>
>
> I’m assuming this is the same issue but if there are any reliable
> workarounds we’re all ears!
>
>
>
> Sincerely,
>
> Rebecca
>
>
>
> Rebecca Bartlett
>
> GIS and Digital Resources Librarian
>
> Research Support Services
>
> Carleton University Library
>
> 613-520-2600 x4084
>
> rebecca.bartlett at carleton.ca
>
>
> ------------------------------
>
> Okay, I've created an issue for this situation. Hopefully I've done it
>
> satisfactorily. Thanks again!
>
>
>
> On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:
>
>
>
> >* Hi,*
>
> >
>
> >* Glad to hear that it helped to run the operations on a closed table.*
>
> >* Nevertheless, as Luigi said, it would make sense to submit an issue report*
>
> >* about this slowness.*
>
> >
>
> >* The other thing one has to keep on mind:*
>
> >
>
> >* The SQLite API is kind of limited. Only until recently, if you wanted to*
>
> >* rename a field, one had to copy the whole table with the renamed field*
>
> >* changed in the new table - then the old table had to be removed (all in the*
>
> >* background), because there was API to rename columns in SQLite. I think*
>
> >* with the very recent version, which is not rolled out into most OS, this*
>
> >* has changed. You'd also need a very recent GDAL/OGR.*
>
> >* Greetings,*
>
> >* Andreas*
>
> >
>
> >* Am 03.10.2018 um 21:20 schrieb Kelley Jabr:*
>
> >
>
> >* Luigi, I'm pretty sure you just solved my problem. I tried a few*
>
> >* operations with the table closed and it was like night and day. That 400k*
>
> >* field calculator finished in mere moments! Thank you!!*
>
> >
>
> >* On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:*
>
> >
>
> >>* can you test doing operations with field calculator without having the*
>
> >>* attribute table opened? just a workaround could be to use a virtual layer*
>
> >>* then save the entire layer later.*
>
> >>
>
> >>* Luigi Pirelli*
>
> >>
>
> >>
>
> >>* ***************************************************************************************************
>
> >>* * LinkedIn: https://www.linkedin.com/in/luigipirelli <https://www.linkedin.com/in/luigipirelli>*
>
> >>* * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli <http://gis.stackexchange.com/users/19667/luigi-pirelli>*
>
> >>* * GitHub: https://github.com/luipir <https://github.com/luipir>*
>
> >>* * Mastering QGIS 2nd Edition:*
>
> >>* **
>
> >>* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition <https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition>*
>
> >>* * Hire me: http://goo.gl/BYRQKg <http://goo.gl/BYRQKg>*
>
> >>
>
> >>* ***************************************************************************************************
>
> >>
>
> >>
>
> >>* On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:*
>
> >>
>
> >>>* Hello,*
>
> >>>
>
> >>>* I'm working with a geopackage on ~530k features, and just about*
>
> >>>* everything I try to do in the table takes way longer than expected.*
>
> >>>
>
> >>>*    - Adding a field took 5 minutes (table now has 22 total fields, so*
>
> >>>*    not massive)*
>
> >>>*    - Field Calculator on ~32k selected records took 8 minutes*
>
> >>>*    (literally all I was putting in the cells was: N)*
>
> >>>*    - Field calculator on ~480k records took so many hours that I had to*
>
> >>>*    leave it running over the weekend and when I came back, the Field*
>
> >>>*    Calculator had "closed unexpectedly"*
>
> >>>
>
> >>>* (I'm running on a linux machine with an Intel Core i7 and 32GB of*
>
> >>>* memory.)*
>
> >>>
>
> >>>* Has anyone else experience similar issues? Any advice or recommendations*
>
> >>>* for how to improve performance here?*
>
> >>>
>
> >>>* Thank you,*
>
> >>>
>
> >>>* --*
>
> >>>
>
> >>>* Kelley Jabr*
>
> >>>
>
> >>>* Analyst, Geographic Data Operations*
>
> >>>
>
> >>>* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >>>
>
> >>>* _______________________________________________*
>
> >>>* Qgis-user mailing list*
>
> >>>* Qgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >>>* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >>>* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >>
>
> >>
>
> >
>
> >* --*
>
> >
>
> >* Kelley Jabr*
>
> >
>
> >* Analyst, Geographic Data Operations*
>
> >
>
> >* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >
>
> >* Indeed - We help people get jobs.*
>
> >
>
> >* Indeed.com <http://www.indeed.com <http://www.indeed.com/>>*
>
> >
>
> >
>
> >
>
> >* _______________________________________________*
>
> >* Qgis-user mailing listQgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >
>
> >
>
> >* _______________________________________________*
>
> >* Qgis-user mailing list*
>
> >* Qgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>
>
>
>
>
>
>
> --
>
>
>
> Kelley Jabr
>
>
>
> Analyst, Geographic Data Operations
>
>
>
> kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>
>
>
> Indeed - We help people get jobs.
>
>
>
> Indeed.com <http://www.indeed.com>
>
> -------------- next part --------------
>
> An HTML attachment was scrubbed...
>
> URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>
>
>
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

-- 
Alexandre Neto
---------------------
@AlexNetoGeo
http://sigsemgrilhetas.wordpress.com
http://gisunchained.wordpress.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181009/13b7c810/attachment-0001.html>