[Qgis-user] Very Slow Table Operations in QGIS 3.2.3

Wed Oct 10 07:19:19 PDT 2018

Hello,

I've found the refactor fields process to be a reliable and fast backup
option (plus being a process it now works in the background on a separate
thread and shows a progress meter so you've a good idea if it's actually
working!) if you're happy to have entirely new fields added.

I've had similar issues recently, tried to add a virtual field onto a
geopackage layer of about ~5000 features consisting of a
attribute(get_feature()) and ended up freezing up with no sign of progress
after 15min. Refactor field did 6 new fields at once in about 20 seconds!

James

On Tue, 9 Oct 2018 at 22:32, Alexandre Neto <senhor.neto at gmail.com> wrote:

> My first suggestion would be for, after importing from csv, save it to
> geopackages for the rest of the operations. Then, in the end export back to
> the necessary file format. I think working directly on text files is not
> optimal.
>
> Best regards,
>
> Alexandre Neto
>
>
> A ter, 9/10/2018, 20:02, Rebecca Bartlett <
> RebeccaBartlett at cunet.carleton.ca> escreveu:
>
>> Hello,
>>
>>
>>
>> A couple of colleagues and I have been wrestling with a similar issue to
>> this and I’ve been hunting around to see if it’s documented, and happily
>> saw this email string. I’ve viewed the bug reports (both Kelley’s and the
>> related #19771) and while they capture most of what we’ve been running
>> into, it’s not an exact match. Here’s our situation and any input would be
>> greatly appreciated – including viable workarounds.
>>
>>
>>
>> 1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File
>> (5721 features; 9 fields; no geometry)
>>
>> 2.       Using Field Calculator via the attribute table to add a Decimal
>> field (length: 10; precision: 2 – the precision is important which is why
>> the virtual field workaround without opening the attribute table hasn’t
>> worked well with its precision of 3, plus the precision isn’t retained upon
>> table export although we may be missing an export option somewhere) and
>> input values from an already-existing text field
>>
>> a.       Copying text field to numeric field for a join
>>
>> 3.       Field calculation takes less than a second
>>
>> 4.       Saving the table doesn’t work. I left it running for 90 minutes
>> the first time and it has crashed QGIS on colleagues’ computers multiple
>> times
>>
>> a.       Notable that it also doesn’t work on 2.18.24 on Windows
>> machines but it’s fine on 2.18 with Macs (ran the process in a class of 20
>> after we had them use 2.18, and only the Windows users had issues)
>>
>> b.       Still takes a few moments to save on Mac 2.18 – but it works!
>>
>>
>>
>> We’ve tried several workarounds and the best one (a.k.a. the one that
>> works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via
>> the Source Fields tab in Properties, but it still takes >6 minutes to save
>> which seems quite long for 5700 features.
>>
>>
>>
>> I’m assuming this is the same issue but if there are any reliable
>> workarounds we’re all ears!
>>
>>
>>
>> Sincerely,
>>
>> Rebecca
>>
>>
>>
>> Rebecca Bartlett
>>
>> GIS and Digital Resources Librarian
>>
>> Research Support Services
>>
>> Carleton University Library
>>
>> 613-520-2600 x4084
>>
>> rebecca.bartlett at carleton.ca
>>
>>
>> ------------------------------
>>
>> Okay, I've created an issue for this situation. Hopefully I've done it
>>
>> satisfactorily. Thanks again!
>>
>>
>>
>> On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:
>>
>>
>>
>> >* Hi,*
>>
>> >
>>
>> >* Glad to hear that it helped to run the operations on a closed table.*
>>
>> >* Nevertheless, as Luigi said, it would make sense to submit an issue report*
>>
>> >* about this slowness.*
>>
>> >
>>
>> >* The other thing one has to keep on mind:*
>>
>> >
>>
>> >* The SQLite API is kind of limited. Only until recently, if you wanted to*
>>
>> >* rename a field, one had to copy the whole table with the renamed field*
>>
>> >* changed in the new table - then the old table had to be removed (all in the*
>>
>> >* background), because there was API to rename columns in SQLite. I think*
>>
>> >* with the very recent version, which is not rolled out into most OS, this*
>>
>> >* has changed. You'd also need a very recent GDAL/OGR.*
>>
>> >* Greetings,*
>>
>> >* Andreas*
>>
>> >
>>
>> >* Am 03.10.2018 um 21:20 schrieb Kelley Jabr:*
>>
>> >
>>
>> >* Luigi, I'm pretty sure you just solved my problem. I tried a few*
>>
>> >* operations with the table closed and it was like night and day. That 400k*
>>
>> >* field calculator finished in mere moments! Thank you!!*
>>
>> >
>>
>> >* On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:*
>>
>> >
>>
>> >>* can you test doing operations with field calculator without having the*
>>
>> >>* attribute table opened? just a workaround could be to use a virtual layer*
>>
>> >>* then save the entire layer later.*
>>
>> >>
>>
>> >>* Luigi Pirelli*
>>
>> >>
>>
>> >>
>>
>> >>* ***************************************************************************************************
>>
>> >>* * LinkedIn: https://www.linkedin.com/in/luigipirelli <https://www.linkedin.com/in/luigipirelli>*
>>
>> >>* * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli <http://gis.stackexchange.com/users/19667/luigi-pirelli>*
>>
>> >>* * GitHub: https://github.com/luipir <https://github.com/luipir>*
>>
>> >>* * Mastering QGIS 2nd Edition:*
>>
>> >>* **
>>
>> >>* https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition <https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition>*
>>
>> >>* * Hire me: http://goo.gl/BYRQKg <http://goo.gl/BYRQKg>*
>>
>> >>
>>
>> >>* ***************************************************************************************************
>>
>> >>
>>
>> >>
>>
>> >>* On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:*
>>
>> >>
>>
>> >>>* Hello,*
>>
>> >>>
>>
>> >>>* I'm working with a geopackage on ~530k features, and just about*
>>
>> >>>* everything I try to do in the table takes way longer than expected.*
>>
>> >>>
>>
>> >>>*    - Adding a field took 5 minutes (table now has 22 total fields, so*
>>
>> >>>*    not massive)*
>>
>> >>>*    - Field Calculator on ~32k selected records took 8 minutes*
>>
>> >>>*    (literally all I was putting in the cells was: N)*
>>
>> >>>*    - Field calculator on ~480k records took so many hours that I had to*
>>
>> >>>*    leave it running over the weekend and when I came back, the Field*
>>
>> >>>*    Calculator had "closed unexpectedly"*
>>
>> >>>
>>
>> >>>* (I'm running on a linux machine with an Intel Core i7 and 32GB of*
>>
>> >>>* memory.)*
>>
>> >>>
>>
>> >>>* Has anyone else experience similar issues? Any advice or recommendations*
>>
>> >>>* for how to improve performance here?*
>>
>> >>>
>>
>> >>>* Thank you,*
>>
>> >>>
>>
>> >>>* --*
>>
>> >>>
>>
>> >>>* Kelley Jabr*
>>
>> >>>
>>
>> >>>* Analyst, Geographic Data Operations*
>>
>> >>>
>>
>> >>>* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >>>
>>
>> >>>* _______________________________________________*
>>
>> >>>* Qgis-user mailing list*
>>
>> >>>* Qgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >>>* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >>>* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >>
>>
>> >>
>>
>> >
>>
>> >* --*
>>
>> >
>>
>> >* Kelley Jabr*
>>
>> >
>>
>> >* Analyst, Geographic Data Operations*
>>
>> >
>>
>> >* kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >
>>
>> >* Indeed - We help people get jobs.*
>>
>> >
>>
>> >* Indeed.com <http://www.indeed.com <http://www.indeed.com/>>*
>>
>> >
>>
>> >
>>
>> >
>>
>> >* _______________________________________________*
>>
>> >* Qgis-user mailing listQgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >
>>
>> >
>>
>> >* _______________________________________________*
>>
>> >* Qgis-user mailing list*
>>
>> >* Qgis-user at lists.osgeo.org <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >* List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>> >* Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://lists.osgeo.org/mailman/listinfo/qgis-user>*
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Kelley Jabr
>>
>>
>>
>> Analyst, Geographic Data Operations
>>
>>
>>
>> kjabr at indeed.com <https://lists.osgeo.org/mailman/listinfo/qgis-user>
>>
>>
>>
>> Indeed - We help people get jobs.
>>
>>
>>
>> Indeed.com <http://www.indeed.com>
>>
>> -------------- next part --------------
>>
>> An HTML attachment was scrubbed...
>>
>> URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>
>>
>>
>>
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> --
> Alexandre Neto
> ---------------------
> @AlexNetoGeo
> http://sigsemgrilhetas.wordpress.com
> http://gisunchained.wordpress.com
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181010/a00ed199/attachment.html>