[Qgis-user] Very Slow Table Operations in QGIS 3.2.3

Tue Oct 9 12:00:38 PDT 2018

Hello,

A couple of colleagues and I have been wrestling with a similar issue to this and I've been hunting around to see if it's documented, and happily saw this email string. I've viewed the bug reports (both Kelley's and the related #19771) and while they capture most of what we've been running into, it's not an exact match. Here's our situation and any input would be greatly appreciated - including viable workarounds.

1.       Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File (5721 features; 9 fields; no geometry)

2.       Using Field Calculator via the attribute table to add a Decimal field (length: 10; precision: 2 - the precision is important which is why the virtual field workaround without opening the attribute table hasn't worked well with its precision of 3, plus the precision isn't retained upon table export although we may be missing an export option somewhere) and input values from an already-existing text field

a.       Copying text field to numeric field for a join

3.       Field calculation takes less than a second

4.       Saving the table doesn't work. I left it running for 90 minutes the first time and it has crashed QGIS on colleagues' computers multiple times

a.       Notable that it also doesn't work on 2.18.24 on Windows machines but it's fine on 2.18 with Macs (ran the process in a class of 20 after we had them use 2.18, and only the Windows users had issues)

b.       Still takes a few moments to save on Mac 2.18 - but it works!

We've tried several workarounds and the best one (a.k.a. the one that works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via the Source Fields tab in Properties, but it still takes >6 minutes to save which seems quite long for 5700 features.

I'm assuming this is the same issue but if there are any reliable workarounds we're all ears!

Sincerely,
Rebecca

Rebecca Bartlett
GIS and Digital Resources Librarian
Research Support Services
Carleton University Library
613-520-2600 x4084
rebecca.bartlett at carleton.ca<mailto:rebecca.bartlett at carleton.ca>

________________________________

Okay, I've created an issue for this situation. Hopefully I've done it

satisfactorily. Thanks again!

On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <a.neumann at carto.net<https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:

> Hi,

>

> Glad to hear that it helped to run the operations on a closed table.

> Nevertheless, as Luigi said, it would make sense to submit an issue report

> about this slowness.

>

> The other thing one has to keep on mind:

>

> The SQLite API is kind of limited. Only until recently, if you wanted to

> rename a field, one had to copy the whole table with the renamed field

> changed in the new table - then the old table had to be removed (all in the

> background), because there was API to rename columns in SQLite. I think

> with the very recent version, which is not rolled out into most OS, this

> has changed. You'd also need a very recent GDAL/OGR.

> Greetings,

> Andreas

>

> Am 03.10.2018 um 21:20 schrieb Kelley Jabr:

>

> Luigi, I'm pretty sure you just solved my problem. I tried a few

> operations with the table closed and it was like night and day. That 400k

> field calculator finished in mere moments! Thank you!!

>

> On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <luipir at gmail.com<https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:

>

>> can you test doing operations with field calculator without having the

>> attribute table opened? just a workaround could be to use a virtual layer

>> then save the entire layer later.

>>

>> Luigi Pirelli

>>

>>

>> **************************************************************************************************

>> * LinkedIn: https://www.linkedin.com/in/luigipirelli

>> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli

>> * GitHub: https://github.com/luipir

>> * Mastering QGIS 2nd Edition:

>> *

>> https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition

>> * Hire me: http://goo.gl/BYRQKg

>>

>> **************************************************************************************************

>>

>>

>> On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <kjabr at indeed.com<https://lists.osgeo.org/mailman/listinfo/qgis-user>> wrote:

>>

>>> Hello,

>>>

>>> I'm working with a geopackage on ~530k features, and just about

>>> everything I try to do in the table takes way longer than expected.

>>>

>>>    - Adding a field took 5 minutes (table now has 22 total fields, so

>>>    not massive)

>>>    - Field Calculator on ~32k selected records took 8 minutes

>>>    (literally all I was putting in the cells was: N)

>>>    - Field calculator on ~480k records took so many hours that I had to

>>>    leave it running over the weekend and when I came back, the Field

>>>    Calculator had "closed unexpectedly"

>>>

>>> (I'm running on a linux machine with an Intel Core i7 and 32GB of

>>> memory.)

>>>

>>> Has anyone else experience similar issues? Any advice or recommendations

>>> for how to improve performance here?

>>>

>>> Thank you,

>>>

>>> --

>>>

>>> Kelley Jabr

>>>

>>> Analyst, Geographic Data Operations

>>>

>>> kjabr at indeed.com<https://lists.osgeo.org/mailman/listinfo/qgis-user>

>>>

>>> _______________________________________________

>>> Qgis-user mailing list

>>> Qgis-user at lists.osgeo.org<https://lists.osgeo.org/mailman/listinfo/qgis-user>

>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user

>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

>>

>>

>

> --

>

> Kelley Jabr

>

> Analyst, Geographic Data Operations

>

> kjabr at indeed.com<https://lists.osgeo.org/mailman/listinfo/qgis-user>

>

> Indeed - We help people get jobs.

>

> Indeed.com <http://www.indeed.com<http://www.indeed.com/>>

>

>

>

> _______________________________________________

> Qgis-user mailing listQgis-user at lists.osgeo.org<https://lists.osgeo.org/mailman/listinfo/qgis-user>

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user

> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

>

>

> _______________________________________________

> Qgis-user mailing list

> Qgis-user at lists.osgeo.org<https://lists.osgeo.org/mailman/listinfo/qgis-user>

> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user

> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

--

Kelley Jabr

Analyst, Geographic Data Operations

kjabr at indeed.com<https://lists.osgeo.org/mailman/listinfo/qgis-user>

Indeed - We help people get jobs.

Indeed.com <http://www.indeed.com<http://www.indeed.com/>>

-------------- next part --------------

An HTML attachment was scrubbed...

URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20181009/ec4bfec4/attachment.html>