<div dir="ltr">Hello,<div><br></div><div>I've found the refactor fields process to be a reliable and fast backup option (plus being a process it now works in the background on a separate thread and shows a progress meter so you've a good idea if it's actually working!) if you're happy to have entirely new fields added.</div><div><br></div><div>I've had similar issues recently, tried to add a virtual field onto a geopackage layer of about ~5000 features consisting of a attribute(get_feature()) and ended up freezing up with no sign of progress after 15min. Refactor field did 6 new fields at once in about 20 seconds!</div><div><br></div><div>James</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 9 Oct 2018 at 22:32, Alexandre Neto <<a href="mailto:senhor.neto@gmail.com" target="_blank">senhor.neto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>My first suggestion would be for, after importing from csv, save it to geopackages for the rest of the operations. Then, in the end export back to the necessary file format. I think working directly on text files is not optimal. </span><div><br></div><div>Best regards, </div><div><br></div><div>Alexandre Neto</div><div><br><div dir="auto"><div dir="auto"><br><div class="gmail_quote"><div dir="ltr">A ter, 9/10/2018, 20:02, Rebecca Bartlett <<a href="mailto:RebeccaBartlett@cunet.carleton.ca" target="_blank">RebeccaBartlett@cunet.carleton.ca</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" id="m_6391222440398778127m_2782308287244269906gmail_block_quote0">
<div lang="EN-CA" link="#0563C1" vlink="#954F72">
<div class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880WordSection1">
<p class="MsoNormal"><span style="color:black">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">A couple of colleagues and I have been wrestling with a similar issue to this and I’ve been hunting around to see if it’s documented, and happily saw this email string. I’ve viewed the bug reports (both Kelley’s
and the related #19771) and while they capture most of what we’ve been running into, it’s not an exact match. Here’s our situation and any input would be greatly appreciated – including viable workarounds.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph"><u></u><span style="color:black"><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Adding a .txt or .csv file to QGIS 3.2.1 as Delimited Text File (5721 features; 9 fields; no geometry)<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph"><u></u><span style="color:black"><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Using Field Calculator via the attribute table to add a Decimal field (length: 10; precision: 2 – the precision is important which is why the virtual field workaround without opening the attribute table
hasn’t worked well with its precision of 3, plus the precision isn’t retained upon table export although we may be missing an export option somewhere) and input values from an already-existing text field
<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph" style="margin-left:72.0pt">
<u></u><span style="color:black"><span>a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Copying text field to numeric field for a join<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph"><u></u><span style="color:black"><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Field calculation takes less than a second<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph"><u></u><span style="color:black"><span>4.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Saving the table doesn’t work. I left it running for 90 minutes the first time and it has crashed QGIS on colleagues’ computers multiple times<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph" style="margin-left:72.0pt">
<u></u><span style="color:black"><span>a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Notable that it also doesn’t work on 2.18.24 on Windows machines but it’s fine on 2.18 with Macs (ran the process in a class of 20 after we had them use 2.18, and only the Windows users had issues)<u></u><u></u></span></p>
<p class="m_6391222440398778127m_2782308287244269906m_-2728571771589215880MsoListParagraph" style="margin-left:72.0pt">
<u></u><span style="color:black"><span>b.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="color:black">Still takes a few moments to save on Mac 2.18 – but it works!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">We’ve tried several workarounds and the best one (a.k.a. the one that works in 3.2.1 and 2.18.24 in Windows) is to use the Field Calculator via the Source Fields tab in Properties, but it still takes >6 minutes
to save which seems quite long for 5700 features.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">I’m assuming this is the same issue but if there are any reliable workarounds we’re all ears!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:black">Sincerely,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black">Rebecca</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Rebecca Bartlett<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">GIS and Digital Resources Librarian<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Research Support Services<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Carleton University Library<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">613-520-2600 x4084<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="mailto:rebecca.bartlett@carleton.ca" target="_blank">rebecca.bartlett@carleton.ca</a>
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:black"><u></u> <u></u></span></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" noshade style="color:black" align="center">
</div>
<pre style="white-space:pre-wrap;font-variant-ligatures:normal;font-variant-caps:normal;text-align:start;text-decoration-style:initial;text-decoration-color:initial;word-spacing:0px"><span style="color:black">Okay, I've created an issue for this situation. Hopefully I've done it<u></u><u></u></span></pre>
<pre><span style="color:black">satisfactorily. Thanks again!<u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">On Wed, Oct 3, 2018 at 2:31 PM Andreas Neumann <<a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">a.neumann at carto.net</a>> wrote:<u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">><i> Hi,<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Glad to hear that it helped to run the operations on a closed table.<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Nevertheless, as Luigi said, it would make sense to submit an issue report<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> about this slowness.<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> The other thing one has to keep on mind:<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> The SQLite API is kind of limited. Only until recently, if you wanted to<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> rename a field, one had to copy the whole table with the renamed field<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> changed in the new table - then the old table had to be removed (all in the<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> background), because there was API to rename columns in SQLite. I think<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> with the very recent version, which is not rolled out into most OS, this<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> has changed. You'd also need a very recent GDAL/OGR.<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Greetings,<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Andreas<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Am 03.10.2018 um 21:20 schrieb Kelley Jabr:<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Luigi, I'm pretty sure you just solved my problem. I tried a few<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> operations with the table closed and it was like night and day. That 400k<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> field calculator finished in mere moments! Thank you!!<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> On Wed, Oct 3, 2018 at 1:42 PM Luigi Pirelli <<a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">luipir at gmail.com</a>> wrote:<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i> can you test doing operations with field calculator without having the<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> attribute table opened? just a workaround could be to use a virtual layer<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> then save the entire layer later.<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i> Luigi Pirelli<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i> **************************************************************************************************<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> * LinkedIn: <a href="https://www.linkedin.com/in/luigipirelli" target="_blank">https://www.linkedin.com/in/luigipirelli</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> * Stackexchange: <a href="http://gis.stackexchange.com/users/19667/luigi-pirelli" target="_blank">http://gis.stackexchange.com/users/19667/luigi-pirelli</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> * GitHub: <a href="https://github.com/luipir" target="_blank">https://github.com/luipir</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> * Mastering QGIS 2nd Edition:<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> *<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> <a href="https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition" target="_blank">https://www.packtpub.com/big-data-and-business-intelligence/mastering-qgis-second-edition</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i> * Hire me: <a href="http://goo.gl/BYRQKg" target="_blank">http://goo.gl/BYRQKg</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i> **************************************************************************************************<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i> On Wed, 3 Oct 2018 at 18:11, Kelley Jabr <<a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">kjabr at indeed.com</a>> wrote:<u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> Hello,<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> I'm working with a geopackage on ~530k features, and just about<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> everything I try to do in the table takes way longer than expected.<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> - Adding a field took 5 minutes (table now has 22 total fields, so<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> not massive)<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> - Field Calculator on ~32k selected records took 8 minutes<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> (literally all I was putting in the cells was: N)<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> - Field calculator on ~480k records took so many hours that I had to<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> leave it running over the weekend and when I came back, the Field<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> Calculator had "closed unexpectedly"<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> (I'm running on a linux machine with an Intel Core i7 and 32GB of<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> memory.)<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> Has anyone else experience similar issues? Any advice or recommendations<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> for how to improve performance here?<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> Thank you,<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> --<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> Kelley Jabr<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> Analyst, Geographic Data Operations<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">kjabr at indeed.com</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>>><i> _______________________________________________<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> Qgis-user mailing list<u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">Qgis-user at lists.osgeo.org</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>>><i> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">>><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> --<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Kelley Jabr<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Analyst, Geographic Data Operations<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">kjabr at indeed.com</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Indeed - We help people get jobs.<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> Indeed.com <<a href="http://www.indeed.com/" target="_blank">http://www.indeed.com</a>><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> _______________________________________________<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Qgis-user mailing <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">listQgis-user at lists.osgeo.org</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i><u></u> <u></u></i></span></pre>
<pre><span style="color:black">><i> _______________________________________________<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Qgis-user mailing list<u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">Qgis-user at lists.osgeo.org</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black">><i> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><u></u><u></u></i></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">-- <u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">Kelley Jabr<u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">Analyst, Geographic Data Operations<u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black"><a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">kjabr at indeed.com</a><u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">Indeed - We help people get jobs.<u></u><u></u></span></pre>
<pre><span style="color:black"><u></u> <u></u></span></pre>
<pre><span style="color:black">Indeed.com <<a href="http://www.indeed.com/" target="_blank">http://www.indeed.com</a>><u></u><u></u></span></pre>
<pre><span style="color:black">-------------- next part --------------<u></u><u></u></span></pre>
<pre><span style="color:black">An HTML attachment was scrubbed...<u></u><u></u></span></pre>
<pre><span style="color:black">URL: <<a href="http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html" target="_blank">http://lists.osgeo.org/pipermail/qgis-user/attachments/20181003/2c54a49e/attachment-0001.html</a>><u></u><u></u></span></pre>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
Qgis-user mailing list<br>
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank">Qgis-user@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a></blockquote></div></div></div></div>-- <br><div dir="ltr" class="m_6391222440398778127m_2782308287244269906gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Alexandre Neto</div><div>---------------------</div><div>@AlexNetoGeo</div><div><a href="http://sigsemgrilhetas.wordpress.com" target="_blank">http://sigsemgrilhetas.wordpress.com</a></div><a href="http://gisunchained.wordpress.com" target="_blank">http://gisunchained.wordpress.com</a><br></div></div>
_______________________________________________<br>
Qgis-user mailing list<br>
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank">Qgis-user@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a></blockquote></div>