<div dir="ltr">Drag-n-drop the file on QGIS canvas!<div><br></div><div>Regards</div><div>Saber</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 10 January 2018 at 11:26, Jeroen Hovens <span dir="ltr"><<a href="mailto:mail@groenebij.nl" target="_blank">mail@groenebij.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">At the moment I make the selection using the Select by expression tool. I then choose one of the Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.<br>
Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.<br>
The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever<br>
<br>
I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused ☹<br>
<br>
<br>
<br>
<br>
-----Oorspronkelijk bericht-----<br>
Van: Richard Duivenvoorde [mailto:<a href="mailto:rdmailings@duif.net">rdmailings@duif.net</a>]<br>
Verzonden: woensdag 10 januari 2018 10:21<br>
Aan: Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>; 'Ramon Andinach' <<a href="mailto:custard@westnet.com.au">custard@westnet.com.au</a>>; 'qgis-user' <<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
Hi Jeroen,<br>
<br>
Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.<br>
<br>
Not sure how you do your 'selecting'?<br>
Is it spatial or based on zip/postcode/buurtnr/wijknr?<br>
<br>
I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.<br>
<br>
IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:<br>
<br>
Eg searching fro 3 zip codes is milliseconds work:<br>
<br>
grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv<br>
<br>
If all fails, give me a call.<br>
<br>
Regards,<br>
<br>
Richard Duivenvoorde<br>
<br>
On 10-01-18 00:33, Jeroen Hovens wrote:<br>
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the<br>
> Edit<br>
>> Copy command.<br>
><br>
><br>
><br>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no<br>
> x-y data.<br>
><br>
> If you want to recreate the issue, the file can be found here:<br>
><br>
> <a href="https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-" rel="noreferrer" target="_blank">https://www.cbs.nl/nl-nl/<wbr>maatwerk/2017/38/buurt-wijk-<wbr>en-gemeente-2017-</a><br>
> voor-postcode-huisnummer<br>
><br>
> I run into this issue when selecting all objects with Attribute Gem217<br>
> = 310<br>
><br>
> 355 also has this issue. 321 and 356 work without a problem.<br>
><br>
><br>
><br>
> I am using qgis here just as a means to chop the csv file into smaller<br>
> selections which I can then use to edit, add attributes and data to<br>
> and make maps of them.<br>
><br>
><br>
><br>
> Gr.<br>
><br>
> Jeroen<br>
><br>
><br>
><br>
> *Van:* Qgis-user [mailto:<a href="mailto:qgis-user-bounces@lists.osgeo.org">qgis-user-bounces@<wbr>lists.osgeo.org</a>] *Namens<br>
> *Ramon Andinach<br>
> *Verzonden:* dinsdag 9 januari 2018 23:35<br>
> *Aan:* qgis-user <<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue<br>
><br>
><br>
><br>
> Just so we’ve got just about everything covered, does this happen for<br>
> “Edit-> Paste As”?<br>
><br>
><br>
><br>
> Also looking at the size of your original CSV, I’d go along with<br>
> Jeroen, and try and import that csv into some other format first.<br>
><br>
> If you can find a way of making the selection without going near the<br>
> attribute table that will help too.<br>
><br>
><br>
><br>
><br>
><br>
> Regards,<br>
><br>
> Ramon.<br>
><br>
> On 10 Jan 2018, at 06:26, Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a><br>
> <mailto:<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>> wrote:<br>
><br>
><br>
><br>
> Thanks for the ideas.<br>
><br>
><br>
><br>
> The option to save the selection as new csv file as Ramon suggested,<br>
> doesn’t work; qgis goes into ‘not responding’ and no csv file is<br>
> added to the layer list. However, a csv file is made, lacking a<br>
> decent number of objects (first test 15095 of 18582 objects and<br>
> second test is 15179 of 18582 objects, so missing about 3,5k<br>
> objects). This csv file can be opened manually in qgis.<br>
><br>
> And I checked: this options does work for selections where the<br>
> ctrl-c option also worked. Then the new cvs layer is added to the<br>
> layer list in no time.<br>
><br>
><br>
><br>
> Is there some sort of log file in qgis that might give clues to the<br>
> ‘not responding?<br>
><br>
><br>
><br>
> @Kirk<br>
><br>
> I did close and reopen qgis multiple times, even shut down the pc<br>
> entirely, but to no effect. Selections that did allow for copying<br>
> earlier, still work: I can make that selection again and ctrl-c<br>
> works fine. Seems to me this rules out memory-issues.<br>
><br>
><br>
><br>
> Gr.<br>
><br>
> Jeroen<br>
><br>
><br>
><br>
><br>
><br>
> *Van:* Qgis-user<br>
> [mailto:<a href="mailto:qgis-user-bounces@lists.osgeo.org">qgis-user-bounces@<wbr>lists.osgeo.org</a>] *Namens *Kirk Schmidt<br>
> *Verzonden:* dinsdag 9 januari 2018 22:20<br>
> *Aan:* <a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.<wbr>org</a>><br>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue<br>
><br>
><br>
><br>
> After a few selections, you may have to close and reopen qgis to<br>
> purge the memory as well. Importing a csv is memory intensive so<br>
> another approach is to try saving as a spatial database (SQLite) or<br>
> even a shapefile. Once written, you can load the spatial file and<br>
> you should have lots of memory to keep selecting points.<br>
><br>
><br>
><br>
> Kirk<br>
><br>
> On 1/9/2018 4:58 PM, Ramon Andinach wrote:<br>
><br>
> I’m not sure what the problem is, but if I were doing that<br>
> repeatedly, I’d probably use Layer -> Save As, tick the Save<br>
> Only Selected Features box and save it to CSV. Then muck about<br>
> with excel later.<br>
><br>
><br>
><br>
> This way ought to be a bit lighter on the memory.<br>
><br>
><br>
><br>
><br>
><br>
> Ramon.<br>
><br>
> On 10 Jan 2018, at 04:25, Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a><br>
> <mailto:<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>> wrote:<br>
><br>
><br>
><br>
> Hi,<br>
><br>
><br>
><br>
> Using qgis 2.14.18 I have been working today with a large<br>
> csv file (7.5 mln objects). I just need selections of this<br>
> database which I can then modify in excel. So I opened the<br>
> attribute table and made a selection based on a specific<br>
> value for an attribute, and copied that selection (about<br>
> 30.000 objects) via the clipboard (ctrl-c) to a new<br>
> Microsoft Excel file, then saved the excel file and repeated<br>
> the process for another selection. This worked perfect for<br>
> some time. However, some selections won’t copy: when I press<br>
> the copy button the programs turn to ‘not responding’ and<br>
> won’t snap out of it anymore. Selection size when this<br>
> happens is about the same (about 20k to 35k objects).<br>
><br>
><br>
><br>
> What could be the problem? Could the data be corrupted? How<br>
> can I check what is wrong?<br>
><br>
><br>
><br>
> Thank you<br>
><br>
> Gr.<br>
><br>
> Jeroen<br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Qgis-user mailing list<br>
> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.<wbr>org</a>><br>
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
> Unsubscribe:<br>
> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
><br>
> Qgis-user mailing list<br>
><br>
> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.<wbr>org</a>><br>
><br>
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
><br>
> Unsubscribe:<br>
> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
><br>
><br>
><br>
><br>
> --<br>
><br>
> Kirk Schmidt, MScF, BScF, RPF<br>
><br>
> General Manager<br>
><br>
> Nortek Resource Solutions Inc.<br>
><br>
> RR # 1<br>
><br>
> Thorburn, NS<br>
><br>
> B0K 1W0<br>
><br>
> Tel (902) 922.3607<br>
><br>
> Email: <a href="mailto:kirk@nortekresources.com">kirk@nortekresources.com</a> <mailto:<a href="mailto:kirk@nortekresources.com">kirk@nortekresources.<wbr>com</a>><br>
><br>
> Web: <a href="http://www.nortekreources.com" rel="noreferrer" target="_blank">www.nortekreources.com</a> <<a href="http://www.nortekreources.com/" rel="noreferrer" target="_blank">http://www.nortekreources.<wbr>com/</a>><br>
><br>
> ______________________________<wbr>_________________<br>
> Qgis-user mailing list<br>
> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.<wbr>org</a>><br>
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
><br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Qgis-user mailing list<br>
> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a><br>
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
><br>
<br>
<br>
______________________________<wbr>_________________<br>
Qgis-user mailing list<br>
<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-user</a></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Saber Razmjooei<br></div><div><a href="http://www.lutraconsulting.co.uk" target="_blank">www.lutraconsulting.co.uk</a><br><span>+44 (0)7568 129733</span><br></div></div></div></div></div></div></div></div></div></div>
</div>