<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.E-mailStijl18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=NL link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='mso-fareast-language:EN-US'>Update: Solution.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>After saving the csv file as a shape-file and then loading the .dbf file into qgis everything worked as intended.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>Although Qgis couldn’t create the .shp file and the .dbf file actually ‘grew’ from 227 MB (.cvs) to 9.4 GB (.dbf), I can now copy any selection without any problem.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>And along the way I discovered the option Split Vector Layer, which is what I was doing, however I was doing it one at the time, instead of creating 388 new layers at once (because I need just a few).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>And Split Vector Layer doesn’t work fort the .csv file or the .dbf by the way, it is probably missing some information.<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'>So, still no idea what is going wrong in the .cvs file, but I have a workable solution.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>Thank you all for your thoughts.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>Gr.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='mso-fareast-language:EN-US'>Jeroen<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><b>Van:</b> Saber Razmjooei [mailto:saber.razmjooei@lutraconsulting.co.uk] <br><b>Verzonden:</b> woensdag 10 januari 2018 12:33<br><b>Aan:</b> Jeroen Hovens <mail@groenebij.nl><br><b>CC:</b> Richard <span style='font-family:"Segoe UI Emoji",sans-serif'>🌍</span> Duivenvoorde <richard@duif.net>; Ramon Andinach <custard@westnet.com.au>; qgis-user <qgis-user@lists.osgeo.org><br><b>Onderwerp:</b> Re: [Qgis-user] ctrl-c selection issue<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Drag-n-drop the file on QGIS canvas!<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Regards<o:p></o:p></p></div><div><p class=MsoNormal>Saber<o:p></o:p></p></div></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On 10 January 2018 at 11:26, Jeroen Hovens <<a href="mailto:mail@groenebij.nl" target="_blank">mail@groenebij.nl</a>> wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><p class=MsoNormal>At the moment I make the selection using the Select by expression tool. I then choose one of the Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.<br>Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.<br>The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever<br><br>I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused <span style='font-family:"Segoe UI Emoji",sans-serif'>☹</span><br><br><br><br><br>-----Oorspronkelijk bericht-----<br>Van: Richard Duivenvoorde [mailto:<a href="mailto:rdmailings@duif.net">rdmailings@duif.net</a>]<br>Verzonden: woensdag 10 januari 2018 10:21<br>Aan: Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>; 'Ramon Andinach' <<a href="mailto:custard@westnet.com.au">custard@westnet.com.au</a>>; 'qgis-user' <<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>Onderwerp: Re: [Qgis-user] ctrl-c selection issue<o:p></o:p></p><div><div><p class=MsoNormal><br><br>Hi Jeroen,<br><br>Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.<br><br>Not sure how you do your 'selecting'?<br>Is it spatial or based on zip/postcode/buurtnr/wijknr?<br><br>I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.<br><br>IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:<br><br>Eg searching fro 3 zip codes is milliseconds work:<br><br>grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv<br><br>If all fails, give me a call.<br><br>Regards,<br><br>Richard Duivenvoorde<br><br>On 10-01-18 00:33, Jeroen Hovens wrote:<br>> Hmm, I can’t use Edit > Paste As, because qgis does not finish the<br>> Edit<br>>> Copy command.<br>><br>><br>><br>> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no<br>> x-y data.<br>><br>> If you want to recreate the issue, the file can be found here:<br>><br>> <a href="https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-" target="_blank">https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-</a><br>> voor-postcode-huisnummer<br>><br>> I run into this issue when selecting all objects with Attribute Gem217<br>> = 310<br>><br>> 355 also has this issue. 321 and 356 work without a problem.<br>><br>><br>><br>> I am using qgis here just as a means to chop the csv file into smaller<br>> selections which I can then use to edit, add attributes and data to<br>> and make maps of them.<br>><br>><br>><br>> Gr.<br>><br>> Jeroen<br>><br>><br>><br>> *Van:* Qgis-user [mailto:<a href="mailto:qgis-user-bounces@lists.osgeo.org">qgis-user-bounces@lists.osgeo.org</a>] *Namens<br>> *Ramon Andinach<br>> *Verzonden:* dinsdag 9 januari 2018 23:35<br>> *Aan:* qgis-user <<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue<br>><br>><br>><br>> Just so we’ve got just about everything covered, does this happen for<br>> “Edit-> Paste As”?<br>><br>><br>><br>> Also looking at the size of your original CSV, I’d go along with<br>> Jeroen, and try and import that csv into some other format first.<br>><br>> If you can find a way of making the selection without going near the<br>> attribute table that will help too.<br>><br>><br>><br>><br>><br>> Regards,<br>><br>> Ramon.<br>><br>> On 10 Jan 2018, at 06:26, Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a><br>> <mailto:<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>> wrote:<br>><br>><br>><br>> Thanks for the ideas.<br>><br>><br>><br>> The option to save the selection as new csv file as Ramon suggested,<br>> doesn’t work; qgis goes into ‘not responding’ and no csv file is<br>> added to the layer list. However, a csv file is made, lacking a<br>> decent number of objects (first test 15095 of 18582 objects and<br>> second test is 15179 of 18582 objects, so missing about 3,5k<br>> objects). This csv file can be opened manually in qgis.<br>><br>> And I checked: this options does work for selections where the<br>> ctrl-c option also worked. Then the new cvs layer is added to the<br>> layer list in no time.<br>><br>><br>><br>> Is there some sort of log file in qgis that might give clues to the<br>> ‘not responding?<br>><br>><br>><br>> @Kirk<br>><br>> I did close and reopen qgis multiple times, even shut down the pc<br>> entirely, but to no effect. Selections that did allow for copying<br>> earlier, still work: I can make that selection again and ctrl-c<br>> works fine. Seems to me this rules out memory-issues.<br>><br>><br>><br>> Gr.<br>><br>> Jeroen<br>><br>><br>><br>><br>><br>> *Van:* Qgis-user<br>> [mailto:<a href="mailto:qgis-user-bounces@lists.osgeo.org">qgis-user-bounces@lists.osgeo.org</a>] *Namens *Kirk Schmidt<br>> *Verzonden:* dinsdag 9 januari 2018 22:20<br>> *Aan:* <a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:qgis-user@lists.osgeo.org">qgis-user@lists.osgeo.org</a>><br>> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue<br>><br>><br>><br>> After a few selections, you may have to close and reopen qgis to<br>> purge the memory as well. Importing a csv is memory intensive so<br>> another approach is to try saving as a spatial database (SQLite) or<br>> even a shapefile. Once written, you can load the spatial file and<br>> you should have lots of memory to keep selecting points.<br>><br>><br>><br>> Kirk<br>><br>> On 1/9/2018 4:58 PM, Ramon Andinach wrote:<br>><br>> I’m not sure what the problem is, but if I were doing that<br>> repeatedly, I’d probably use Layer -> Save As, tick the Save<br>> Only Selected Features box and save it to CSV. Then muck about<br>> with excel later.<br>><br>><br>><br>> This way ought to be a bit lighter on the memory.<br>><br>><br>><br>><br>><br>> Ramon.<br>><br>> On 10 Jan 2018, at 04:25, Jeroen Hovens <<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a><br>> <mailto:<a href="mailto:mail@groenebij.nl">mail@groenebij.nl</a>>> wrote:<br>><br>><br>><br>> Hi,<br>><br>><br>><br>> Using qgis 2.14.18 I have been working today with a large<br>> csv file (7.5 mln objects). I just need selections of this<br>> database which I can then modify in excel. So I opened the<br>> attribute table and made a selection based on a specific<br>> value for an attribute, and copied that selection (about<br>> 30.000 objects) via the clipboard (ctrl-c) to a new<br>> Microsoft Excel file, then saved the excel file and repeated<br>> the process for another selection. This worked perfect for<br>> some time. However, some selections won’t copy: when I press<br>> the copy button the programs turn to ‘not responding’ and<br>> won’t snap out of it anymore. Selection size when this<br>> happens is about the same (about 20k to 35k objects).<br>><br>><br>><br>> What could be the problem? Could the data be corrupted? How<br>> can I check what is wrong?<br>><br>><br>><br>> Thank you<br>><br>> Gr.<br>><br>> Jeroen<br>><br>><br>><br>> _______________________________________________<br>> Qgis-user mailing list<br>> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a>><br>> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>> Unsubscribe:<br>> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>><br>><br>><br>><br>><br>><br>><br>> _______________________________________________<br>><br>> Qgis-user mailing list<br>><br>> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a>><br>><br>> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>><br>> Unsubscribe:<br>> <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>><br>><br>><br>><br>> --<br>><br>> Kirk Schmidt, MScF, BScF, RPF<br>><br>> General Manager<br>><br>> Nortek Resource Solutions Inc.<br>><br>> RR # 1<br>><br>> Thorburn, NS<br>><br>> B0K 1W0<br>><br>> Tel (902) 922.3607<br>><br>> Email: <a href="mailto:kirk@nortekresources.com">kirk@nortekresources.com</a> <mailto:<a href="mailto:kirk@nortekresources.com">kirk@nortekresources.com</a>><br>><br>> Web: <a href="http://www.nortekreources.com" target="_blank">www.nortekreources.com</a> <<a href="http://www.nortekreources.com/" target="_blank">http://www.nortekreources.com/</a>><br>><br>> _______________________________________________<br>> Qgis-user mailing list<br>> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a> <mailto:<a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a>><br>> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>><br>><br>><br>><br>><br>> _______________________________________________<br>> Qgis-user mailing list<br>> <a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a><br>> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>><br><br><br>_______________________________________________<br>Qgis-user mailing list<br><a href="mailto:Qgis-user@lists.osgeo.org">Qgis-user@lists.osgeo.org</a><br>List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br>Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank">https://lists.osgeo.org/mailman/listinfo/qgis-user</a><o:p></o:p></p></div></div></blockquote></div><p class=MsoNormal><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <o:p></o:p></p><div><div><div><div><div><div><div><div><div><div><p class=MsoNormal>Saber Razmjooei<o:p></o:p></p></div><div><p class=MsoNormal><a href="http://www.lutraconsulting.co.uk" target="_blank">www.lutraconsulting.co.uk</a><br>+44 (0)7568 129733<o:p></o:p></p></div></div></div></div></div></div></div></div></div></div></div></div></body></html>