[Qgis-user] ctrl-c selection issue
Jeroen Hovens
mail at groenebij.nl
Wed Jan 10 05:58:46 PST 2018
Update: Solution.
After saving the csv file as a shape-file and then loading the .dbf file into qgis everything worked as intended.
Although Qgis couldn’t create the .shp file and the .dbf file actually ‘grew’ from 227 MB (.cvs) to 9.4 GB (.dbf), I can now copy any selection without any problem.
And along the way I discovered the option Split Vector Layer, which is what I was doing, however I was doing it one at the time, instead of creating 388 new layers at once (because I need just a few).
And Split Vector Layer doesn’t work fort the .csv file or the .dbf by the way, it is probably missing some information.
So, still no idea what is going wrong in the .cvs file, but I have a workable solution.
Thank you all for your thoughts.
Gr.
Jeroen
Van: Saber Razmjooei [mailto:saber.razmjooei at lutraconsulting.co.uk]
Verzonden: woensdag 10 januari 2018 12:33
Aan: Jeroen Hovens <mail at groenebij.nl>
CC: Richard 🌍 Duivenvoorde <richard at duif.net>; Ramon Andinach <custard at westnet.com.au>; qgis-user <qgis-user at lists.osgeo.org>
Onderwerp: Re: [Qgis-user] ctrl-c selection issue
Drag-n-drop the file on QGIS canvas!
Regards
Saber
On 10 January 2018 at 11:26, Jeroen Hovens <mail at groenebij.nl <mailto:mail at groenebij.nl> > wrote:
At the moment I make the selection using the Select by expression tool. I then choose one of the Gem2017 values (each object is an address, and has one of 388 unique Gem217 values). I want to select all objects bwith one unique Gem2017 value.
Making the selection is no problem, I can make any selection I want. Making the selection usually takes less then a minute.
The problem is in copying specific selections to the clipboard or saving a selection as a new file. Copying a selection of about 20.000 objects from the 7,5 mln takes about a second. Unless the issue arises, then the copy process takes forever
I have no experience with Postgis or using databases. I tried saving the csv file as a sqlite, but then had no idea how to re-open that file. The dialogbox got me all confused ☹
-----Oorspronkelijk bericht-----
Van: Richard Duivenvoorde [mailto:rdmailings at duif.net <mailto:rdmailings at duif.net> ]
Verzonden: woensdag 10 januari 2018 10:21
Aan: Jeroen Hovens <mail at groenebij.nl <mailto:mail at groenebij.nl> >; 'Ramon Andinach' <custard at westnet.com.au <mailto:custard at westnet.com.au> >; 'qgis-user' <qgis-user at lists.osgeo.org <mailto:qgis-user at lists.osgeo.org> >
Onderwerp: Re: [Qgis-user] ctrl-c selection issue
Hi Jeroen,
Loading a csv in QGIS is not optimal. I think csv does not have an index, so when you create a small selection, QGIS has to run over all those 7,5 mln object every time.
Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?
I think I would either load it in Postgis (if available) or save as GeoPackage (better use QGIS 2.99 for that) or if all fails, create a shapefile for it. AND then create indexes on desired columns.
IF you are searching for strings like zipcodes in that list, I think tools like grep (available in OSGeo4W too!) work best:
Eg searching fro 3 zip codes is milliseconds work:
grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
If all fails, give me a call.
Regards,
Richard Duivenvoorde
On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> Edit
>> Copy command.
>
>
>
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> x-y data.
>
> If you want to recreate the issue, the file can be found here:
>
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> voor-postcode-huisnummer
>
> I run into this issue when selecting all objects with Attribute Gem217
> = 310
>
> 355 also has this issue. 321 and 356 work without a problem.
>
>
>
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to
> and make maps of them.
>
>
>
> Gr.
>
> Jeroen
>
>
>
> *Van:* Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org <mailto:qgis-user-bounces at lists.osgeo.org> ] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <qgis-user at lists.osgeo.org <mailto:qgis-user at lists.osgeo.org> >
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
>
>
>
> Also looking at the size of your original CSV, I’d go along with
> Jeroen, and try and import that csv into some other format first.
>
> If you can find a way of making the selection without going near the
> attribute table that will help too.
>
>
>
>
>
> Regards,
>
> Ramon.
>
> On 10 Jan 2018, at 06:26, Jeroen Hovens <mail at groenebij.nl <mailto:mail at groenebij.nl>
> <mailto:mail at groenebij.nl <mailto:mail at groenebij.nl> >> wrote:
>
>
>
> Thanks for the ideas.
>
>
>
> The option to save the selection as new csv file as Ramon suggested,
> doesn’t work; qgis goes into ‘not responding’ and no csv file is
> added to the layer list. However, a csv file is made, lacking a
> decent number of objects (first test 15095 of 18582 objects and
> second test is 15179 of 18582 objects, so missing about 3,5k
> objects). This csv file can be opened manually in qgis.
>
> And I checked: this options does work for selections where the
> ctrl-c option also worked. Then the new cvs layer is added to the
> layer list in no time.
>
>
>
> Is there some sort of log file in qgis that might give clues to the
> ‘not responding?
>
>
>
> @Kirk
>
> I did close and reopen qgis multiple times, even shut down the pc
> entirely, but to no effect. Selections that did allow for copying
> earlier, still work: I can make that selection again and ctrl-c
> works fine. Seems to me this rules out memory-issues.
>
>
>
> Gr.
>
> Jeroen
>
>
>
>
>
> *Van:* Qgis-user
> [mailto:qgis-user-bounces at lists.osgeo.org <mailto:qgis-user-bounces at lists.osgeo.org> ] *Namens *Kirk Schmidt
> *Verzonden:* dinsdag 9 januari 2018 22:20
> *Aan:* qgis-user at lists.osgeo.org <mailto:qgis-user at lists.osgeo.org> <mailto:qgis-user at lists.osgeo.org>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
>
>
>
> After a few selections, you may have to close and reopen qgis to
> purge the memory as well. Importing a csv is memory intensive so
> another approach is to try saving as a spatial database (SQLite) or
> even a shapefile. Once written, you can load the spatial file and
> you should have lots of memory to keep selecting points.
>
>
>
> Kirk
>
> On 1/9/2018 4:58 PM, Ramon Andinach wrote:
>
> I’m not sure what the problem is, but if I were doing that
> repeatedly, I’d probably use Layer -> Save As, tick the Save
> Only Selected Features box and save it to CSV. Then muck about
> with excel later.
>
>
>
> This way ought to be a bit lighter on the memory.
>
>
>
>
>
> Ramon.
>
> On 10 Jan 2018, at 04:25, Jeroen Hovens <mail at groenebij.nl <mailto:mail at groenebij.nl>
> <mailto:mail at groenebij.nl <mailto:mail at groenebij.nl> >> wrote:
>
>
>
> Hi,
>
>
>
> Using qgis 2.14.18 I have been working today with a large
> csv file (7.5 mln objects). I just need selections of this
> database which I can then modify in excel. So I opened the
> attribute table and made a selection based on a specific
> value for an attribute, and copied that selection (about
> 30.000 objects) via the clipboard (ctrl-c) to a new
> Microsoft Excel file, then saved the excel file and repeated
> the process for another selection. This worked perfect for
> some time. However, some selections won’t copy: when I press
> the copy button the programs turn to ‘not responding’ and
> won’t snap out of it anymore. Selection size when this
> happens is about the same (about 20k to 35k objects).
>
>
>
> What could be the problem? Could the data be corrupted? How
> can I check what is wrong?
>
>
>
> Thank you
>
> Gr.
>
> Jeroen
>
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> <mailto:Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> >
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
>
>
> _______________________________________________
>
> Qgis-user mailing list
>
> Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> <mailto:Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> >
>
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
> Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
> --
>
> Kirk Schmidt, MScF, BScF, RPF
>
> General Manager
>
> Nortek Resource Solutions Inc.
>
> RR # 1
>
> Thorburn, NS
>
> B0K 1W0
>
> Tel (902) 922.3607
>
> Email: kirk at nortekresources.com <mailto:kirk at nortekresources.com> <mailto:kirk at nortekresources.com <mailto:kirk at nortekresources.com> >
>
> Web: www.nortekreources.com <http://www.nortekreources.com> <http://www.nortekreources.com/>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> <mailto:Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org> >
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
>
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
_______________________________________________
Qgis-user mailing list
Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
--
Saber Razmjooei
www.lutraconsulting.co.uk <http://www.lutraconsulting.co.uk>
+44 (0)7568 129733
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20180110/3297e118/attachment.html>
More information about the Qgis-user
mailing list