[Qgis-user] ctrl-c selection issue

Richard Duivenvoorde rdmailings at duif.net
Wed Jan 10 01:21:23 PST 2018


Hi Jeroen,

Loading a csv in QGIS is not optimal. I think csv does not have an
index, so when you create a small selection, QGIS has to run over all
those 7,5 mln object every time.

Not sure how you do your 'selecting'?
Is it spatial or based on zip/postcode/buurtnr/wijknr?

I think I would either load it in Postgis (if available) or save as
GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
shapefile for it. AND then create indexes on desired columns.

IF you are searching for strings like zipcodes in that list, I think
tools like grep (available in OSGeo4W too!) work best:

Eg searching fro 3 zip codes is milliseconds work:

grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv

If all fails, give me a call.

Regards,

Richard Duivenvoorde

On 10-01-18 00:33, Jeroen Hovens wrote:
> Hmm, I can’t use Edit > Paste As, because qgis does not finish the Edit
>> Copy command.
> 
>  
> 
> The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no x-y
> data.
> 
> If you want to recreate the issue, the file can be found here:
> 
> https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-voor-postcode-huisnummer
> 
> I run into this issue when selecting all objects with Attribute Gem217 = 310
> 
> 355 also has this issue. 321 and 356 work without a problem.
> 
>  
> 
> I am using qgis here just as a means to chop the csv file into smaller
> selections which I can then use to edit, add attributes and data to and
> make maps of them.
> 
>  
> 
> Gr.
> 
> Jeroen
> 
>  
> 
> *Van:* Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org] *Namens
> *Ramon Andinach
> *Verzonden:* dinsdag 9 januari 2018 23:35
> *Aan:* qgis-user <qgis-user at lists.osgeo.org>
> *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> 
>  
> 
> Just so we’ve got just about everything covered, does this happen for
> “Edit-> Paste As”?
> 
>  
> 
> Also looking at the size of your original CSV, I’d go along with Jeroen,
> and try and import that csv into some other format first. 
> 
> If you can find a way of making the selection without going near the
> attribute table that will help too.
> 
>  
> 
>  
> 
> Regards,
> 
> Ramon.
> 
>     On 10 Jan 2018, at 06:26, Jeroen Hovens <mail at groenebij.nl
>     <mailto:mail at groenebij.nl>> wrote:
> 
>      
> 
>     Thanks for the ideas.
> 
>      
> 
>     The option to save the selection as new csv file as Ramon suggested,
>     doesn’t work; qgis goes into ‘not responding’ and no csv file is
>     added to the layer list. However, a csv file is made, lacking a
>     decent number of objects (first test 15095 of 18582 objects and
>     second test is 15179 of 18582 objects, so missing about 3,5k
>     objects). This csv file can be opened manually in qgis.
> 
>     And I checked: this options does work for selections where the
>     ctrl-c option also worked. Then the new cvs layer is added to the
>     layer list in no time.
> 
>      
> 
>     Is there some sort of log file in qgis that might give clues to the
>     ‘not responding?
> 
>      
> 
>     @Kirk
> 
>     I did close and reopen qgis multiple times, even shut down the pc
>     entirely, but to no effect. Selections that did allow for copying
>     earlier, still work: I can make that selection again and ctrl-c
>     works fine. Seems to me this rules out memory-issues.
> 
>      
> 
>     Gr.
> 
>     Jeroen
> 
>      
> 
>      
> 
>     *Van:* Qgis-user
>     [mailto:qgis-user-bounces at lists.osgeo.org] *Namens *Kirk Schmidt
>     *Verzonden:* dinsdag 9 januari 2018 22:20
>     *Aan:* qgis-user at lists.osgeo.org <mailto:qgis-user at lists.osgeo.org>
>     *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> 
>      
> 
>     After a few selections, you may have to close and reopen qgis to
>     purge the memory as well. Importing a csv is memory intensive so
>     another approach is to try saving as a spatial database (SQLite) or
>     even a shapefile.  Once written, you can load the spatial file and
>     you should have lots of memory to keep selecting points. 
> 
> 
> 
>     Kirk
> 
>     On 1/9/2018 4:58 PM, Ramon Andinach wrote:
> 
>         I’m not sure what the problem is, but if I were doing that
>         repeatedly, I’d probably use Layer -> Save As, tick the Save
>         Only Selected Features box and save it to CSV.  Then muck about
>         with excel later. 
> 
>          
> 
>         This way ought to be a bit lighter on the memory. 
> 
>          
> 
>          
> 
>         Ramon.
> 
>             On 10 Jan 2018, at 04:25, Jeroen Hovens <mail at groenebij.nl
>             <mailto:mail at groenebij.nl>> wrote:
> 
>              
> 
>             Hi,
> 
>              
> 
>             Using qgis 2.14.18 I have been working today with a large
>             csv file (7.5 mln objects). I just need selections of this
>             database which I can then modify in excel. So I opened the
>             attribute table and made a selection based on a specific
>             value for an attribute, and copied that selection (about
>             30.000 objects) via the clipboard (ctrl-c) to a new
>             Microsoft Excel file, then saved the excel file and repeated
>             the process for another selection. This worked perfect for
>             some time. However, some selections won’t copy: when I press
>             the copy button the programs turn to ‘not responding’ and
>             won’t snap out of it anymore.  Selection size when this
>             happens is about the same (about 20k to 35k objects).
> 
>              
> 
>             What could be the problem? Could the data be corrupted? How
>             can I check what is wrong?
> 
>              
> 
>             Thank you
> 
>             Gr.
> 
>             Jeroen 
> 
>              
> 
>             _______________________________________________
>             Qgis-user mailing list
>             Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
>             List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>             Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
>          
> 
> 
> 
> 
> 
>         _______________________________________________
> 
>         Qgis-user mailing list
> 
>         Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> 
>         List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
>         Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
> 
> 
> 
>     -- 
> 
>     Kirk Schmidt, MScF, BScF, RPF
> 
>     General Manager
> 
>     Nortek Resource Solutions Inc.
> 
>     RR # 1
> 
>     Thorburn, NS
> 
>     B0K 1W0
> 
>     Tel (902) 922.3607
> 
>     Email: kirk at nortekresources.com <mailto:kirk at nortekresources.com>
> 
>     Web: www.nortekreources.com <http://www.nortekreources.com/>
> 
>     _______________________________________________
>     Qgis-user mailing list
>     Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
>     List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>     Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 
>  
> 
> 
> 
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> 




More information about the Qgis-user mailing list