[Qgis-user] ctrl-c selection issue
Saber Razmjooei
saber.razmjooei at lutraconsulting.co.uk
Wed Jan 10 03:32:44 PST 2018
Drag-n-drop the file on QGIS canvas!
Regards
Saber
On 10 January 2018 at 11:26, Jeroen Hovens <mail at groenebij.nl> wrote:
> At the moment I make the selection using the Select by expression tool. I
> then choose one of the Gem2017 values (each object is an address, and has
> one of 388 unique Gem217 values). I want to select all objects bwith one
> unique Gem2017 value.
> Making the selection is no problem, I can make any selection I want.
> Making the selection usually takes less then a minute.
> The problem is in copying specific selections to the clipboard or saving a
> selection as a new file. Copying a selection of about 20.000 objects from
> the 7,5 mln takes about a second. Unless the issue arises, then the copy
> process takes forever
>
> I have no experience with Postgis or using databases. I tried saving the
> csv file as a sqlite, but then had no idea how to re-open that file. The
> dialogbox got me all confused ☹
>
>
>
>
> -----Oorspronkelijk bericht-----
> Van: Richard Duivenvoorde [mailto:rdmailings at duif.net]
> Verzonden: woensdag 10 januari 2018 10:21
> Aan: Jeroen Hovens <mail at groenebij.nl>; 'Ramon Andinach' <
> custard at westnet.com.au>; 'qgis-user' <qgis-user at lists.osgeo.org>
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
>
>
> Hi Jeroen,
>
> Loading a csv in QGIS is not optimal. I think csv does not have an index,
> so when you create a small selection, QGIS has to run over all those 7,5
> mln object every time.
>
> Not sure how you do your 'selecting'?
> Is it spatial or based on zip/postcode/buurtnr/wijknr?
>
> I think I would either load it in Postgis (if available) or save as
> GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
> shapefile for it. AND then create indexes on desired columns.
>
> IF you are searching for strings like zipcodes in that list, I think tools
> like grep (available in OSGeo4W too!) work best:
>
> Eg searching fro 3 zip codes is milliseconds work:
>
> grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
>
> If all fails, give me a call.
>
> Regards,
>
> Richard Duivenvoorde
>
> On 10-01-18 00:33, Jeroen Hovens wrote:
> > Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> > Edit
> >> Copy command.
> >
> >
> >
> > The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> > x-y data.
> >
> > If you want to recreate the issue, the file can be found here:
> >
> > https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> > voor-postcode-huisnummer
> >
> > I run into this issue when selecting all objects with Attribute Gem217
> > = 310
> >
> > 355 also has this issue. 321 and 356 work without a problem.
> >
> >
> >
> > I am using qgis here just as a means to chop the csv file into smaller
> > selections which I can then use to edit, add attributes and data to
> > and make maps of them.
> >
> >
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> > *Van:* Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org] *Namens
> > *Ramon Andinach
> > *Verzonden:* dinsdag 9 januari 2018 23:35
> > *Aan:* qgis-user <qgis-user at lists.osgeo.org>
> > *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> >
> >
> >
> > Just so we’ve got just about everything covered, does this happen for
> > “Edit-> Paste As”?
> >
> >
> >
> > Also looking at the size of your original CSV, I’d go along with
> > Jeroen, and try and import that csv into some other format first.
> >
> > If you can find a way of making the selection without going near the
> > attribute table that will help too.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Ramon.
> >
> > On 10 Jan 2018, at 06:26, Jeroen Hovens <mail at groenebij.nl
> > <mailto:mail at groenebij.nl>> wrote:
> >
> >
> >
> > Thanks for the ideas.
> >
> >
> >
> > The option to save the selection as new csv file as Ramon suggested,
> > doesn’t work; qgis goes into ‘not responding’ and no csv file is
> > added to the layer list. However, a csv file is made, lacking a
> > decent number of objects (first test 15095 of 18582 objects and
> > second test is 15179 of 18582 objects, so missing about 3,5k
> > objects). This csv file can be opened manually in qgis.
> >
> > And I checked: this options does work for selections where the
> > ctrl-c option also worked. Then the new cvs layer is added to the
> > layer list in no time.
> >
> >
> >
> > Is there some sort of log file in qgis that might give clues to the
> > ‘not responding?
> >
> >
> >
> > @Kirk
> >
> > I did close and reopen qgis multiple times, even shut down the pc
> > entirely, but to no effect. Selections that did allow for copying
> > earlier, still work: I can make that selection again and ctrl-c
> > works fine. Seems to me this rules out memory-issues.
> >
> >
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> >
> >
> > *Van:* Qgis-user
> > [mailto:qgis-user-bounces at lists.osgeo.org] *Namens *Kirk Schmidt
> > *Verzonden:* dinsdag 9 januari 2018 22:20
> > *Aan:* qgis-user at lists.osgeo.org <mailto:qgis-user at lists.osgeo.org>
> > *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> >
> >
> >
> > After a few selections, you may have to close and reopen qgis to
> > purge the memory as well. Importing a csv is memory intensive so
> > another approach is to try saving as a spatial database (SQLite) or
> > even a shapefile. Once written, you can load the spatial file and
> > you should have lots of memory to keep selecting points.
> >
> >
> >
> > Kirk
> >
> > On 1/9/2018 4:58 PM, Ramon Andinach wrote:
> >
> > I’m not sure what the problem is, but if I were doing that
> > repeatedly, I’d probably use Layer -> Save As, tick the Save
> > Only Selected Features box and save it to CSV. Then muck about
> > with excel later.
> >
> >
> >
> > This way ought to be a bit lighter on the memory.
> >
> >
> >
> >
> >
> > Ramon.
> >
> > On 10 Jan 2018, at 04:25, Jeroen Hovens <mail at groenebij.nl
> > <mailto:mail at groenebij.nl>> wrote:
> >
> >
> >
> > Hi,
> >
> >
> >
> > Using qgis 2.14.18 I have been working today with a large
> > csv file (7.5 mln objects). I just need selections of this
> > database which I can then modify in excel. So I opened the
> > attribute table and made a selection based on a specific
> > value for an attribute, and copied that selection (about
> > 30.000 objects) via the clipboard (ctrl-c) to a new
> > Microsoft Excel file, then saved the excel file and repeated
> > the process for another selection. This worked perfect for
> > some time. However, some selections won’t copy: when I press
> > the copy button the programs turn to ‘not responding’ and
> > won’t snap out of it anymore. Selection size when this
> > happens is about the same (about 20k to 35k objects).
> >
> >
> >
> > What could be the problem? Could the data be corrupted? How
> > can I check what is wrong?
> >
> >
> >
> > Thank you
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> > List info: https://lists.osgeo.org/
> mailman/listinfo/qgis-user
> > Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> >
> > Qgis-user mailing list
> >
> > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> >
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> > Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> > --
> >
> > Kirk Schmidt, MScF, BScF, RPF
> >
> > General Manager
> >
> > Nortek Resource Solutions Inc.
> >
> > RR # 1
> >
> > Thorburn, NS
> >
> > B0K 1W0
> >
> > Tel (902) 922.3607
> >
> > Email: kirk at nortekresources.com <mailto:kirk at nortekresources.com>
> >
> > Web: www.nortekreources.com <http://www.nortekreources.com/>
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
--
Saber Razmjooei
www.lutraconsulting.co.uk
+44 (0)7568 129733
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20180110/1bf3836f/attachment.html>
More information about the Qgis-user
mailing list