[Qgis-user] ctrl-c selection issue
Fernando M. Roxo da Motta
petro at roxo.org
Wed Jan 10 06:12:52 PST 2018
On Wed, 10 Jan 2018 12:26:42 +0100, "Jeroen Hovens" <mail at groenebij.nl>
wrote:
Hi Jeroen,
If you allow, I will suggest you two possible approach. You didn't
specify your OS platform. I tested on Linux but the first approach
should work anywhere.
First one using qgis only. Load your CSV file in qgis as a table
only, no geom. At least I did not found one. After loading save it
as a SQLite database (save as...). It should sabe and load the
database as a table. Unload the CSV file.
At this point double click the sqlite layer and got to general tab.
There click on Query Builder button and use an expression like:
"Gem2017" = 310
You can just apply this selection or, if you wish, test it to get the
number of lines selected. When you apply a selection like this your
layer is composed of only those selected one. Just save it. For
example save as a "MS Office Open XML spreadsheet", that is one of the
offered formats in my Qgis.
After that you just change the desired value and so on.
Another approach is pure shell approach.
I am attaching to the message a shell script that will:
1- Convert the CVS from DOS (CRLF) to Linux (LF) format
2- Sort it by the 5th column (the Gem2017)
3- Get the unique values for that column
4- Iterate over the CSV file extracting all lines with
each unique Gem2017, saving it with the name:
Gem2017-<GemValue>.csv
I think it is possible to bach convert them to Excel or LibreOffice
calc format, just did not looked at it.
If you are using Windows, you can try this in a Cygwin or even the
Linux subsystem for Windows 10 (I never saw this last one, it is just a
guess).
Hope this helps.
P.S. - If you prefer to deal with database it self, a SQL like:
select * from pc6hnr20170801_gwb where gem2017 = 310;
Will get you all entries with that value of gem2017.
> At the moment I make the selection using the Select by expression
> tool. I then choose one of the Gem2017 values (each object is an
> address, and has one of 388 unique Gem217 values). I want to select
> all objects bwith one unique Gem2017 value. Making the selection is
> no problem, I can make any selection I want. Making the selection
> usually takes less then a minute. The problem is in copying specific
> selections to the clipboard or saving a selection as a new file.
> Copying a selection of about 20.000 objects from the 7,5 mln takes
> about a second. Unless the issue arises, then the copy process takes
> forever
>
> I have no experience with Postgis or using databases. I tried saving
> the csv file as a sqlite, but then had no idea how to re-open that
> file. The dialogbox got me all confused ☹
>
>
>
>
> -----Oorspronkelijk bericht-----
> Van: Richard Duivenvoorde [mailto:rdmailings at duif.net]
> Verzonden: woensdag 10 januari 2018 10:21
> Aan: Jeroen Hovens <mail at groenebij.nl>; 'Ramon Andinach'
> <custard at westnet.com.au>; 'qgis-user' <qgis-user at lists.osgeo.org>
> Onderwerp: Re: [Qgis-user] ctrl-c selection issue
>
>
> Hi Jeroen,
>
> Loading a csv in QGIS is not optimal. I think csv does not have an
> index, so when you create a small selection, QGIS has to run over all
> those 7,5 mln object every time.
>
> Not sure how you do your 'selecting'?
> Is it spatial or based on zip/postcode/buurtnr/wijknr?
>
> I think I would either load it in Postgis (if available) or save as
> GeoPackage (better use QGIS 2.99 for that) or if all fails, create a
> shapefile for it. AND then create indexes on desired columns.
>
> IF you are searching for strings like zipcodes in that list, I think
> tools like grep (available in OSGeo4W too!) work best:
>
> Eg searching fro 3 zip codes is milliseconds work:
>
> grep '2022ZJ\|2211ZE\|1016XX' pc6hnr20170801_gwb.csv
>
> If all fails, give me a call.
>
> Regards,
>
> Richard Duivenvoorde
>
> On 10-01-18 00:33, Jeroen Hovens wrote:
> > Hmm, I can’t use Edit > Paste As, because qgis does not finish the
> > Edit
> >> Copy command.
> >
> >
> >
> > The csv file is about 270 MB, 7,5 mln objects with 5 attributes, no
> > x-y data.
> >
> > If you want to recreate the issue, the file can be found here:
> >
> > https://www.cbs.nl/nl-nl/maatwerk/2017/38/buurt-wijk-en-gemeente-2017-
> > voor-postcode-huisnummer
> >
> > I run into this issue when selecting all objects with Attribute
> > Gem217 = 310
> >
> > 355 also has this issue. 321 and 356 work without a problem.
> >
> >
> >
> > I am using qgis here just as a means to chop the csv file into
> > smaller selections which I can then use to edit, add attributes and
> > data to and make maps of them.
> >
> >
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> > *Van:* Qgis-user [mailto:qgis-user-bounces at lists.osgeo.org] *Namens
> > *Ramon Andinach
> > *Verzonden:* dinsdag 9 januari 2018 23:35
> > *Aan:* qgis-user <qgis-user at lists.osgeo.org>
> > *Onderwerp:* Re: [Qgis-user] ctrl-c selection issue
> >
> >
> >
> > Just so we’ve got just about everything covered, does this happen
> > for “Edit-> Paste As”?
> >
> >
> >
> > Also looking at the size of your original CSV, I’d go along with
> > Jeroen, and try and import that csv into some other format first.
> >
> > If you can find a way of making the selection without going near
> > the attribute table that will help too.
> >
> >
> >
> >
> >
> > Regards,
> >
> > Ramon.
> >
> > On 10 Jan 2018, at 06:26, Jeroen Hovens <mail at groenebij.nl
> > <mailto:mail at groenebij.nl>> wrote:
> >
> >
> >
> > Thanks for the ideas.
> >
> >
> >
> > The option to save the selection as new csv file as Ramon
> > suggested, doesn’t work; qgis goes into ‘not responding’ and no csv
> > file is added to the layer list. However, a csv file is made,
> > lacking a decent number of objects (first test 15095 of 18582
> > objects and second test is 15179 of 18582 objects, so missing about
> > 3,5k objects). This csv file can be opened manually in qgis.
> >
> > And I checked: this options does work for selections where the
> > ctrl-c option also worked. Then the new cvs layer is added to
> > the layer list in no time.
> >
> >
> >
> > Is there some sort of log file in qgis that might give clues to
> > the ‘not responding?
> >
> >
> >
> > @Kirk
> >
> > I did close and reopen qgis multiple times, even shut down the
> > pc entirely, but to no effect. Selections that did allow for copying
> > earlier, still work: I can make that selection again and ctrl-c
> > works fine. Seems to me this rules out memory-issues.
> >
> >
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> >
> >
> > *Van:* Qgis-user
> > [mailto:qgis-user-bounces at lists.osgeo.org] *Namens *Kirk Schmidt
> > *Verzonden:* dinsdag 9 januari 2018 22:20
> > *Aan:* qgis-user at lists.osgeo.org
> > <mailto:qgis-user at lists.osgeo.org> *Onderwerp:* Re: [Qgis-user]
> > ctrl-c selection issue
> >
> >
> >
> > After a few selections, you may have to close and reopen qgis to
> > purge the memory as well. Importing a csv is memory intensive so
> > another approach is to try saving as a spatial database
> > (SQLite) or even a shapefile. Once written, you can load the
> > spatial file and you should have lots of memory to keep selecting
> > points.
> >
> >
> >
> > Kirk
> >
> > On 1/9/2018 4:58 PM, Ramon Andinach wrote:
> >
> > I’m not sure what the problem is, but if I were doing that
> > repeatedly, I’d probably use Layer -> Save As, tick the Save
> > Only Selected Features box and save it to CSV. Then muck
> > about with excel later.
> >
> >
> >
> > This way ought to be a bit lighter on the memory.
> >
> >
> >
> >
> >
> > Ramon.
> >
> > On 10 Jan 2018, at 04:25, Jeroen Hovens
> > <mail at groenebij.nl <mailto:mail at groenebij.nl>> wrote:
> >
> >
> >
> > Hi,
> >
> >
> >
> > Using qgis 2.14.18 I have been working today with a
> > large csv file (7.5 mln objects). I just need selections of this
> > database which I can then modify in excel. So I opened
> > the attribute table and made a selection based on a specific
> > value for an attribute, and copied that selection (about
> > 30.000 objects) via the clipboard (ctrl-c) to a new
> > Microsoft Excel file, then saved the excel file and
> > repeated the process for another selection. This worked perfect for
> > some time. However, some selections won’t copy: when I
> > press the copy button the programs turn to ‘not responding’ and
> > won’t snap out of it anymore. Selection size when this
> > happens is about the same (about 20k to 35k objects).
> >
> >
> >
> > What could be the problem? Could the data be corrupted?
> > How can I check what is wrong?
> >
> >
> >
> > Thank you
> >
> > Gr.
> >
> > Jeroen
> >
> >
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org
> > <mailto:Qgis-user at lists.osgeo.org> List info:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> >
> > Qgis-user mailing list
> >
> > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> >
> > List info:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> > Unsubscribe:
> > https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> > --
> >
> > Kirk Schmidt, MScF, BScF, RPF
> >
> > General Manager
> >
> > Nortek Resource Solutions Inc.
> >
> > RR # 1
> >
> > Thorburn, NS
> >
> > B0K 1W0
> >
> > Tel (902) 922.3607
> >
> > Email: kirk at nortekresources.com
> > <mailto:kirk at nortekresources.com>
> >
> > Web: www.nortekreources.com <http://www.nortekreources.com/>
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org <mailto:Qgis-user at lists.osgeo.org>
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
> >
> >
> >
> >
> > _______________________________________________
> > Qgis-user mailing list
> > Qgis-user at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
Roxo
--
---------------- Non luctari, ludare -------------------+ WYSIWYG
Fernando M. Roxo da Motta <petro at roxo.org> | Editor?
Except where explicitly stated I speak on my own behalf.| VI !!
( Usuário Linux registrado #39505 ) | I see text,
------------ Quis custodiet ipsos custodes?-------------+ I get text!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: doit.sh
Type: application/x-shellscript
Size: 630 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20180110/321bb77d/attachment.bin>
More information about the Qgis-user
mailing list