[Qgis-user] editing and saving very large table

[Gordon Wakelin-King]

Surely this is something that can be done easily with an SQL Select  with Order-by, in one of the SQL variants that QGIS can use?

I’m asking for my own benefit too. Having come from the MapInfo environment I would reach for an SQL Select to do this task, but I’m less familiar with SQL in QGIS.

Gordon Wakelin-King



From: QGIS-User <qgis-user-bounces at lists.osgeo.org> On Behalf Of chris hermansen via QGIS-User
Sent: Wednesday, March 13, 2024 8:02 AM
To: Michael.Dodd <michael.dodd at open.ac.uk>
Cc: Frederik-Matti Bartels (CAPSA) <bartels at capsa-building.org>; qgis-user at lists.osgeo.org
Subject: Re: [Qgis-user] editing and saving very large table

Michael and list,

On Tue, Mar 12, 2024 at 11:48 AM Michael.Dodd <michael.dodd at open.ac.uk<mailto:michael.dodd at open.ac.uk>> wrote:
In this particular instance the data are not spatial data, at this stage at least. There is no link to location.

The data does have a hierarchy according to several columns, by sorting I can put the data into a suitable order for editing in further information into an extra column. Without sorting the places where the extra info and which specific extra info are impossible to find and check as they are scattered all over.

I tried putting the information in using the field calculator and even though the simple expression seemed correct and there was no error message it did not put the information in. So I was even thinking of putting the information in by hand once sorted although suspect there may be issues with this too.

I was wondering about sorting the whole file then copying out/exporting chunks of a million rows at a time to go into a normal spreadsheet. If the copying out/export would not work directly then saving the sorted version of the data and chopping up using e.g. notepad++ and editing with spreadsheet. It is important to do the sorting before chopping up into the 5 separate files.

It is only later that spatial information is attached, that is not relevant at this stage as the spatial data will not be attached to all rows.

Ok so you are sorting a text file, presumably something like a .CSV.  Perhaps you are working on Windows.

What are your options to sort the file before you put it in QGIS?  It seems your spreadsheet won't accommodate the number of lines in your file; what about the "sort" utility in say Windows PowerShell?

Here's an article that talks about sorting .CSV files that way.

https://stackoverflow.com/questions/23603397/sorting-csv-using-windows-sort which in turn refers to this article

https://devblogs.microsoft.com/scripting/hey-scripting-guy-how-can-i-use-windows-powershell-to-sort-a-csv-file/

Alternatively, if you're using Linux or (I would imagine) OS/X, you can use the Posix "sort" utility.
--
Chris Hermansen · clhermansen "at" gmail "dot" com

C'est ma façon de parler.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20240312/39a4eb90/attachment.htm>