[Qgis-user] editing and saving very large table

[Michael.Dodd]

In this particular instance the data are not spatial data, at this stage at least. There is no link to location.

The data does have a hierarchy according to several columns, by sorting I can put the data into a suitable order for editing in further information into an extra column. Without sorting the places where the extra info and which specific extra info are impossible to find and check as they are scattered all over.

I tried putting the information in using the field calculator and even though the simple expression seemed correct and there was no error message it did not put the information in. So I was even thinking of putting the information in by hand once sorted although suspect there may be issues with this too.

I was wondering about sorting the whole file then copying out/exporting chunks of a million rows at a time to go into a normal spreadsheet. If the copying out/export would not work directly then saving the sorted version of the data and chopping up using e.g. notepad++ and editing with spreadsheet. It is important to do the sorting before chopping up into the 5 separate files.

It is only later that spatial information is attached, that is not relevant at this stage as the spatial data will not be attached to all rows.

From: chris hermansen <clhermansen at gmail.com>
Sent: Tuesday, March 12, 2024 2:49 PM
To: Michael.Dodd <michael.dodd at open.ac.uk>
Cc: Frederik-Matti Bartels (CAPSA) <bartels at capsa-building.org>; qgis-user at lists.osgeo.org
Subject: Re: [Qgis-user] editing and saving very large table


External email: if the sender or content looks suspicious, please click the Report Message icon, or forward it to report-phishing
Michael and list,

On Tue, Mar 12, 2024 at 5:31 AM Michael.Dodd via QGIS-User <qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>> wrote:
Thanks, can you suggest how as the refactor fields help is not clear, to say the least, about saving after sorting.

I sorted the data (5 million rows) by a couple of the columns, just by clicking on them, then wanted to save in the new order but this 'save in the new order' seems impossible.

Michael, I for one am puzzled as to what you sorted.  Was this a point, polygon, line attribute table?  Or some .csv layer?  Or...?

And how did you sort it?  Opened it in the attribute table editor and clicked on columns?  If so, as far as I know, sorting in this editor is just for convenience in reviewing data.  The physical data order is determined by the link to the spatial features.

Maybe you could also tell us what you're trying to achieve by sorting?

--
Chris Hermansen · clhermansen "at" gmail "dot" com

C'est ma façon de parler.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20240312/44320678/attachment-0001.htm>