[gdal-dev] Info about technical details of loading massive data

[Gerald Nelson]

I’m an end user, of software for which gdal is a critical input, in particular the R package terra, the successor to the raster package. I’m using it with very large climate data files. Terra (and raster) are built to deal with memory space available and the read and write from disk to temporary files. So in principle you can work with very large data sets on a machine with relatively little ram. Depending on your disk speed this can be tolerable or extremely slow.

But my ad hoc experimentation with a mac book pro with 32 gb of ram and an intel processor and a linux box with 64 gb has shown some remarkable differences. The bottom line is that the mac’s memory management is incredible. I can look at the system monitor on the amount ram an rsession is using and see 50 gb, with no memory pressure indicated and no temporary files being written by terra. On the linux box, the same R code crashes the machine. And code that takes x time on the mac takes 2-3 x time on the linux box.

And this is before the new Apple silicon, which by all accounts speeds up things dramatically. I’m saving my pennies.

Gerald C. Nelson
Professor Emeritus, UIUC
+1 217-390-7888 (cell)
+1 970-639-2079 (land line)
Skype: jerrynelson
http://bit.ly/1arho7d

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Andrew C Aitchison <andrew at aitchison.me.uk>
Date: Thursday, February 11, 2021 at 4:06 AM
To: Richard Duivenvoorde <rdmailings at duif.net>
Cc: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
Subject: Re: [gdal-dev] Info about technical details of loading massive data

On Thu, 11 Feb 2021, Richard Duivenvoorde wrote:

Hi Dev's,

I had a discussion with a friend about the sometimes hard times a
GIS-person has when handling/loading/viewing (using QGIS/GDAL)
massive (vector/raster) datasets, versus the R/Data-mangling
community.

Ending with a conclusion that it seemed (to us) that data-scientists
try to load as much (clean objects/multi dimensional arrays) data in
memory as possible, while GIS peeps always use the 'let's make it
some kind of feature object from first, and do lazy loading' way
use.

BUT I'm not sure about this, so: is there maybe somebody who held a
presentation or wrote a paper on how, for example gdal, handles a
huge point file vs R (memory/disk/io wise)?

While historically the 'Simple Features'-paradigm has be VERY
valuable for us, I'm questioning myself if there could be some 'more
efficient' way of handling the every time growing datasets we have
to handle... I envision a super fast memory-data viewer or so, so I
can quickly view my 16 Million points in my Postgis DB easily (mmm
probably have to fork QGIS 0.1 for this... QGIS started of as a
'simple' postgis viewer :-) )

My experience is limited to file-based data and machines
have grown to the point where the files will fit in memory.

I have written a couple of device drivers (not yet released)
for raster file formats which seem designed for memory-mapped
read access. Although functions like VSIFReadL support
reading from memory-based files, I have not found a way to
use memory-mapping in a driver.
This makes me wonder whether I end up with three copies of the map in
memory in addition to whatever is needed for the screen display;
one in the Linux file-system cache, one in the driver
and one (or perhaps two) in the gdal library and QGIS ?

I haven't looked, and perhaps should, to see whether QGIS
reads a map once into whatever format it finds best (possibly compressed),
keeps each map open and reads areas as needs, or repeatedly opens, reads
and closes each map.
Without knowing that, it isn't clear which map decompressions and
memory to memory copies are necessary.

--
Andrew C. Aitchison                                                                        Kendal, UK
                                                andrew at aitchison.me.uk<mailto:andrew at aitchison.me.uk>
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20210211/77bb4728/attachment.html>