<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">I’m an end user, of software for which gdal is a critical input, in particular the R package terra, the successor to the raster package. I’m using it with very large climate data files. Terra (and raster) are built to deal with memory space
available and the read and write from disk to temporary files. So in principle you can work with very large data sets on a machine with relatively little ram. Depending on your disk speed this can be tolerable or extremely slow.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But my ad hoc experimentation with a mac book pro with 32 gb of ram and an intel processor and a linux box with 64 gb has shown some remarkable differences. The bottom line is that the mac’s memory management is incredible. I can look at
the system monitor on the amount ram an rsession is using and see 50 gb, with no memory pressure indicated and no temporary files being written by terra. On the linux box, the same R code crashes the machine. And code that takes x time on the mac takes 2-3
x time on the linux box. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And this is before the new Apple silicon, which by all accounts speeds up things dramatically. I’m saving my pennies.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="text-align:justify">Gerald C. Nelson<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify">Professor Emeritus, UIUC<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify">+1 217-390-7888 (cell)<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify">+1 970-639-2079 (land line)<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify">Skype: jerrynelson<o:p></o:p></p>
<p class="MsoNormal" style="text-align:justify">http://bit.ly/1arho7d<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">gdal-dev <gdal-dev-bounces@lists.osgeo.org> on behalf of Andrew C Aitchison <andrew@aitchison.me.uk><br>
<b>Date: </b>Thursday, February 11, 2021 at 4:06 AM<br>
<b>To: </b>Richard Duivenvoorde <rdmailings@duif.net><br>
<b>Cc: </b>"gdal-dev@lists.osgeo.org" <gdal-dev@lists.osgeo.org><br>
<b>Subject: </b>Re: [gdal-dev] Info about technical details of loading massive data<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On Thu, 11 Feb 2021, Richard Duivenvoorde wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #B5C4DF 4.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-right:0in" id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE">
<div>
<p class="MsoNormal">Hi Dev's,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I had a discussion with a friend about the sometimes hard times a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">GIS-person has when handling/loading/viewing (using QGIS/GDAL)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">massive (vector/raster) datasets, versus the R/Data-mangling<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">community.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ending with a conclusion that it seemed (to us) that data-scientists<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">try to load as much (clean objects/multi dimensional arrays) data in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">memory as possible, while GIS peeps always use the 'let's make it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">some kind of feature object from first, and do lazy loading' way<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">use.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">BUT I'm not sure about this, so: is there maybe somebody who held a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">presentation or wrote a paper on how, for example gdal, handles a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">huge point file vs R (memory/disk/io wise)?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">While historically the 'Simple Features'-paradigm has be VERY<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">valuable for us, I'm questioning myself if there could be some 'more<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">efficient' way of handling the every time growing datasets we have<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to handle... I envision a super fast memory-data viewer or so, so I<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">can quickly view my 16 Million points in my Postgis DB easily (mmm<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">probably have to fork QGIS 0.1 for this... QGIS started of as a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">'simple' postgis viewer :-) )<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">My experience is limited to file-based data and machines<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">have grown to the point where the files will fit in memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have written a couple of device drivers (not yet released)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">for raster file formats which seem designed for memory-mapped<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">read access. Although functions like VSIFReadL support<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">reading from memory-based files, I have not found a way to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">use memory-mapping in a driver.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This makes me wonder whether I end up with three copies of the map in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">memory in addition to whatever is needed for the screen display;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">one in the Linux file-system cache, one in the driver<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and one (or perhaps two) in the gdal library and QGIS ?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I haven't looked, and perhaps should, to see whether QGIS<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">reads a map once into whatever format it finds best (possibly compressed),<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">keeps each map open and reads areas as needs, or repeatedly opens, reads<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and closes each map.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Without knowing that, it isn't clear which map decompressions and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">memory to memory copies are necessary.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Andrew C. Aitchison<span class="apple-tab-span">
</span>Kendal, UK<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span><a href="mailto:andrew@aitchison.me.uk">andrew@aitchison.me.uk</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">gdal-dev mailing list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>