<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#888888'>Daniele,<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#888888'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#888888'>Yeah, I’m not a Java programmer
so I guess I should not be giving advice, but I happen to remember similar issues
when using gdal from jni in another application I worked on a year or two
ago. Basically, I think the goal is to minimize the memory used by the
JRE and run gdal processes as out-of-process exe programs so they can have as
much memory as needed to perform memory allocations. It’s either
that or let the JRE consume all the memory…not recommended…or get
more memory…which is always recommended. </span><span style='font-family:
Wingdings;color:#888888'>J</span><span style='font-size:11.0pt;font-family:
"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Best regards,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Martin Chapman<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Chief Software Architect<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Fortified Datacom Inc.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>http://www.fortifieddatacom.com<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>mchapman@fortifieddatacom.com<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>303-324-1065<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Daniele Romagnoli
[mailto:dany.geotools@gmail.com] <br>
<b>Sent:</b> Friday, January 04, 2008 10:26 AM<br>
<b>To:</b> mchapman@texelinc.com; Simone Giannecchini;
gdal-dev@lists.osgeo.org; Frank Warmerdam<br>
<b>Subject:</b> Re: [gdal-dev] In Memory Raster - allocation run out of memory
onbig sizes.<o:p></o:p></span></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>Sorry Martin,<br>
I will rectify the statement in my previous reply: "I guess that
increasing the xmx, "steals" memory which could be used by native
items."<br>
I'm saying that I have noticed this:<br>
when increasing xmx settings, the out of memory errors in the native method
occurs with a smaller requested amount of data with respect to the case of a
lower xmx set.<br>
As an instance, Xmx1024 allows the native method to allocate only <550
MB memory while Xmx16M allows to allocate up to ~820MB. <br>
<br>
Thanks for your help.<br>
<br>
Regards,<br>
Daniele<br>
<br>
<o:p></o:p></p>
<div>
<p class=MsoNormal>On Jan 4, 2008 6:05 PM, Daniele Romagnoli <<a
href="mailto:dany.geotools@gmail.com">dany.geotools@gmail.com</a>> wrote:<o:p></o:p></p>
<p class=MsoNormal>Hi Martin,<br>
I'm using GDAL through JNI (by means of the SWIG's generated java bindings). <br>
However, I guess that increasing the xmx, "steals" memory which could
be used by native items.<br>
<br>
Regards, <br>
<span style='color:#888888'>Daniele.</span><o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p>
<div>
<p class=MsoNormal>On Jan 4, 2008 5:32 PM, <<a
href="mailto:mchapman@texelinc.com" target="_blank">mchapman@texelinc.com</a>>
wrote: <o:p></o:p></p>
<p class=MsoNormal>Daniele,<br>
<br>
Are you using gdal api through jni or are you calling one of gdals exe
programs? Java limits the total memory for in-process dll's to whatever
the xmx is set to minus whatever the jre needs. I have dealt with this
using java before. You can either increase the xmx setting or call gdal
from outside the java runtime, like a c++ exe. In any case, you will
always be limited to the amount of memory on your machine. <br>
P.S. A good driver will use memory mapped file io (file channels in java
NIO) for reading and writing which gives you a max file size of 1.8 Exobytes on
windows and will consume minimal memory during execution...like a couple of MB.
Unfortunately, most drivers don't use file channels. :) <br>
<br>
Martin<br>
<br>
Sent via BlackBerry by AT&T<o:p></o:p></p>
<div>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
-----Original Message-----<br>
From: Frank Warmerdam <<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>><br>
<br>
Date: Fri, 04 Jan 2008 11:09:19<br>
To:Daniele Romagnoli <<a href="mailto:dany.geotools@gmail.com"
target="_blank">dany.geotools@gmail.com</a>><br>
<a href="mailto:Cc:gdal-dev@lists.osgeo.org" target="_blank">Cc:gdal-dev@lists.osgeo.org
</a><br>
Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on <br>
big sizes.<br>
<br>
<br>
Daniele Romagnoli wrote:<br>
> Hi list,<br>
> I'm trying to build a very big In Memory Raster Dataset (~1 GB) using<br>
> SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram) <br>
><br>
> When calling the java method memoryDriver.create(...) the following<br>
> error appears "ERROR 2: Unable to create band arrays ... out of
memory"<br>
> which is thrown when the underlying calloc method can't successfully <br>
> allocate memory.<br>
> I have performed some tests and tunings with the following conclusions:<br>
> - reducing the JVM Heap Size (using -Xmx command) allows to allocate a<br>
> bigger native memory portion (using -Xmx1024M allows me to allocate only <br>
> <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)<br>
> [Java Heap space and "Native" Heap space are managed separately]<br>
> - increasing the virtual memory, the result is the same (out of memory). <br>
> - killing all unuseful processes -> freeing more physical memory (~1.2<br>
> GB), the result is the same (out of memory))<br>
> Any setting allows me to allocate a max of ~840000 KBytes.<br>
><br>
> Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual<br>
> memory setting (with ~2GB free) allows me only to create always ~840000<br>
> KBytes?<br>
> Does anyone know how memory is managed, how the calloc method operates <br>
> and whether an OS limitation exist?<br>
> Thank you very much.<br>
<br>
Daniele,<br>
<br>
I have no reason to believe this is a GDAL specific problem. I presume
you<br>
are running 32bit Windows Vista? If so, I believe any one process can
have <br>
at most 2GB of heap space which in your case would presumably be split<br>
between the JVM and non Java memory in the same process.<br>
<br>
I think you could allocate larger in memory rasters if you were running Win64<br>
which has a very large potential virtual memory size per process.<br>
<br>
PS. I started out this reply to tell you to check the ulimit command output<br>
which reports per-process memory limits on linux before I realized you were <br>
using windows. I don't know if there is any equivelent thing on windows.<br>
<br>
Best regards,<br>
--<br>
---------------------------------------+--------------------------------------<br>
I set the clouds in motion - turn up | Frank Warmerdam, <a
href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>
light and sound - activate the windows | <a href="http://pobox.com/%7Ewarmerdam"
target="_blank">http://pobox.com/~warmerdam</a><br>
and watch the world go round - Rush | President OSGeo, <a
href="http://osgeo.org" target="_blank">http://osgeo.org</a><o:p></o:p></p>
</div>
</div>
<p class=MsoNormal>_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org
</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></p>
</div>
<p class=MsoNormal><br>
<br clear=all>
<o:p></o:p></p>
</div>
</div>
<p class=MsoNormal>-- <br>
------------------------------------------------------- <o:p></o:p></p>
<div>
<div>
<p class=MsoNormal><br>
Eng. Daniele Romagnoli <br>
Software Engineer<br>
<br>
GeoSolutions S.A.S.<br>
Via Carignoni 51<br>
55041 Camaiore (LU)<br>
Italy<br>
<br>
phone: +39 0584983027<br>
fax: +39 0584983027<br>
mob: +39 328 0559267<br>
<br>
<br>
<a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a><br>
<br>
-------------------------------------------------------<o:p></o:p></p>
</div>
</div>
</div>
<p class=MsoNormal><br>
<br clear=all>
<br>
-- <br>
-------------------------------------------------------<br>
Eng. Daniele Romagnoli <br>
Software Engineer<br>
<br>
GeoSolutions S.A.S.<br>
Via Carignoni 51<br>
55041 Camaiore (LU) <br>
Italy<br>
<br>
phone: +39 0584983027<br>
fax: +39 0584983027<br>
mob: +39 328 0559267<br>
<br>
<br>
<a href="http://www.geo-solutions.it">http://www.geo-solutions.it</a><br>
<br>
------------------------------------------------------- <o:p></o:p></p>
</div>
</body>
</html>