Sorry Martin,<br>I will rectify the statement in my previous reply: "I guess that increasing the xmx, "steals" memory which could be used by native items."<br>I'm saying that I have noticed this:<br>
when increasing xmx settings, the out of memory errors in the native method occurs with a smaller requested amount of data with respect to the case of a lower xmx set.<br>As an instance, Xmx1024 allows the native method to allocate only <550 MB memory while Xmx16M allows to allocate up to ~820MB.
<br><br>Thanks for your help.<br><br>Regards,<br>Daniele<br><br><br><div class="gmail_quote">On Jan 4, 2008 6:05 PM, Daniele Romagnoli <<a href="mailto:dany.geotools@gmail.com">dany.geotools@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Martin,<br>I'm using GDAL through JNI (by means of the SWIG's generated java bindings).
<br>However, I guess that increasing the xmx, "steals" memory which could be used by native items.<br><br>Regards,
<br><font color="#888888">Daniele.</font><div><div></div><div class="Wj3C7c"><br><br><div class="gmail_quote">On Jan 4, 2008 5:32 PM, <<a href="mailto:mchapman@texelinc.com" target="_blank">mchapman@texelinc.com</a>> wrote:
<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Daniele,<br><br>Are you using gdal api through jni or are you calling one of gdals exe programs? Java limits the total memory for in-process dll's to whatever the xmx is set to minus whatever the jre needs. I have dealt with this using java before. You can either increase the xmx setting or call gdal from outside the java runtime, like a c++ exe. In any case, you will always be limited to the amount of memory on your machine.
<br>P.S. A good driver will use memory mapped file io (file channels in java NIO) for reading and writing which gives you a max file size of 1.8 Exobytes on windows and will consume minimal memory during execution...like a couple of MB. Unfortunately, most drivers don't use file channels. :)
<br><br>Martin<br><br>Sent via BlackBerry by AT&T<br><div><div></div><div><br>-----Original Message-----<br>From: Frank Warmerdam <<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a>><br>
<br>Date: Fri, 04 Jan 2008 11:09:19<br>To:Daniele Romagnoli <<a href="mailto:dany.geotools@gmail.com" target="_blank">dany.geotools@gmail.com</a>><br><a href="mailto:Cc:gdal-dev@lists.osgeo.org" target="_blank">Cc:gdal-dev@lists.osgeo.org
</a><br>Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on
<br> big sizes.<br><br><br>Daniele Romagnoli wrote:<br>> Hi list,<br>> I'm trying to build a very big In Memory Raster Dataset (~1 GB) using<br>> SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
<br>><br>> When calling the java method memoryDriver.create(...) the following<br>> error appears "ERROR 2: Unable to create band arrays ... out of memory"<br>> which is thrown when the underlying calloc method can't successfully
<br>> allocate memory.<br>> I have performed some tests and tunings with the following conclusions:<br>> - reducing the JVM Heap Size (using -Xmx command) allows to allocate a<br>> bigger native memory portion (using -Xmx1024M allows me to allocate only
<br>> <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)<br>> [Java Heap space and "Native" Heap space are managed separately]<br>> - increasing the virtual memory, the result is the same (out of memory).
<br>> - killing all unuseful processes -> freeing more physical memory (~1.2<br>> GB), the result is the same (out of memory))<br>> Any setting allows me to allocate a max of ~840000 KBytes.<br>><br>> Anyway, why having almost
1.2 GB free physical memory and a 3GB virtual<br>> memory setting (with ~2GB free) allows me only to create always ~840000<br>> KBytes?<br>> Does anyone know how memory is managed, how the calloc method operates
<br>
> and whether an OS limitation exist?<br>> Thank you very much.<br><br>Daniele,<br><br>I have no reason to believe this is a GDAL specific problem. I presume you<br>are running 32bit Windows Vista? If so, I believe any one process can have
<br>at most 2GB of heap space which in your case would presumably be split<br>between the JVM and non Java memory in the same process.<br><br>I think you could allocate larger in memory rasters if you were running Win64<br>
which has a very large potential virtual memory size per process.<br><br>PS. I started out this reply to tell you to check the ulimit command output<br>which reports per-process memory limits on linux before I realized you were
<br>using windows. I don't know if there is any equivelent thing on windows.<br><br>Best regards,<br>--<br>---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam,
<a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>light and sound - activate the windows | <a href="http://pobox.com/%7Ewarmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | President OSGeo,
<a href="http://osgeo.org" target="_blank">http://osgeo.org</a><br><br></div></div>_______________________________________________<br>gdal-dev mailing list<br><a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org
</a><br><a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></blockquote></div><br><br clear="all"><br></div></div>-- <br>-------------------------------------------------------
<div><div></div><div class="Wj3C7c"><br>Eng. Daniele Romagnoli <br>Software Engineer<br><br>GeoSolutions S.A.S.<br>Via Carignoni 51<br>55041 Camaiore (LU)<br>Italy<br><br>phone: +39 0584983027<br>fax: +39 0584983027<br>
mob: +39 328 0559267<br><br><br>
<a href="http://www.geo-solutions.it" target="_blank">http://www.geo-solutions.it</a><br><br>-------------------------------------------------------<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>-------------------------------------------------------<br>Eng. Daniele Romagnoli <br>Software Engineer<br><br>GeoSolutions S.A.S.<br>Via Carignoni 51<br>55041 Camaiore (LU)
<br>Italy<br><br>phone: +39 0584983027<br>fax: +39 0584983027<br>mob: +39 328 0559267<br><br><br><a href="http://www.geo-solutions.it">http://www.geo-solutions.it</a><br><br>-------------------------------------------------------
<br>