[gdal-dev] In Memory Raster - allocation run out of memory onbig
sizes.
Martin Chapman
mchapman at texelinc.com
Fri Jan 4 13:04:45 EST 2008
Daniele,
Yeah, I'm not a Java programmer so I guess I should not be giving advice,
but I happen to remember similar issues when using gdal from jni in another
application I worked on a year or two ago. Basically, I think the goal is
to minimize the memory used by the JRE and run gdal processes as
out-of-process exe programs so they can have as much memory as needed to
perform memory allocations. It's either that or let the JRE consume all the
memory.not recommended.or get more memory.which is always recommended. J
Best regards,
Martin Chapman
Chief Software Architect
Fortified Datacom Inc.
http://www.fortifieddatacom.com
mchapman at fortifieddatacom.com
303-324-1065
From: Daniele Romagnoli [mailto:dany.geotools at gmail.com]
Sent: Friday, January 04, 2008 10:26 AM
To: mchapman at texelinc.com; Simone Giannecchini; gdal-dev at lists.osgeo.org;
Frank Warmerdam
Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory
onbig sizes.
Sorry Martin,
I will rectify the statement in my previous reply: "I guess that increasing
the xmx, "steals" memory which could be used by native items."
I'm saying that I have noticed this:
when increasing xmx settings, the out of memory errors in the native method
occurs with a smaller requested amount of data with respect to the case of a
lower xmx set.
As an instance, Xmx1024 allows the native method to allocate only <550 MB
memory while Xmx16M allows to allocate up to ~820MB.
Thanks for your help.
Regards,
Daniele
On Jan 4, 2008 6:05 PM, Daniele Romagnoli <dany.geotools at gmail.com> wrote:
Hi Martin,
I'm using GDAL through JNI (by means of the SWIG's generated java bindings).
However, I guess that increasing the xmx, "steals" memory which could be
used by native items.
Regards,
Daniele.
On Jan 4, 2008 5:32 PM, <mchapman at texelinc.com> wrote:
Daniele,
Are you using gdal api through jni or are you calling one of gdals exe
programs? Java limits the total memory for in-process dll's to whatever the
xmx is set to minus whatever the jre needs. I have dealt with this using
java before. You can either increase the xmx setting or call gdal from
outside the java runtime, like a c++ exe. In any case, you will always be
limited to the amount of memory on your machine.
P.S. A good driver will use memory mapped file io (file channels in java
NIO) for reading and writing which gives you a max file size of 1.8 Exobytes
on windows and will consume minimal memory during execution...like a couple
of MB. Unfortunately, most drivers don't use file channels. :)
Martin
Sent via BlackBerry by AT&T
-----Original Message-----
From: Frank Warmerdam <warmerdam at pobox.com>
Date: Fri, 04 Jan 2008 11:09:19
To:Daniele Romagnoli <dany.geotools at gmail.com>
Cc:gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on
big sizes.
Daniele Romagnoli wrote:
> Hi list,
> I'm trying to build a very big In Memory Raster Dataset (~1 GB) using
> SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
>
> When calling the java method memoryDriver.create(...) the following
> error appears "ERROR 2: Unable to create band arrays ... out of memory"
> which is thrown when the underlying calloc method can't successfully
> allocate memory.
> I have performed some tests and tunings with the following conclusions:
> - reducing the JVM Heap Size (using -Xmx command) allows to allocate a
> bigger native memory portion (using -Xmx1024M allows me to allocate only
> <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)
> [Java Heap space and "Native" Heap space are managed separately]
> - increasing the virtual memory, the result is the same (out of memory).
> - killing all unuseful processes -> freeing more physical memory (~1.2
> GB), the result is the same (out of memory))
> Any setting allows me to allocate a max of ~840000 KBytes.
>
> Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual
> memory setting (with ~2GB free) allows me only to create always ~840000
> KBytes?
> Does anyone know how memory is managed, how the calloc method operates
> and whether an OS limitation exist?
> Thank you very much.
Daniele,
I have no reason to believe this is a GDAL specific problem. I presume you
are running 32bit Windows Vista? If so, I believe any one process can have
at most 2GB of heap space which in your case would presumably be split
between the JVM and non Java memory in the same process.
I think you could allocate larger in memory rasters if you were running
Win64
which has a very large potential virtual memory size per process.
PS. I started out this reply to tell you to check the ulimit command output
which reports per-process memory limits on linux before I realized you were
using windows. I don't know if there is any equivelent thing on windows.
Best regards,
--
---------------------------------------+------------------------------------
--
I set the clouds in motion - turn up | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
<http://pobox.com/%7Ewarmerdam>
and watch the world go round - Rush | President OSGeo, http://osgeo.org
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev
--
-------------------------------------------------------
Eng. Daniele Romagnoli
Software Engineer
GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy
phone: +39 0584983027
fax: +39 0584983027
mob: +39 328 0559267
http://www.geo-solutions.it
-------------------------------------------------------
--
-------------------------------------------------------
Eng. Daniele Romagnoli
Software Engineer
GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy
phone: +39 0584983027
fax: +39 0584983027
mob: +39 328 0559267
http://www.geo-solutions.it
-------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20080104/e69f7ea4/attachment.html
More information about the gdal-dev
mailing list