[gdal-dev] In Memory Raster - allocation run out of memory onbig sizes.

Fri Jan 4 15:31:47 EST 2008

Simone,

I didn't know the JVM process heap is split like that. Thanks for that
information.

Best regards,

Martin Chapman
Chief Software Architect
Fortified Datacom Inc.
http://www.fortifieddatacom.com
mchapman at fortifieddatacom.com
303-324-1065

-----Original Message-----
From: Simone Giannecchini [mailto:simboss1 at gmail.com] 
Sent: Friday, January 04, 2008 12:54 PM
To: mchapman at texelinc.com
Cc: Frank Warmerdam; gdal-dev-bounces at lists.osgeo.org; Daniele Romagnoli;
gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory
onbig sizes.

Ciao Martin, (ciao daniele :-) )
sorry to chime in late in the discussion.

It is actually incorrect saying that "Java limits the total memory for
in-process dll's to whatever the xmx is set to minus whatever the jre
needs".

Actually a JVM process created using java or javaw is a normal process
itself and therefore it is (more or less :-) ) subject to the OS rules
for virtual memory usage.
Internally the JVM process heap is split between the GC managed Heap,
which size and layout can be controlled using the options you propose
plus a bunch more, and the native heap which is used by the JVM itseld
for its own  purposes but which is also used by whichever in-process
native code allocates memory dynamically like gdal would do using
*alloc functions.

On windows 32 bit a user process can *normally* use no more than 2 GB
of virtual space because the other 2 are *normally* reserved for OS
(kernel) usage. In practice I have never been able to make the JVM
process work with more than 1.5 gb maximum total memory which will be
shared between the native and gc heap.ù

In the end raising the -Xmx memory limit should not only be
ineffective in this case, but should rather be countereffective since
it would *steal* heap space to the native space from the total quota
available to te JVM process in favour to the GC heap space.

Ciao,
Simone.

PS
Of course, feel free to correct in case I made some mistakes. I am
still trying to recover from the amount of alcohol and food I have
been drinking/eating latetly :-).

On Jan 4, 2008 5:32 PM,  <mchapman at texelinc.com> wrote:
> Daniele,
>
> Are you using gdal api through jni or are you calling one of gdals exe
programs?  Java limits the total memory for in-process dll's to whatever the
xmx is set to minus whatever the jre needs.  I have dealt with this using
java before.  You can either increase the xmx setting or call gdal from
outside the java runtime, like a c++ exe.  In any case, you will always be
limited to the amount of memory on your machine.
> P.S.  A good driver will use memory mapped file io (file channels in java
NIO) for reading and writing which gives you a max file size of 1.8 Exobytes
on windows and will consume minimal memory during execution...like a couple
of MB. Unfortunately, most drivers don't use file channels. :)
>
> Martin
>
> Sent via BlackBerry by AT&T
>
>
> -----Original Message-----
> From: Frank Warmerdam <warmerdam at pobox.com>
>
> Date: Fri, 04 Jan 2008 11:09:19
> To:Daniele Romagnoli <dany.geotools at gmail.com>
> Cc:gdal-dev at lists.osgeo.org
> Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on
>         big sizes.
>
>
> Daniele Romagnoli wrote:
> > Hi list,
> > I'm trying to build a very big In Memory Raster Dataset (~1 GB) using
> > SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
> >
> > When calling the java method memoryDriver.create(...) the following
> > error appears "ERROR 2: Unable to create band arrays ... out of memory"
> > which is thrown when the underlying calloc method can't successfully
> > allocate memory.
> > I have performed some tests and tunings with the following conclusions:
> > - reducing the JVM Heap Size (using -Xmx command) allows to allocate a
> > bigger native memory portion (using -Xmx1024M allows me to allocate only
> > <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)
> > [Java Heap space and "Native" Heap space are managed separately]
> > - increasing the virtual memory, the result is the same (out of memory).
> > - killing all unuseful processes -> freeing more physical memory (~1.2
> > GB), the result is the same (out of memory))
> > Any setting allows me to allocate a max of ~840000 KBytes.
> >
> > Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual
> > memory setting (with ~2GB free) allows me only to create always ~840000
> > KBytes?
> > Does anyone know how memory is managed, how the calloc method operates
> > and whether an OS limitation exist?
> > Thank you very much.
>
> Daniele,
>
> I have no reason to believe this is a GDAL specific problem.  I presume
you
> are running 32bit Windows Vista?  If so, I believe any one process can
have
> at most 2GB of heap space which in your case would presumably be split
> between the JVM and non Java memory in the same process.
>
> I think you could allocate larger in memory rasters if you were running
Win64
> which has a very large potential virtual memory size per process.
>
> PS. I started out this reply to tell you to check the ulimit command
output
> which reports per-process memory limits on linux before I realized you
were
> using windows.  I don't know if there is any equivelent thing on windows.
>
> Best regards,
> --
>
---------------------------------------+------------------------------------
--
> I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | President OSGeo, http://osgeo.org
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>

-- 
-------------------------------------------------------
Eng. Simone Giannecchini
President /CEO GeoSolutions S.A.S.
Via Carignoni 51
55041  Camaiore (LU)
Italy

phone: +39 0584983027
fax:      +39 0584983027
mob:    +39 333 8128928

http://www.geo-solutions.it

-------------------------------------------------------