[STATSGRASS] Re: memory issues

[Rohan Sadler]

Hi Daniel,

I've been dealing with large data issues on single 32-bit machines 
successfully using  fortran (alternatively C) code dynamically loaded into 
R. In my application I've had to rapidly locate minute bits of information 
in a 6.5 Gb library of sets of pixel locations corresponding to 
parameterised shapes. I use fortran loaded into R because R by itself is an 
interpretive language (like Python) that in parsing the code holds objects 
in memory (therefore no writing and dumping of memory objects) until parsing 
is complete, instead of being fast wired as compiled machine code. From the 
size of your data (equivalent to 2000x2000) my opinion is that it is the 
only way if you want to do it in R on 32 bit machines. I don't know what 
your options are with r.mapcalc.

The steps I visualise would be:
1) use R to import single band layers and then write as text files.
2) Write fortran (f77) code to process files but call from within R.
3) import output text file into R and export to GRASS.

If the processing is complex, and wishes to use sophisticated R functions 
then the code becomes a hybrid of fortran subroutines and wrapped by R code 
(e.g. to modify parameter values from logicals passed from fortran 
subroutines to R to then be put into another iteration of a fortran 
subroutine).

Also, if you are able to have it all work from a clean workspace 
add --vanilla to a R CMD BATCH command.

With the dynamically loaded script it is more involved but worth it in the 
end. Limited help is found in the R documentation on writing R extensions.

Regards
Rohan Sadler

Ecosystems Research Group M090
The University of Western Australia
35 Stirling Hwy, Crawley, 6008
ph:       +61 8 6488 7914
fax:      +61 8 6488 7925
mobile: +61 0433 192 600
email:    rsadler at cyllene.uwa.edu.au

Affiliations:
Bushfire CRC
School of Mathematics and Statistics, UWA
>
> Message: 5
> Date: Wed, 05 Apr 2006 19:54:30 +0100
> From: Daniel McInerney <daniel.mcinerney at ucd.ie>
> To: statsgrass at grass.itc.it
> Subject: [STATSGRASS] memory issues.
>
> Hello list,
>
> I've been using R/GRASS for satellite
> image classification and I've been content
> with it's capabilities. I recently changed
> study area and increased the number
> of image bands being used for classification
> and began to encounter a series of memory
> related issues during R sessions.
>
> GRASS rasters are read in using
> 'readCELL6sp' and then stored as a data.frame
> i.e.
>>tmpimg <- data.frame(readCELL6sp("rasterlayer")[,1])
>
> The current study region is composed of approximately
> 3989808 pixels and up to 15 bands. The .RData file
> is in excess of 700Mb.
>
> Various error messages have been returned during the
> process.
>
> The following when reading in the grass rasters
> $>"cannot open file '/tmp/RtmpRAYfK2/file74b0dc51', reason
>        'No such file or directory"
>
> There is no shortage of space available in the partition
> within which /tmp resides.
>
> The following during image classification. It occurs
> during the  second iteration after modification of
> parameters and the first classification has been
> written to GRASS with 'writeRast6':
>
> $>Error: cannot allocate vector of size 187022 Kb
>
> When trying to edit functions after above errors
> occurred.
> $>Error in edit(name, file, title, editor) :
>        problem with running editor vi
>
>
> I've included gc() in my functions as well as removing
> temporary objects during runtime. I recompiled R to
> enable linux-lfs and 64bit libs. I've also started R with
> various combinations of --max/min-vsize and --max/min-nsize
> However, none of these have greatly improved the situation.
>
> I'm writing to know whether anyone has encountered
> something similar and whether a work around exists? Am
> I asking too much of R or am I overlooking something?
>
> Any advice or suggestions would be greatly appreciated.
>
> Thanks in advance,
>
> Regards,
> Daniel.
>
>
>
> System
> -------
>
> Debian 2.6.8-11-amd64
> R 2.2.1
> sp 0.8-9
> spgrass6 0.1.3
> spGDAL 0.1.1
> spproj 0.3-2
> GRASS 6.1 cvs 25.02.06
>
>
>
> --__--__--
>
> Message: 8
> Date: Thu, 06 Apr 2006 11:10:49 +0100
> From: "Edzer J. Pebesma" <e.pebesma at geo.uu.nl>
> To: Daniel McInerney <daniel.mcinerney at ucd.ie>
> Cc: statsgrass at grass.itc.it
> Subject: Re: [STATSGRASS] memory issues.
>
> Daniel, it seems you have a setup that anticipated large arrays.
> What happens if you I give in R:
>
> > object.size(as.numeric(1:1e9))
>
> I get:
>
> Error: cannot allocate vector of size 3906250 Kb
> In addition: Warning messages:
> 1: Reached total allocation of 1023Mb: see help(memory.size)
> 2: Reached total allocation of 1023Mb: see help(memory.size)
>
> but if your R is really 64 bits, this should work.
> --
> Edzer
>
> Daniel McInerney wrote:
>
>>Hello list,
>>
>>I've been using R/GRASS for satellite
>>image classification and I've been content
>>with it's capabilities. I recently changed
>>study area and increased the number
>>of image bands being used for classification
>>and began to encounter a series of memory
>>related issues during R sessions.
>>
>>GRASS rasters are read in using
>>'readCELL6sp' and then stored as a data.frame
>>i.e.
>>
>>
>>>tmpimg <- data.frame(readCELL6sp("rasterlayer")[,1])
>>>
>>>
>>
>>The current study region is composed of approximately
>>3989808 pixels and up to 15 bands. The .RData file
>>is in excess of 700Mb.
>>
>>Various error messages have been returned during the
>>process.
>>
>>The following when reading in the grass rasters
>>$>"cannot open file '/tmp/RtmpRAYfK2/file74b0dc51', reason
>>        'No such file or directory"
>>
>>There is no shortage of space available in the partition
>>within which /tmp resides.
>>
>>The following during image classification. It occurs
>>during the  second iteration after modification of
>>parameters and the first classification has been
>>written to GRASS with 'writeRast6':
>>
>>$>Error: cannot allocate vector of size 187022 Kb
>>
>>When trying to edit functions after above errors
>>occurred.
>>$>Error in edit(name, file, title, editor) :
>>        problem with running editor vi
>>
>>
>>I've included gc() in my functions as well as removing
>>temporary objects during runtime. I recompiled R to
>>enable linux-lfs and 64bit libs. I've also started R with
>>various combinations of --max/min-vsize and --max/min-nsize
>>However, none of these have greatly improved the situation.
>>
>>I'm writing to know whether anyone has encountered
>>something similar and whether a work around exists? Am
>>I asking too much of R or am I overlooking something?
>>
>>Any advice or suggestions would be greatly appreciated.
>>
>>Thanks in advance,
>>
>>Regards,
>>Daniel.
>>
>>
>>
>>System
>>-------
>>
>>Debian 2.6.8-11-amd64
>>R 2.2.1
>>sp 0.8-9
>>spgrass6 0.1.3
>>spGDAL 0.1.1
>>spproj 0.3-2
>>GRASS 6.1 cvs 25.02.06
>>