[GRASS-dev] r.proj.seg disk space usage problem

[Glynn Clements]

Hamish wrote:

> > I had a chance today to try a VERY large raster with r.proj.seg.   
> > 2.7GB raster, 55000x47000 cells.  The memory issue of r.proj have now 
> > been moved to a HD space issue.  With 7.75GB free, it chokes at about 
> > 75% in the allocating memory stage, while free space slowly drops to  
> > zero, with an "Error writing segment file".
> > It seems r.proj.seg, for its segmentation, creates an uncompressed  
> > copy of the whole raster on the HD (instead of in memory).  I'm  
> > guessing this has something to do with random access speed in rasters?
> > 
> > So I'm back to projecting pieces of the raster, but now so they fit  
> > in my disk free space, and patching them together.  Disappointing,  
> > but if that's the way it must be, I can live with it.
> 
> did you pump the memory= option up to a few hundred MB smaller than your
> installed RAM?

That won't affect disk usage, unless he has enough RAM to hold the
entire uncompressed map (~10GB in this case).

You need to be able to hold the entire uncompressed map (4 bytes per
cell) either in RAM or on disk. If it won't fit entirely in RAM, the
entire map is stored on disk, and portions are copied to RAM as
required. The portions which are in RAM are still kept on disk; there
isn't any "swapping".

You could conceivably use system swap as "RAM", but that would result
in taking physical RAM from any other applications on the system. At
best, the rest of the system is likely to become rather unresponsive
in that kind of situation; at worst, r.proj.seg and the rest of the
system will end up fighting it out for physical RAM, resulting in
"thrashing".

Also, that approach requires a 64-bit CPU and OS (in the sense of
providing a 64-bit address space). A 32-bit address space limits you
to no more than 4GB of "memory" (physical RAM, swap, mmap()d files
etc) per process, and the maximum size of the heap will probably be
significantly less than that 4GB.

-- 
Glynn Clements <glynn at gclements.plus.com>