[GRASS-user] Large File Support (LFS)

[Hamish]

John wrote:
> g.region -p
...
> rows:       14000
> cols:       11000
> cells:      154000000

for type = float this will use 154000000 * 4 / 1024000 ~= 600mb
RAM for method = min. (floats are 4 bytes each) It will be
twice that for mean. so memory and the percent= option are not
an issue here. (memory use is dependent on the region rows x
columns, not the number of input points)


> 2.1Gb file
> r.in.xyz -s -g input=/home/bob2/rawdata/clean/test4.asc 
>  output=test4 method=min type=FCELL fs=, x=6 y=7 z=8 zscale=1.0
>  percent=100
> Unable to open input file </home/bob2/rawdata/clean/test4.asc>
...
> So anything above 2Gb fails, even using percent=25, which for > the 2.1Gb file should have had an effect if memory was an
> issue.

as Glynn explained, I was wrong- glibc's fopen() does care if it
is not used on a fully 64bit software stack.

if you have a 32bit machine, the solution/workaround for this is
to pipe from stdin instead.  try this:

cat /home/bob2/rawdata/clean/test4.asc | r.in.xyz input=- \
  output=test4 method=min fs=, x=6 y=7 z=8

or this:

r.in.xyz input=- output=test4 method=min fs=, x=6 y=7 z=8 < \
  /home/bob2/rawdata/clean/test4.asc


> It does not really use the memory (or swap file) when using
> '-s' flag.

right, the scan flag only has to remember a single set of
min/max values for x,y,z. that's just a few bytes. if it sees
a smaller/larger value it can forget the old one and just
remember the current leader.

> Memory hangs around 45%, swap file=0% but the CPU hits 100%
> varying between processors (watched in system monitor).
> If '-s' flag is removed to create a raster then it uses 100%
> of memory

of, then you should employ the percent= option until it does
not. (start with percent=50 to make it run in 2 passes)

> and <2Gb files start to process fine (didn't continue
> processing, to save time in testing), but >2Gb give the same
> message: 'Unable to open input file...'

try sending the data via stdin rather than reading the file
directly, as in the examples above.

> This 32bit machine only has 1Gb RAM (the 64bit machine has
> 2Gb RAM and never had a problem), but (as observed in system
> monitor) I don't think memory is an issue for scanning the
> data with the '-s' flag.

right, as scanning doesn't have to remember much at all.


> I compiled and installed yesterday (20/03/10) following the
> specific Ubuntu page, but added yes to largefile support (and
> ignored the slight typo over folder locations, 'grass_current
> / grass_trunk'.)

you mean the wiki page?



... so changing to "input=-" should make it work on the 32bit
machine.



regards,
Hamish