[GRASS-dev] Re: grass-dev Digest, Vol 43, Issue 8

[Markus Metz]

Hi Laura,


Laura Toma wrote:
> Hi Markus,
>
> How much memory was available  on the machine?  
8 GB
> If the machine had more than 512MB  RAM,  it is not fair to run
> terracost running with mem=400MB, and compare it with an algorithm
> that can use more memory.
I don't understand, both modules can use more than 400MB of memory. I
set r.terracost to use 400MB max and r.cost to use 135MB max. If
anything, this is not fair for r.cost and gives an advantage to
r.terracost. I did not mention that I gave r.terracost another advantage
by assigning the temporary directories to a folder on a separate, very
fast hard drive that had nothing else to do but manage the temp files of
r.terracost. The temp files of r.cost are in the standard grass
.tmp/$HOST directory, in my case that (slower) hard drive also had other
things to do than just manage r.cost's temp files. I really tried to
give r.terracost a head start ;-)
>
> However, I am surprised that withnumtiles=1,  it was slower than
> r.cost.   
???
cut'n paste from below
r.terracost numtiles=1
real    0m17.969s
user    0m17.276s
sys    0m0.500s

r.cost in grass7, percent_memory=20% # that's not percent of available
memory, that's percent of the region to keep in memory, for the test
region this was max 135 MB
real    0m55.349s
user    0m53.360s
sys    0m1.797s

-> r.terracost with numtiles=1 is much faster than r.cost

> That's something I'd like to  look into.  Would you mind sharing the
> raster with me, and sending me the exact commands that you ran?
Is in the message you replied to, see below.
cut'n paste from below

North Carolina sample dataset

g.region rast=elev_state_500m at PERMANENT res=100
# gives about 28 million cells
v.to.rast nc_state at PERMANENT use=val val=500.0 out=cost --o
v.to.rast urbanarea at PERMANENT use=val val=1 out=urbanarea --o
r.cost in=cost start_rast=urbanarea out=dist_urban percent_memory=20 --o

r.terracost in=cost start_rast=urbanarea out=dist_urban_terracost -i
gives
[...]
TILESIZE: nc_spm_08 N=28064550 elements, M=419430400 bytes, optimal
numtiles=1870

# test command for r.terracost, same as above, now with optimal
numtiles=1870
r.terracost in=cost start_rast=urbanarea out=dist_urban_terracost
numtiles=1870

>
> A grid with 29M points is pretty small, for today's machines.  I
> suggest running on something 10 times larger.
Will do.
>   And use a lot of sources, that makes the data access  pattern  less
> local.
Yes, I thought about that too, best would be a lot of evenly distributed
start points, that should force r.cost to do highly random and scattered
disk access. My theory however predicts that the current r.cost will
always be faster than the current r.terracost if r.terracost is run with
numtiles>1. We will see.

BTW, I took the liberty to fix r.terracost, it works now with
numtiles>1. See changelog for r39684
https://trac.osgeo.org/grass/changeset/39684

Markus M

>
> -Laura
>
>
>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Fri, 06 Nov 2009 09:50:02 +0100
>> From: Markus Metz <markus.metz.giswork at googlemail.com>
>> Subject: [GRASS-dev] comparing r.cost and r.terracost
>> To: GRASS developers list <grass-dev at lists.osgeo.org>
>> Message-ID: <4AF3E33A.1090208 at googlemail.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Same test region as before
>>
>> North Carolina sample dataset
>>
>> g.region rast=elev_state_500m at PERMANENT res=100
>> # gives about 28 million cells
>> v.to.rast nc_state at PERMANENT use=val val=500.0 out=cost --o
>> v.to.rast urbanarea at PERMANENT use=val val=1 out=urbanarea --o
>> r.cost in=cost start_rast=urbanarea out=dist_urban percent_memory=20 --o
>>
>> grass7 r.cost
>> real    0m55.349s
>> user    0m53.360s
>> sys    0m1.797s
>>
>> grass65 r.cost
>> real    26m35.166s
>> user    2m55.612s
>> sys    23m37.921s
>>
>> r.terracost: check optimal tile size for 400MB memory (default setting;
>> r.cost in grass7 used 135MB with 20% of maps in memory)
>> r.terracost in=cost start_rast=urbanarea out=dist_urban_terracost -i
>> [...]
>> TILESIZE: nc_spm_08 N=28064550 elements, M=419430400 bytes, optimal
>> numtiles=1870
>>
>> r.terracost numtiles=1870, intermediate data are stored on disk as
>> r.cost does
>> real    25m13.593s
>> user    22m46.978s
>> sys    1m8.059s
>>
>> r.terracost numtiles=1, all in memory (just fits into 400MB)
>> real    0m17.969s
>> user    0m17.276s
>> sys    0m0.500s
>>
>> According to Laura Toma, when comparing r.cost with r.terracost,
>> numtiles must be >1 for r.terracost in order to compare disk I/O
>> algorithms [1]
>>
>> With these test settings that intentionally reduced memory consumption
>> in order to test disk I/O performance, r.terracost is not really faster
>> than r.cost in grass65 and much slower than r.cost in grass7.
>>
>> Markus M
>>
>> [1] http://lists.osgeo.org/pipermail/grass-dev/2009-July/045157.html
>>
>>
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> grass-dev mailing list
>> grass-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/grass-dev
>>
>> End of grass-dev Digest, Vol 43, Issue 8
>> ****************************************
>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev