[GRASS-dev] Re: [GRASS-user] problems using r.proj with large data set

[Dylan Beaudette]

On Tuesday 19 December 2006 05:19, Glynn Clements wrote:
> Glynn Clements wrote:
> > > > Assuming that it works out okay, I'll probably end up merging
> > > > r.proj.seg into r.proj, with the macro handling the details of
> > > > whether to read from a tile cache (large maps) or from a single block
> > > > of memory (small maps).
> > >
> > > This makes sense. For small maps is it much faster to not use the
> > > cache?
> >
> > I haven't performed timings; to be of much use, they would need to be
> > done with "real-world" cases.
> >
> > As it stands, even if you made the cache large enough to hold the
> > entire map, it would currently get written out to a temporary file
> > then read back in on demand.
> >
> > There's bound to be some overhead to using an extra level of
> > indirection, but it's probably trivial compared to the projection
> > calculations (and particularly compared to bicubic interpolation).
> >
> > One of the next things which needs to be done to r.proj.seg is to add
> > an option to set the cache size. Once that's done, I'll look into
> > trapping the case where the cache can hold the entire map, and
> > eliminate the temporary file in that situation.
>
> I've done that; some (not exactly representative) results reprojecting
> GTOPO30 to spearfish (30m, 634 cols x 477 rows):
>
> $ r.proj input=w140n90 location=global mapset=glynn output=proj
> method=nearest
>
> real	0m2.971s
> user	0m2.922s
> sys	0m0.023s
> $ r.proj.seg memory=100 input=w140n90 location=global mapset=glynn
> output=proj method=nearest
>
> real	0m2.929s
> user	0m2.891s
> sys	0m0.018s
> $ r.proj input=w140n90 location=global mapset=glynn output=proj
> method=bilinear
>
> real	0m3.419s
> user	0m3.280s
> sys	0m0.106s
> $ r.proj.seg input=w140n90 location=global mapset=glynn output=proj
> method=bilinear memory=100
>
> real	0m3.461s
> user	0m3.346s
> sys	0m0.088s
> $ r.proj input=w140n90 location=global mapset=glynn output=proj
> method=cubic
>
> real	0m3.877s
> user	0m3.731s
> sys	0m0.111s
> $ r.proj.seg memory=100 input=w140n90 location=global mapset=glynn
> output=proj method=cubic
>
> real	0m4.169s
> user	0m4.058s
> sys	0m0.093s
>
> The speed of r.proj.seg relative to r.proj ranges from ~1.4% faster
> (nearest) to ~7.5% slower (bicubic). If those results hold up for
> representative data, r.proj.seg can replace r.proj.

i had similar tming results for my data:

Input Projection Parameters: +proj=longlat +no_defs +a=6378137 
+rf=298.257222101 +towgs84=0.000,0.000,0.000
Input Unit Factor: 1

Output Projection Parameters: +proj=aea +lat_1=20 +lat_2=60 +lat_0=40 
+lon_0=-96 +x_0=0 +y_0=0 +no_defs +a=6378137 +rf=298.257222101 
+towgs84=0.000,0.000,0.000
Output Unit Factor: 1
Input:
Cols: 3500 (3500)
Rows: 2500 (2500)
North: 40.000000 (40.000000)
South: 37.500000 (37.500000)
West: -123.000000 (-123.000000)
East: -119.500000 (-119.500000)
ew-res: 0.001000
ns-res: 0.001000

Output:
Cols: 3934 (7340)
Rows: 3994 (13894)
North: 306990.000000 (560610.000000)
South: -52470.000000 (-689850.000000)
West: -2223720.000000 (-2236860.000000)
East: -1869660.000000 (-1576260.000000)
ew-res: 90.000000
ns-res: 90.000000

---------------- results --------------------
r.proj:
real    0m32.558s
user    0m31.690s
sys     0m0.336s

r.proj.seg:
real    0m34.995s
user    0m32.978s
sys     0m0.952s


the output from r.proj and r.proj.seg give about 99.99% identical results. I 
have attached the output from r.report run on the difference between the two 
maps.


it looks like a couple of us have confirmed these results. Replacing r.proj 
with the new r.proj.seg seems like a good idea: 

1. fixes the nasty bilinear bug (yikes)
2. handles very large files (great!)

great job glynn!

Cheers,


-- 
Dylan Beaudette
Soils and Biogeochemistry Graduate Group
University of California at Davis
530.754.7341
-------------- next part --------------
+-----------------------------------------------------------------------------+
|                         RASTER MAP CATEGORY REPORT                          |
|LOCATION: casoilresource                             Tue Dec 19 11:54:31 2006|
|-----------------------------------------------------------------------------|
|          north: -63025.00495929    east: -1909837.87169447                  |
|REGION    south: -98868.38404437    west: -1936490.64075773                  |
|          res:       29.99445949    res:        30.01437958                  |
|-----------------------------------------------------------------------------|
|MASK:none                                                                    |
|-----------------------------------------------------------------------------|
|MAP: (untitled) (e.diff in PERMANENT)                                        |
|-----------------------------------------------------------------------------|
|                        Category Information                         |   cell|
|                    #|description                                    |  count|
|-----------------------------------------------------------------------------|
|           0-1.780694|from  to . . . . . . . . . . . . . . . . . . . | 642694|
|    1.780694-3.561387|from  to . . . . . . . . . . . . . . . . . . . |      2|
|    3.561387-5.342081|from  to . . . . . . . . . . . . . . . . . . . |      2|
|    5.342081-7.122775|from  to . . . . . . . . . . . . . . . . . . . |      2|
|    7.122775-8.903469|from  to . . . . . . . . . . . . . . . . . . . |      2|
|   8.903469-10.684162|from  to . . . . . . . . . . . . . . . . . . . |      2|
|  10.684162-12.464856|from  to . . . . . . . . . . . . . . . . . . . |      1|
|  17.806937-19.587631|from  to . . . . . . . . . . . . . . . . . . . |      1|
|  19.587631-21.368325|from  to . . . . . . . . . . . . . . . . . . . |      2|
|    28.4911-30.271794|from  to . . . . . . . . . . . . . . . . . . . |      1|
|  51.640119-53.420812|from  to . . . . . . . . . . . . . . . . . . . |      1|
|  60.543587-62.324281|from  to . . . . . . . . . . . . . . . . . . . |      1|
|  73.008443-74.789137|from  to . . . . . . . . . . . . . . . . . . . |      1|
|156.701049-158.481743|from  to . . . . . . . . . . . . . . . . . . . |      1|
| 259.981286-261.76198|from  to . . . . . . . . . . . . . . . . . . . |      1|
|452.296211-454.076904|from  to . . . . . . . . . . . . . . . . . . . |      1|
|                    *|no data. . . . . . . . . . . . . . . . . . . . | 418445|
|-----------------------------------------------------------------------------|
|TOTAL                                                                |1061160|
+-----------------------------------------------------------------------------+