[gdal-dev] Re: Strange things with gdalwarp ...

[Rahkonen Jukka]

Even Rouault wrote:
 
> Quoting Jukka Rahkonen <jukka.rahkonen at mmmtike.fi>:
> 
> > Hermann Peifer <peifer <at> gmx.eu> writes:
> >
> > >
> > > Jukka wrote
> >
> > > >
> > > > Did you try to use gdalwarp also for the second step with 
> > > > -srcnodata
> > value
> > [value...]?  I believe you did set
> > > some NODATA value in step 1 for the intermediate images.
> > > >
> > > > -Jukka Rahkonen-
> > >
> > > I did. However, the main problem with gdalwarp is its 
> performance. 
> > > So my
> > options seem to be:
> > >
> > > 1.
> > > gdalwarping 1500 ASTER tiles in 1 run into a 4GB 100m raster 
> > > output.tif = 5-10 days processing (my best estimate)
> > >
> > > 2.
> > > gdalwarping 1500 ASTER tiles 1-by-1 as an intermediate step, then 
> > > gdal_merge or gdal_translate to mosaic warped tiles into 
> output.tif 
> > > = smaller or larger NODATA areas in output.tif, along tile borders
> >
> > Hi,
> >
> > I decided to do a little test in order to see how slow 
> gdalwarp in in 
> > this kind of usage with my own imagery. My test plan was like this:
> >
> > 1. Original images: 199 JPEG2000 images, with 3 bands and 10000 by 
> > 10000 pixels each. All together make 28 gigabytes on disk 
> in lossless 
> > JPEG2000 format, or 60 GB uncompressed.
> >
> > 2. My first try with gdalbuildvtr
> > gdalbuildvrt -input_file_list gdalwarptest.txt gdalwarptest.vrt 
> > 0...10...20...30...40...50...60...70...80...90...100 - done.
> > Nice experience, I like this little tool.
> >
> > 3. A try to make a tiff file from this virtual raster with gdalwarp 
> > gdalwarp -co BIGTIFF=YES -srcnodata 0 gdalwarptest.vrt 
> > gdalwarptest.tif Creating output file that is 182026P x 191204L.
> >
> >
> > Gdalwarp has now been running for 27 hours. Process is taking 70 
> > megabytes of memory and 5-15 percent of processor time. As 
> a result I 
> > have now output tiff file that is very, very slowly growing 
> in size. 
> > Now, after 27-hour run the filesize is 13 gigabytes. Judged by the 
> > filesize my new tiff file might be ready after 4-5 more days.  
> > However, I am not sure if this estimate is reliable.
> > The
> > gdalwarp status indicator is still showing just zero for me:
> > Processing input file gdalwarptest.vrt.
> > 0
> >
> > Conclusion: Too slow to be useful with my settings.
> > Question 1: Should it work this way?
> > Question 2: If yes, is it worth trying to find the limits 
> when it goes 
> > too slow and perhaps file a ticket?
> 
> Jukka, you can probably speed up considerably the gdalwarp 
> process by adding "-co TILED=YES" to the gdalwarp 
> commandline. For such a big output file, using the default 
> stripped structure of a TIFF will be very inefficient because 
> the way gdalwarp proceeds (by sort of tiles) causes a lot of 
> cache thrashing and I/O. A little of tuning with the "-wm 
> XXX", "-config GDAL_CACHEMAX YYY" and -multi options might also help.

I can have a try. I want to change just on option at a time and now it
is running with -co TILED=YES option. I will see tomorrow how did it do.
Now I can say just that in the beginning things looked much better.
Memory consumption and processor load were much higher and disk unit
went more silently. But after some half an hour the conversion  seems to
have almost stopped at 2,302,729,094 bytes output file size.  Gdalwarp
reserves 390 megabytes of memory and utilises 5-15 percent of processor
time.

Can you suggest any reasonable values for other parameters? This old
machine is running on Windows XP Pro and it has only 1 GB memory and one
3 GHz processor.
I do not have myself a real need for making large mosaics through .vrt
files at the moment, but as the set-up is ready and the computer
otherwise idle I can well let it run and gather some hopefully
comparable numbers. I have also a bit more powerful computer with 8
cores that I can use for final run if there is a reason to believe that
gdalwarp can do the job with some settings. I guess -multi stands for
multithreading and it may suit well the 8-core system.

-Jukka-