[gdal-dev] Re: Strange things with gdalwarp ...

[Even Rouault]

Quoting Jukka Rahkonen <jukka.rahkonen at mmmtike.fi>:

> Hermann Peifer <peifer <at> gmx.eu> writes:
>
> >
> > Jukka wrote
>
> > >
> > > Did you try to use gdalwarp also for the second step with -srcnodata
> value
> [value...]?  I believe you did set
> > some NODATA value in step 1 for the intermediate images.
> > >
> > > -Jukka Rahkonen-
> >
> > I did. However, the main problem with gdalwarp is its performance. So my
> options seem to be:
> >
> > 1.
> > gdalwarping 1500 ASTER tiles in 1 run into a 4GB 100m raster output.tif
> > = 5-10 days processing (my best estimate)
> >
> > 2.
> > gdalwarping 1500 ASTER tiles 1-by-1 as an intermediate step, then
> > gdal_merge or gdal_translate to mosaic warped tiles into output.tif
> > = smaller or larger NODATA areas in output.tif, along tile borders
>
> Hi,
>
> I decided to do a little test in order to see how slow gdalwarp in in this
> kind
> of usage with my own imagery. My test plan was like this:
>
> 1. Original images: 199 JPEG2000 images, with 3 bands and 10000 by 10000
> pixels
> each. All together make 28 gigabytes on disk in lossless JPEG2000 format, or
> 60
> GB uncompressed.
>
> 2. My first try with gdalbuildvtr
> gdalbuildvrt -input_file_list gdalwarptest.txt gdalwarptest.vrt
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> Nice experience, I like this little tool.
>
> 3. A try to make a tiff file from this virtual raster with gdalwarp
> gdalwarp -co BIGTIFF=YES -srcnodata 0 gdalwarptest.vrt gdalwarptest.tif
> Creating output file that is 182026P x 191204L.
>
>
> Gdalwarp has now been running for 27 hours. Process is taking 70 megabytes of
> memory and 5-15 percent of processor time. As a result I have now output tiff
> file that is very, very slowly growing in size. Now, after 27-hour run the
> filesize is 13 gigabytes. Judged by the filesize my new tiff file might be
> ready
> after 4-5 more days.  However, I am not sure if this estimate is reliable.
> The
> gdalwarp status indicator is still showing just zero for me:
> Processing input file gdalwarptest.vrt.
> 0
>
> Conclusion: Too slow to be useful with my settings.
> Question 1: Should it work this way?
> Question 2: If yes, is it worth trying to find the limits when it goes too
> slow
> and perhaps file a ticket?

Jukka, you can probably speed up considerably the gdalwarp process by adding
"-co TILED=YES" to the gdalwarp commandline. For such a big output file, using
the default stripped structure of a TIFF will be very inefficient because the
way gdalwarp proceeds (by sort of tiles) causes a lot of cache thrashing and
I/O. A little of tuning with the "-wm XXX", "-config GDAL_CACHEMAX YYY" and
-multi options might also help.

>
> -Jukka Rahkonen-
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>