[gdal-dev] Re: Strange things with gdalwarp ...

[Even Rouault]

Quoting Rahkonen Jukka <Jukka.Rahkonen at mmmtike.fi>:
> >
> > Jukka, you can probably speed up considerably the gdalwarp
> > process by adding "-co TILED=YES" to the gdalwarp
> > commandline. For such a big output file, using the default
> > stripped structure of a TIFF will be very inefficient because
> > the way gdalwarp proceeds (by sort of tiles) causes a lot of
> > cache thrashing and I/O. A little of tuning with the "-wm
> > XXX", "-config GDAL_CACHEMAX YYY" and -multi options might also help.
>
> I can have a try. I want to change just on option at a time and now it
> is running with -co TILED=YES option. I will see tomorrow how did it do.
> Now I can say just that in the beginning things looked much better.
> Memory consumption and processor load were much higher and disk unit
> went more silently. But after some half an hour the conversion  seems to
> have almost stopped at 2,302,729,094 bytes output file size.  Gdalwarp
> reserves 390 megabytes of memory and utilises 5-15 percent of processor
> time.

I have no explanation in mind why it seems to have stalled at 2.3 GB. I would
let it go and see what happens a few hours after.

>
> Can you suggest any reasonable values for other parameters? This old
> machine is running on Windows XP Pro and it has only 1 GB memory and one
> 3 GHz processor.

The -wm defaults to 64 MB and GDAL_CACHEMAX to 40 MB. So increasing both to 300
for example might work (it will let 1000-2*300=400 MB for the rest of the OS and
other buffers of gdalwarp). If you see the computer starts swapping, those
values should be reduced. See http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp
for a discussion on those parameters.

> I do not have myself a real need for making large mosaics through .vrt
> files at the moment, but as the set-up is ready and the computer
> otherwise idle I can well let it run and gather some hopefully
> comparable numbers. I have also a bit more powerful computer with 8
> cores that I can use for final run if there is a reason to believe that
> gdalwarp can do the job with some settings. I guess -multi stands for
> multithreading and it may suit well the 8-core system.

-multi can also be helpfull with a single core system. It uses a thread for I/O
and another one for computation. The I/O thread is mostly I/O bound and doesn't
generally consumes much CPU, except if you use a compressed input or output file
(which might be the case with your JP2K data). But having more than 2 cores
won't help much.

>
> -Jukka-
>