[gdal-dev] Re: Strange things with gdalwarp ...

[Peter J Halls]

Just a thought, but 2.3GB is remarkably like the maximum size of a positive 
32bit integer.  Is the file being written on Windows or some other 32bit system?

Peter

Even Rouault wrote:
> Quoting Rahkonen Jukka <Jukka.Rahkonen at mmmtike.fi>:
>>> Jukka, you can probably speed up considerably the gdalwarp
>>> process by adding "-co TILED=YES" to the gdalwarp
>>> commandline. For such a big output file, using the default
>>> stripped structure of a TIFF will be very inefficient because
>>> the way gdalwarp proceeds (by sort of tiles) causes a lot of
>>> cache thrashing and I/O. A little of tuning with the "-wm
>>> XXX", "-config GDAL_CACHEMAX YYY" and -multi options might also help.
>> I can have a try. I want to change just on option at a time and now it
>> is running with -co TILED=YES option. I will see tomorrow how did it do.
>> Now I can say just that in the beginning things looked much better.
>> Memory consumption and processor load were much higher and disk unit
>> went more silently. But after some half an hour the conversion  seems to
>> have almost stopped at 2,302,729,094 bytes output file size.  Gdalwarp
>> reserves 390 megabytes of memory and utilises 5-15 percent of processor
>> time.
> 
> I have no explanation in mind why it seems to have stalled at 2.3 GB. I would
> let it go and see what happens a few hours after.
> 
>> Can you suggest any reasonable values for other parameters? This old
>> machine is running on Windows XP Pro and it has only 1 GB memory and one
>> 3 GHz processor.
> 
> The -wm defaults to 64 MB and GDAL_CACHEMAX to 40 MB. So increasing both to 300
> for example might work (it will let 1000-2*300=400 MB for the rest of the OS and
> other buffers of gdalwarp). If you see the computer starts swapping, those
> values should be reduced. See http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp
> for a discussion on those parameters.
> 
>> I do not have myself a real need for making large mosaics through .vrt
>> files at the moment, but as the set-up is ready and the computer
>> otherwise idle I can well let it run and gather some hopefully
>> comparable numbers. I have also a bit more powerful computer with 8
>> cores that I can use for final run if there is a reason to believe that
>> gdalwarp can do the job with some settings. I guess -multi stands for
>> multithreading and it may suit well the 8-core system.
> 
> -multi can also be helpfull with a single core system. It uses a thread for I/O
> and another one for computation. The I/O thread is mostly I/O bound and doesn't
> generally consumes much CPU, except if you use a compressed input or output file
> (which might be the case with your JP2K data). But having more than 2 cores
> won't help much.
> 
>> -Jukka-
>>
> 
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
--------------------------------------------------------------------------------
Peter J Halls, GIS Advisor, University of York
Telephone: 01904 433806     Fax: 01904 433740
Snail mail: Computing Service, University of York, Heslington, York YO10 5DD
This message has the status of a private and personal communication
--------------------------------------------------------------------------------