[Gdal-dev] Re: gdal fails on vrt (virtual raster) file

[Frank Warmerdam]

Even Rouault wrote:
> Answering to my self and after a few more investigations, we can read in 
> jpeglib.h :
> 
>   /* Limit on memory allocation for this JPEG object.  (Note that this is
>    * merely advisory, not a guaranteed maximum; it only affects the space
>    * used for virtual-array buffers.)  May be changed by outer application
>    * after creating the JPEG object.
>    */
>   long max_memory_to_use;
> 
> 
> So, an alternative to define JPEGMEM (which seems to do what I thought it 
> should do in my previous message) or  could be to add the following lines :
>     if (getenv("JPEGMEM") == NULL)
>        poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
> just after creating the decompression object done by :
>     jpeg_create_decompress( &(poDS->sDInfo) ) ;
> 
> This way, we don't need to patch libjpeg and if the user wants to define it's 
> own value, he's still able to do so.

Even,

I think this is a good idea, and I'd appreciate your adding it trunk.  But
I wonder if jpeg_create_decompress() doesn't already do a bunch of work that
might start allocating memory / creating this spill file.

> FYI, on my Ubuntu system, I found out that libjpeg is build by default with 
> 1GB for DEFAULT_MAX_MEM.
> I've tried to define JPEGMEM to a small value but couldn't reproduce 
> Wolfgang's crash. I think it's because, in jmemansi.c, we can see that the 
> temporary file is created by the C library "tmpfile()" call. On Linux, I 
> think this file is created in /tmp : doing a "lsof | grep gdalinfo" with 
> small values of JPEGMEM on a 10000x10000 progressive jpeg seems to confirm 
> this intuition. I don't know where the MS C library tries to create the 
> temporary file. The answer to this question could show a need for a specific 
> patch in libjpeg for FWTool to create the temporary file in an appropriate 
> directory.
> 
> To answer one of Wolfgang's question, it may be surprising that gdalinfo is 
> affected by this issue. But in JPGDataset::Open, we call 
> jpeg_start_decompress that is really slow when I set a small value for 
> JPEGMEM. Then we read image size, color space and other information. The 
> question is : do we really need to do jpeg_start_decompress to get this 
> values ? Probably, but I we don't, it could speed up gdalinfo.

I don't know either.  If we don't that's great, lets fix it.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org