[Gdal-dev] Re: gdal fails on vrt (virtual raster) file

[Even Rouault]

Wolfgang, for your information, I've commited in GDAL trunk a variant of the 
fix for the JPEGMEM issue (see ticket 1796 on GDAL trac)

I've also just commited an improvement that should speed up the opening of 
jpeg datasets when just doing gdalinfo on them (see ticket 1798)

Best regards

On Wednesday 05 September 2007 21:58:27 Frank Warmerdam wrote:
> Even Rouault wrote:
> > Answering to my self and after a few more investigations, we can read in
> > jpeglib.h :
> >
> >   /* Limit on memory allocation for this JPEG object.  (Note that this is
> >    * merely advisory, not a guaranteed maximum; it only affects the space
> >    * used for virtual-array buffers.)  May be changed by outer
> > application * after creating the JPEG object.
> >    */
> >   long max_memory_to_use;
> >
> >
> > So, an alternative to define JPEGMEM (which seems to do what I thought it
> > should do in my previous message) or  could be to add the following lines
> > : if (getenv("JPEGMEM") == NULL)
> >        poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
> > just after creating the decompression object done by :
> >     jpeg_create_decompress( &(poDS->sDInfo) ) ;
> >
> > This way, we don't need to patch libjpeg and if the user wants to define
> > it's own value, he's still able to do so.
>
> Even,
>
> I think this is a good idea, and I'd appreciate your adding it trunk.  But
> I wonder if jpeg_create_decompress() doesn't already do a bunch of work
> that might start allocating memory / creating this spill file.
>
> > FYI, on my Ubuntu system, I found out that libjpeg is build by default
> > with 1GB for DEFAULT_MAX_MEM.
> > I've tried to define JPEGMEM to a small value but couldn't reproduce
> > Wolfgang's crash. I think it's because, in jmemansi.c, we can see that
> > the temporary file is created by the C library "tmpfile()" call. On
> > Linux, I think this file is created in /tmp : doing a "lsof | grep
> > gdalinfo" with small values of JPEGMEM on a 10000x10000 progressive jpeg
> > seems to confirm this intuition. I don't know where the MS C library
> > tries to create the temporary file. The answer to this question could
> > show a need for a specific patch in libjpeg for FWTool to create the
> > temporary file in an appropriate directory.
> >
> > To answer one of Wolfgang's question, it may be surprising that gdalinfo
> > is affected by this issue. But in JPGDataset::Open, we call
> > jpeg_start_decompress that is really slow when I set a small value for
> > JPEGMEM. Then we read image size, color space and other information. The
> > question is : do we really need to do jpeg_start_decompress to get this
> > values ? Probably, but I we don't, it could speed up gdalinfo.
>
> I don't know either.  If we don't that's great, lets fix it.
>
> Best regards,