[Gdal-dev] Re: gdal fails on vrt (virtual raster) file

Even Rouault even.rouault at mines-paris.org
Wed Sep 5 15:14:27 EDT 2007


Answering to my self and after a few more investigations, we can read in 
jpeglib.h :

  /* Limit on memory allocation for this JPEG object.  (Note that this is
   * merely advisory, not a guaranteed maximum; it only affects the space
   * used for virtual-array buffers.)  May be changed by outer application
   * after creating the JPEG object.
   */
  long max_memory_to_use;


So, an alternative to define JPEGMEM (which seems to do what I thought it 
should do in my previous message) or  could be to add the following lines :
    if (getenv("JPEGMEM") == NULL)
       poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
just after creating the decompression object done by :
    jpeg_create_decompress( &(poDS->sDInfo) ) ;

This way, we don't need to patch libjpeg and if the user wants to define it's 
own value, he's still able to do so.

FYI, on my Ubuntu system, I found out that libjpeg is build by default with 
1GB for DEFAULT_MAX_MEM.
I've tried to define JPEGMEM to a small value but couldn't reproduce 
Wolfgang's crash. I think it's because, in jmemansi.c, we can see that the 
temporary file is created by the C library "tmpfile()" call. On Linux, I 
think this file is created in /tmp : doing a "lsof | grep gdalinfo" with 
small values of JPEGMEM on a 10000x10000 progressive jpeg seems to confirm 
this intuition. I don't know where the MS C library tries to create the 
temporary file. The answer to this question could show a need for a specific 
patch in libjpeg for FWTool to create the temporary file in an appropriate 
directory.

To answer one of Wolfgang's question, it may be surprising that gdalinfo is 
affected by this issue. But in JPGDataset::Open, we call 
jpeg_start_decompress that is really slow when I set a small value for 
JPEGMEM. Then we read image size, color space and other information. The 
question is : do we really need to do jpeg_start_decompress to get this 
values ? Probably, but I we don't, it could speed up gdalinfo.


On Wednesday 05 September 2007 20:17:16 Even Rouault wrote:
> Hi,
>
> I've just had a quick look into libjpeg. You may not need to patch the
> #define DEFAULT_MAX_MEM, as the only indirect use of this I can see is in
> jmemmgr.c where it	can be overloaded by the JPEGMEM environment variable.
> So, even with FWTools, if you do "JPEGMEM=500M gdalinfo blah.jpeg", you
> should have the same behaviour as the gdal SVN patched libjpeg.
> That's just theory. I haven't tested anything.
>
> Best regards
>
> On Wednesday 05 September 2007 18:04:47 Frank Warmerdam wrote:
> > WolfgangZ wrote:
> > > this patch is probably not jet applied in the FWTools. But I'm
> > > wondering why even gdalinfo needs to create a temporary file? Why is
> > > there the need from gdalinfo or gdal_translate to read in the full
> > > image? Do I understand that that limit equals the maximum file size
> > > that can be handled? Where is this temporary file usually generated?
> > > I'm working on W2k.
> >
> > Wolfgang,
> >
> > I agree that the patch is likely not in FWTools because FWTools uses
> > a stock external libjpeg instead of the adjusted GDAL libjpeg.
> >
> > I'm not exactly clear what the temp file is for.  I have the vague
> > impression it is only needed with progressive jpegs, but I could be
> > wrong.
> >
> > I don't know for sure where it is created.
> >
> > Best regards,
> >
> > -
> > ---------------------------------------+---------------------------------
> >-- --- I set the clouds in motion - turn up   | Frank Warmerdam,
> > warmerdam at pobox.com light and sound - activate the windows |
> > http://pobox.com/~warmerdam and watch the world go round - Rush    |
> > President OSGeo, http://osgeo.org
> >
> > _______________________________________________
> > Gdal-dev mailing list
> > Gdal-dev at lists.maptools.org
> > http://lists.maptools.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> Gdal-dev mailing list
> Gdal-dev at lists.maptools.org
> http://lists.maptools.org/mailman/listinfo/gdal-dev





More information about the Gdal-dev mailing list