[Gdal-dev] Re: gdal fails on vrt (virtual raster) file

[WolfgangZ]

Even Rouault schrieb:
> Answering to my self and after a few more investigations, we can read in 
> jpeglib.h :
> 
>   /* Limit on memory allocation for this JPEG object.  (Note that this is
>    * merely advisory, not a guaranteed maximum; it only affects the space
>    * used for virtual-array buffers.)  May be changed by outer application
>    * after creating the JPEG object.
>    */
>   long max_memory_to_use;
> 
> 
> So, an alternative to define JPEGMEM (which seems to do what I thought it 
> should do in my previous message) or  could be to add the following lines :
>     if (getenv("JPEGMEM") == NULL)
>        poDS->sDInfo.mem->max_memory_to_use = 500 * 1024 * 1024;
> just after creating the decompression object done by :
>     jpeg_create_decompress( &(poDS->sDInfo) ) ;
> 
> This way, we don't need to patch libjpeg and if the user wants to define it's 
> own value, he's still able to do so.
> 
> FYI, on my Ubuntu system, I found out that libjpeg is build by default with 
> 1GB for DEFAULT_MAX_MEM.
> I've tried to define JPEGMEM to a small value but couldn't reproduce 
> Wolfgang's crash. I think it's because, in jmemansi.c, we can see that the 
> temporary file is created by the C library "tmpfile()" call. On Linux, I 
> think this file is created in /tmp : doing a "lsof | grep gdalinfo" with 
> small values of JPEGMEM on a 10000x10000 progressive jpeg seems to confirm 
> this intuition. I don't know where the MS C library tries to create the 
> temporary file. The answer to this question could show a need for a specific 
> patch in libjpeg for FWTool to create the temporary file in an appropriate 
> directory.
> 
> To answer one of Wolfgang's question, it may be surprising that gdalinfo is 
> affected by this issue. But in JPGDataset::Open, we call 
> jpeg_start_decompress that is really slow when I set a small value for 
> JPEGMEM. Then we read image size, color space and other information. The 
> question is : do we really need to do jpeg_start_decompress to get this 
> values ? Probably, but I we don't, it could speed up gdalinfo.
> 
> 
> On Wednesday 05 September 2007 20:17:16 Even Rouault wrote:
>> Hi,
>>
>> I've just had a quick look into libjpeg. You may not need to patch the
>> #define DEFAULT_MAX_MEM, as the only indirect use of this I can see is in
>> jmemmgr.c where it	can be overloaded by the JPEGMEM environment variable.
>> So, even with FWTools, if you do "JPEGMEM=500M gdalinfo blah.jpeg", you
>> should have the same behaviour as the gdal SVN patched libjpeg.
>> That's just theory. I haven't tested anything.
>>
>> Best regards
>>
>> On Wednesday 05 September 2007 18:04:47 Frank Warmerdam wrote:
>>> WolfgangZ wrote:
>>>> this patch is probably not jet applied in the FWTools. But I'm
>>>> wondering why even gdalinfo needs to create a temporary file? Why is
>>>> there the need from gdalinfo or gdal_translate to read in the full
>>>> image? Do I understand that that limit equals the maximum file size
>>>> that can be handled? Where is this temporary file usually generated?
>>>> I'm working on W2k.
>>> Wolfgang,
>>>
>>> I agree that the patch is likely not in FWTools because FWTools uses
>>> a stock external libjpeg instead of the adjusted GDAL libjpeg.
>>>
>>> I'm not exactly clear what the temp file is for.  I have the vague
>>> impression it is only needed with progressive jpegs, but I could be
>>> wrong.
>>>
>>> I don't know for sure where it is created.
>>>
>>> Best regards,
>>>

I will try tomorrow to convert these data on my linux machine (with 
FWTools) again. Furthermore I will try a "gdal_translate -of JPEG 
input.jpg out.jpg" on all tiles to get non progressive jpgs (hopefully).
At least a more clear error would be nice to get a better hint where the 
error is. Would it also be possible to add to gdalinfo which compression 
is used (ie tiff has several possibilities (JPEG, LZW,...) and jpeg has 
a progressive and nonprogressive mode). I would suggest a --verbose mode 
for gdalinfo.

Thanks for your help so far!

Kind regards
Wolfgang