[gdal-dev] Very large time difference using gdal_translate from Jpeg2000 to different formats using JP2KAK driver

[Frank Warmerdam]

On Tue, Apr 17, 2012 at 10:04 PM, Adam Brown <adam.brown at arcitecta.com> wrote:
> On 15/04/12 2:48 PM, Frank Warmerdam wrote:
>>
>> On 12-04-12 05:53 PM, ahbrown wrote:
>>>
>>> Hi,
>>>
>>> I am wondering why there is such a large difference in the time to
>>> convert a
>>> large O(250MB) Jpeg2000 image to GeoTiff format in comparison with
>>> conversion to other formats (e.g. jpeg, png)?
>>>
>>> I am using the Kakadu Jpeg2000 driver. With a 43221 x 36021 Jpeg2000
>>> image.
>>>
>>> Results of timed conversions are as follows:
>>>
>>>
>>>
>>> Using RasterIO directly (via the Java wrapper) the conversion takes
>>> around 6
>>> seconds.
>>
>>
>> Adam,
>>
>> Many drivers implement the CreateCopy() rather than Create() method. In
>> these
>> cases the input dataset is passed to the output driver and they pull data
>> in
>> whatever way is convenient.
>>
>> The PNG and JPEG drivers are "CreateCopy" drivers and likely request data
>> one scanline at a time.  In this situation the JP2KAK driver actually gets
>> asked for the data in moderate sized tiles which turns out not to be very
>> efficient for the Kakadu library as there is a high setup cost for each
>> windowed read.
>>
>> The GeoTIFF format support Create(), so gdal_translate is able to more
>> closely manage the chunking of reads and writes and tries to do it in
>> large chunks.  The JP2KAK driver handles large reads through IRasterIO()
>> as single windowed reads which are more efficient than the smaller tile
>> reads.
>>
>> So I think the bulk of the difference in performance is related to
>> different
>> IO strategies that apply against the JP2KAK driver.  The variation between
>> JPEG and PNG times is likely different costs on the compression side.
>>
>> PS. it appears posts through nabble do not preserve some components, like
>> the fixed font timings text.  I encourage direct use of the mailing list
>> for posts.
>>
>> Best regards,
>
> Hi Frank and Chaitanya,
>
> Thanks for the quick response.
>
> What doesn't make sense to me is that I can perform the same JP2->PNG
> translation benchmarked below in a little over 6 seconds using the GDAL Java
> wrappers.
> Most of this is spent in the read/subsample operation. The write to the
> 200x200 png takes only 20 ms.
>
> So gdal_translate is adding 8-12s of overhead somewhere.
>
> I've posted the timings again below as they were stripped out from the
> original post:
>
> time gdal_translate -of GTiff -outsize 200 200 master_original.jp2
> master_original.tif
> real    0m2.325s
>
> time gdal_translate -of JPEG -outsize 200 200 master_original.jp2
> master_original.jpg
> real    0m18.317s
>
> time gdal_translate -of PNG -outsize 200 200 master_original.jp2
> master_original.png
> user    0m14.588s
>
> GDAL Java wrapper (same JP2 to 200x200 PNG): approx 6s

Adam,

Completely different mechanisms are being used.  I imagine in your
Java case you are doing a 200x200 read from the source file and
writing that to PNG, right?  gdal_translate is creating a VRT wrapper
for the downsampling and letting the PNG driver pull from it scanline
by scanline.

Anyways you haven't been specific about what you are doing, but the
devil is in the details.

Best regards,
Frank

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer