<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Even,<div class="">This looks great and I’m really looking for point 4 and 5.</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">Le 3 mai 2019 à 04:04, Even Rouault <<a href="mailto:even.rouault@spatialys.com" class="">even.rouault@spatialys.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi,<br class=""><br class="">I wanted to mention COG related enhancements (*) that I will work on in GDAL <br class="">in the coming weeks, so interested parties are aware of them and can <br class="">potentially react.<br class=""><br class="">1) Creation of a dedicated COG creation-only driver simplifying the creation <br class="">workflow. Currently, creating a COG involves a number of steps, using gdaladdo <br class="">and gdal_translate with the right arguments. For very large COG files, <br class="">invoking gdaladdo in an efficient way can be tricky (.ovr.ovr trick: https://<br class=""><a href="http://github.com/OSGeo/gdal/issues/1442" class="">github.com/OSGeo/gdal/issues/1442</a>). The driver will take care of creating <br class="">needed temporary overviews.<br class=""><br class="">2) The driver will offer integrated reprojection capabilities, and in <br class="">particular a WebMercator/GoogleMapsCompatible tiling scheme profile (as <br class="">defined in WMTS), so that TIFF tiles exactly match GoogleMapsCompatible ones. <br class="">This will be similar to the corresponding option of GeoPackage. With a <br class="">subtelty that due to how GeoTIFF overviews work, it is not possible to have <br class="">this alignment on the tiling scheme for all zoom levels. So the user will <br class="">define how many zoom levels starting from the full resolution image must be <br class="">aligned (if N is the number of aligned levels, up to 2^N padding tiles in <br class="">horizontal and vertical dimensions are needed for the full resolution image, <br class="">so N should be kept reasonably small)<br class=""><br class="">3) gdalwarp will be enhanced to allow output to drivers that have only <br class="">CreateCopy() capabilities such as the COG driver. It will try to avoid <br class="">materializing the intermediate file when possible by using VRT capabilities, <br class="">otherwise it will have to create a temporary TIFF file before creating <br class="">CreateCopy()<br class=""></div></div></blockquote><div><br class=""></div><div>About point <b class="">1, 2, 3</b> I have mixed feeling because to me it seems that we will introduce a new driver to replace the combination of gdal commands (disclaimer as one of the creator of rio-cogeo I may not be fully objective here). </div><div class=""><br class=""></div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div class="">4) Optimizations specific to JPEG-compressed imagery (YCbCr color space) with <br class="">a 1-bit transparency channel, to minimize the number of HTTP range requests <br class="">needed to read them.<br class="">As JPEG compression cannot include the transparency information, two TIFF IFD <br class="">have to be created: one for YCbCr, and another one for alpha. Currently the <br class="">COPY_SRC_OVERVIEWS=YES creation option of the GeoTIFF driver separates data <br class="">for all the tiles of the color channels from data for all the tiles of the <br class="">transparency channel. In practice, readers will generally want to access, for <br class="">a same location, to data of both color and transparency channels. I will <br class="">modify the writer to interleave blocks so that color and transparency <br class="">information are contiguous. If COLOR_X_Y designates the tile with color <br class="">information at coordinates X,Y (in tile coordinate space), the layout of data <br class="">in the file will be: COLOR_0_0, TRANSPARENCY_0_0, COLOR_1_0, TRANSPARENCY_1_0, <br class="">etc. The GeoTIFF driver will be improved to fetch together the color and <br class="">transparency channel when such a layout is detected.<br class=""></div></div></blockquote><div><br class=""></div><div>Why this is specific to JPEG compression, what about other compressed format with internal mask ? </div><br class=""><blockquote type="cite" class=""><div class=""><div class="">A further improvement is to be able to avoid completely to read the <br class="">TileByteCount array of the color channel, and the TileByteCount & TileOffset <br class="">arrays of the transparency channel. The trick is to reserve 4 bytes before the <br class="">start of each COLOR_X_Y tile to indicate its size (those bytes will be <br class="">'ghost', that is not in the range of data pointed by TileByCount&TileOffset). <br class="">An optimized reader wanting to read tile i=Y*nb_tiles_in_width+X will start by <br class="">reading the offsets of tile i and i+1: TileOffset_color[i] and <br class="">TileOffset_color[i+1]. It will then seek to TileOffset_color[i] – 4 and read 4 <br class="">+ TileOffset_color[i+1] – TileOffset_color[i] bytes in a buffer. The first 4 <br class="">bytes of this buffer will indicate the number of bytes of the color tile, and <br class="">thus it is possible to deduce the offset and size of the mask tile that is <br class="">located at the end of the buffer. A TIFF metadata item will be written to <br class="">indicate that such layout has been used (with an indication of the file size <br class="">so as to be able to detect if the file has been later be altered in a non-<br class="">optimized way), so that optimized readers can adopt the above described <br class="">behavior. This will require to extend the libtiff interface so that the user <br class="">can directly provide the input buffer to decompress.<br class="">As the file will remain fully TIFF/BIGTIFF compliant, non-optimized readers <br class="">(such as newer GDAL builds against an older external libtiff version, or <br class="">previous GDAL versions) will still be able read it, loading values from the 4 <br class="">arrays instead of just one. <br class="">Note: for other compressions types, a simpler version of the above <br class="">optimization can still be done, by using TileOffset[i] and TileOffset[i+1], <br class="">and saving the read of TileByteCount[i]<br class="">To sum up, with the improvements of this task, once the initial loading of <br class="">metadata has been done, a GDAL ReadBlock(x,y) request will cause only two <br class="">networks range requests: one to read TileOffset[i] and TileOffset[i+1] <br class="">(potentially already cached if neighboring tiles have been previously accessed <br class="">in the same process), and another one to read the imagery (+mask) data. <br class="">Whereas currently, 6 might be needed for JPEG YcbCr+mask.<br class=""><br class="">5) Optimizing the layout of the header of a COG file<br class=""><br class="">The current layout of the header part of COG file is:<br class="">- TIFF / BigTIFF signature, followed by the offset of the first IFD (Image <br class="">File Directory)<br class="">- IFD of full resolution image, that is the list of the tags and their value <br class="">when it consists of a single numeric value, followed by the offset of the next <br class="">- IFD. Its size is 2 + number_of_tags * 12 + 4 (or 2 + number_of_tags * 20 + <br class="">8) bytes, so typically 200 bytes maximum<br class="">- Values of TIFF tags that don't fit inline in the IFD directory, such as <br class="">TileOffsets and TileByteCounts arrays and GeoTIFF keys <br class="">- IFD of first overview (typically subsampled by a factor of 2)<br class="">- Values of its tags that don't fit inline <br class="">- ...<br class="">-IFD of last overview<br class="">- Values of its tags that don't fit inline <br class=""><br class="">When the COG file is not too large, the fact of having the TileOffsets and <br class="">TileByteCounts between IFD descriptors is not an issue since they are not too <br class="">large, and most TIFF readers will load their values when opening the IFD. But <br class="">for an optimized reader such as GDAL with internal libtiff support (or with <br class="">external libtiff after the optimization of task 4), loading the values of the <br class="">TileOffsets/TileByteCounts arrays is only needed when accessing imagery.<br class=""><br class="">A more efficient layout for network access is :<br class="">- TIFF / BigTIFF signature, followed by the offset of the first IFD<br class="">- IFD of full resolution image, followed by the value of its non-inline tags, <br class="">except TileOffsets/TileByteCounts<br class="">- IFD of first overview followed by the value of its non-inline tags, except <br class="">- TileOffsets/TileByteCounts<br class="">- IFD of last overview followed by the value of its non-inline tags, except <br class="">TileOffsets/TileByteCounts<br class="">- Values of the TileOffsets/TileByteCounts arrays of IFD of full resolution <br class="">image<br class="">- Values of the TileOffsets/TileByteCounts arrays of IFD of first overview<br class="">- ...<br class="">- Values of the TileOffsets/TileByteCounts arrays of IFD of last overview<br class=""><br class="">With such a structure, the initial reading of 16 KB at the start of the file <br class="">will be able to load the IFD descriptors of all overviews (and masks, which <br class="">are actually interleaved in between when present). So, combined together with <br class="">task 4, a cold read of a tile at any zoom level (ie opening the file + tile <br class="">request) could result in just 3 network range requests: one to get the IFD <br class="">descriptors at the start of the file, one to read the location of the tile <br class="">from the TileOffsets array and one to read the tile data.<br class="">The proposed structure itself is still fully TIFF compliant. The script that <br class="">validates the COG structure will be adapted to accept that new variant of the <br class="">header structure.<br class=""><br class="">Even<br class=""><br class="">(*) Funding by Land Information New Zealand / <a href="https://www.linz.govt.nz/" class="">https://www.linz.govt.nz/</a><br class=""><br class="">-- <br class="">Spatialys - Geospatial professional services<br class=""><a href="http://www.spatialys.com" class="">http://www.spatialys.com</a><br class="">_______________________________________________<br class="">gdal-dev mailing list<br class="">gdal-dev@lists.osgeo.org<br class="">https://lists.osgeo.org/mailman/listinfo/gdal-dev</div></div></blockquote></div><br class=""></div></body></html>