[gdal-dev] Compression Artifacts After Merging Imagery

[Jacob Adams]

Oh ye great minds of the GDAL world, I've run into a problem that I've not been able to solve.

I have aerial imagery comprised of about 1500 jpegs that I am trying to merge together into three large (~18gb) jpeg-compressed geotiffs to be served by Geoserver. The images are tiled such that they shift down about 100 pixels (out of ~7000) every so often. With a pure gdal_merge or gdalwarp mosaic, this leaves compression artifacts in the nodata spaces along the borders. The cause of these artifacts seems to be explained in Evan's comments on this ticket: https://trac.osgeo.org/gdal/ticket/5102).  

I tried pre-processing a handful of the tilesinto geotiffs with the following three commands based on a similar discussion in February (http://lists.osgeo.org/pipermail/gdal-dev/2017-February/046027.html):
    nearblack -co TILED=YES -of Gtiff -nb 0 -near 0 -setmask -o temp1.tif Tile1.jpeg
    gdalwarp -t_srs EPSG:3560 -dstalpha -co TILED=YES temp1.tif temp2.tif
    gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif 

However, I still get the artifacts when I then try to mosaic the resulting tiffs using either gdalwarp or gdal_merge.py:
    python gdal_merge.py -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR --config GDAL_TIFF_INTERNAL_MASK YES -o merged.tif Tile1_out.tif Tile2_out.tif (etc)

Leaving the compression out of the gdal_translate call (leaving all the compression to the gdal_merge command) does not make a difference in the artifacts.

I've also tried to merge the images and cut the resulting compressed tiffs with cutlines,:
    python gdal_merge.py -ot Byte -of Gtiff -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES -o "f:\imagery\trimmed_merged_jpegs\0.tif" --optfile f:\imagery\0.txt
    gdalwarp.exe -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES -cutline f:\imagery_footprints\threeinch0.shp -crop_to_cutline "f:\imagery\trimmed_merged_jpegs\0.tif" "f:\imagery\trimmed_merged_jpegs\trimmed0.tif"

But this also fails to remove the artifacts.

My understanding of the problem is that the jpeg compression occurs as the last step of any command, and thus even when fed a properly-masked file the final compression still mangles the output.

Is there a way to remove these artifacts in a format amenable to Geoserver while preserving the compression levels achievable with ycbcr-based jpeg compression?

Thanks,

Jacob Adams
GIS Specialist, Cache County