[gdal-dev] JPEG compressed GeoTIFF ignores Nodata

[Marius Jigmond]

Just realized that earlier I hit the wrong reply button (apologies).

The COPY_SRC_OVERVIEWS yielded a wild result. Here's what I did:
1. gdaladdo -ro s70rgb321.tif 2 4 8 16 32 64 128 (note: input GTiff is 3
band with nodata=0 and no mask)
2. gdal_translate -co COMPRESS=JPEG -co TILED=YES -co JPEG_QUALITY=90
-co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES s70rgb321.tif test1.tif

The result: http://tinypic.com/r/110gig4/7

The top (~)quarter of the original image is replicated downwards up to
the original extent of the image. I would have expected the artifacts
but not this :). According to the http://www.gdal.org/frmt_gtiff.html
COPY_SRC_OVERVIEWS is applied as long as no general options are passed.

-marius


-- 
Sent from Ubuntu



On Sat, 2011-01-29 at 21:04 +0100, Even Rouault wrote: 
> Le samedi 29 janvier 2011 20:28:02, Marius Jigmond a écrit :
> > Thanks Even. I ran some more tests but that didn't quite work for me. I
> > still get those noisy pixels after gdal_translate (in QGIS I set value 0
> > for transparency and those pixels stand out). However, I noticed the
> > following:
> > 
> > * nearblack -setmask shrinks the significant pixels area (expected?)
> > * the near black noise pixels seem to follow a stair step pattern
> > similar to the edge of the significant area but at a different scale
> > (larger steps, if what I just said makes sense) and extend past the
> > significant data areas in both masked and unmasked datasets.
> 
> Ah yes, nearblack has some default parameters that can account for this. See 
> the -near and -nb parameters of http://gdal.org/nearblack.html
> 
> You can set them to 0 to have strict nearblack behaviour.
> 
> > > 
> > I was aware of the fact that nodata values are band specific but I had a
> > little brain glitch from working with gdaladdo (which honors nodata only
> > as a triplet nodata). That's how I actually ran into the problem. I am
> > trying to compress a RGB GeoTIFF using JPEG and then generate compressed
> > overviews. We plan to serve some imagery through Geoserver and would
> > like to stick to open-source solutions. The compressed datasets are
> > significantly smaller and with overviews they are extremely responsive,
> > rendering very fast). OpenJPEG v2 looks great but I haven't found
> > support for it in Geoserver, yet.
> 
> To limit the issues with artifacts, I would compute the overviews on the 
> uncompressed dataset and then translate the whole full resolution + overviews 
> into a JPEG compressed TIFF with the new COPY_SRC_OVERVIEWS option of the GTiff 
> driver.
> 
> I'm not sure that JPEG2000 would solve the issue. This would need to be 
> checked if alpha channel is strictly preserved.
> 
> For the record, MapServer is able to use nodata mask band as transparency 
> channel.
> 
> > 
> > -marius
> > 
> > On Sat, 2011-01-29 at 16:13 +0100, Even Rouault wrote:
> > > Le samedi 29 janvier 2011 15:47:43, Marius Jigmond a écrit :
> > > > Hi Everyone,
> > > > 
> > > > I am having some trouble figuring out why adding JPEG compression to a
> > > > RGB GeoTIFF results in Nodata pixel triplets becoming data pixel
> > > > triplets. Is this expected behavior due to the noisy nature of JPEG
> > > > compression or to the fact that these are border Nodata triplets? I
> > > > know certain algorithms process the boundary between data and Nodata
> > > > differently.
> > > 
> > > Marius,
> > > 
> > > Yes you did find the cause : the pixels maching nodata values will be
> > > JPEG compressed, so you have no guarantee that the 0 value is preserved
> > > through compression/decompression. I'd also note that the nodata concept
> > > in GDAL is a per-band value (so it should not be interpreted as a nodata
> > > "triplet")
> > > 
> > > A possibility to workaround this would be to create a "nodata mask band"
> > > (see http://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask), but I'm
> > > pretty sure that QGIS ignores those mask bands.
> > > 
> > > Anyway, if you want to try and create one, you can :
> > > 
> > > 1) Use nearblack to create a nodata mask band based on (0,0,0) triplet
> > > nearblack -setmask -of GTiff -o out_with_mask.tif in.tif
> > > 
> > > (the -setmask option is new in GDAL 1.8.0)
> > > 
> > > 2) Then compress the dataset with
> > > gdal_translate -co COMPRESS=JPEG -co TILED=YES -co JPEG_QUALITY=90 -co
> > > PHOTOMETRIC=YCBCR out_with_mask.tif out_compressed.tif
> > > 
> > > Only the RGB channels will be JPEG compressed : the mask band will use
> > > lossless compression so you won't have artifacts at the borders between
> > > significant areas and nodata areas.
> > > 
> > > (You can add --config GDAL_TIFF_INTERNAL_MASK YES to make the nodata mask
> > > internal to the GTiff file, instead of having a .tif.msk external file,
> > > which will not make any difference for software using GDAL API but can
> > > be more convenient to have a sinle file)
> > > 
> > > FYI, you can later reconvert the dataset into an uncompressed RGBA
> > > dataset :
> > > 
> > > gdal_translate out_compressed.tif out_rgba.tif -b 1 -b 2 -b 3 -b mask
> > > 
> > > Best regards,
> > > 
> > > Even
>