[gdal-dev] JPEG compressed GeoTIFF ignores Nodata

[Marius Jigmond]

The link should work for another day or so (automatic server purge). I
can re-upload it if you don't get the chance to download it. Warning:
1.2GB.
https://www.twdb.state.tx.us/filetxfr/emaillinkdownload.aspx?id=&FileTransferID=289562464

-marius

-- 
Sent from Ubuntu



On Sat, 2011-01-29 at 22:25 +0100, Even Rouault wrote: 
> Le samedi 29 janvier 2011 22:17:45, Marius Jigmond a écrit :
> > Just realized that earlier I hit the wrong reply button (apologies).
> > 
> > The COPY_SRC_OVERVIEWS yielded a wild result. Here's what I did:
> > 1. gdaladdo -ro s70rgb321.tif 2 4 8 16 32 64 128 (note: input GTiff is 3
> > band with nodata=0 and no mask)
> > 2. gdal_translate -co COMPRESS=JPEG -co TILED=YES -co JPEG_QUALITY=90
> > -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES s70rgb321.tif test1.tif
> > 
> > The result: http://tinypic.com/r/110gig4/7
> > 
> > The top (~)quarter of the original image is replicated downwards up to
> > the original extent of the image. I would have expected the artifacts
> > but not this :). According to the http://www.gdal.org/frmt_gtiff.html
> > COPY_SRC_OVERVIEWS is applied as long as no general options are passed.
> 
> This is weird. Do you have a link so I can download s70rgb321.tif and test ?
> 
> > 
> > -marius
> > 
> > > Le samedi 29 janvier 2011 20:28:02, Marius Jigmond a écrit :
> > > > Thanks Even. I ran some more tests but that didn't quite work for me. I
> > > > still get those noisy pixels after gdal_translate (in QGIS I set value
> > > > 0 for transparency and those pixels stand out). However, I noticed the
> > > > following:
> > > > 
> > > > * nearblack -setmask shrinks the significant pixels area (expected?)
> > > > * the near black noise pixels seem to follow a stair step pattern
> > > > similar to the edge of the significant area but at a different scale
> > > > (larger steps, if what I just said makes sense) and extend past the
> > > > significant data areas in both masked and unmasked datasets.
> > > 
> > > Ah yes, nearblack has some default parameters that can account for this.
> > > See the -near and -nb parameters of http://gdal.org/nearblack.html
> > > 
> > > You can set them to 0 to have strict nearblack behaviour.
> > > 
> > > > I was aware of the fact that nodata values are band specific but I had
> > > > a little brain glitch from working with gdaladdo (which honors nodata
> > > > only as a triplet nodata). That's how I actually ran into the problem.
> > > > I am trying to compress a RGB GeoTIFF using JPEG and then generate
> > > > compressed overviews. We plan to serve some imagery through Geoserver
> > > > and would like to stick to open-source solutions. The compressed
> > > > datasets are significantly smaller and with overviews they are
> > > > extremely responsive, rendering very fast). OpenJPEG v2 looks great
> > > > but I haven't found support for it in Geoserver, yet.
> > > 
> > > To limit the issues with artifacts, I would compute the overviews on the
> > > uncompressed dataset and then translate the whole full resolution +
> > > overviews into a JPEG compressed TIFF with the new COPY_SRC_OVERVIEWS
> > > option of the GTiff driver.
> > > 
> > > I'm not sure that JPEG2000 would solve the issue. This would need to be
> > > checked if alpha channel is strictly preserved.
> > > 
> > > For the record, MapServer is able to use nodata mask band as transparency
> > > channel.
> > > 
> > > > -marius
> > > > 
> > > > On Sat, 2011-01-29 at 16:13 +0100, Even Rouault wrote:
> > > > > Le samedi 29 janvier 2011 15:47:43, Marius Jigmond a écrit :
> > > > > > Hi Everyone,
> > > > > > 
> > > > > > I am having some trouble figuring out why adding JPEG compression
> > > > > > to a RGB GeoTIFF results in Nodata pixel triplets becoming data
> > > > > > pixel triplets. Is this expected behavior due to the noisy nature
> > > > > > of JPEG compression or to the fact that these are border Nodata
> > > > > > triplets? I know certain algorithms process the boundary between
> > > > > > data and Nodata differently.
> > > > > 
> > > > > Marius,
> > > > > 
> > > > > Yes you did find the cause : the pixels maching nodata values will be
> > > > > JPEG compressed, so you have no guarantee that the 0 value is
> > > > > preserved through compression/decompression. I'd also note that the
> > > > > nodata concept in GDAL is a per-band value (so it should not be
> > > > > interpreted as a nodata "triplet")
> > > > > 
> > > > > A possibility to workaround this would be to create a "nodata mask
> > > > > band" (see http://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask), but
> > > > > I'm pretty sure that QGIS ignores those mask bands.
> > > > > 
> > > > > Anyway, if you want to try and create one, you can :
> > > > > 
> > > > > 1) Use nearblack to create a nodata mask band based on (0,0,0)
> > > > > triplet nearblack -setmask -of GTiff -o out_with_mask.tif in.tif
> > > > > 
> > > > > (the -setmask option is new in GDAL 1.8.0)
> > > > > 
> > > > > 2) Then compress the dataset with
> > > > > gdal_translate -co COMPRESS=JPEG -co TILED=YES -co JPEG_QUALITY=90
> > > > > -co PHOTOMETRIC=YCBCR out_with_mask.tif out_compressed.tif
> > > > > 
> > > > > Only the RGB channels will be JPEG compressed : the mask band will
> > > > > use lossless compression so you won't have artifacts at the borders
> > > > > between significant areas and nodata areas.
> > > > > 
> > > > > (You can add --config GDAL_TIFF_INTERNAL_MASK YES to make the nodata
> > > > > mask internal to the GTiff file, instead of having a .tif.msk
> > > > > external file, which will not make any difference for software using
> > > > > GDAL API but can be more convenient to have a sinle file)
> > > > > 
> > > > > FYI, you can later reconvert the dataset into an uncompressed RGBA
> > > > > dataset :
> > > > > 
> > > > > gdal_translate out_compressed.tif out_rgba.tif -b 1 -b 2 -b 3 -b mask
> > > > > 
> > > > > Best regards,
> > > > > 
> > > > > Even
>