[GRASS-dev] Re: [GRASS GIS] #73: r.out.gdal tiff output does not
work
GRASS GIS
trac at osgeo.org
Sat Oct 25 04:14:15 EDT 2008
#73: r.out.gdal tiff output does not work
--------------------------+-------------------------------------------------
Reporter: helena | Owner: grass-dev at lists.osgeo.org
Type: defect | Status: new
Priority: critical | Milestone: 6.4.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: r.out.gdal, tiff
Platform: Unspecified | Cpu: Unspecified
--------------------------+-------------------------------------------------
Comment (by hamish):
Replying to [comment:11 mmetz] (and others):
> No color table was specified with r.colors, so what data visualization
scheme is to be expected?
if none is specified (no colr/ files exists for the raster) the default
"rainbow" is used. GRASS's libgis does this automatically, AFAIK the
module never knows that r.colors hasn't been run.
> Gimp and many other image viewers only support 8 bit per channel,
cinepaint e.g. supports 32 bit.
this is a CELL map exported with type=Byte.
> Most of them can not display tiff files with anything else but Byte:
only tiff images, not tiff datasets.
the spearfish example I gave was Byte-only.
> Elevation is a dataset, not an image.
True, but if elev data is put in the correct range, Gimp neither knows nor
cares what the levels represent. I was just using it as random values, not
anything meaningful.
Spearfish's elevation.dted would be a better sample dataset to use here,
it is CELL 0-255. or convert spot.image into RGB bands.
Other test datasets are the spearfish imagery60 data, and the North
Carolina dataset lsat7 images.
> Still, datatypes other than Byte can only be read with spatial data
viewers that could make use of this colour table, but not all of them are
actually able to do so.
Important to know when this is happening, not really our problem, and not
much we can do about it as long as we are writing metadata to spec.
Hamish:
> > gdalinfo confirms palette info was not written to file:
> > ... COLOR_TABLE_RULES_COUNT=0 ... 0: 0,0,0,255 ... 255: 0,0,0,255
MM:
> But the color palette was indeed written to the file.
An empty one was written while within GRASS it had real colors.
For the expected colors run:
{{{
seq 0 255 | r.what.color -i in=elev_0
0: 255:255:0
1: 254:255:0
2: 252:255:0
...
253: 0:255:161
254: 0:255:163
255: 0:255:165
}}}
> It would help a lot to have a list with these data types and their
ranges in the man page so that the appropriate data type for file export
can be chosen.
Yes. Can someone in the know supply some suggested text?
> Maybe the color table is not so important, different people have
different preferences
The color table will be carefully constructed/chosen by some people and
they may wish to keep it. e.g. r.colors.stddev addon script, r.cpt2grass
addon script, and NASA color tables given for MODIS sat data (see grass's
MODIS wiki page).
These data are real (calibrated into sea surface temperature or mg/mL
chlorophyll) and also have specially designed color tables (incl log-
scale) which are nice to keep.
> Suggested data visualization with an embedded color table is nice to
have, but coloring schemes are a matter of taste
geotiff has a method for assigning color palettes, all I suggest is
populating that correctly, apparently we are currently (or were) failing
at.
> There is still the issue of nodata: a nodata value must fall into the
range of the chosen datatype, otherwise this information gets lost.
r.out.gdal does not check if a supplied nodata value is valid.
Hopefully better in SVN now. If the user manually specifies an out of
range value, I don't mind respecting that.
> For UINT16, the color table in the tiff file will always have 65535
entries, gdal does that.
perhaps this should be addressed in GDAL then; only write palette entries
which are given.
> Have an option in r.out.gdal to choose whether a color table should be
written to the tiff file?
For Uint16, how to just get the "COLOR_TABLE_RULE_RGB_0" part of this
without the "(RGB with 66535 entries)" part?
r.out.gdal geotiff from 0-255 (black & white) ortho photo with grass's
sepia color table:
{{{
Metadata:
COLOR_TABLE_RULES_COUNT=10
COLOR_TABLE_RULE_RGB_0=0.000000e+00 2.550000e+01 0 0 0 34 23 9
COLOR_TABLE_RULE_RGB_1=2.550000e+01 5.100000e+01 34 23 9 68 47 19
COLOR_TABLE_RULE_RGB_2=5.100000e+01 7.650000e+01 68 47 19 101 72 33
COLOR_TABLE_RULE_RGB_3=7.650000e+01 1.020000e+02 101 72 33 130 96 49
COLOR_TABLE_RULE_RGB_4=1.020000e+02 1.275000e+02 130 96 49 158 122 71
COLOR_TABLE_RULE_RGB_5=1.275000e+02 1.530000e+02 158 122 71 183 147 96
COLOR_TABLE_RULE_RGB_6=1.530000e+02 1.785000e+02 183 147 96 206 174
126
COLOR_TABLE_RULE_RGB_7=1.785000e+02 2.040000e+02 206 174 126 225 200
162
COLOR_TABLE_RULE_RGB_8=2.040000e+02 2.295000e+02 225 200 162 242 227
204
COLOR_TABLE_RULE_RGB_9=2.295000e+02 2.550000e+02 242 227 204 255 254
251
Color Table (RGB with 256 entries)
0: 0,0,0,255
1: 1,0,0,255
2: 2,1,0,255
3: 4,2,1,255
4: 5,3,1,255
5: 6,4,1,255
...
}}}
> r.out.gdal modified: - if a nodata value is given, it is adjusted to the
range of the selected datatype if it is out of range
It is better to respect the user's instructions. Give them a warning if
you like, but do what they say; don't quietly "fix" it for them.
> basic statistics (min, max, mean, stddev) are calculated for the current
region, replacing NULL cells with nodata value (default or specified) and
then exported with GDALSetRasterStatistics()
how is this better than "gdalinfo -stats" ?
I don't understand, you convert NULL to nodata value in stats? So for a
CELL map with lots of NULL areas, if I make nodata=100 (well above the
number of used palette entries) or leave it at 255, the stats will be
skewed towards 100 or 255 instead of just considering the real data? huh?
"gdalinfo -stats" seems to do the right thing here.
> colortable is only exported on request (new flag needs to be set) and
only if there is an entry in directories colr or colr2
This is wrong, libgis is in charge of handling colors. Use the API not raw
file access. For one thing, otherwise you miss out on the default mapping
if there isn't a colr/ file.
> Additionally, a warning would be useful if the selected datatype does
not cover the range of raster values to be exported
i.e. if the map was CELL 0-744 as in my elev_0 example earlier in this
ticket, but the user asked for Byte (0-255). I think that's a very good
idea.
Start by moving range maxes into header #defines for each type from the
currently scattered "nodataval = -1E37f;" etc lines in the r.out.gdal
source code.
> Some other, commercial, software doesn't like GeoTIFFs with colortables
either, so maybe it is a good idea not to export colortables, or only on
request with a warning?
If other software ignore valid GeoTIFF metadata tags, then that's their
problem and their responsibility to provide an alternative for their
users. All we can do is correctly implement the spec. If other software
read but incorrectly handle valid GeoTIFF metadata, then that's really
their bug to deal with.
> Of the three packages tested here, only QGIS makes proper use of the
colortable, and QGIS is too close to GRASS to be regarded as independent
testing suite.
The test suite you can't argue with is the data in the file. It either
follows the geotiff spec or does not. (no idea how free-form or tight that
is in this regard)
> Apparently COLOR_TABLE_RULE entries are custom metadata not read by
other software.
Any confirmation on this? It is custom to GRASS or custom to OSGeo family
(GDAL dependents) or suggested by the geotiff spec or ...???
Even if 66535 rules are turned off by default by r.out.gdal for UInt16,
these range rules would be nice to keep as they are rarely very long.
> current region restored on exit
module should not have to do that, it should set WIND_OVERRIDE variable or
so, so if module is interrupted by ctrl-C or whatever the temporary change
does not become permanent. (but as mentioned earlier, this violates
grass's raster module conventions and the feature should not be applied in
SVN)
> raster band statistics (min, max, mean, stddev) written to metadata
perhaps nice for casual browsing, but again what's wrong with "gdalinfo
-stats" or similar scanning for that task?
> if supported, GDAL decides
what does that mean? when is min/max etc not supported??
> Tests were performed with the elevation raster map layer (range within
Byte, FCELL map) in the North Carolina sample dataset, creating a MASK for
elevation values <= 70m, needed for testing of nodata handling with
nodata=0. The elevation raster was exported as Byte, UInt16, Int32, and
Float32, once with colortable (Byte and UInt16 only, colortable export for
other data types not supported by gdal), once without colortable, always
as GeoTIFF.
Please supply map names and exact commands so we can replicate.
> QGIS-0.9.1 on Linux 64bit
> QGIS-0.11.1 on XP
> ...
> ESRI ArcMap 9.2 on XP
this is handy information for users to be aware of, but (again) not much
we can do about 3rd party support beyond coding to geotiff
spec/recomendations.
It is nice to see QGIS do well. If this is filtered through a GeoTIFF and
not using the GRASS native format plugin, then it is a good sign that we
are all coding to the same tune. I don't think the code itself has much
cut & paste between the projects, and GDAL I/O is independently well-
tested.
> If nodata specification is not supported, applications return the actual
cell value, in this case 0, instead of "NoData?", "NULL", "" or similar.
The files are still perfectly readable, only nodata assignment doesn't
happen.
this is as expected. If you don't know the tag, ignore it.
> I would prefer to use r.out.gmt to get the GRASS color rules into an
extra file (GMT .cpt). This would also be a workaround for export file
types other than Byte and UInt16 where GDAL does not support writing out a
colortable. But then I don't know how widely GMT .cpt files are supported.
It could be nice to have both internal and .cpt file saved, like current
world-file support. But from GRASS or (like world files) from GDAL?
FWIW, and it has been a while so I don't remember so well; I don't think I
spent much time making r.out.gmt's .cpt output very robust. AFAIR it was
a quick hack.
Since it has been some months since the original bug report, and the code
has changed, could we please restate what this bug actually is (rather
than fuzzy improved r.out.gdal niceness), and some spearfish or NC sample-
dataset instructions for recreating it? After studying the ticket from
beginning to end I can't say where the GRASS bug is, beyond:
* provide warning if map data overflows requested data type=
* type limits to #defines in the code, and human readable to the help
page
* how to make GDAL stop writing color rules for UInt16 once Maximum= is
reached (instead of writing out 66535-Max empty rules)
what else?
And we must remember that r.out.gdal is not just for GeoTIFF files..
best,
Hamish
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/73#comment:18>
GRASS GIS <http://grass.osgeo.org>
More information about the grass-dev
mailing list