[gdal-dev] gdal_merge: stacking and band statistics

[Jason Roberts]

Peter,

Thanks for your comments. I did not mean to suggest that GDAL produced
different pixel values or statistical values than ArcGIS. In my experience,
they are the same (barring this GDAL bug that I found
http://trac.osgeo.org/gdal/ticket/3151 a while back, which has since been
fixed).

>     When any image is opened from disk in ArcGIS the default display is a 
> black-to-white stretch: it is necessary to go to the Symbology menu and
select a 
> more appropriate scheme.  However, the internal Statistics tool in ArcMap
sets a 
> categorical contrast colour scheme by default.  I think this is the
difference 
> you are observing.

I am not sure if the mechanism that you described here (the internal
Statistics tool in ArcMap sets a categorical contrast colour scheme by
default) is how it works, but the end effect you describe is what I'm
seeing. When you execute the "Calculate Statistics" tool in ArcGIS (in
ArcMap or ArcCatalog), it writes something to integer raster files that
cause them to be displayed with a random colorful categorical scheme by
default, rather than the black-and-white stretch. This seems to be an
additional side effect besides computing the statistical values. (Note that
this is not a color map; the colors are selected randomly every time the
file is loaded by the GUI).

GDAL seems to compute the statistical values just fine but does not perform
that side effect. I like that side effect because it makes it much easier to
distinguish arbitrary categorical values than a black-to-white stretch does.
But I have not figured out how to reproduce that side effect with GDAL. It
may be something that is ESRI-specific.

In the comments I extracted from my code (in my original reply to Matt), I
noted that the problem was "I do not know how to get GDAL to build an
ArcGIS-compatible histogram". That makes it sound like I had narrowed down
the side effect somewhat. Perhaps I did, but I do not have good notes from
those experiments. It sounds like I noticed that calling
GDALRasterBand::SetStatistics() was not enough to produce the side effect
but that the ArcGIS Calculate Statistics tool also happened to compute a
histogram that could be retrieved by GDALRasterBand::GetDefaultHistogram().
I probably concluded that the histogram was what enabled the side effect,
but then found that calling GDALRasterBand::SetDefaultHistogram() did not
produce the side effect either, and therefore that "GDAL does not build an
ArcGIS-compatible histogram".

Anyway, if you or anyone else can demonstrate how to produce an integer
raster with GDAL that displays by default in ArcGIS with the random colorful
categorical scheme, I would greatly appreciate that knowledge!

Best,
Jason


-----Original Message-----
From: Peter J Halls [mailto:P.Halls at york.ac.uk] 
Sent: Thursday, October 21, 2010 2:12 AM
To: Jason Roberts
Cc: 'Matt Hanneman'; gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] gdal_merge: stacking and band statistics

Jason,

    as a heavy ArcGIS user and teacher, as well as GDAL, I think you may
have 
missed something about ArcGIS.  I have not tested the *outputs* of
statistics 
calculations between GDAL and ArcGIS, however what you describe seems to be
a 
display default.

    When any image is opened from disk in ArcGIS the default display is a 
black-to-white stretch: it is necessary to go to the Symbology menu and
select a 
more appropriate scheme.  However, the internal Statistics tool in ArcMap
sets a 
categorical contrast colour scheme by default.  I think this is the
difference 
you are observing.

    Whether the two results are equivalent or different cannot really be 
determined by display in this way, rather use the ArcGIS Raster Calculator
to 
subtract one image from the other and then examine the residual results: if
the 
images are equivalent the output cells should have the same value - although

this value may not be zero.

Best wishes,

Peter

Jason Roberts wrote:
> Matt,
> 
>  
> 
> Although I have not worked with multi-band HFA format, I have experimented
> fairly extensively with single-band, GDAL-produced, HFA-format files in
> ArcGIS 9.3.1 SP1. Maybe my experience will be interesting to you anyway.
> 
>  
> 
> My experience was that calculating the statistics with GDAL and with
ArcGIS
> was not equivalent. For example, when I calculated the statistics with
GDAL
> for integer rasters, they did not show up with a random color
classification
> in ArcGIS by default. They appeared to use a black-to-white stretch
instead.
> This was not desirable for categorical data (e.g. landcover values). But
if
> I calculated the statistics using ArcGIS, they did show up with random
> colors.
> 
>  
> 
> Unfortunately I did not take extensive notes on this, summarizing my
> experience in my own code with this comment:
> 
>  
> 
>         # Do not use GDAL to calculate statistics.
> 
>         # Even though GDAL can do it faster than the geoprocessor, we
> 
>         # prefer to use the geoprocessor to maximize the compatibility
> 
>         # of the output raster with ArcGIS. For example, I do not know
> 
>         # how to get GDAL to build an ArcGIS-compatible histogram.
> 
>         # Without the histogram ArcGIS defaults to displaying integer
> 
>         # rasters using a black-to-white stretch rather than a unique
> 
>         # value classifier with random colors. ArcGIS users will
> 
>         # expect colors; to get the necessary histogram, we have to
> 
>         # calculate statistics with the geoprocessor.
> 
>  
> 
> I also recall there being some other differences, possibly with floating
> point files, too. I'm sorry I did not take better notes.
> 
>  
> 
> I found the whole thing a bit bizarre because I thought that ArcGIS 9.3.1
> used GDAL internally to manipulate HFA files. But maybe not in all
> circumstances. And 9.3.1 has an ESRI-customized version of GDAL 1.4, so it
> is not like they are running the latest GDAL code.
> 
>  
> 
> Anyway, hope this helps,
> 
>  
> 
> Jason
> 
>  
> 
> From: gdal-dev-bounces at lists.osgeo.org
> [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Matt Hanneman
> Sent: Wednesday, October 20, 2010 4:51 PM
> To: gdal-dev at lists.osgeo.org
> Subject: [gdal-dev] gdal_merge: stacking and band statistics
> 
>  
> 
> Hi,
> 
> Can anyone help a newbie? I have been trying to calculate statistics with
> gdal_merge when merging/stacking several tiffs together into a HFA format
> (see below for syntax). However, when loaded into ArcGIS it reports that
> there are no statistics calculated for any of the bands. Will the
> 'STATISTICS' option not work when stacking multiple bands? As you can
> probably tell I am fairly new to this, so forgive my ignorance.
> 
> gdal_merge -co STATISTICS=YES -v -o "E:/Temp/A2E.img" -of HFA -separate
> "A.TIF" "B.TIF" "C.TIF" "D.TIF" "E.TIF" "F.TIF"
> 
> Thanks,
> 
> Matt 
> 
>  
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
----------------------------------------------------------------------------
----
Peter J Halls, GIS Advisor, University of York
Telephone: 01904 323806     Fax: 01904 323740
Snail mail: IT Services, University of York, Heslington, York YO10 5DD
This message has the status of a private and personal communication
----------------------------------------------------------------------------
----