[gdal-dev] how can a Geotiff occupy three times the disk space of an ASCII Grid?

Norman Vine nhv at cape.com
Sat Jul 5 14:42:10 PDT 2014


My mistake

please disregard

Norman

On Jul 5, 2014, at 3:49 PM, Norman Vine <nhv at cape.com> wrote:

> This question is better addressed to the QGis-dev list
> as QGis is promoting the Float32 type to be Float64 type
> 
> https://github.com/qgis/QGIS/blob/master/src/providers/gdal/qgsgdalprovider.cpp#L1103
> 
> 
>  Jul 5, 2014, at 3:43 PM, "G. Allegri" <giohappy at gmail.com> wrote:
> 
>> ldd tells me that both QGIS and gdalinfo point to the same gdal libs
>> 
>> ldd /usr/bin/qgis.bin | grep gdal
>> libgdal.so.1 => /usr/lib/libgdal.so.1 (0x0284b000)
>> 
>> ldd /usr/bin/gdalinfo | grep gdal
>> libgdal.so.1 => /usr/lib/libgdal.so.1 (0x00c5f000)
>> 
>> But I must correct about Python. Running it against the right modules it reports raster band data type as GDT_Float32.
>> 
>> In QGIS Float64 comes out here: https://github.com/qgis/QGIS/blob/master/src/providers/gdal/qgsgdalprovider.cpp#L1076
>> 
>> giovanni
>> 
>> 
>> 
>> 
>> 2014-07-05 21:02 GMT+02:00 Even Rouault <even.rouault at mines-paris.org>:
>> Le samedi 05 juillet 2014 20:35:45, G. Allegri a écrit :
>> > Given that an ASCII Grid doesn't provide a data type, maybe the driver
>> > choose the larger one, while the utilities guess it trying to fit the
>> > values into a smaller type?
>> 
>> No, GDAL does not do that sort of magic. Are you sure the GDAL version used by
>> Python is the same as by the utilities ?
>> 
>> >
>> > giovanni
>> >
>> > 2014-07-05 20:31 GMT+02:00 G. Allegri <giohappy at gmail.com>:
>> > > I've debugged it, and it gets Float64 directly from
>> > > GDALGetRasterDataType( GDALRasterBandH ).
>> > > I've also tried opening it with Python GDAL and I get Float64.
>> > > gdalinfo tells me Float32.
>> > > I will try to send you a subset of it. It's not easy, as it is a heavy
>> > > ASCII Grid and I don't want to alter it.
>> > >
>> > > giovanni
>> > >
>> > >
>> > > 2014-07-05 20:22 GMT+02:00 Even Rouault <even.rouault at mines-paris.org>:
>> > >
>> > > Le samedi 05 juillet 2014 20:18:50, G. Allegri a écrit :
>> > >> > Finally I've reached the point: GDAL's driver is opening the ASCII as
>> > >> > Float64, consequently QGIS treats it this way.
>> > >> > I wonder why gdalinfo and gdal_translate treat it as Float32
>> > >> > instead....
>> > >>
>> > >> If gdalinfo opens it as Float32, then it is Float32 for everybody. I
>> > >> guess QGIS probably promotes Float32 to Float64.
>> > >>
>> > >> > giovanni
>> > >> >
>> > >> > 2014-07-05 18:53 GMT+02:00 G. Allegri <giohappy at gmail.com>:
>> > >> > > The problem was simpler then it appeared: I didn't realize that QGIS
>> > >> > > output is Float64.
>> > >> > > Yet I don't know why QGIS chosed to use this data type...
>> > >> > >
>> > >> > > giovanni
>> > >> > >
>> > >> > >
>> > >> > > 2014-07-05 17:56 GMT+02:00 Even Rouault
>> > >> > > <even.rouault at mines-paris.org
>> > >> > >
>> > >> > > Le samedi 05 juillet 2014 17:25:48, G. Allegri a écrit :
>> > >> > >> > > QGIS is usually just calls to gdal, which makes this even more
>> > >> > >> > > mysterious.
>> > >> > >> >
>> > >> > >> > Yes, in the end it uses gdal, but it chooses the way blocks are
>> > >>
>> > >> read
>> > >>
>> > >> > >> > and written.
>> > >> > >> > If I remember correctly geotiff final sizes may depend on block
>> > >> > >> > ordering and memory alignment.
>> > >> > >>
>> > >> > >> In that instance, the target geotiff has a natural block dimension
>> > >>
>> > >> which
>> > >>
>> > >> > >> is a
>> > >> > >> raster line.
>> > >> > >> If QGIS writes the geotiff by 256x256 blocks (this is just a guess.
>> > >>
>> > >> I've
>> > >>
>> > >> > >> not
>> > >> > >> verified), the same raster line will be written several times. As
>> > >> > >> it
>> > >>
>> > >> is
>> > >>
>> > >> > >> a compressed geotiff, the resulting raster line will be each time
>> > >>
>> > >> being
>> > >>
>> > >> > >> bigger
>> > >> > >> since the initial zeros will be replaced by actual values. And if
>> > >> > >> the new size
>> > >> > >> of the line is bigger than its previous size, the new line will be
>> > >> > >> rewritten
>> > >> > >> at the end of the file, losing the space previously occupied.
>> > >> > >>
>> > >> > >> > Maybe this is the case, QGIS raster provider not doing the best
>> > >> > >> > at this level? Don't know, but this discussion is for the QGIS
>> > >> > >> > ml ;)
>> > >> > >> >
>> > >> > >> > giovanni
>> > >> > >> >
>> > >> > >> > > On 7/5/2014 9:09 AM, G. Allegri wrote:
>> > >> > >> > > > I agree with you David, I'm surprised too.
>> > >> > >> > > > Anyway, gdal_translate is run without compression options.
>> > >> > >> > > > I've written to the QGIS devs (it was the software) to verify
>> > >> > >> > > > what's happening with its raster file writer code...
>> > >> > >>
>> > >> > >> --
>> > >> > >> Geospatial professional services
>> > >> > >> http://even.rouault.free.fr/services.html
>> > >> > >
>> > >> > > --
>> > >> > > Giovanni Allegri
>> > >> > > http://about.me/giovanniallegri
>> > >> > > Twitter: https://twitter.com/_giohappy_
>> > >> > > blog: http://blog.spaziogis.it
>> > >> > > GEO+ geomatica in Italia http://bit.ly/GEOplus
>> > >>
>> > >> --
>> > >> Geospatial professional services
>> > >> http://even.rouault.free.fr/services.html
>> > >
>> > > --
>> > > Giovanni Allegri
>> > > http://about.me/giovanniallegri
>> > > Twitter: https://twitter.com/_giohappy_
>> > > blog: http://blog.spaziogis.it
>> > > GEO+ geomatica in Italia http://bit.ly/GEOplus
>> 
>> --
>> Geospatial professional services
>> http://even.rouault.free.fr/services.html
>> 
>> 
>> 
>> -- 
>> Giovanni Allegri
>> http://about.me/giovanniallegri
>> Twitter: https://twitter.com/_giohappy_
>> blog: http://blog.spaziogis.it
>> GEO+ geomatica in Italia http://bit.ly/GEOplus
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> 
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20140705/717ee5cc/attachment.html>


More information about the gdal-dev mailing list