[postgis-devel] [WKT Raster] Random bands truncation

Wed Apr 8 09:54:44 PDT 2009

Folks,

I have submitted fix for this bug (r3987).
It seems to work, but it's possible there are more similar
places around that will leak at some point.

Generally, it's not safe to assume this:

assert(0 == (((uint64_t)ptr) % 8) );

Best regards,

> -----Original Message-----
> From: postgis-devel-bounces at postgis.refractions.net [mailto:postgis-
> devel-bounces at postgis.refractions.net] On Behalf Of Mateusz Loskot
> Sent: Wednesday, April 01, 2009 2:54 PM
> To: PostGIS Development Discussion
> Subject: [postgis-devel] [WKT Raster] Random bands truncation
> 
> Hi,
> 
> Yesterday, when I was playing with loading hundreds and thousands big,
> small and very tiles and a mysterious bug has revealed. I've made
> detailed tests which seem to confirm there must be a bug in rt_core or
> rt_pg modules.
> 
> Below, I explain how to reproduce the problem:
> 
> 0. 32-bit Linux (Ubuntu 8.10)
> 
> 1. Simple table
> 
> CREATE TABLE car5
> (
>   rid serial NOT NULL,
>   rast raster
> );
> 
> 2. Simple RGB raster: 5 x 5 with 3 bands, each of pixel type of PT_8BUI
> (0x04) and NODATA = 0x00
> 
> 3. Insert the same raster 3 times and check pixel types of all bands
> after every insert
> 
> rtest=# INSERT INTO car5 ( rast ) VALUES (
> ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F0761
> 9564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9
> F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D5
> 2594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C
> 3E')::raster ); INSERT 0 1
> 
> rtest=# select rid, rt_width(rast), rt_height(rast),
> rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2),
> rt_bandpixeltype(rast,3) from car5;
> 
>  rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype |
> rt_bandpixeltype
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+---
>   1 |        5 |         5 | 8BUI             | 1BB              | 1BB
> (1 row)
> 
> rtest=# INSERT INTO car5 ( rast ) VALUES (
> ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F0761
> 9564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9
> F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D5
> 2594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C
> 3E')::raster ); INSERT 0 1
> 
> rtest=# select rid, rt_width(rast), rt_height(rast),
> rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2),
> rt_bandpixeltype(rast,3) from car5;
> 
> rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype |
> rt_bandpixeltype
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+---
>   1 |        5 |         5 | 8BUI             | 1BB              | 1BB
>   2 |        5 |         5 | 8BUI             | 8BUI             | 8BUI
> (2 rows)
> 
> rtest=# INSERT INTO car5 ( rast ) VALUES (
> ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F0761
> 9564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9
> F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D5
> 2594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C
> 3E')::raster ); INSERT 0 1
> 
> rtest=# select rid, rt_width(rast), rt_height(rast),
> rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2),
> rt_bandpixeltype(rast,3) from car5;
> 
> rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype |
> rt_bandpixeltype
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+----
> -
> -----+----------+-----------+------------------+------------------+---
>   1 |        5 |         5 | 8BUI             | 1BB              | 1BB
>   2 |        5 |         5 | 8BUI             | 8BUI             | 8BUI
>   3 |        5 |         5 | 8BUI             | 1BB              | 1BB
> (3 rows)
> 
> rtest=#
> 
> 
> 4. The result is obviously and mysteriously wrong. Rasters with rid 1
> and 3 have invalid pixel types of 2nd and 3rd bands.
> If I issue the inserts more times the pattern repeats.
> 
> 5. Rasters WKB examination shows what's wrong with the data - bands
> with incorrectly reported pixel types above are truncated.
> However, the total length of raster data is correct.
> 
> The format of raster hex dump below is:
> 1st row - raster header
> 2nd - first band
> 3rd - second band
> 4th - third band
> 
> Hopefully mailmain nor my mail client won't mess up th formatting, so
> places of bands truncation is well visible.
> 
> rtest=# select rid, rast from car5;
> 
>  rid | rast
> -----+--------------------------------------------
> 1 |
> 01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F076195
> 64100000000000000000000000000000000FFFFFFFF05000500
>    0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB00000000
>    04004D4A4B505946444850514B4754524D52594F4A444E00000000
>    04003F363B41403536383A3F36333E413F4546403E3746
> 
> 2 |
> 01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F076195
> 64100000000000000000000000000000000FFFFFFFF05000500
>    0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB
>    04004D4A4B505946444850514B4754524D52594F4A444E504D514D
>    04003F363B41403536383A3F36333E413F4546403E3746453D3C3E
> 
> 3 |
> 01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F076195
> 64100000000000000000000000000000000FFFFFFFF05000500
>    0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB00000000
>    04004D4A4B505946444850514B4754524D52594F4A444E00000000
>    04003F363B41403536383A3F36333E413F4546403E3746
> (3 rows)
> rtest=#
> 
> 
> In order to simplify testing this problem, I have added dedicated test
> case for test/regress [1] package:
> # bug_test_car5.sql
> # bug_test_car5_expected
> 
> [1] http://svn.osgeo.org/postgis//spike/wktraster/test/regress/
> 
> 
> I've run this test case on both architectures, 32-bit and 64-bit,
> on Linux. The bug leaks only on 32-bit. On 64-bit everything works
> well.
> 
> What's more funky, is that the problem is quite random and leaks
> depending on size of tiles.
> For instance, I cut GDAL's small_world.tif [2] to tiles of
> 20x20 pixels and 10x10 pixels.
> The first set of tiles works well.
> The second one does not even load into table:
> ...
> INSERT 0 1
> INSERT 0 1
> psql:sw10.sql:804: ERROR:  Unknown pixeltype 12
> ROLLBACK
> 
> This is also related to the problem I'm explaining, pixel type
> is broken because of bands truncation.
> The raster used in car5 test is just "lucky" so pixel type
> is determined as Zero (1BB = code 0).
> The bands of small_world are being truncated, so the
> pixel type is randomly read as value 12 (likely
> a part of raster data value, of a some pixel).
> 
> [2] http://download.osgeo.org/gdal/data/gtiff/
> 
> 
> This issue needs more testing, so I'd appreciate if more people
> could try to run WKT Raster regression tests and report here:
> 1. CPU (32 or 64 bit)
> 2. Operating System (32 or 64 bit)
> 3. Result of test.
> 
> How to run the test (Unix-like only):
> 
> $ Build PostGIS from SVN trunk
> $ svn co http://svn.osgeo.org/postgis//spike/wktraster/ wktraster
> $ cd wktraster
> $ ./autogen.sh
> $ ./configure --with-postgis-source=/path/to/postgis/trunk
> $ make
> $ make check
> $ sudo make install
> $ make post-install-check
> 
> Check the tests report if bug_test_car5 failed or not.
> 
> Best regards,
> --
> Mateusz Loskot
> Senior Programmer, Cadcorp
> http://www.cadcorp.com