[postgis-devel] [WKT Raster] Random bands truncation

[Mateusz Loskot]

Hi,

Yesterday, when I was playing with loading hundreds and
thousands big, small and very tiles and a mysterious bug
has revealed. I've made detailed tests which seem to confirm
there must be a bug in rt_core or rt_pg modules.

Below, I explain how to reproduce the problem:

0. 32-bit Linux (Ubuntu 8.10)

1. Simple table

CREATE TABLE car5
(
  rid serial NOT NULL,
  rast raster
);

2. Simple RGB raster: 5 x 5 with 3 bands,
each of pixel type of PT_8BUI (0x04) and NODATA = 0x00

3. Insert the same raster 3 times and check pixel types
of all bands after every insert

rtest=# INSERT INTO car5 ( rast ) VALUES ( ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D52594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C3E')::raster );
INSERT 0 1

rtest=# select rid, rt_width(rast), rt_height(rast), rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2), rt_bandpixeltype(rast,3) from car5;

 rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype | rt_bandpixeltype 
-----+----------+-----------+------------------+------------------+------------------
  1 |        5 |         5 | 8BUI             | 1BB              | 1BB
(1 row)
 
rtest=# INSERT INTO car5 ( rast ) VALUES ( ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D52594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C3E')::raster );
INSERT 0 1

rtest=# select rid, rt_width(rast), rt_height(rast), rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2), rt_bandpixeltype(rast,3) from car5;

rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype | rt_bandpixeltype 
-----+----------+-----------+------------------+------------------+------------------
  1 |        5 |         5 | 8BUI             | 1BB              | 1BB
  2 |        5 |         5 | 8BUI             | 8BUI             | 8BUI
(2 rows)
 
rtest=# INSERT INTO car5 ( rast ) VALUES ( ('01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF050005000400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB04004D4A4B505946444850514B4754524D52594F4A444E504D514D04003F363B41403536383A3F36333E413F4546403E3746453D3C3E')::raster );
INSERT 0 1

rtest=# select rid, rt_width(rast), rt_height(rast), rt_bandpixeltype(rast,1), rt_bandpixeltype(rast,2), rt_bandpixeltype(rast,3) from car5;

rid | rt_width | rt_height | rt_bandpixeltype | rt_bandpixeltype | rt_bandpixeltype 
-----+----------+-----------+------------------+------------------+------------------
  1 |        5 |         5 | 8BUI             | 1BB              | 1BB
  2 |        5 |         5 | 8BUI             | 8BUI             | 8BUI
  3 |        5 |         5 | 8BUI             | 1BB              | 1BB
(3 rows)
 
rtest=#


4. The result is obviously and mysteriously wrong. Rasters with
rid 1 and 3 have invalid pixel types of 2nd and 3rd bands.
If I issue the inserts more times the pattern repeats.

5. Rasters WKB examination shows what's wrong with the data - bands
with incorrectly reported pixel types above are truncated.
However, the total length of raster data is correct.

The format of raster hex dump below is:
1st row - raster header 
2nd - first band
3rd - second band
4th - third band

Hopefully mailmain nor my mail client won't mess up th formatting,
so places of bands truncation is well visible.

rtest=# select rid, rast from car5;

 rid | rast                                                                                                                                             
-----+--------------------------------------------
1 | 
01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF05000500
   0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB00000000
   04004D4A4B505946444850514B4754524D52594F4A444E00000000
   04003F363B41403536383A3F36333E413F4546403E3746
 
2 | 
01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF05000500
   0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB
   04004D4A4B505946444850514B4754524D52594F4A444E504D514D
   04003F363B41403536383A3F36333E413F4546403E3746453D3C3E
 
3 | 
01000003009A9999999999A93F9A9999999999A9BF000000E02B274A41000000F07619564100000000000000000000000000000000FFFFFFFF05000500
   0400E5E7F5FAF3F9F9F8FBFDF3EEF5FBFEF9FCFBFBFEFCFBFCFAFB00000000
   04004D4A4B505946444850514B4754524D52594F4A444E00000000
   04003F363B41403536383A3F36333E413F4546403E3746
(3 rows)
rtest=#


In order to simplify testing this problem, I have added dedicated test
case for test/regress [1] package:
# bug_test_car5.sql
# bug_test_car5_expected

[1] http://svn.osgeo.org/postgis//spike/wktraster/test/regress/


I've run this test case on both architectures, 32-bit and 64-bit,
on Linux. The bug leaks only on 32-bit. On 64-bit everything works well.

What's more funky, is that the problem is quite random and leaks
depending on size of tiles.
For instance, I cut GDAL's small_world.tif [2] to tiles of
20x20 pixels and 10x10 pixels.
The first set of tiles works well.
The second one does not even load into table:
...
INSERT 0 1
INSERT 0 1
psql:sw10.sql:804: ERROR:  Unknown pixeltype 12
ROLLBACK

This is also related to the problem I'm explaining, pixel type
is broken because of bands truncation.
The raster used in car5 test is just "lucky" so pixel type
is determined as Zero (1BB = code 0).
The bands of small_world are being truncated, so the
pixel type is randomly read as value 12 (likely
a part of raster data value, of a some pixel).

[2] http://download.osgeo.org/gdal/data/gtiff/


This issue needs more testing, so I'd appreciate if more people
could try to run WKT Raster regression tests and report here:
1. CPU (32 or 64 bit)
2. Operating System (32 or 64 bit)
3. Result of test.

How to run the test (Unix-like only):

$ Build PostGIS from SVN trunk
$ svn co http://svn.osgeo.org/postgis//spike/wktraster/ wktraster
$ cd wktraster
$ ./autogen.sh
$ ./configure --with-postgis-source=/path/to/postgis/trunk
$ make
$ make check
$ sudo make install
$ make post-install-check

Check the tests report if bug_test_car5 failed or not.

Best regards,
--
Mateusz Loskot
Senior Programmer, Cadcorp
http://www.cadcorp.com