[postgis-tickets] [PostGIS] #2359: raster2pgsql - large collection of examples of different bugs in -l (overviews)

PostGIS trac at osgeo.org
Thu Jun 13 02:14:12 PDT 2013 

#2359: raster2pgsql - large collection of examples of different bugs in -l
(overviews)
------------------------------------+---------------------------------------
 Reporter:  gbell                   |       Owner:  dustymugs    
     Type:  defect                  |      Status:  new          
 Priority:  high                    |   Milestone:  PostGIS 2.1.0
Component:  raster                  |     Version:  2.0.x        
 Keywords:  raster2pgsql overviews  |  
------------------------------------+---------------------------------------
 The big picture here is that the behaviour of the current stable version
 of raster2pgsql is unpredictable and highly prone to error (both in
 raster2pgsql and in the resulting SQL) when overview layers are specified.

 Since the raster2pgsql overview factor is specified as the log_2() of the
 scale rather than as the scale, it is very likely that users will
 encounter these bugs.

 The general theme of these errors are:

 a)  when an overview is created at too high a level, rather than generate
 a small picture, or refuse to generate the overview, raster2pgsql either
 segfaults in various ways, or its emitted SQL attempts to generate the
 overview and fails in a variety of ways, which share a theme of being
 untidy crashes.


 b) When someone tries to add a raster as a table that already exists,
 unhelpful and messy errors are caused by the generated SQL rather than a
 clean failure explaining the problem.


 >raster2pgsql
 RELEASE: 2.0.3 GDAL_VERSION=110 (r11128)


 Using some examples taken from the recommended samples at
 http://trac.osgeo.org/geotiff/:


 ftp://ftp.remotesensing.org/pub/geotiff/samples/made_up/bogota.tif


 ftp://ftp.remotesensing.org/pub/geotiff/samples/made_up/ilatlon_float.tif


 0a. Base case: both import correctly without overviews, with or without
 -Y.


 >raster2pgsql -t 100x100 -I -C -M bogota.tif  temp.bogota | psql

 OK.

 (drop table)


 >raster2pgsql -t 100x100 -I -C -M -Y bogota.tif  temp.bogota

 OK.

 (drop table)

 >raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif  temp.bogota | psql

 OK.

 (drop table)

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif  temp.bogota | psql

 OK.

 (drop table)


 0c. Trying to add without dropping the table. Messy failure.

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif  temp.bogota | psql

 OK.

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif  temp.bogota | psql

 ...

 ERROR:  current transaction is aborted, commands ignored until end of
 transaction block

 invalid command \.

 ERROR:  syntax error at or near "0100000100622255302"

 LINE 1: 0100000100622255302AA9333F622255302AA933BFA311BC0512A85EC000...
         ^
 ERROR:  current transaction is aborted, commands ignored until end of
 transaction block
 ...


 1. Import with overview 2 and "-Y"

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -l 2 -Y temp.bogota | psql

 OUTPUT

 Processing 1/1: bogota.tif
 ...
 CREATE TABLE
 ERROR:  rt_band_from_wkb: Premature end of WKB on band data reading
 (rt_api.c:6611)
 CONTEXT:  COPY o_2_bogota, line 9, column rast:
 "01000001000000000000005E400000000000005EC000000000405D1C4100000000008EF24000000000000000000000000000..."


 2. Import with overview 2 and without "-Y"

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -l 2 temp.bogota | psql

 OUTPUT

 ...
 LINE 1: INSERT INTO "temp"."o_2_bogota" ("rast") VALUES ('0100000100...

 (no tables added)


 3. Import with overview 16 and with "-Y"

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -l 16 -Y temp.bogota | psql

 OUTPUT

 ....

 ERROR:  unterminated quoted string at or near "'"
 LINE 1: SELECT AddOverviewConstraints('temp','o_16_bogota','


 4. Import with overviews 2 and 16 and with -Y

 >raster2pgsql -t 100x100 -I -C -M bogota.tif -l 2,16 -Y temp.bogota | psql

 OUTPUT

 ...
 CREATE TABLE
 ERROR:  rt_raster_from_hexwkb: Raster HEXWKB input must have an even
 number of characters
 CONTEXT:  COPY o_16_bogota, line 1, column rast:
 "01000001000000000000008E400000000000008EC00000000040E61A4100000000006AF84000000000000000000000000000..."


 5. Import with overviews 2,16,32 and with or without -Y

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -Y -l 2,16,1000 temp.bogota
 | psql

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -l 2,16,1000 temp.bogota |
 psql


 OUTPUT:

 ...
 CREATE TABLE
 ERROR:  rt_band_from_wkb: Premature end of WKB on band data reading
 (rt_api.c:6611)
 CONTEXT:  COPY o_32_bogota, line 1, column rast:
 "01000001000000000000009E400000000000009EC00000000040E61A4100000000006AF84000000000000000000000000000..."


 6.  Import with overviews 2,16 and without -Y

 > raster2pgsql -t 100x100 -I -C -M bogota.tif -Y -l 2,16 temp.bogota

 OUTPUT:

 ...
 ERROR:  rt_raster_from_hexwkb: Raster HEXWKB input must have an even
 number of characters
 CONTEXT:  COPY o_16_bogota, line 1, column rast:
 "01000001000000000000008E400000000000008EC00000000040E61A4100000000006AF84000000000000000000000000000..."


 7. Add a tiff, then try again. Bad things happen.

 > raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 4 temp.bogota |
 psql sl

 (works)

 > raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 4 temp.bogota |
 psql sl

 (fails horribly)

 ERROR:  relation "bogota" already exists
 ERROR:  current transaction is aborted, commands ignored until end of
 transaction block
 ERROR:  current transaction is aborted, commands ignored until end of
 transaction block
 invalid command \.

 ERROR:  syntax error at or near "0100000100622255302"
 LINE 1: 0100000100622255302AA9333F622255302AA933BFA311BC0512A85EC000...
         ^
 *** glibc detected *** raster2pgsql: corrupted double-linked list:
 0x0000000001e209a0 ***

 ======= Backtrace: =========
 /lib64/libc.so.6[0x3d15a7c1a2]
 /lib64/libc.so.6[0x3d15a7c988]
 /usr/local/lib/libgdal.so.1(+0x5837c3)[0x7fc5cdba57c3]
 /usr/local/lib/libgdal.so.1(+0x58383f)[0x7fc5cdba583f]
 /usr/local/lib/libgdal.so.1(+0x3485fb)[0x7fc5cd96a5fb]
 /usr/local/lib/libgdal.so.1(_ZN10GDALDriverD1Ev+0x2a)[0x7fc5cdb7264a]
 /usr/local/lib/libgdal.so.1(_ZN10GDALDriverD0Ev+0x9)[0x7fc5cdb72689]
 /usr/local/lib/libgdal.so.1(_ZN17GDALDriverManagerD1Ev+0xd2)[0x7fc5cdb74b62]
 /usr/local/lib/libgdal.so.1(_ZN17GDALDriverManagerD0Ev+0x9)[0x7fc5cdb74c89]
 /usr/local/lib/libgdal.so.1(+0x2a28e2)[0x7fc5cd8c48e2]
 /lib64/ld-linux-x86-64.so.2[0x3d1520f5f7]
 ...


 8. Variation in overview scale. A works, B doesn't. (overview 2 vs.
 overview 100)

 A

 > raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 1000
 temp.bogota | psql sl

 B

 > raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 1000
 temp.bogota | psql sl

 OUTPUT:

 A

 ...
 COMMIT
 VACUUM
 VACUUM

 B

 ...
 ERROR:  rt_raster_from_hexwkb: Raster HEXWKB input must have an even
 number of characters
 CONTEXT:  COPY o_100_bogota, line 1, column rast:
 "0100000100B90585EB51B89E3FB90585EB51B89EBFA311BC0512A85EC0000000000000434000000000000000000000000000..."


 9. A closer look at the STDOUT output of:

 >raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 100 temp.bogota

 OUTPUT:

 ...
 VACUUM ANALYZE "temp"."bogota";
 VACUUM ANALYZE "temp"."o_100_bogota";
 Segmentation fault


 10. Various errors depending on how many times the same overview is
 requested:

 (asking for -2 once)

 >raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 2 temp.bogota

 *** glibc detected *** raster2pgsql: corrupted double-linked list:
 0x00000000025719a0 ***
 ======= Backtrace: =========
 /lib64/libc.so.6[0x3d15a7c1a2]
 /lib64/libc.so.6[0x3d15a7c988]
 /usr/local/lib/libgdal.so.1(+0x5837c3)[0x7fdfbc89b7c3]
 /usr/local/lib/libgdal.so.1(+0x58383f)[0x7fdfbc89b83f]
 /usr/local/lib/libgdal.so.1(+0x3485fb)[0x7fdfbc6605fb]
 /usr/local/lib/libgdal.so.1(_ZN10GDALDriverD1Ev+0x2a)[0x7fdfbc86864a]
 /usr/local/lib/libgdal.so.1(_ZN10GDALDriverD0Ev+0x9)[0x7fdfbc868689]
 /usr/local/lib/libgdal.so.1(_ZN17GDALDriverManagerD1Ev+0xd2)[0x7fdfbc86ab62]
 /usr/local/lib/libgdal.so.1(_ZN17GDALDriverManagerD0Ev+0x9)[0x7fdfbc86ac89]
 /usr/local/lib/libgdal.so.1(+0x2a28e2)[0x7fdfbc5ba8e2]
 /lib64/ld-linux-x86-64.so.2[0x3d1520f5f7]


 11. add another ,2

 >raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -Y -l 2,2 temp.bogota

 OUTPUT:

 ...
 SELECT
 AddRasterConstraints('temp','bogota','rast',TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,FALSE,TRUE,TRUE,TRUE,TRUE,TRUE);
 SELECT
 AddRasterConstraints('temp','o_2_bogota','rast',TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,FALSE,TRUE,TRUE,TRUE,TRUE,TRUE);
 SELECT
 AddRasterConstraints('temp','o_2_bogota','rast',TRUE,TRUE,TRUE,TRUE,TRUE,TRUE,FALSE,TRUE,TRUE,TRUE,TRUE,TRUE);
 Segmentation fault


 12. and without -y

 >raster2pgsql -t 100x100 -I -C -M ilatlon_float.tif -l 2,2 temp.bogota

 OUTPUT:

 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'::raster);
 Segmentation fault
 >


 13. Trying to add a non-integer overview level. This is not detected.

 OUTPUT:

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif -l 2.5 temp.bogota

 SELECT
 AddOverviewConstraints('temp','o_2_bogota','rast','temp','bogota','rast',2);
 END;
 VACUUM ANALYZE "temp"."bogota";
 VACUUM ANALYZE "temp"."o_2_bogota";
 Segmentation fault


 14. Forgetting to specify the overview factor produces a misleading error
 message

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif -l temp.bogota

 OUTPUT:

 ERROR: Overview factor 0 is not between 2 and 1000

 (should be: Overview factor not specified)


 15. Large numbers uncover another bug. Maybe a BASH bug.

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif -l 100000000000
 temp.bogota

 OUTPUT:

 ERROR: Overview factor 1215752192 is not between 2 and 1000


 16. Specifying -l multiple times causes segfault (without and without
 duplication of overview factor)

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif -l 2 -l 4
 temp.bogota

 >raster2pgsql -t 100x100 -I -C -M -Y ilatlon_float.tif -l 2 -l 2
 temp.bogota

 OUTPUT:

 VACUUM ANALYZE "temp"."bogota";
 VACUUM ANALYZE "temp"."o_2_bogota";
 Segmentation fault

-- 
Ticket URL: <http://trac.osgeo.org/postgis/ticket/2359>
PostGIS <http://trac.osgeo.org/postgis/>
The PostGIS Trac is used for bug, enhancement & task tracking, a user and developer wiki, and a view into the subversion code repository of PostGIS project.

More information about the postgis-tickets mailing list