[postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Pierre Racine
Pierre.Racine at sbf.ulaval.ca
Wed Jul 28 06:14:49 PDT 2010
A table of tiles form a large coverage as one. Every process might be done one tile at a time but SQL GROUP BY statements help regrouping things together if needed.
Pierre
From: Peter Baumann [mailto:p.baumann at jacobs-university.de]
Sent: 28 juillet 2010 04:27
To: Pierre Racine
Cc: PostGIS Users Discussion
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Pierre,
admittedly this was a little inexact, and I certainly will correct it. A more accurate phrasing I guess is "does not support retrieval on a raster array consisting of a set of tiles, rather tiles have to be treated individually."
>From what I understand WKT Raster can tile an incoming image, but cannot treat a set of tiles making up a larger coverage as one.
Would you agree to that?
-Peter
On 07/27/2010 05:52 PM, Pierre Racine wrote:
BTW Peter,
in your site (http://kahlua.eecs.jacobs-university.de/trac/rasdaman/wiki/WhyRasdaman) you say: "First and foremost, it (PostGIS WKT Raster) does not support tiling (partitioning) of raster arrays." This statement is false since WKT Raster actually does support tiling of large rasters. You can partition any raster coverage by importing it as separate tiles or by using the -k option of gdal2wktraster.py. Each tile become a separate row in the table and the whole table is the coverage. The table is spatially indexed using the PostgreSQL GiST index.
I think it would be just fair if you could rectify this statement.
Thanks,
Pierre
From: Peter Baumann [mailto:p.baumann at jacobs-university.de]
Sent: 21 juillet 2010 17:28
To: Pierre Racine
Cc: PostGIS Users Discussion
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Pierre-
rasdaman per se is domain agnostic - you can just as well maintain human brain images (as we did). The petascope layer, which is part of the distro, adds geo semantics by providing OGC interfaces.
PostgreSQL integration is not yet done, we are working on this; same for GDAL / MapServer.
-Peter
On 07/21/2010 08:29 PM, Pierre Racine wrote:
Peter,
Are rasters in rasdaman georeferenced? How do I convert my vectorial PostGIS points coordinates into rasdaman coordinates? Do I have to do all the math myself?
Pierre
From: Peter Baumann [mailto:p.baumann at jacobs-university.de]
Sent: 21 juillet 2010 04:41
To: Pierre Racine
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Pierre-
let's see. First, I would not store 100k raster points in the relational table as its terribly inefficient. We do have input data formats, though, which are similar to what a PG dump of such a table would deliver. We import this into an array and then can query it. Import via rasql or through an import tool which internally generates rasql calls.
Then, for corner points (x0,y0) and (x1,y1) we retrieve as
select png( M[x0:y1,y0:y1] ) from MyMap as M
...in PNG format via the rasql cmd line utility or the C++ or Java API.
Alternatively, sending this this query over the API gets you the data into main memory ready for nested for loops = in the main memory format of your client:
select M[x0:y1,y0:y1] from MyMap as M
-Peter
On 07/20/2010 10:03 PM, Pierre Racine wrote:
So if I already have a table of, say 100 000 points, uploaded in PostGIS, how do I use rasdaman to get the raster values of the pixels located under the points?
Pierre
From: Peter Baumann [mailto:p.baumann at jacobs-university.de]
Sent: 20 juillet 2010 12:08
To: PostGIS Users Discussion
Cc: Pierre Racine
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
On 07/20/2010 05:47 PM, Pierre Racine wrote:
Since we have this bug with big rasters,
proven, any-size, ... ;-)
Who is proven? Well, running a dozen-TB seamless mosaic on PostgreSQL for many years, having online-demos available since years, etc... you decide, in comparison.
-Peter
I would suggest you split your raster with something like GDAL gdal_retile.py and then import your tiles into a single WKT Raster table following the instruction provided in our tutorial. Each tile will be stored in a column cell of type RASTER similar to the PostGIS GEOMETRY type.
As I said previously you can then query the raster in SQL like this:
SELECT ST_Value(rast, ST_Geomfromtext('Point(-78.1 58.1)', 4326))
FROM srtm_tiled_100x100
WHERE ST_Intersects(rast::geometry, ST_Geomfromtext('Point(-78.1 58.1)', 4326)) AND whatever you want.
Basically WKT Raster is the first true SQL interface with which is is simple to do such things... It's proven, any-size, cloud-scalable, and open source :) Maybe not that prooven. But who is?
There is also rasdaman but I don't think you can use SQL. It would be nice if you could compare both... I can't find time for this. Jorge has started comparing PostGIS WKT Raster with Oracle Georaster though.
Pierre
From: postgis-users-bounces at postgis.refractions.net<mailto:postgis-users-bounces at postgis.refractions.net> [mailto:postgis-users-bounces at postgis.refractions.net] On Behalf Of Sebastian E. Ovide
Sent: 20 juillet 2010 10:46
To: PostGIS Users Discussion
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
yes... with png worked... but it was a different png (a smaller one)...
so I've converted the ESRI into a png and tried to import it... and it didn't work neither...
so this is the situation:
A have huge rusters (from 150kx150k).. In Oracle I would just load it (the huge raster) in a single row of a GeoRaster table and then Oracle GeoRaster would split it in small tiles and store one tile per line of another table (Raster Data Table)... then I can run a query similar to this one: SELECT getcellvalue(t.rastercolumn,x,y) from GeoRasterTable t where t.id<http://t.id>=1; and GeoRaster will query automatically the spatial indexes and the Raster Data Table and it will find the right tile etc...
I do not know how WKTRaster works.... If I cannot import a such big image, of course I can split it in smaller georeferenced tiles... (how?)... but.... My main question is: after that, how will the table look like ? how can I do the same query (where id=1 or where name="UK" etc...) ?
On Tue, Jul 20, 2010 at 2:31 PM, Pierre Racine <Pierre.Racine at sbf.ulaval.ca<mailto:Pierre.Racine at sbf.ulaval.ca>> wrote:
Wait. You first said the png was working. Now it's not? Did you try gdal_translate with the ESRI grid? For sure I haven't test yet with such big rasters. Is this the result of a merge or all your original raster are all this size? The point is that with WKT Raster you don't have to merge your raster first into a gigantic raster in order to get it store in a unique table like with Oracle Spatial.
Pierre
From: postgis-users-bounces at postgis.refractions.net<mailto:postgis-users-bounces at postgis.refractions.net> [mailto:postgis-users-bounces at postgis.refractions.net<mailto:postgis-users-bounces at postgis.refractions.net>] On Behalf Of Sebastian E. Ovide
Sent: 20 juillet 2010 05:51
To: PostGIS Users Discussion
Subject: Re: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Hi Pierre,
Does gdal2wktraster.py have any limitation on the maximum number of columnsxrows ?
in my case, my raster is 107759 x 168633...
gdal works well:
sebas at SeansPC:~/rasters$ gdal_translate -of PNG raster/ test.png
Input file size is 107759, 168633
0...10...20...30...40...50...60...70...80...90...100 - done.
sebas at SeansPC:~/rasters$ python gdal2wktraster.py -r raster/ -t sebastable -o ok.sql
gdal2wktraster.py:695: DeprecationWarning: 'H' format requires 0 <= number <= 65535
hexwkb += wkblify('H', xsize)
gdal2wktraster.py:696: DeprecationWarning: 'H' format requires 0 <= number <= 65535
hexwkb += wkblify('H', ysize)
gdal2wktraster.py:727: DeprecationWarning: integer argument expected, got float
hexwkb += wkblify(pt2fmt(pixtype), nodata)
Traceback (most recent call last):
File "gdal2wktraster.py", line 1013, in <module>
main()
File "gdal2wktraster.py", line 976, in main
wkblify_raster(opts, filename, i)
File "gdal2wktraster.py", line 921, in wkblify_raster
summary = wkblify_raster_level(options, ds, options.overview_level, band_range, infile, i)
File "gdal2wktraster.py", line 888, in wkblify_raster_level
hexwkb += wkblify_band(options, band, level, xoff, yoff, read_block_size, block_size, infile, b)
File "gdal2wktraster.py", line 777, in wkblify_band
target_block_size[0], target_block_size[1])
File "/usr/lib/python2.6/dist-packages/osgeo/gdal.py", line 895, in ReadAsArray
buf_xsize, buf_ysize, buf_obj )
File "/usr/lib/python2.6/dist-packages/osgeo/gdal_array.py", line 228, in BandReadAsArray
ar = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)
MemoryError
sebas at SeansPC:~/rasters$ python gdal2wktraster.py -r test.png -t sebastable -o ok.sql
gdal2wktraster.py:695: DeprecationWarning: 'H' format requires 0 <= number <= 65535
hexwkb += wkblify('H', xsize)
gdal2wktraster.py:696: DeprecationWarning: 'H' format requires 0 <= number <= 65535
hexwkb += wkblify('H', ysize)
gdal2wktraster.py:727: DeprecationWarning: integer argument expected, got float
hexwkb += wkblify(pt2fmt(pixtype), nodata)
Traceback (most recent call last):
File "gdal2wktraster.py", line 1013, in <module>
main()
File "gdal2wktraster.py", line 976, in main
wkblify_raster(opts, filename, i)
File "gdal2wktraster.py", line 921, in wkblify_raster
summary = wkblify_raster_level(options, ds, options.overview_level, band_range, infile, i)
File "gdal2wktraster.py", line 888, in wkblify_raster_level
hexwkb += wkblify_band(options, band, level, xoff, yoff, read_block_size, block_size, infile, b)
File "gdal2wktraster.py", line 777, in wkblify_band
target_block_size[0], target_block_size[1])
File "/usr/lib/python2.6/dist-packages/osgeo/gdal.py", line 895, in ReadAsArray
buf_xsize, buf_ysize, buf_obj )
File "/usr/lib/python2.6/dist-packages/osgeo/gdal_array.py", line 228, in BandReadAsArray
ar = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)
MemoryError
On Mon, Jul 19, 2010 at 5:56 PM, Pierre Racine <Pierre.Racine at sbf.ulaval.ca<mailto:Pierre.Racine at sbf.ulaval.ca>> wrote:
Hi Sebastian,
I can convert ESRI Grid file to .sql without problem using gdal2wktraster.py and the same parameters as you. I can do both integer and floating point rasters.
Maybe this is a GDAL problem. Try to convert it using gdal_translate (to tiff for example). This would be a better test than just gdalinfo.
Could you provide us with a file sample?
Pierre
From: postgis-users-bounces at postgis.refractions.net<mailto:postgis-users-bounces at postgis.refractions.net> [mailto:postgis-users-bounces at postgis.refractions.net<mailto:postgis-users-bounces at postgis.refractions.net>] On Behalf Of Sebastian E. Ovide
Sent: 19 juillet 2010 12:28
To: PostGIS Users Discussion
Subject: [postgis-users] WKTRaster : gdal2wktraster.py cannot read AIG/Arc/Info Binary Grid
Hi All,
trying to create a SQL with gdal2wktraster.py. It works on PNG but it doesn't on AIG files...
Note: As Gdal works fine.
C:\Program Files\PostgreSQL\8.4\bin>gdalinfo c:\tmp\raster
Driver: AIG/Arc/Info Binary Grid
Files: c:\tmp\raster
c:\tmp\raster\dblbnd.adf
c:\tmp\raster\hdr.adf
c:\tmp\raster\metadata.xml
c:\tmp\raster\prj.adf
c:\tmp\raster\sta.adf
c:\tmp\raster\vat.adf
c:\tmp\raster\w001000.adf
c:\tmp\raster\w001000x.adf
c:\tmp\raster\w001001.adf
c:\tmp\raster\w001001x.adf
c:\tmp\raster\z001001.adf
c:\tmp\raster\z001001x.adf
c:\tmp\raster\z001002.adf
c:\tmp\raster\z001002x.adf
c:\tmp\raster\z001003.adf
c:\tmp\raster\z001003x.adf
c:\tmp\raster\z001004.adf
c:\tmp\raster\z001004x.adf
c:\tmp\raster\z001005.adf
c:\tmp\raster\z001005x.adf
c:\tmp\raster\z001006.adf
c:\tmp\raster\z001006x.adf
c:\tmp\raster\z001007.adf
c:\tmp\raster\z001007x.adf
c:\tmp\raster\z001008.adf
c:\tmp\raster\z001008x.adf
c:\tmp\raster\z001009.adf
c:\tmp\raster\z001009x.adf
c:\tmp\raster\z001010.adf
c:\tmp\raster\z001010x.adf
c:\tmp\raster\z001011.adf
c:\tmp\raster\z001011x.adf
c:\tmp\raster\z001012.adf
c:\tmp\raster\z001012x.adf
c:\tmp\raster\z001013.adf
c:\tmp\raster\z001013x.adf
c:\tmp\raster\z001014.adf
c:\tmp\raster\z001014x.adf
c:\tmp\raster\z001015.adf
c:\tmp\raster\z001015x.adf
Size is 107759, 168633
Coordinate System is:
PROJCS["unnamed",
GEOGCS["Unknown datum based upon the Airy 1830 ellipsoid",
DATUM["Not_specified_based_on_Airy_1830_ellipsoid",
SPHEROID["Airy 1830",6377563.396,299.3249646,
AUTHORITY["EPSG","7001"]],
AUTHORITY["EPSG","6001"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.01745329251994328,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4001"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",49],
PARAMETER["central_meridian",-2],
PARAMETER["scale_factor",0.9996012717],
PARAMETER["false_easting",400000],
PARAMETER["false_northing",-100000],
UNIT["METERS",1]]
Origin = (128110.000000000000000,813270.000000000000000)
Pixel Size = (5.000000000000000,-5.000000000000000)
Corner Coordinates:
Upper Left ( 128110.000, 813270.000) ( 6d29'37.32"W, 57d 7'47.53"N)
Lower Left ( 128110.000, -29895.000) ( 5d45'40.00"W, 49d34'10.24"N)
Upper Right ( 666905.000, 813270.000) ( 2d24'41.72"E, 57d 7'58.04"N)
Lower Right ( 666905.000, -29895.000) ( 1d41'32.29"E, 49d34'18.23"N)
Center ( 397507.500, 391687.500) ( 2d 2'15.04"W, 53d25'18.19"N)
Band 1 Block=256x4 Type=Byte, ColorInterp=Undefined
Min=1.000 Max=4.000
NoData Value=255
C:\Program Files\PostgreSQL\8.4\bin>python gdal2wktraster.py -r c:\tmp\raster -t sebastable -o c:\tmp\sebas.sql
gdal2wktraster.py:644: DeprecationWarning: 'H' format requires 0 <= number <= 65535
hexstr = binascii.hexlify(struct.pack(fmt_little, data)).upper()
gdal2wktraster.py:644: DeprecationWarning: integer argument expected, got float
hexstr = binascii.hexlify(struct.pack(fmt_little, data)).upper()
ERROR 2: Multiplication overflow : 107759 * 168633 * 1
Traceback (most recent call last):
File "gdal2wktraster.py", line 1013, in <module>
main()
File "gdal2wktraster.py", line 976, in main
wkblify_raster(opts, filename, i)
File "gdal2wktraster.py", line 921, in wkblify_raster
summary = wkblify_raster_level(options, ds, options.overview_level, band_range, infile, i)
File "gdal2wktraster.py", line 888, in wkblify_raster_level
hexwkb += wkblify_band(options, band, level, xoff, yoff, read_block_size, block_size, infile, b)
File "gdal2wktraster.py", line 777, in wkblify_band
target_block_size[0], target_block_size[1])
File "C:\OSGeo4W\apps\gdal-16\pymod\osgeo\gdal.py", line 835, in ReadAsArray
buf_xsize, buf_ysize, buf_obj )
File "C:\OSGeo4W\apps\gdal-16\pymod\osgeo\gdal_array.py", line 140, in BandReadAsArray
ar = numpy.reshape(ar, [buf_ysize,buf_xsize])
File "C:\OSGeo4W\apps\Python25\lib\site-packages\numpy\core\fromnumeric.py", line 116, in reshape
return reshape(newshape, order=order)
ValueError: total size of new array must be unchanged
Any ideas ?
--
Sebastian E. Ovide
_______________________________________________
postgis-users mailing list
postgis-users at postgis.refractions.net<mailto:postgis-users at postgis.refractions.net>
http://postgis.refractions.net/mailman/listinfo/postgis-users
--
Sebastian E. Ovide
skype: sebastian.ovide
+353 (0) 87 6340149
_______________________________________________
postgis-users mailing list
postgis-users at postgis.refractions.net<mailto:postgis-users at postgis.refractions.net>
http://postgis.refractions.net/mailman/listinfo/postgis-users
--
Sebastian E. Ovide
--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
www.faculty.jacobs-university.de/pbaumann<http://www.faculty.jacobs-university.de/pbaumann>
mail: p.baumann at jacobs-university.de<mailto:p.baumann at jacobs-university.de>
tel: +49-421-200-3178, fax: +49-421-200-493178
- Executive Director, rasdaman GmbH Bremen (HRB 147737)
www.rasdaman.com<http://www.rasdaman.com>, mail: baumann at rasdaman.com<mailto:baumann at rasdaman.com>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 10xx)
--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
www.faculty.jacobs-university.de/pbaumann<http://www.faculty.jacobs-university.de/pbaumann>
mail: p.baumann at jacobs-university.de<mailto:p.baumann at jacobs-university.de>
tel: +49-421-200-3178, fax: +49-421-200-493178
- Executive Director, rasdaman GmbH Bremen (HRB 147737)
www.rasdaman.com<http://www.rasdaman.com>, mail: baumann at rasdaman.com<mailto:baumann at rasdaman.com>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 10xx)
--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
www.faculty.jacobs-university.de/pbaumann<http://www.faculty.jacobs-university.de/pbaumann>
mail: p.baumann at jacobs-university.de<mailto:p.baumann at jacobs-university.de>
tel: +49-421-200-3178, fax: +49-421-200-493178
- Executive Director, rasdaman GmbH Bremen (HRB 147737)
www.rasdaman.com<http://www.rasdaman.com>, mail: baumann at rasdaman.com<mailto:baumann at rasdaman.com>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 10xx)
--
Dr. Peter Baumann
- Professor of Computer Science, Jacobs University Bremen
www.faculty.jacobs-university.de/pbaumann<http://www.faculty.jacobs-university.de/pbaumann>
mail: p.baumann at jacobs-university.de<mailto:p.baumann at jacobs-university.de>
tel: +49-421-200-3178, fax: +49-421-200-493178
- Executive Director, rasdaman GmbH Bremen (HRB 147737)
www.rasdaman.com<http://www.rasdaman.com>, mail: baumann at rasdaman.com<mailto:baumann at rasdaman.com>
tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 10xx)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20100728/43d53dae/attachment.html>
More information about the postgis-users
mailing list