[gdal-dev] COG via HTTP POST
Korver, Mark
mkorver at amazon.com
Wed Jan 3 13:16:09 PST 2024
Hi Javier,
A POST url to COG content only makes sense if the intent is to support full download of the file.
POST is normally for adding/updating data.
I took a quick look and yes no way to directly access the COGs other than use the search UI and get a list.
We see this many times, the good news is that they are formatted as COGs, but the value prop of COGs, being able to use them directly via range-gets is not there. We are hoping to make this easier for our public sector customers this year by simplifying STAC deployments on top serverless.
I did notice that download UI is very slow to load but the WMTS service used by the Viewer tool is performant.
https://www.ign.es/wmts/pnoa-ma?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=OI.OrthoimageCoverage&STYLE=default&FORMAT=image/jpeg&TILEMATRIXSET=EPSG:4326&TILEMATRIX=EPSG:4326:18&TILEROW=77554&TILECOL=257526
Depending on what you are looking to do, it might actually be faster to use GDAL or GDAL tools via QGIS to point a the relevant data using the WMTS service and output COG.
Of course you would lose any meta data in the source COGS. WMS I did not try.
Mark
Date: Tue, 2 Jan 2024 13:14:09 +0100
From: Javier Jimenez Shaw <j1 at jimenezshaw.com <mailto:j1 at jimenezshaw.com>>
To: gdal dev <gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>>
Subject: [gdal-dev] COG via HTTP POST
Message-ID:
<CADRrdKvusz8UsaHyVMGLXN+zR8+7Q=sg-qQVVGDb9hMNAZSD=A at mail.gmail.com <mailto:A at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"
Hi
Does it make any sense to provide COG via HTTP POST method?
If I understand correctly, the "magic" of downloading only what is needed
from the server is done with HTTP Range requests. But I am not sure if that
works in POST method (it does work in GET method, when enabled ;).
In case it could work with POST, I do not know how to tell GDAL or QGIS to
use a POST method.
Why am I asking this? the CNIG (Natial Center of Geographic Information in
Spain) is doing it.
https://centrodedescargas.cnig.es/CentroDescargas/index.jsp <https://centrodedescargas.cnig.es/CentroDescargas/index.jsp> The link is
just the starting point. Unfortunately they do not provide link to the
final page where you can download the files (yes, ugly).
So for now I have to download the full file, and do whatever I want later.
Thanks
Javier
.___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/cedd5d7b/attachment-0001.htm> <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/cedd5d7b/attachment-0001.htm>>
------------------------------
Message: 3
Date: Tue, 2 Jan 2024 13:22:00 +0100
From: Javier Jimenez Shaw <j1 at jimenezshaw.com <mailto:j1 at jimenezshaw.com>>
To: Even Rouault <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>>
Cc: gdal dev <gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] GDAL 3.8.3 RC1 is available, and motion to
approve it
Message-ID:
<CADRrdKtF+dGX0JiB75dLD3SwQiOvi96969MEAzLuasnubygodQ at mail.gmail.com <mailto:CADRrdKtF+dGX0JiB75dLD3SwQiOvi96969MEAzLuasnubygodQ at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"
+1 Javier
On Tue, 2 Jan 2024, 12:56 Even Rouault via gdal-dev, <
gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>> wrote:
> Hi,
>
> I have prepared an advanced-of-time GDAL/OGR 3.8.3 release candidate,
> mostly
> to fix a 3.8.0 regression that has been reported lately
> (https://github.com/OSGeo/gdal/issues/8998 <https://github.com/OSGeo/gdal/issues/8998>), that causes content of
> boolean
> fields in GeoPackage or FlatGeobuf datasets to be incorrectly read when
> using
> the ArrowArray interface, for example when used as source datasets in
> ogr2ogr
> workflows.
>
> Pick up an archive among the following ones (by ascending size):
>
> https://download.osgeo.org/gdal/3.8.3/gdal-3.8.3rc1.tar.xz <https://download.osgeo.org/gdal/3.8.3/gdal-3.8.3rc1.tar.xz>
> https://download.osgeo.org/gdal/3.8.3/gdal-3.8.3rc1.tar.gz <https://download.osgeo.org/gdal/3.8.3/gdal-3.8.3rc1.tar.gz>
> https://download.osgeo.org/gdal/3.8.3/gdal383rc1.zip <https://download.osgeo.org/gdal/3.8.3/gdal383rc1.zip>
>
> A snapshot of the gdalautotest suite is also available:
>
> https://download.osgeo.org/gdal/3.8.3/gdalautotest-3.8.3rc1.tar.gz <https://download.osgeo.org/gdal/3.8.3/gdalautotest-3.8.3rc1.tar.gz>
> https://download.osgeo.org/gdal/3.8.3/gdalautotest-3.8.3rc1.zip <https://download.osgeo.org/gdal/3.8.3/gdalautotest-3.8.3rc1.zip>
>
> The NEWS file is here:
>
> https://github.com/OSGeo/gdal/blob/v3.8.3RC1/NEWS.md <https://github.com/OSGeo/gdal/blob/v3.8.3RC1/NEWS.md>
>
> Motion: adopt 3.8.3 RC1 as final 3.8.3 release
>
> Starting with my +1,
>
> Best regards,
>
> Even
>
> --
> http://www.spatialys.com <http://www.spatialys.com>
> My software is free, but my time generally not.
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/4928e12e/attachment-0001.htm> <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/4928e12e/attachment-0001.htm>>
------------------------------
Message: 4
Date: Tue, 2 Jan 2024 13:39:45 +0100
From: Even Rouault <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>>
To: Javier Jimenez Shaw <j1 at jimenezshaw.com <mailto:j1 at jimenezshaw.com>>, gdal dev
<gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] COG via HTTP POST
Message-ID: <d6946b88-f82c-4cee-a3c5-56cd0e479897 at spatialys.com <mailto:d6946b88-f82c-4cee-a3c5-56cd0e479897 at spatialys.com>>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Javier,
According to https://httpwg.org/specs/rfc7233.html#header.range <https://httpwg.org/specs/rfc7233.html#header.range> , HTTP
Range requests only apply to GET requests.? So COGs delivered through
POST can indeed only be usable after full download.
BTW, someone should ask CNIG to include GeoTIFF SRS tags. At least, for
file
pnt_sentinel2_2020_invierno_mosaico_islas_canarias_b843_hu28_8bits.tif
which has only ModelTiepointTag and ModelPixelScaleTag geotiff keys
giving the geospatial extent, but no CRS keys.
Even
Le 02/01/2024 ? 13:14, Javier Jimenez Shaw via gdal-dev a ?crit?:
> Hi
>
> Does it make any sense to provide COG via HTTP POST method?
> If I understand correctly, the "magic" of downloading only what is
> needed from the server is done with HTTP Range requests. But I am not
> sure if that works in POST method (it does work in GET method, when
> enabled ;).
> In case it could work with POST, I do not know how to tell GDAL or
> QGIS to use a POST method.
>
> Why am I asking this? the CNIG (Natial Center of Geographic
> Information in Spain) is doing it.
> https://centrodedescargas.cnig.es/CentroDescargas/index.jsp <https://centrodedescargas.cnig.es/CentroDescargas/index.jsp> The link
> is just the starting point. Unfortunately they do not provide link to
> the final page where you can download the files (yes, ugly).
>
> So for now I have to download the full file, and do whatever I want later.
>
> Thanks
> Javier
> .___ ._ ..._ .. . ._.? .___ .. __ . _. . __..? ... .... ._ .__
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
--
http://www.spatialys.com <http://www.spatialys.com>
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/94ae3cae/attachment-0001.htm> <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/94ae3cae/attachment-0001.htm>>
------------------------------
Message: 5
Date: Tue, 2 Jan 2024 13:43:59 +0100
From: Javier Jimenez Shaw <j1 at jimenezshaw.com <mailto:j1 at jimenezshaw.com>>
To: Even Rouault <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>>
Cc: gdal dev <gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] COG via HTTP POST
Message-ID:
<CADRrdKtWswA+eokeQwtQWDuVmhjAD641SGOZ8WDk=fbQKot9Xg at mail.gmail.com <mailto:fbQKot9Xg at mail.gmail.com>>
Content-Type: text/plain; charset="utf-8"
Thanks Even.
I will tell them about the missing SRS
On Tue, 2 Jan 2024, 13:39 Even Rouault, <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>> wrote:
> Javier,
>
> According to https://httpwg.org/specs/rfc7233.html#header.range <https://httpwg.org/specs/rfc7233.html#header.range> , HTTP
> Range requests only apply to GET requests. So COGs delivered through POST
> can indeed only be usable after full download.
>
> BTW, someone should ask CNIG to include GeoTIFF SRS tags. At least, for
> file pnt_sentinel2_2020_invierno_mosaico_islas_canarias_b843_hu28_8bits.tif
> which has only ModelTiepointTag and ModelPixelScaleTag geotiff keys giving
> the geospatial extent, but no CRS keys.
>
> Even
> Le 02/01/2024 ? 13:14, Javier Jimenez Shaw via gdal-dev a ?crit :
>
> Hi
>
> Does it make any sense to provide COG via HTTP POST method?
> If I understand correctly, the "magic" of downloading only what is needed
> from the server is done with HTTP Range requests. But I am not sure if that
> works in POST method (it does work in GET method, when enabled ;).
> In case it could work with POST, I do not know how to tell GDAL or QGIS to
> use a POST method.
>
> Why am I asking this? the CNIG (Natial Center of Geographic Information in
> Spain) is doing it.
> https://centrodedescargas.cnig.es/CentroDescargas/index.jsp <https://centrodedescargas.cnig.es/CentroDescargas/index.jsp> The link is
> just the starting point. Unfortunately they do not provide link to the
> final page where you can download the files (yes, ugly).
>
> So for now I have to download the full file, and do whatever I want later.
>
> Thanks
> Javier
> .___ ._ ..._ .. . ._. .___ .. __ . _. . __.. ... .... ._ .__
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orgh <mailto:listgdal-dev at lists.osgeo.orgh>ttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com <http://www.spatialys.com>
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/8627e23b/attachment-0001.htm> <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/8627e23b/attachment-0001.htm>>
------------------------------
Message: 6
Date: Tue, 2 Jan 2024 15:09:23 +0100
From: leo.fuhrmann.lgln at icloud.com <mailto:leo.fuhrmann.lgln at icloud.com>
To: gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
Subject: [gdal-dev] Error when importing GeoPackage into PostgreSQL
database ("Worker thread task has not expected m_iStartShapeId value")
Message-ID: <A0357B4D-4DFB-4070-86F4-EB7975C4140F at icloud.com <mailto:A0357B4D-4DFB-4070-86F4-EB7975C4140F at icloud.com>>
Content-Type: text/plain; charset="utf-8"
Hi,
I?m using ogr2ogr 3.8.2 to import a (layer of a) GeoPackage into a PostGIS database:
```
ogr2ogr --debug ON -f PostgreSQL "PG:dbname=XX host=XX port=XX user=XX password=XX? <path/to/gpkg.gpkg> <layer>
```
However, I?m seeing an error in the logs that I cannot sort out (see mail subject and/or logs below). Despite the error, the data gets written to the database, yet I?m worried that something is messed up along the way.
Using the official docker image to execute the command (from a MacBook with M1 Pro) works without an error. Then again, using a cloud service (IBM Code Engine in this case) with the official image also throws the error. I tried with different image versions but found no difference.
I tried to track down the error and found https://github.com/OSGeo/gdal/blob/1df855eb676e17278c337e9b14774d1bb298df90/ogr/ogrsf_frmts/gpkg/ogrgeopackagetablelayer.cpp#L8352 <https://github.com/OSGeo/gdal/blob/1df855eb676e17278c337e9b14774d1bb298df90/ogr/ogrsf_frmts/gpkg/ogrgeopackagetablelayer.cpp#L8352>
The comment suggests that I might have messed with GetNextFeature(), but I?m sure I haven?t.
Can someone help me to make sense of this error? I?m sorry to say that I can?t share the GeoPackage at the moment.
Thanks, Leo
Here are the logs:
GPKG: GeoPackage v1.2.0
GDAL: GDALOpen(<path/to/gpkg.gpkg>, this=0x14c053e00) succeeds as GPKG.
GDAL: GDALDriver::Create(PostgreSQL,PG:dbname=XX host=XX port=XX user=XX password=XX
PG: Client encoding: 'UTF8'
PG: PostGIS schema: 'public'
PG: PostgreSQL version string : 'PostgreSQL 16.1 (Debian 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit'
PG: PostGIS version string : '3.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1'
PG: Could not retrieve table oid for <layer>
GDALVectorTranslate: Using FID=fid and -preserve_fid
PG: Could not retrieve table oid for <layer>
OGR2OGR: Using WriteArrowBatch()
GPKG: GeoPackage v1.2.0
GPKG: GeoPackage v1.2.0
GPKG: GeoPackage v1.2.0
ERROR 1: Worker thread task has not expected m_iStartShapeId value
PG: PQputCopyEnd()
GDAL: GDALClose(<path/to/gpkg.gpkg>, this=0x14c053e00)
GDAL: GDALClose(PG:dbname=XX host=XX port=XX user=XX password=XX this=0x14b721fb0)
GDAL: In GDALDestroy - unloading GDAL shared library.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/3eca1d4d/attachment.htm> <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240102/3eca1d4d/attachment.htm>>
------------------------------
Subject: Digest Footer
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev>
------------------------------
End of gdal-dev Digest, Vol 236, Issue 1
****************************************
More information about the gdal-dev
mailing list