[gdal-dev] Directories and Parquet

Fri Jul 28 15:14:01 PDT 2023

Here's fetching the first admin boundary:

docker run --rm -it ghcr.io/osgeo/gdal:ubuntu-full-latest ogrinfo /vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins/  -where "type='administrativeBoundary'" -al  -fid 1
Warning 1: Field names of unhandled type map<string, list<array_element: map<string, string ('array_element')>> ('names')> ignored
Warning 1: Field sources of unhandled type list<array_element: map<string, string ('array_element')>> ignored
INFO: Open of `/vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins/'
      using driver `Parquet' successful.

Layer name: theme=admins
Geometry: Unknown (any)
Feature Count: 96455
Extent: (-180.000000, -89.999900) - (180.000000, 89.999900)
Layer SRS WKT:
(unknown)
Geometry Column = geometry
id: String (0.0)
updatetime: String (0.0)
version: Integer (0.0)
adminlevel: Integer (0.0)
maritime: String (0.0)
subtype: String (0.0)
localitytype: String (0.0)
context: String (0.0)
isocountrycodealpha2: String (0.0)
isosubcountrycode: String (0.0)
defaultlanugage: String (0.0)
drivingside: String (0.0)
bbox.minx: Real (0.0)
bbox.maxx: Real (0.0)
bbox.miny: Real (0.0)
bbox.maxy: Real (0.0)
type: String (0.0)
OGRFeature(theme=admins):1
  id (String) = 85186a93fffffff1100000000000007
  updatetime (String) = 2023-05-28T00:18:08.062Z
  version (Integer) = 0
  adminlevel (Integer) = 4
  maritime (String) = (null)
  subtype (String) = (null)
  localitytype (String) = (null)
  context (String) = (null)
  isocountrycodealpha2 (String) = (null)
  isosubcountrycode (String) = (null)
  defaultlanugage (String) = (null)
  drivingside (String) = (null)
  bbox.minx (Real) = -0.6814001
  bbox.maxx (Real) = -0.6561592
  bbox.miny (Real) = 46.686869
  bbox.maxy (Real) = 46.7005651
  type (String) = administrativeBoundary
  LINESTRING (-0.6561592 46.7005651,-0.6576952 46.7001611,-0.6596517 46.6992552,-0.6607547 46.6986231,-0.661421 46.6981174,-0.6642916 46.6959388,-0.6664351 46.6949608,-0.6700295 46.6934809,-0.670524 46.6932773,-0.6719283 46.6922006,-0.6738641 46.6916239,-0.6749166 46.6910205,-0.6764343 46.6894853,-0.6767317 46.6893071,-0.6784728 46.688264,-0.6797574 46.6881753,-0.6807998 46.6873219,-0.6811768 46.6870573,-0.6814001 46.686869)

Mike

On 7/28/23, 6:05 PM, "Michael Smith" <michael.smith.erdc at gmail.com <mailto:michael.smith.erdc at gmail.com>> wrote:

Paul,

I had to use 3.7.1 to have it read the geometry properly, 3.7.0 would read the directories.

docker run --rm -it ghcr.io/osgeo/gdal:ubuntu-full-latest ogrinfo /vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins -al -so
Warning 1: Field names of unhandled type map<string, list<array_element: map<string, string ('array_element')>> ('names')> ignored
Warning 1: Field sources of unhandled type list<array_element: map<string, string ('array_element')>> ignored
INFO: Open of `/vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins'
using driver `Parquet' successful.

Layer name: theme=admins
Geometry: Unknown (any)
Feature Count: 99403
Extent: (-180.000000, -89.999900) - (180.000000, 89.999900)
Layer SRS WKT:
(unknown)
Geometry Column = geometry
id: String (0.0)
updatetime: String (0.0)
version: Integer (0.0)
adminlevel: Integer (0.0)
maritime: String (0.0)
subtype: String (0.0)
localitytype: String (0.0)
context: String (0.0)
isocountrycodealpha2: String (0.0)
isosubcountrycode: String (0.0)
defaultlanugage: String (0.0)
drivingside: String (0.0)
bbox.minx: Real (0.0)
bbox.maxx: Real (0.0)
bbox.miny: Real (0.0)
bbox.maxy: Real (0.0)
type: String (0.0)

Mike

-- 

Michael Smith 
Remote Sensing/GIS Center 
US Army Corps of Engineers 

On 7/28/23, 5:50 PM, "gdal-dev on behalf of Paul Ramsey" <gdal-dev-bounces at lists.osgeo.org <mailto:gdal-dev-bounces at lists.osgeo.org> <mailto:gdal-dev-bounces at lists.osgeo.org <mailto:gdal-dev-bounces at lists.osgeo.org>> on behalf of pramsey at cleverelephant.ca <mailto:pramsey at cleverelephant.ca> <mailto:pramsey at cleverelephant.ca <mailto:pramsey at cleverelephant.ca>>> wrote:

I am playing with accessing the Overture data dump, which is multiple
files in each S3 bucket. The doco says that as of 3.6 (check!), with
the libarrow_dataset build (check) I should be able to read directory
collections as layers, but no luck thus far:

AWS_REGION=us-west-2 AWS_NO_SIGN_REQUEST=YES ./ogrinfo
/vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins/type=administrativeBoundary/
AWS_REGION=us-west-2 AWS_NO_SIGN_REQUEST=YES ./ogrinfo
/vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins/type=administrativeBoundary/

ogrinfo failed - unable to open
'/vsis3/overturemaps-us-west-2/release/2023-07-26-alpha.0/theme=admins/type=administrativeBoundary/'.

If I pick one of the constituent files in that directory, ogr can open
it up just fine, and we just have the array/map/list type issue that
Scott has raised in another thread. Is this expected, a result of the
access being via S3 rather than an actual filesystem directory?
Something else?

Thx,
P
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org> <mailto:gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>>
https://lists.osgeo.org/mailman/listinfo/gdal-dev <https://lists.osgeo.org/mailman/listinfo/gdal-dev> <https://lists.osgeo.org/mailman/listinfo/gdal-dev> <https://lists.osgeo.org/mailman/listinfo/gdal-dev>>