[gdal-dev] Passing open options along dataset name in a string ?

Tue Nov 17 16:24:08 PST 2020

Hi Jukka,

On Tue, Nov 17, 2020 at 2:57 AM jratike80 <
jukka.rahkonen at maanmittauslaitos.fi> wrote:

> Hi,
>
> I have done some helpdesk work within the GDAL community and I know well
> that the open options and config options are confusing. I also know that
> they exists for a reason but simplified and uniform way to use them would
> be
> nice.
>

I'm glad to read that you're interested! I believe there is tension between
simple and uniform and we'll need wide input to find a good balance.

I've been using QGIS today for the first time in a while and I think that
application could benefit from this as well. A "URL" for datasets could
remove some of the complexity of QGIS dialog boxes for creating or
configuring connections.

> Some comments on comments:
>
> >> gdalinfo my.tif -oo GEOREF_SOURCES=WORLDFILE,PAM
> >>
>
> > Ideally this would be baked into the format, but, yes, I think we've got
> a
> > bead on dataset open options.
>
> I don't know how it could be baked into the format. The option gives user
> an
> option to override wrong GeoTIFF georeferencing with wordfile, for example.
>

Yes.

> >> gdalinfo BAG:"data/test_vr.bag":supergrid:0:1
> >>
>
> > DRIVER:"file":something
>
> > Right. This will require some work because of multiple colons. Though
> I've
> > never seen BAG driver data in the wild. Is this a real live format?
>
> As far as I know BAG is the hdf of bathymetry and widely used in that
> context.
>
> >> gdalinfo data/test_vr.bag -oo MODE=RESAMPLED_GRID -oo
> SUPERGRIDS_MASK=YES
> >> gdalinfo HDF5:"d:\foo.he5"://HDFEOS/SWATHS/foo/bar
> >>
>
> > HDF5 driver, filename using Windows drive, and UNC path within it. This
> is
> > marginal, right?
>
> The part beginning with // is not UNC path but the name of the subdataset
> within hdf5 file https://gdal.org/drivers/raster/hdf5.html. Not more
> marginal than HDF5 itself.
>
>
Thank you for explaining.

> >> ogrinfo OCI:warmerda/password at .dreadfest
>
> >Wat?
>
> Text has just been formatted into email link because of the @ sign that
> belongs to the Oracle connection string "username" / "password" @ "the name
> of the Oracle database as it appears in the tnsnames.ora file". Let's see
> if
> formatting happens again when I send this from Nabble:
>
> OCI:warmerda/password at gdal800.dreadfest.com abc.shp
>

Thanks.

> -Jukka Rahkonen-
>
>
>
>
> Sean Gillies-3 wrote
> > Hi Even,
> >
> > On Wed, Nov 4, 2020 at 9:01 AM Even Rouault <
>
> > even.rouault@
>
> > >
> > wrote:
> >
> >> > > Another particularity we have in GDAL is that the dataset name might
> >> be
> >> > > almost
> >> > > anything. Most of the time, it is a regular file path, or some /vsi
> >> path.
> >> > > But
> >> > > sometimes, it can be JSON content (the GeoJSON driver accepts the
> >> content
> >> > > to
> >> > > be directly provided as the dataset name), or XML (VRT, WMS
> drivers).
> >> > > We have also the subdataset syntax "HDF5:foo.hdf:my_variable"
> >> >
> >> > Could VRT XML and JSON be exempted? We already have a way to embed
> open
> >> > options in the XML.
> >>
> >> If the gdn: mechanism is a new possibility offered that doesn't exclude
> >> existing ones (otherwise that would be a pretty big breaking change), we
> >> could
> >> possibly exempt the odd cases I mentioned (or have some quoting/escaping
> >> rules
> >> to enable that payload to be seen as a file), which generally don't need
> >> a
> >> "permanent" way of refering to the dataset like gdn: would offer, since
> >> this
> >> is content often generated programatically or retrieved dynamically.
> >>
> >> Covering subdataset would be a more important use case. Something that
> >> would
> >> have to be decided if the way we express subdatasets would be somehow
> >> standardized or if it would be a black-box string for the gdn:
> >> encapsulation.
> >> For a black-box approach, we would have to define some escaping/quoting
> >> rules
> >> to avoid any potential issue with separators of the gdn syntax. If we
> >> decide
> >> that the subdataset syntax is part of what is standardized by GDN that
> >> would
> >> be a more challenging exercice, because the subdataset syntax varies
> from
> >> driver to driver.
> >>
> >
> > The variation of subdataset syntax among drivers is a bug, let's try to
> > fix
> > this.
> >
> > It seems to me that the internet way to address subdatasets would be to
> > use
> > a # URL fragment. But since most of our formats and the servers that
> serve
> > files of these formats are not aware, we may have to come up with
> > something
> > different. We may need to consider making subdatasets a layer opening
> > option?
> >
> > pending on how we design things, that might impact between:
> >> - just GDALOpen() generic code if GDALOpen() decodes the gdn: string to
> >> decompose it into 'classic' dataset names and open options
> >> - all drivers if the gdn: string would be passed to each
> >> GDALDriver::pfnOpen()
> >> implementation
> >> - intermediate situation if we decide to drop (at least for future
> >> drivers)
> >> per-driver subdataset syntax (which has deficiencies has the quoting
> >> rules
> >> to
> >> separate the filename from the non-filename component vary from driver
> to
> >> driver, and are most of the time not defined) to come up with something
> >> more
> >> standardized
> >>
> >> To help brainstorming, a non-exhaustive overview of a few situations
> >> mixing
> >> driver prefixing, subdataset syntax and open options:
> >>
> >> gdalinfo my.tif
> >>
> >
> > Yes. We have to handle bare paths to local dataset files.
> >
> >
> >> gdalinfo my.tif -oo GEOREF_SOURCES=WORLDFILE,PAM
> >>
> >
> > Ideally this would be baked into the format, but, yes, I think we've got
> a
> > bead on dataset open options.
> >
> >
> >> gdalinfo GTIFF_DIR:0:d:\my.tif
> >>
> >
> > WTF is this? :)
> >
> >
> >> gdalinfo EEDAI:my/asset
> >> gdalinfo EEDAI: -oo ASSET=my/asset
> >> gdalinfo EEDAI:my/asset:band1, band2
> >> gdalinfo EEDAI: -oo ASSET=my/asset -oo BANDS=band1,band2
> >>
> >
> > Never seen these.
> >
> >
> >> gdalinfo BAG:"data/test_vr.bag":supergrid:0:1
> >>
> >
> > DRIVER:"file":something
> >
> > Right. This will require some work because of multiple colons. Though
> I've
> > never seen BAG driver data in the wild. Is this a real live format?
> >
> >
> >> gdalinfo data/test_vr.bag -oo MODE=RESAMPLED_GRID -oo
> SUPERGRIDS_MASK=YES
> >> gdalinfo HDF5:"d:\foo.he5"://HDFEOS/SWATHS/foo/bar
> >>
> >
> > HDF5 driver, filename using Windows drive, and UNC path within it. This
> is
> > marginal, right?
> >
> >
> >> gdalinfo netCDF:"/vsicurl/http://example.com/my.nc":my_var
> >>
> >
> > This looks less complicated than some of the examples above.
> >
> >
> >> ogrinfo "PG:dbname=testdb user=foo"
> >> ogrinfo "mySQL:testdb,user=foo"
> >>
> >
> > These seem like they could be driver specific, but generalized key-value
> > parameters.
> >
> >
> >> ogrinfo OCI:warmerda/
>
> > password at .dreadfest
>
> >
> >
> > Wat?
> >
> >
> >> GDALOpen() is not even aware that HDF5:bla means that the dataset will
> be
> >> recognized by the HDF5 driver
> >>
> >>
> > Wait what?
> >
> > --
> > Sean Gillies
> >
> > _______________________________________________
> > gdal-dev mailing list
>
> > gdal-dev at .osgeo
>
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
>
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20201117/c344b98f/attachment.html>