[OSGeo-Standards] [OSGeo-Discuss] WFS 3.0 and SpatioTemporal Asset Catalog sprint in Colorado

Wed May 22 07:13:52 PDT 2019

The forthcoming OGC API Hackathon should help clarify much of these discussions.
Agree with Chris' endpoint patterns.  I would probably update the CSW
endpoints like:

/collections/search
/search

To be continued -- stay tuned!

..Tom

On Sat, May 18, 2019 at 4:27 PM Stefan Keller <sfkeller at gmail.com> wrote:
>
> Dear Chris (dear all)
>
> Thanks, Chris, again for your detailed answer and explanation.
>
> We've implemented a tiny WFS3 server in Go and I'm just stumbled over
> the response requiring a “self-link” (rel=“self” which includes the
> collectionId).
> Is that really mandatory - and what's it used for?
>
> (Tell me where to move with this discussion if this list isn't the right place).
>
> :Stefan
>
>
> Am Fr., 3. Mai 2019 um 16:41 Uhr schrieb Chris Holmes <cholmes at radiant.earth>:
> >
> > Hey Stefan, I composed a pretty long message on my view on ideal relationship between STAC and CSW, I'll paste it in below. The main conceptual thing to me is that most CSW's have been about 'layer' level search, and image / 'asset' level search is actually different. In OGC history the 'ebrim profile' of CSW actually did the image / asset level search well. But in my opinion they are different enough that we shouldn't group them all in one thing. An asset level catalog could/should be equivalent to a 'data cube', and so should slot at the 'layer' level.
> >
> > As for schema.org, we've made attempts to align with it, but at the level where STAC is transformed into HTML output. See https://github.com/radiantearth/stac-spec/blob/dev/catalog-spec/catalog-best-practices.md#stac-on-the-web for info on how we think about HTML & STAC, and in particular the sub-section on schema.org, etc.
> >
> > The in depth view I wrote on csw and stac is:
> >
> >
> > The ideal relationship between CSW, WFS and STAC is that WFS _is_ the api that all three use. STAC and CSW should just be specialized content - they use the same API response, but the JSON responses must comply with particular JSON Schemas. With STAC the core is id, geometry, datetime, links and assets - https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#item-fields Any WFS Item that meets the STAC Schema could be considered part of STAC. STAC also includes an ecosystem of more particular types that build on that base. So the Electro-Optical extension has more fields that comply to a schema that can be added and queried. The goal is to encourage more bottom up communities of interest. But any of those can be queried from the STAC endpoint. That's the vision at least, and we're trying to align with WFS to do it, but there's more work needed, which I'll detail below.
> >
> > Then to me CSW should be a set of fields that describe a collection of data. So it'd have like title, license, keywords, spatial and temporal extents, etc. Could theoretically just use the same fields that CSW 3.0 defined, and make those a JSON content type. Define the schema, and then a 'CSW 4.0' would just be a WFS 3 that serves up that content. WFS 3 still hasn't quite specified its query language afaik, so maybe CSW4 requires a particular richer query language. But it should be the same query language that is at least an option for WFS. Thus any WFS client could naively query a catalog, requesting the fields it needs and getting the response. A CSW 4 client would be an extension to the WFS one that perhaps takes advantage of some more expectations. It could be possible that you would add like 'harvesting' extensions for CSW 4, but those should be generic WFS API mechanisms. So a CSW is a WFS endpoint that meets a set schema definition - whatever flavor of dublin core works best - plus it may also specify some required WFS API extensions that should be used.
> >
> > I do think there is an opportunity to combine WFS and CSW even more, by aligning the schema of CSW with the response in the /collection/{collectionId} endpoint in WFS. Like the way it defines extents, title, etc. should be the same as the CSW JSON response. Ideally CSW defines additional fields that are useful for defining a dataset, and then the WFS /collection/{collectionId} endpoint can use those same fields. This would make it so every WFS is 'harvestable' by a CSW. And then any WFS could supply a special /collections/csw endpoint, that exposes the metadata of all the other collections that are in that WFS.
> >
> > So basically CSW should be the summary of dataset / collection level metadata, available through the WFS API. And then STAC is 'asset' level metadata - references to other types of geographic files that can be downloaded. So one could easily see a service that implements all three:
> >
> > /collections/buildings/ (a standard WFS layer - a set of vectors)
> > /collections/landsat/ (a STAC (and EO Extension) compliant WFS layer that has every landsat scene available for search)
> > /collections/sentinel/ (another STAC & eo compliant WFS layer)
> > /collections/csw/ (the CSW endpoint, that has 3 records - the collection level metadata for buildings, landsat and sentinel)
> > /stac/ - the browse endpoint for STAC, that returns STAC catalogs that navigate down to individual records)
> > /stac/landsat/L8/166/077/  returns all a json STAC Catalog that returns links to individual feature records in /collections/landsat/
> > /stac/search/ - the endpoint that does cross collection search, returning results from both landsat and sentinel.
> >
> > The 'collection level metadata' of the /collections/csw/items/ would be the exact same fields returned at /collections/buildings, etc.
> >
> > In an ideal world we probably wouldn't even have /stac/search - that would just be a standard WFS search endpoint that can do cross collection search, which it sounds like people are maybe working on?
> >
> > That highlights that in STAC we do a few things that should just be done in the WFS spec. Indeed any type of API construct, like querying and transactions (which we do have in STAC), should be defined in WFS. STAC could choose to use a particular set of API constructs, but they should be more generic components at the WFS level. We have /stac/search/ because WFS doesn't yet have cross collection search. We have a query language because WFS's doesn't have one that meets our needs yet, etc. The other thing WFS doesn't have is the 'static' browse ability, that we have in /stac/, but I do think it would be a good addition, and then we wouldn't really need to do anything 'special' - STAC would just be a set of schemas to be used in WFS.
> >
> > We have actually aligned STAC even more with WFS, with our STAC Collection using the same fields as a WFS collection. Though there were a couple little things off about the WFS Collections. The goal to me is to make a 'collection definition' of some set of core required fields, plus additional dublin core/etc metadata fields that are optional. And that is returned at /collections/{collectionId} but is also the same fields for a CSW search, but even also defines coverages or other data types as well - you have the same core construct of how you describe a 'layer'.
> >
> > On Fri, May 3, 2019 at 6:14 AM Stefan Keller <sfkeller at gmail.com> wrote:
> >>
> >> Thx for the quick info.
> >> Now I'm keen to learn more about SpatioTemporal Asset Catalog (STAC)
> >> and it's relationship to existing metadata standards (geo and general
> >> non-geo).
> >> :Stefan
> >>
> >> Am Fr., 3. Mai 2019 um 14:55 Uhr schrieb Scott Simmons
> >> <ssimmons at opengeospatial.org>:
> >> >
> >> > Hi Stefan,
> >> >
> >> > I can talk to WFS3. We now have a roadmap for all OGC candidate standards under development (not fully populated, but those closest to approval and a few others are in the roadmap):
> >> > http://www.opengeospatial.org/roadmap
> >> >
> >> > This roadmap will be more formally promoted in the coming weeks once we get it fully populated.
> >> >
> >> > WFS3 (now being referred to as OGC API Features) is to be briefed to both ISO / TC 211 and OGC in June in advance of approval voting in each organization. No guarantees, but the OGC approval is forecast to occur after our June Technical Committee Meeting (24-27 June) in Leuven, Belgium. Voting takes about 2 months in total.
> >> >
> >> > Best Regards,
> >> > Scott
> >> >
> >> > Scott Simmons
> >> > Chief Operations Officer
> >> > Open Geospatial Consortium (OGC)
> >> > tel +1 970 682 1922
> >> > mob +1 970 214 9467
> >> > ssimmons at opengeospatial.org
> >> >
> >> > The OGC: Making Location Count…
> >> > www.opengeospatial.org
> >> >
> >> > On May 3, 2019, at 6:44 AM, Stefan Keller <sfkeller at gmail.com> wrote:
> >> >
> >> > Hi Chris
> >> >
> >> > Let me chime in here with two specific question about my understanding
> >> > of WFS3 and of STAC..
> >> > A. When will WFS3 become an OGC standard? Is there a road a map? (I
> >> > know about the London Hackathon mid year)
> >> > B. What is the relationship of STAC with Catalog Service for Web
> >> > (CS-W) and metadata (schema.org) in general?
> >> >
> >> > :Stefan
> >> >
> >> > Am Do., 1. März 2018 um 14:33 Uhr schrieb Angelos Tzotsos
> >> > <gcpp.kalxas at gmail.com>:
> >> >
> >> >
> >> > Thank you Chris for the information.
> >> >
> >> > Best regards,
> >> > Angelos
> >> >
> >> > On 03/01/2018 02:17 PM, Chris Holmes wrote:
> >> >
> >> > Just realized I forgot to reply to this, but that's a great idea. I'll be
> >> > in Orleans for the OGC meetings, so will be on the same time zone. And
> >> > there's definitely some STAC collaborators who will be in Bonn.
> >> >
> >> > Note also that we're organizing a remote participation option for next
> >> > week. GeoSolutions is going to be working on implementing WFS 3 in
> >> > GeoServer and Even Rouault is going to be looking in to GDAL bindings for
> >> > WFS 3.0 and STAC. Would be great for others in Europe to work on OSGeo
> >> > projects while we're working away in Ft. Collins. Obviously any WFS 3 /
> >> > STAC implementation & spec feedback in the next few weeks would be great,
> >> > but we're hoping to be able to show lots of diverse progress next week.
> >> >
> >> > For more info see
> >> > https://medium.com/@cholmes/participate-remotely-in-the-wfs-hackathon-next-week-d11a99eb510b
> >> > and please sign up to participate at https://goo.gl/forms/v8hyeJvd2yudYvZS2
> >> >
> >> > best regards,
> >> >
> >> > Chris
> >> >
> >> >
> >> >
> >> > On Thu, Feb 15, 2018 at 8:15 PM, Angelos Tzotsos <gcpp.kalxas at gmail.com>
> >> > wrote:
> >> >
> >> > Thank you Chris for organizing this event and for reaching out to this
> >> > mailing list.
> >> >
> >> > I feel there is strong interest from our community to participate on the
> >> > development of these new standards, and I know that there is ongoing work
> >> > towards this from some OSGeo members, so I am sure you will see some of us
> >> > in Ft Collins.
> >> >
> >> > Let's make an OGC/WFS/STAC Catalog session during the Bonn Code Sprint: I
> >> > realized that the Orleans OGC TC meeting is on the same week as the Bonn
> >> > code sprint. Perhaps we could all join a video call session during that
> >> > week?
> >> >
> >> > Best,
> >> > Angelos
> >> >
> >> >
> >> >
> >> > On 02/13/2018 03:34 AM, Chris Holmes wrote:
> >> >
> >> > Hello OSGeo-ers!
> >> >
> >> > I just wanted to make sure everyone knew about an event [1] I'm helping
> >> > organize, to bring together developers to give feedback on the evolving WFS
> >> > 3.0 specification. It's organized by Open Geospatial Consortium (OGC), and
> >> > sponsored by USGS, but the goal is make it different than a typical OGC
> >> > event. Indeed the direct inspiration is the code sprints that OSGeo runs,
> >> > but to center it around a standard and give developers an opportunity to
> >> > actually code against the spec _before_ it becomes an official standard,
> >> > and to have any feedback incorporated.
> >> >
> >> > I'm not sure if many people have been following the progress of WFS 3.0,
> >> > but it's been made more like how we build open source software, with an
> >> > open repository on github [2], and management of the spec through issues
> >> > and pull requests. And it's JSON and RESTful at the core, aiming to be much
> >> > easier to implement than existing OGC specs. For more backstory on it and
> >> > my take on its potential see my blog posts [3].
> >> >
> >> > I'd love to see a number of OSGeo people come, it's on March 6 & 7th in Ft.
> >> > Collins, Colorado. Contributing I believe will help show OGC that there is
> >> > interest in the wider developer and open source world if they do things
> >> > differently, and help evolve how they create standards to be more
> >> > compatible with how developers work. I'm also organizing a day on March 8th
> >> > on SpatioTemporal Asset Catalog[4] [5], an open spec a group of us started
> >> > working on to search satellite imagery archives, that I'm hoping we can
> >> > align with the WFS specification. If you're interested in WFS and/or STAC
> >> > just email me and I can give you more details, or just fill out the form athttps://goo.gl/forms/RqQtNbfdEOHuLE272 - there are some limited travel
> >> > grants if that helps get there.
> >> >
> >> > I know travel may be tough for those also going to Bonn, as the OSGeo
> >> > sprint there is two weeks later. I am figuring out if we will do a remote
> >> > participation option for Colorado, so email me if you're interested in
> >> > that. And Angelos had a great idea[6], of organizing a session on WFS/STAC
> >> > at Bonn, which I'd love to see. Drop me a line if you are attending the
> >> > OSGeo code sprint and interested in attending (or leading :) a session
> >> > there.
> >> >
> >> > best regards,
> >> >
> >> > Chris
> >> >
> >> >
> >> > [1]https://medium.com/@cholmes/wfs-3-0-and-spatiotemporal-asset-catalog-stac-in-person-collaboration-609e10d7f714
> >> > [2] https://github.com/opengeospatial/WFS_FES
> >> > [3] https://medium.com/tag/wfs-3/latest
> >> > [4] https://github.com/radiantearth/stac-spec/
> >> > [5] https://medium.com/tag/stac-spec/latest
> >> > [6] https://twitter.com/tzotsos/status/963081024187060225
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Discuss mailing listDiscuss at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/discuss
> >> >
> >> >
> >> > --
> >> > Angelos Tzotsos, PhD
> >> > Charter Member
> >> > Open Source Geospatial Foundationhttp://users.ntua.gr/tzotsos
> >> >
> >> >
> >> >
> >> > --
> >> > Angelos Tzotsos, PhD
> >> > Charter Member
> >> > Open Source Geospatial Foundation
> >> > http://users.ntua.gr/tzotsos
> >> >
> >> > _______________________________________________
> >> > Standards mailing list
> >> > Standards at lists.osgeo.org
> >> > https://lists.osgeo.org/mailman/listinfo/standards
> >> >
> >> > _______________________________________________
> >> > Standards mailing list
> >> > Standards at lists.osgeo.org
> >> > https://lists.osgeo.org/mailman/listinfo/standards
> >> >
> >> >
> _______________________________________________
> Standards mailing list
> Standards at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/standards