request for comment: adding xslt processing to mapserver
Paul Spencer
pspencer at DMSOLUTIONS.CA
Sat Mar 24 17:14:17 EDT 2007
Steve,
I will add some notes about the streaming xslt thing. This seems
like it could be a showstopper. I was not able to find anything
useful on streaming XLST in a quick search, I will continue to search
though. Of course, if no one is able to do streaming transformations
then this isn't a strong argument against including the ability to do
transforms in mapserver since any external implementation would have
similar problems. In the case of per-layer transforms, it might
actually be more efficient to do it in mapserver since you would only
need to process the node containing the tree for that layer, one at a
time.
re wfs schema, good point. I'll include a note about schemas too.
re queries ... I had not thought about processing incoming XML data.
I'll have to add something about that too.
Cheers
Paul
On 23-Mar-07, at 8:12 PM, Steve Lime wrote:
> Paul: I did look at libxml2/libxslt and it's XSLT support before
> doing any of the
> work on GML transformation last summer. At that time I decided
> against it
> because:
>
> - performance, libxslt required an entire document be memory resident
> before transforming. You had to write vanilla GML to memory first
> (as DOM
> object) and for large datasets that wouldn't be straightforward).
> Streaming
> transformations were needed and, at the time, only available via
> java (and
> experimental at that), I don't know if that has changed or not.
>
> - with WFS, what you produce you need to document. So for the
> transformed
> output you'd need to produce a corresponding schema, yet another
> XSLT file.
> Referencing an external schema (e.g. some standard application
> schema) is
> one workaround but I don't know that it is a perfect solution.
>
> - with WFS, what you produce you also need to support queries
> against, so
> now you need to transform an incoming filter into something that
> that can be
> applied against the native data store, basically inverting the
> transformation
> applied in the first place. Then you also have to deal with non-xml
> request
> encoding.
>
> In the end it was all too much to swallow, not to mention that XSLT
> isn't exactly
> trivial for users. Not that configuring GML transformations through
> metadata is
> easy- it ain't, it sucks and overly complicates a mapfile.
>
> Still there's an elegance to only having to write construct (and
> support!) vanilla
> GML and then using external transformations. If the issues above
> could be addressed
> I would support the idea and might go one step further and suggest
> yanking out
> the metadata configured tranformation.
>
> Steve
>
>
>>>> Paul Spencer <pspencer at DMSOLUTIONS.CA> 03/22/07 2:50 PM >>>
> MapServer devs ...
>
> DMSG has a contract with Environment Canada/Tom Kralidis to implement
> some changes to the Sensor Observation Service.
>
> Part of this contract is to write a technical report on the
> feasibility of introducing the ability to transform XML coming out of
> MapServer's various OGC interfaces (specifically WMS GetFeatureInfo,
> WFS GetFeature and SOS GetObservation) using an XSLT directly in
> MapServer. I (with help from Assefa) have prepared a draft of this
> report structured more-or-less like an RFC.
>
> One of the requirements is to 'gauge the reaction of the TSC to this
> proposal', which is the purpose of this email. If the general
> feedback from you is positive, I believe Tom will then submit this as
> an RFC for approval and, if approved, will seek funding to implement
> the changes.
>
> Please give the attached proposal a read over and let me know:
>
> * would you be for or against this capability being introduced into
> mapserver?
>
> * if against, would any rewording or additional detail change your
> mind?
>
> * is there enough technical detail in the proposal?
>
> * is there anything obvious missing (I know I may have missed some
> files that would need to be modified)?
>
> * any other comments?
>
> I know this is last minute, but the report is due for mid next week
> so getting your feedback ASAP is highly desirable.
>
> Cheers
>
> Paul
>
> +-----------------------------------------------------------------+
> |Paul Spencer pspencer at dmsolutions.ca |
> +-----------------------------------------------------------------+
> |Chief Technology Officer |
> |DM Solutions Group Inc http://www.dmsolutions.ca/ |
> +-----------------------------------------------------------------+
>
> Mapserver Output Transformation Extension (MOTE)
>
> Motivation
>
> The purpose of MOTE is to allow for the manipulation of XML output
> from MapServer through an XSLT transformation. Existing interfaces
> that output XML are the WMS GetFeatureInfo (with appropriate mime
> type), WFS server and SOS server.
>
> Solutions external to MapServer are possible using other technology.
> This can be undesirable for several reasons:
>
> * portability
> * performance
> * additional points of failure for a service
> * additional complexity for IT installation, setup, and maintenance
> * potential exposure of the original XML if clients can connect
> directly to MapServer
>
> This type of conversion could also be handled in the WxS scripting
> interface of mapscript. As with an external solution, this requires
> additional complexity, overhead and skill sets beyond the minimum
> required to implement this kind of transformation.
>
> Introducing this capability directly in MapServer avoids these
> problems.
>
> Technical approach
>
> This work will be done by adding the option to include libxslt -
> http://xmlsoft.org/XSLT/index.html - in the configure/build stage and
> by adding map and/or layer level metadata that specifies an external
> XSLT file to use for processing the final output.
>
> The work will require that all existing, related XML output methods
> be converted from msIO_printf to libxml2. Note that the SOS Server
> already uses libxml2 and would not need to be converted.
>
> In addition to the mandatory changes to functions that output XML
> that would need to be transformed, it may be desirable to convert
> other functions that output XML to use libxml2 and to share/reuse
> common code (such as exception output).
>
> The potential to apply XSLT transformations would be introduced into
> WMS GetFeatureInfo (for the appropriate mime types), WFS GetFeature,
> and SOS GetObservation service capabilities. In each of these
> components, as each layer is processed, the resulting xml node may be
> transformed if an XSLT is specified at the layer level. The final
> step before outputting the root node as XML would be to apply a map
> level transform if an XSLT is specified at the web metadata level.
>
> An XSLT document may be referenced in the file system using either a
> relative path (relative to the map file) or an absolute path, or via
> a URL.
>
> Error Handling
>
> Suitable error handling will be specified for the following
> conditions:
>
> * the XSLT document could not be loaded because it does not exist at
> the specified location. This should be a non-fatal error condition
> that results in a notice or warning being emitted via normal logging
> functions
> * the XSLT document could not be loaded because it is invalid. This
> is caused by a syntax error in the XSLT document that prevents
> libxslt from loading the XSLT document. This should be a non-fatal
> error condition that results in a notice or warning being emitted via
> normal logging functions
> * an exception occurred processing the XSLT document. Where
> possible, exceptions will be trapped and converted into meaningful
> error messages emitted via normal logging channels and processing
> should continue with untransformed output.
>
> All error messages shall include as much contextual information and
> information from the underlying subsystem as possible to assist in
> fixing problems with XSLT transformations.
>
> Additional Libraries
>
> A dependency on libxslt (with a prerequisite to have libxml2) would
> be introduced for WMS, WFS and SOS builds.
>
> Files and objects affected
>
> The following files will be affected by this work.
>
> configure/make
>
> The mapserver build system for windows and linux will be modified to
> include a dependency on libxslt if appropriately configured.
>
> mapwfs.c
>
> Only msWFSGetFeature would require conversion, although other
> functions could be converted for consistency.
>
> mapgml.c
>
> All functions need to be converted. Existing functions that call
> msIO_printf would create or add to an xmlNode object and return the
> result.
>
> mapwms.c
>
> Only msWMSFeatureInfo would require conversion, although other
> functions could be converted for consistency.
>
> mapwcs.c
>
> No functions in mapwcs.c require conversion, but it may be desirable
> to convert them for consistency with other OGC services.
>
> Backwards compatibility issues
>
> There are no backwards compatibility issues since adding libxslt is a
> build-time configuration option and there are no new or changed
> keywords in the mapfile syntax.
>
> Implementation issues
>
> Documentation and examples will be provided as part of the
> implementation of this work, in the form of updates to existing How
> To documents for WMS, WFS and SOS, and a new How To dealing
> specifically with the XSLT component.
>
> There should be mapscript no implications.
>
> There may be build considerations for php_mapscript versions if PHP
> is loading libxslt as well.
>
> Test suite
>
> msautotest scripts will be extended to include tests for this
> functionality, including normal processing cases (without XSLT) and
> each combination of possibilities of XSLT processing including all
> known sources of erroneous input.
>
> Examples
>
> Sample XML Output
>
> <Xxml version='1.0' encoding="ISO-8859-1" ?>
> <owscat:msFeatureCollection
> version="1.0.0"
> xmlns:owscat="http://www.ec.gc.ca/owscat"
> xmlns:gml="http://www.opengis.net/gml"
> xmlns:ogc="http://www.opengis.net/ogc"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.ec.gc.ca/owscat http://
> devgeo.cciw.ca/cgi-bin/mapserv/owscat?
> SERVICE=WFS&VERSION=1.0.0&REQUEST=DescribeFeatureType&TYPENA
> ME=service_resources&OUTPUTFORMAT=SFE_XMLSCHEMA">
> <gml:boundedBy>
> <gml:Envelope srsName="EPSG:4326">
> <gml:pos>-180.000000 -90.000000</gml:pos>
> <gml:pos>180.000000 90.000000</gml:pos>
> </gml:Envelope>
> </gml:boundedBy>
> <gml:featureMember>
>
> <owscat:service_resources>
> <gml:boundedBy>
> <gml:Envelope srsName="EPSG:4326">
> <gml:pos>-180.000000 -90.000000</gml:pos>
> <gml:pos>180.000000 90.000000</gml:pos>
> </gml:Envelope>
> </gml:boundedBy>
> <owscat:msGeometry>
>
> <gml:Polygon srsName="EPSG:4326">
> <gml:exterior>
> <gml:LinearRing>
> <gml:posList srsDimension="2">-180.000000 -90.000000
> -180.000000 90.000000 180.000000 90.000000 180.000000 -90.000000
> -180.000000 -90.000000 </gml:posList>
> </gml:LinearRing>
> </gml:exterior>
> </gml:Polygon>
> </owscat:msGeometry>
>
> <owscat:lang>en-CA</owscat:lang>
> <owscat:organization>NASA</owscat:organization>
> <owscat:logourl></owscat:logourl>
> <owscat:endpoint_getresource>http://aes.gsfc.nasa.gov/cgi-
> bin/wms?</owscat:endpoint_getresource>
> <owscat:service_type>OGC:WMS</owscat:service_type>
> <owscat:service_version>1.1.1</owscat:service_version>
>
> <owscat:resource_id>3572</owscat:resource_id>
> <owscat:service_id>111</owscat:service_id>
> <owscat:name>2916_21230</owscat:name>
> <owscat:title>Complete Earth (2048x1024 Image)</owscat:title>
> <owscat:abstract>This image of Earth's city lights
> was created with data from the Defense Meteorological Satellite
> Program (DMSP) Operational Linescan System (OLS). Originally designed
> to view clouds by moonlight, the OLS is also used to map the
> locations of permanent lights on the Earth's surface.
> The brightest areas of the Earth are the most urbanized, but not
> necessarily the most populated. (Compare western Europe with China
> and India.) Cities tend to grow along coastlines and transportation
> networks. Even without the underlying map, the outlines of many
> continents would still be visible. The United States interstate
> highway system appears as a lattice connecting the brighter dots of
> city centers. In Russia, the Trans-Siberian railroad is a thin line
> stretching from Moscow through the center of Asia to Vladivostok. The
> Nile River, from the Aswan Dam to the Mediterranean Sea, is another
> bright thread through an otherwise dark region.
> Even more than 100 years after the invention of the electric light,
> some regions remain thinly populated and unlit. Antarctica is
> entirely dark. The interior jungles of Africa and South America are
> mostly dark, but lights are beginning to appear there. Deserts in
> Africa, Arabia, Australia, Mongolia, and the United States are poorly
> lit as well (except along the coast), along with the boreal forests
> of Canada and Russia, and the great mountains of the Himalaya.</
> owscat:abstract>
>
> <owscat:keywords>Human geography,Lights,Urbanization,EARTH
> SCIENCE:Human Dimensions:Population:Population Distribution</
> owscat:keywords>
> <owscat:dataurl></owscat:dataurl>
> <owscat:metadataurl>http://svs.gsfc.nasa.gov/vis/a000000/
> a002900/a002916/a002916.fgdc</owscat:metadataurl>
> <owscat:legendurl></owscat:legendurl>
> <owscat:scale_min>-99</owscat:scale_min>
> <owscat:scale_max>-99</owscat:scale_max>
> <owscat:srs>EPSG:4326</owscat:srs>
>
> <owscat:format_list>image/png</owscat:format_list>
> <owscat:style_list>opaque</owscat:style_list>
> <owscat:time_extent></owscat:time_extent>
> <owscat:queryable>0</owscat:queryable>
> </owscat:service_resources>
> </gml:featureMember>
> </owscat:msFeatureCollection>
>
> Sample XLST
>
> <Xxml version="1.0" encoding="iso-8859-1"?>
> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/
> Transform" xmlns:owscat="http://www.ec.gc.ca/owscat"
> xmlns:gml="http://www.opengis.net/gml">
>
> <xsl:output indent="yes" medhod="xml"/>
>
> <xsl:template match="/">
> <records>
> <xsl:apply-templates/>
> </records>
> </xsl:template>
>
> <xsl:template match="gml:featureMember/owscat:service_resources">
> <record>
> <name><xsl:value-of select="owscat:name"/></name>
> <title><xsl:value-of select="owscat:title"/></title>
> <abstract><xsl:value-of select="owscat:abstract"/></abstract>
> </record>
> </xsl:template>
>
> </xsl:stylesheet>
>
> Example Result
>
> <Xxml version="1.0"?>
> <records xmlns:owscat="http://www.ec.gc.ca/owscat" xmlns:gml="http://
> www.opengis.net/gml">
> <record><name>2916_21230</name><title>Complete Earth
> (2048x1024 Image)</title><abstract>This image of Earth's
> city lights was created with data from the Defense Meteorological
> Satellite Program (DMSP) Operational Linescan System (OLS).
> Originally designed to view clouds by moonlight, the OLS is also used
> to map the locations of permanent lights on the Earth's
> surface.
> The brightest areas of the Earth are the most urbanized, but not
> necessarily the most populated. (Compare western Europe with China
> and India.) Cities tend to grow along coastlines and transportation
> networks. Even without the underlying map, the outlines of many
> continents would still be visible. The United States interstate
> highway system appears as a lattice connecting the brighter dots of
> city centers. In Russia, the Trans-Siberian railroad is a thin line
> stretching from Moscow through the center of Asia to Vladivostok. The
> Nile River, from the Aswan Dam to the Mediterranean Sea, is another
> bright thread through an otherwise dark region.
> Even more than 100 years after the invention of the electric light,
> some regions remain thinly populated and unlit. Antarctica is
> entirely dark. The interior jungles of Africa and South America are
> mostly dark, but lights are beginning to appear there. Deserts in
> Africa, Arabia, Australia, Mongolia, and the United States are poorly
> lit as well (except along the coast), along with the boreal forests
> of Canada and Russia, and the great mountains of the Himalaya.</
> abstract></record>
> </records>
>
+-----------------------------------------------------------------+
|Paul Spencer pspencer at dmsolutions.ca |
+-----------------------------------------------------------------+
|Chief Technology Officer |
|DM Solutions Group Inc http://www.dmsolutions.ca/ |
+-----------------------------------------------------------------+
More information about the mapserver-dev
mailing list