request for comment: adding xslt processing to mapserver

Paul Spencer pspencer at DMSOLUTIONS.CA
Sat Mar 24 17:14:17 EDT 2007


Steve,

I will add some notes about the streaming xslt thing.  This seems  
like it could be a showstopper.  I was not able to find anything  
useful on streaming XLST in a quick search, I will continue to search  
though.  Of course, if no one is able to do streaming transformations  
then this isn't a strong argument against including the ability to do  
transforms in mapserver since any external implementation would have  
similar problems.  In the case of per-layer transforms, it might  
actually be more efficient to do it in mapserver since you would only  
need to process the node containing the tree for that layer, one at a  
time.

re wfs schema, good point.  I'll include a note about schemas too.

re queries ... I had not thought about processing incoming XML data.   
I'll have to add something about that too.

Cheers

Paul


On 23-Mar-07, at 8:12 PM, Steve Lime wrote:

> Paul: I did look at libxml2/libxslt and it's XSLT support before  
> doing any of the
> work on GML transformation last summer. At that time I decided  
> against it
> because:
>
>  - performance, libxslt required an entire document be memory resident
> before transforming. You had to write vanilla GML to memory first  
> (as DOM
> object) and for large datasets that wouldn't be straightforward).  
> Streaming
> transformations were  needed and, at the time, only available via  
> java (and
> experimental at that), I don't know if that has changed or not.
>
>  - with WFS, what you produce you need to document. So for the  
> transformed
> output you'd need to produce a corresponding schema, yet another  
> XSLT file.
> Referencing an external schema (e.g. some standard application  
> schema) is
> one workaround but I don't know that it is a perfect solution.
>
>  - with WFS, what you produce you also need to support queries  
> against, so
> now you need to transform an incoming filter into something that  
> that can be
> applied against the native data store, basically inverting the  
> transformation
> applied in the first place. Then you also have to deal with non-xml  
> request
> encoding.
>
> In the end it was all too much to swallow, not to mention that XSLT  
> isn't exactly
> trivial for users. Not that configuring GML transformations through  
> metadata is
> easy- it ain't, it sucks and overly complicates a mapfile.
>
> Still there's an elegance to only having to write construct (and  
> support!) vanilla
> GML and then using external transformations. If the issues above  
> could be addressed
> I would support the idea and might go one step further and suggest  
> yanking out
> the metadata configured tranformation.
>
> Steve
>
>
>>>> Paul Spencer <pspencer at DMSOLUTIONS.CA> 03/22/07 2:50 PM >>>
> MapServer devs ...
>
> DMSG has a contract with Environment Canada/Tom Kralidis to implement
> some changes to the Sensor Observation Service.
>
> Part of this contract is to write a technical report on the
> feasibility of introducing the ability to transform XML coming out of
> MapServer's various OGC interfaces (specifically WMS GetFeatureInfo,
> WFS GetFeature and SOS GetObservation) using an XSLT directly in
> MapServer.  I (with help from Assefa) have prepared a draft of this
> report structured more-or-less like an RFC.
>
> One of the requirements is to 'gauge the reaction of the TSC to this
> proposal', which is the purpose of this email. If the general
> feedback from you is positive, I believe Tom will then submit this as
> an RFC for approval and, if approved, will seek funding to implement
> the changes.
>
> Please give the attached proposal a read over and let me know:
>
> * would you be for or against this capability being introduced into
> mapserver?
>
> * if against, would any rewording or additional detail change your  
> mind?
>
> * is there enough technical detail in the proposal?
>
> * is there anything obvious missing (I know I may have missed some
> files that would need to be modified)?
>
> * any other comments?
>
> I know this is last minute, but the report is due for mid next week
> so getting your feedback ASAP is highly desirable.
>
> Cheers
>
> Paul
>
> +-----------------------------------------------------------------+
> |Paul Spencer                          pspencer at dmsolutions.ca    |
> +-----------------------------------------------------------------+
> |Chief Technology Officer                                         |
> |DM Solutions Group Inc                http://www.dmsolutions.ca/ |
> +-----------------------------------------------------------------+
>
> Mapserver Output Transformation Extension (MOTE)
>
> Motivation
>
> The purpose of MOTE is to allow for the manipulation of XML output
> from MapServer through an XSLT transformation.  Existing interfaces
> that output XML are the WMS GetFeatureInfo (with appropriate mime
> type), WFS server and SOS server.
>
> Solutions external to MapServer are possible using other technology.
> This can be undesirable for several reasons:
>
> * portability
> * performance
> * additional points of failure for a service
> * additional complexity for IT installation, setup, and maintenance
> * potential exposure of the original XML if clients can connect
> directly to MapServer
>
> This type of conversion could also be handled in the WxS scripting
> interface of mapscript.   As with an external solution, this requires
> additional complexity, overhead and skill sets beyond the minimum
> required to implement this kind of transformation.
>
> Introducing this capability directly in MapServer avoids these  
> problems.
>
> Technical approach
>
> This work will be done by adding the option to include libxslt -
> http://xmlsoft.org/XSLT/index.html - in the configure/build stage and
> by adding map and/or layer level metadata that specifies an external
> XSLT file to use for processing the final output.
>
> The work will require that all existing, related XML output methods
> be converted from msIO_printf to libxml2.  Note that the SOS Server
> already uses libxml2 and would not need to be converted.
>
> In addition to the mandatory changes to functions that output XML
> that would need to be transformed, it may be desirable to convert
> other functions that output XML to use libxml2 and to share/reuse
> common code (such as exception output).
>
> The potential to apply XSLT transformations would be introduced into
> WMS GetFeatureInfo (for the appropriate mime types), WFS GetFeature,
> and SOS GetObservation service capabilities.  In each of these
> components, as each layer is processed, the resulting xml node may be
> transformed if an XSLT is specified at the layer level.  The final
> step before outputting the root node as XML would be to apply a map
> level transform if an XSLT is specified at the web metadata level.
>
> An XSLT document may be referenced in the file system using either a
> relative path (relative to the map file) or an absolute path, or via
> a URL.
>
> Error Handling
>
> Suitable error handling will be specified for the following  
> conditions:
>
> * the XSLT document could not be loaded because it does not exist at
> the specified location.  This should be a non-fatal error condition
> that results in a notice or warning being emitted via normal logging
> functions
> * the XSLT document could not be loaded because it is invalid.  This
> is caused by a syntax error in the XSLT document that prevents
> libxslt from loading the XSLT document.  This should be a non-fatal
> error condition that results in a notice or warning being emitted via
> normal logging functions
> * an exception occurred processing the XSLT document.  Where
> possible, exceptions will be trapped and converted into meaningful
> error messages emitted via normal logging channels and processing
> should continue with untransformed output.
>
> All error messages shall include as much contextual information and
> information from the underlying subsystem as possible to assist in
> fixing problems with XSLT transformations.
>
> Additional Libraries
>
> A dependency on libxslt (with a prerequisite to have libxml2) would
> be introduced for WMS, WFS and SOS builds.
>
> Files and objects affected
>
> The following files will be affected by this work.
>
> configure/make
>
> The mapserver build system for windows and linux will be modified to
> include a dependency on libxslt if appropriately configured.
>
> mapwfs.c
>
> Only msWFSGetFeature would require conversion, although other
> functions could be converted for consistency.
>
> mapgml.c
>
> All functions need to be converted.  Existing functions that call
> msIO_printf would create or add to an xmlNode object and return the
> result.
>
> mapwms.c
>
> Only msWMSFeatureInfo would require conversion, although other
> functions could be converted for consistency.
>
> mapwcs.c
>
> No functions in mapwcs.c require conversion, but it may be desirable
> to convert them for consistency with other OGC services.
>
> Backwards compatibility issues
>
> There are no backwards compatibility issues since adding libxslt is a
> build-time configuration option and there are no new or changed
> keywords in the mapfile syntax.
>
> Implementation issues
>
> Documentation and examples will be provided as part of the
> implementation of this work, in the form of updates to existing How
> To documents for WMS, WFS and SOS, and a new How To dealing
> specifically with the XSLT component.
>
> There should be mapscript no implications.
>
> There may be build considerations for php_mapscript versions if PHP
> is loading libxslt as well.
>
> Test suite
>
> msautotest scripts will be extended to include tests for this
> functionality, including normal processing cases (without XSLT) and
> each combination of possibilities of XSLT processing including all
> known sources of erroneous input.
>
> Examples
>
> Sample XML Output
>
> <Xxml version='1.0' encoding="ISO-8859-1" ?>
> <owscat:msFeatureCollection
>     version="1.0.0"
>     xmlns:owscat="http://www.ec.gc.ca/owscat"
>     xmlns:gml="http://www.opengis.net/gml"
>     xmlns:ogc="http://www.opengis.net/ogc"
>     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>     xsi:schemaLocation="http://www.ec.gc.ca/owscat http://
> devgeo.cciw.ca/cgi-bin/mapserv/owscat?
> SERVICE=WFS&VERSION=1.0.0&REQUEST=DescribeFeatureType&TYPENA
> ME=service_resources&OUTPUTFORMAT=SFE_XMLSCHEMA">
>        <gml:boundedBy>
>             <gml:Envelope srsName="EPSG:4326">
>                  <gml:pos>-180.000000 -90.000000</gml:pos>
>                  <gml:pos>180.000000 90.000000</gml:pos>
>             </gml:Envelope>
>        </gml:boundedBy>
>      <gml:featureMember>
>
>        <owscat:service_resources>
>          <gml:boundedBy>
>               <gml:Envelope srsName="EPSG:4326">
>                    <gml:pos>-180.000000 -90.000000</gml:pos>
>                    <gml:pos>180.000000 90.000000</gml:pos>
>               </gml:Envelope>
>          </gml:boundedBy>
>          <owscat:msGeometry>
>
>            <gml:Polygon srsName="EPSG:4326">
>              <gml:exterior>
>                <gml:LinearRing>
>                  <gml:posList srsDimension="2">-180.000000 -90.000000
> -180.000000 90.000000 180.000000 90.000000 180.000000 -90.000000
> -180.000000 -90.000000 </gml:posList>
>                </gml:LinearRing>
>              </gml:exterior>
>            </gml:Polygon>
>          </owscat:msGeometry>
>
>          <owscat:lang>en-CA</owscat:lang>
>          <owscat:organization>NASA</owscat:organization>
>          <owscat:logourl></owscat:logourl>
>          <owscat:endpoint_getresource>http://aes.gsfc.nasa.gov/cgi-
> bin/wms?</owscat:endpoint_getresource>
>          <owscat:service_type>OGC:WMS</owscat:service_type>
>          <owscat:service_version>1.1.1</owscat:service_version>
>
>          <owscat:resource_id>3572</owscat:resource_id>
>          <owscat:service_id>111</owscat:service_id>
>          <owscat:name>2916_21230</owscat:name>
>          <owscat:title>Complete Earth (2048x1024 Image)</owscat:title>
>          <owscat:abstract>This image of Earth&#039;s city lights
> was created with data from the Defense Meteorological Satellite
> Program (DMSP) Operational Linescan System (OLS). Originally designed
> to view clouds by moonlight, the OLS is also used to map the
> locations of permanent lights on the Earth&#039;s surface.
> The brightest areas of the Earth are the most urbanized, but not
> necessarily the most populated. (Compare western Europe with China
> and India.) Cities tend to grow along coastlines and transportation
> networks. Even without the underlying map, the outlines of many
> continents would still be visible. The United States interstate
> highway system appears as a lattice connecting the brighter dots of
> city centers. In Russia, the Trans-Siberian railroad is a thin line
> stretching from Moscow through the center of Asia to Vladivostok. The
> Nile River, from the Aswan Dam to the Mediterranean Sea, is another
> bright thread through an otherwise dark region.
> Even more than 100 years after the invention of the electric light,
> some regions remain thinly populated and unlit. Antarctica is
> entirely dark. The interior jungles of Africa and South America are
> mostly dark, but lights are beginning to appear there. Deserts in
> Africa, Arabia, Australia, Mongolia, and the United States are poorly
> lit as well (except along the coast), along with the boreal forests
> of Canada and Russia, and the great mountains of the Himalaya.</
> owscat:abstract>
>
>          <owscat:keywords>Human geography,Lights,Urbanization,EARTH
> SCIENCE:Human Dimensions:Population:Population Distribution</
> owscat:keywords>
>          <owscat:dataurl></owscat:dataurl>
>          <owscat:metadataurl>http://svs.gsfc.nasa.gov/vis/a000000/
> a002900/a002916/a002916.fgdc</owscat:metadataurl>
>          <owscat:legendurl></owscat:legendurl>
>          <owscat:scale_min>-99</owscat:scale_min>
>          <owscat:scale_max>-99</owscat:scale_max>
>          <owscat:srs>EPSG:4326</owscat:srs>
>
>          <owscat:format_list>image/png</owscat:format_list>
>          <owscat:style_list>opaque</owscat:style_list>
>          <owscat:time_extent></owscat:time_extent>
>          <owscat:queryable>0</owscat:queryable>
>        </owscat:service_resources>
>      </gml:featureMember>
> </owscat:msFeatureCollection>
>
> Sample XLST
>
> <Xxml version="1.0" encoding="iso-8859-1"?>
> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/
> Transform" xmlns:owscat="http://www.ec.gc.ca/owscat"
> xmlns:gml="http://www.opengis.net/gml">
>
> <xsl:output indent="yes" medhod="xml"/>
>
> <xsl:template match="/">
>   <records>
>      <xsl:apply-templates/>
>   </records>
> </xsl:template>
>
> <xsl:template match="gml:featureMember/owscat:service_resources">
>    <record>
>     <name><xsl:value-of select="owscat:name"/></name>
>     <title><xsl:value-of select="owscat:title"/></title>
>     <abstract><xsl:value-of select="owscat:abstract"/></abstract>
>    </record>
> </xsl:template>
>
> </xsl:stylesheet>
>
> Example Result
>
> <Xxml version="1.0"?>
> <records xmlns:owscat="http://www.ec.gc.ca/owscat" xmlns:gml="http://
> www.opengis.net/gml">
>        <record><name>2916_21230</name><title>Complete Earth
> (2048x1024 Image)</title><abstract>This image of Earth&#039;s
> city lights was created with data from the Defense Meteorological
> Satellite Program (DMSP) Operational Linescan System (OLS).
> Originally designed to view clouds by moonlight, the OLS is also used
> to map the locations of permanent lights on the Earth&#039;s
> surface.
> The brightest areas of the Earth are the most urbanized, but not
> necessarily the most populated. (Compare western Europe with China
> and India.) Cities tend to grow along coastlines and transportation
> networks. Even without the underlying map, the outlines of many
> continents would still be visible. The United States interstate
> highway system appears as a lattice connecting the brighter dots of
> city centers. In Russia, the Trans-Siberian railroad is a thin line
> stretching from Moscow through the center of Asia to Vladivostok. The
> Nile River, from the Aswan Dam to the Mediterranean Sea, is another
> bright thread through an otherwise dark region.
> Even more than 100 years after the invention of the electric light,
> some regions remain thinly populated and unlit. Antarctica is
> entirely dark. The interior jungles of Africa and South America are
> mostly dark, but lights are beginning to appear there. Deserts in
> Africa, Arabia, Australia, Mongolia, and the United States are poorly
> lit as well (except along the coast), along with the boreal forests
> of Canada and Russia, and the great mountains of the Himalaya.</
> abstract></record>
> </records>
>

+-----------------------------------------------------------------+
|Paul Spencer                          pspencer at dmsolutions.ca    |
+-----------------------------------------------------------------+
|Chief Technology Officer                                         |
|DM Solutions Group Inc                http://www.dmsolutions.ca/ |
+-----------------------------------------------------------------+



More information about the mapserver-dev mailing list