request for comment: adding xslt processing to mapserver

[Stephen Woodbridge]

Paul Spencer wrote:
> Thanks for your feedback, Steve.  A couple of comments inline.
> 
> On 24-Mar-07, at 11:14 AM, Stephen Woodbridge wrote:
> 
>> Paul,
>>
>> I am somewhat neutral on this proposal, since I don't see a general 
>> need for it. I can see how it might have some benefits for what I 
>> think is a generally small audience.
>>
>> My basic concerns are:
>>
>> 1) this seems to add significant complexity for the user
>>    - mitigated by optional inclusion
>>    - mitigated partially by the additional documentation
>>    - technically this does not seem overly complex to add
> 
> only for those that want to use it.  I would consider this a feature 
> that is not for the faint of heart!
> 
>> 2) limited audience (at least that is my perception)
> 
> Anyone using MapServer for OGC services could be potentially benefited.  
> I'm not sure how many people that is, but its a non-trivial number :)

Ahh, that should be Mapserver OGC users that are faint of heart! :) but 
this is directly to my point of needing some concrete examples.

>> 3) additional drag on the overall development/test/release process
>>    - mitigated partially by the added msautotest
> 
> Not sure I understand your concern on this?  Do you think it would be 
> more difficult to build code using libxml2 rather than using 
> msIO_printf?  I believe that it will result in less erroneous output 
> since the structure and formatting of the output will be handled in a 
> consistent and reliable manner.  The current approach is a ripe breeding 
> ground for all kinds of problems.  Once there is a sufficient base of 
> code built on libxml2, I think it will be quite easy to work with for 
> most of the core developers.

All I mean here is that every feature that is added to mapserver has to 
be tested on every future release, coded around or coded into all future 
developments that rub up against it, etc for every future releases of 
mapserver. This complexity and additional cost does not go away with 
conditional builds. If the feature being added is complex, difficult to 
understand, etc then is slows down future progress of mapserver when we 
want to add mainstream features. "We" typically (explicitly or not) 
weigh these costs against the benefits of the feature to all users.

Anyway, you make good points of the potential code benefits and I will 
deffer to the core developers on this point. I just wanted to get it on 
the table for discussion.

>> 4) Steve L raised some issues around WFS that I would like to see 
>> addressed in the RFC
> 
> I am adding notes about:
> 
> * limitations of using XSLT (entire doc in memory)
> * requiring incoming transformations for WFS GetFeature
> * that schemas for transformed WFS GetFeature results would be handled 
> by the implementor (i.e. they provide the xslt or external reference to 
> the schema)
> 
>> 5) Would you be targeting this for 5.0 release? What if any impact 
>> would you see to the release if this was included?
> 
> Given that MapServer 5.0 is currently looking at a May timeframe, I 
> doubt this would be targetted for 5.0.  This effort is to build a 
> reasonable case that can be used to obtain funding post March 31st.  
> Given the way funding and budgets go, I would suspect it would be 6-12 
> months before this would be brought to implementation assuming it is 
> technically approved.

OK, this makes sense, I just thought it might make sense to include 
comment on the expected time frame realizing that it might change.

>> I don't think I would veto this.
>>
>> Something that you could add to the RFC that might get me more excited 
>> is something like examples/benefits that are more concrete. Like what 
>> this would give a mapserver user that they don't have today. Adding 
>> embedded XSLT to transform your output is pretty abstract conceptually.
>>
>> How might this allow a user to do something neat with OpenLayers and 
>> WFS that I can't easily do with the existing code? Or something else 
>> along the lines of showing benefits to the users.
> 
> Good points.  I won't have anything like this before I hand the results 
> off, but I will include a recommendation that concrete examples be 
> provided before submitting the RFC.

Great, I think you will find that it is easier to move forward with an 
idea that people can see has benefits and the broader the benefits the 
better. If the primary benefit is easier, faster OGC standards support 
from the core developers, then lets state that and get it validated by 
them. On the other hand if it will only help a handful of "not faint of 
heart" XSLT experts, then I'm not so sure.

BTW, Good job on the initial write up.

-Steve W

>>
>> -Steve W
>>
>> Paul Spencer wrote:
>>> MapServer devs ...
>>> DMSG has a contract with Environment Canada/Tom Kralidis to implement 
>>> some changes to the Sensor Observation Service.
>>> Part of this contract is to write a technical report on the 
>>> feasibility of introducing the ability to transform XML coming out of 
>>> MapServer's various OGC interfaces (specifically WMS GetFeatureInfo, 
>>> WFS GetFeature and SOS GetObservation) using an XSLT directly in 
>>> MapServer.  I (with help from Assefa) have prepared a draft of this 
>>> report structured more-or-less like an RFC.
>>> One of the requirements is to 'gauge the reaction of the TSC to this 
>>> proposal', which is the purpose of this email. If the general 
>>> feedback from you is positive, I believe Tom will then submit this as 
>>> an RFC for approval and, if approved, will seek funding to implement 
>>> the changes.
>>> Please give the attached proposal a read over and let me know:
>>> * would you be for or against this capability being introduced into 
>>> mapserver?
>>> * if against, would any rewording or additional detail change your mind?
>>> * is there enough technical detail in the proposal?
>>> * is there anything obvious missing (I know I may have missed some 
>>> files that would need to be modified)?
>>> * any other comments?
>>> I know this is last minute, but the report is due for mid next week 
>>> so getting your feedback ASAP is highly desirable.
>>> Cheers
>>> Paul
>>> +-----------------------------------------------------------------+
>>> |Paul Spencer                          pspencer at dmsolutions.ca    |
>>> +-----------------------------------------------------------------+
>>> |Chief Technology Officer                                         |
>>> |DM Solutions Group Inc                http://www.dmsolutions.ca/ |
>>> +-----------------------------------------------------------------+
>>> Mapserver Output Transformation Extension (MOTE)
>>> Motivation
>>> The purpose of MOTE is to allow for the manipulation of XML output 
>>> from MapServer through an XSLT transformation.  Existing interfaces 
>>> that output XML are the WMS GetFeatureInfo (with appropriate mime 
>>> type), WFS server and SOS server.
>>> Solutions external to MapServer are possible using other technology.  
>>> This can be undesirable for several reasons:
>>> * portability
>>> * performance
>>> * additional points of failure for a service
>>> * additional complexity for IT installation, setup, and maintenance
>>> * potential exposure of the original XML if clients can connect 
>>> directly to MapServer
>>> This type of conversion could also be handled in the WxS scripting 
>>> interface of mapscript.   As with an external solution, this requires 
>>> additional complexity, overhead and skill sets beyond the minimum 
>>> required to implement this kind of transformation.
>>> Introducing this capability directly in MapServer avoids these problems.
>>> Technical approach
>>> This work will be done by adding the option to include libxslt - 
>>> http://xmlsoft.org/XSLT/index.html - in the configure/build stage and 
>>> by adding map and/or layer level metadata that specifies an external 
>>> XSLT file to use for processing the final output.
>>> The work will require that all existing, related XML output methods 
>>> be converted from msIO_printf to libxml2.  Note that the SOS Server 
>>> already uses libxml2 and would not need to be converted.
>>> In addition to the mandatory changes to functions that output XML 
>>> that would need to be transformed, it may be desirable to convert 
>>> other functions that output XML to use libxml2 and to share/reuse 
>>> common code (such as exception output).
>>> The potential to apply XSLT transformations would be introduced into 
>>> WMS GetFeatureInfo (for the appropriate mime types), WFS GetFeature, 
>>> and SOS GetObservation service capabilities.  In each of these 
>>> components, as each layer is processed, the resulting xml node may be 
>>> transformed if an XSLT is specified at the layer level.  The final 
>>> step before outputting the root node as XML would be to apply a map 
>>> level transform if an XSLT is specified at the web metadata level.
>>> An XSLT document may be referenced in the file system using either a 
>>> relative path (relative to the map file) or an absolute path, or via 
>>> a URL.
>>> Error Handling
>>> Suitable error handling will be specified for the following conditions:
>>> * the XSLT document could not be loaded because it does not exist at 
>>> the specified location.  This should be a non-fatal error condition 
>>> that results in a notice or warning being emitted via normal logging 
>>> functions
>>> * the XSLT document could not be loaded because it is invalid.  This 
>>> is caused by a syntax error in the XSLT document that prevents 
>>> libxslt from loading the XSLT document.  This should be a non-fatal 
>>> error condition that results in a notice or warning being emitted via 
>>> normal logging functions
>>> * an exception occurred processing the XSLT document.  Where 
>>> possible, exceptions will be trapped and converted into meaningful 
>>> error messages emitted via normal logging channels and processing 
>>> should continue with untransformed output.
>>> All error messages shall include as much contextual information and 
>>> information from the underlying subsystem as possible to assist in 
>>> fixing problems with XSLT transformations.
>>> Additional Libraries
>>> A dependency on libxslt (with a prerequisite to have libxml2) would 
>>> be introduced for WMS, WFS and SOS builds.
>>> Files and objects affected
>>> The following files will be affected by this work.
>>> configure/make
>>> The mapserver build system for windows and linux will be modified to 
>>> include a dependency on libxslt if appropriately configured.
>>> mapwfs.c
>>> Only msWFSGetFeature would require conversion, although other 
>>> functions could be converted for consistency.
>>> mapgml.c
>>> All functions need to be converted.  Existing functions that call 
>>> msIO_printf would create or add to an xmlNode object and return the 
>>> result.
>>> mapwms.c
>>> Only msWMSFeatureInfo would require conversion, although other 
>>> functions could be converted for consistency.
>>> mapwcs.c
>>> No functions in mapwcs.c require conversion, but it may be desirable 
>>> to convert them for consistency with other OGC services.
>>> Backwards compatibility issues
>>> There are no backwards compatibility issues since adding libxslt is a 
>>> build-time configuration option and there are no new or changed 
>>> keywords in the mapfile syntax.
>>> Implementation issues
>>> Documentation and examples will be provided as part of the 
>>> implementation of this work, in the form of updates to existing How 
>>> To documents for WMS, WFS and SOS, and a new How To dealing 
>>> specifically with the XSLT component.
>>> There should be mapscript no implications.
>>> There may be build considerations for php_mapscript versions if PHP 
>>> is loading libxslt as well.
>>> Test suite
>>> msautotest scripts will be extended to include tests for this 
>>> functionality, including normal processing cases (without XSLT) and 
>>> each combination of possibilities of XSLT processing including all 
>>> known sources of erroneous input.
>>> Examples
>>> Sample XML Output
>>> <?xml version='1.0' encoding="ISO-8859-1" ?>
>>> <owscat:msFeatureCollection
>>>    version="1.0.0"
>>>    xmlns:owscat="http://www.ec.gc.ca/owscat"
>>>    xmlns:gml="http://www.opengis.net/gml"
>>>    xmlns:ogc="http://www.opengis.net/ogc"
>>>    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>    xsi:schemaLocation="http://www.ec.gc.ca/owscat 
>>> http://devgeo.cciw.ca/cgi-bin/mapserv/owscat?SERVICE=WFS&amp;VERSION=1.0.0&amp;REQUEST=DescribeFeatureType&amp;TYPENAME=service_resources&amp;OUTPUTFORMAT=SFE_XMLSCHEMA">       
>>> <gml:boundedBy>
>>>            <gml:Envelope srsName="EPSG:4326">
>>>                 <gml:pos>-180.000000 -90.000000</gml:pos>
>>>                 <gml:pos>180.000000 90.000000</gml:pos>
>>>            </gml:Envelope>
>>>       </gml:boundedBy>
>>>     <gml:featureMember>
>>>       <owscat:service_resources>
>>>         <gml:boundedBy>
>>>              <gml:Envelope srsName="EPSG:4326">
>>>                   <gml:pos>-180.000000 -90.000000</gml:pos>
>>>                   <gml:pos>180.000000 90.000000</gml:pos>
>>>              </gml:Envelope>
>>>         </gml:boundedBy>
>>>         <owscat:msGeometry>
>>>           <gml:Polygon srsName="EPSG:4326">
>>>             <gml:exterior>
>>>               <gml:LinearRing>
>>>                 <gml:posList srsDimension="2">-180.000000 -90.000000 
>>> -180.000000 90.000000 180.000000 90.000000 180.000000 -90.000000 
>>> -180.000000 -90.000000 </gml:posList>
>>>               </gml:LinearRing>
>>>             </gml:exterior>
>>>           </gml:Polygon>
>>>         </owscat:msGeometry>
>>>         <owscat:lang>en-CA</owscat:lang>
>>>         <owscat:organization>NASA</owscat:organization>
>>>         <owscat:logourl></owscat:logourl>
>>>         
>>> <owscat:endpoint_getresource>http://aes.gsfc.nasa.gov/cgi-bin/wms?</owscat:endpoint_getresource>         
>>> <owscat:service_type>OGC:WMS</owscat:service_type>
>>>         <owscat:service_version>1.1.1</owscat:service_version>
>>>         <owscat:resource_id>3572</owscat:resource_id>
>>>         <owscat:service_id>111</owscat:service_id>
>>>         <owscat:name>2916_21230</owscat:name>
>>>         <owscat:title>Complete Earth (2048x1024 Image)</owscat:title>
>>>         <owscat:abstract>This image of Earth&amp;#039;s city lights 
>>> was created with data from the Defense Meteorological Satellite 
>>> Program (DMSP) Operational Linescan System (OLS). Originally designed 
>>> to view clouds by moonlight, the OLS is also used to map the 
>>> locations of permanent lights on the Earth&amp;#039;s surface.
>>> The brightest areas of the Earth are the most urbanized, but not 
>>> necessarily the most populated. (Compare western Europe with China 
>>> and India.) Cities tend to grow along coastlines and transportation 
>>> networks. Even without the underlying map, the outlines of many 
>>> continents would still be visible. The United States interstate 
>>> highway system appears as a lattice connecting the brighter dots of 
>>> city centers. In Russia, the Trans-Siberian railroad is a thin line 
>>> stretching from Moscow through the center of Asia to Vladivostok. The 
>>> Nile River, from the Aswan Dam to the Mediterranean Sea, is another 
>>> bright thread through an otherwise dark region.
>>> Even more than 100 years after the invention of the electric light, 
>>> some regions remain thinly populated and unlit. Antarctica is 
>>> entirely dark. The interior jungles of Africa and South America are 
>>> mostly dark, but lights are beginning to appear there. Deserts in 
>>> Africa, Arabia, Australia, Mongolia, and the United States are poorly 
>>> lit as well (except along the coast), along with the boreal forests 
>>> of Canada and Russia, and the great mountains of the 
>>> Himalaya.</owscat:abstract>
>>>         <owscat:keywords>Human geography,Lights,Urbanization,EARTH 
>>> SCIENCE:Human Dimensions:Population:Population 
>>> Distribution</owscat:keywords>
>>>         <owscat:dataurl></owscat:dataurl>
>>>         
>>> <owscat:metadataurl>http://svs.gsfc.nasa.gov/vis/a000000/a002900/a002916/a002916.fgdc</owscat:metadataurl>         
>>> <owscat:legendurl></owscat:legendurl>
>>>         <owscat:scale_min>-99</owscat:scale_min>
>>>         <owscat:scale_max>-99</owscat:scale_max>
>>>         <owscat:srs>EPSG:4326</owscat:srs>
>>>         <owscat:format_list>image/png</owscat:format_list>
>>>         <owscat:style_list>opaque</owscat:style_list>
>>>         <owscat:time_extent></owscat:time_extent>
>>>         <owscat:queryable>0</owscat:queryable>
>>>       </owscat:service_resources>
>>>     </gml:featureMember>
>>> </owscat:msFeatureCollection>
>>> Sample XLST
>>> <?xml version="1.0" encoding="iso-8859-1"?>
>>> <xsl:stylesheet version="1.0" 
>>> xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
>>> xmlns:owscat="http://www.ec.gc.ca/owscat" 
>>> xmlns:gml="http://www.opengis.net/gml">
>>> <xsl:output indent="yes" medhod="xml"/>
>>> <xsl:template match="/">
>>>  <records>
>>>     <xsl:apply-templates/>
>>>  </records>
>>> </xsl:template>
>>> <xsl:template match="gml:featureMember/owscat:service_resources">
>>>   <record>
>>>    <name><xsl:value-of select="owscat:name"/></name>
>>>    <title><xsl:value-of select="owscat:title"/></title>
>>>    <abstract><xsl:value-of select="owscat:abstract"/></abstract>
>>>   </record>
>>> </xsl:template>
>>> </xsl:stylesheet>
>>> Example Result
>>> <?xml version="1.0"?>
>>> <records xmlns:owscat="http://www.ec.gc.ca/owscat" 
>>> xmlns:gml="http://www.opengis.net/gml">
>>>       <record><name>2916_21230</name><title>Complete Earth (2048x1024 
>>> Image)</title><abstract>This image of Earth&amp;#039;s city lights 
>>> was created with data from the Defense Meteorological Satellite 
>>> Program (DMSP) Operational Linescan System (OLS). Originally designed 
>>> to view clouds by moonlight, the OLS is also used to map the 
>>> locations of permanent lights on the Earth&amp;#039;s surface.
>>> The brightest areas of the Earth are the most urbanized, but not 
>>> necessarily the most populated. (Compare western Europe with China 
>>> and India.) Cities tend to grow along coastlines and transportation 
>>> networks. Even without the underlying map, the outlines of many 
>>> continents would still be visible. The United States interstate 
>>> highway system appears as a lattice connecting the brighter dots of 
>>> city centers. In Russia, the Trans-Siberian railroad is a thin line 
>>> stretching from Moscow through the center of Asia to Vladivostok. The 
>>> Nile River, from the Aswan Dam to the Mediterranean Sea, is another 
>>> bright thread through an otherwise dark region.
>>> Even more than 100 years after the invention of the electric light, 
>>> some regions remain thinly populated and unlit. Antarctica is 
>>> entirely dark. The interior jungles of Africa and South America are 
>>> mostly dark, but lights are beginning to appear there. Deserts in 
>>> Africa, Arabia, Australia, Mongolia, and the United States are poorly 
>>> lit as well (except along the coast), along with the boreal forests 
>>> of Canada and Russia, and the great mountains of the 
>>> Himalaya.</abstract></record>
>>> </records>
>>
> 
> +-----------------------------------------------------------------+
> |Paul Spencer                          pspencer at dmsolutions.ca    |
> +-----------------------------------------------------------------+
> |Chief Technology Officer                                         |
> |DM Solutions Group Inc                http://www.dmsolutions.ca/ |
> +-----------------------------------------------------------------+