request for comment: adding xslt processing to mapserver
Paul Spencer
pspencer at DMSOLUTIONS.CA
Sat Mar 24 17:23:46 EDT 2007
Thanks for your feedback, Steve. A couple of comments inline.
On 24-Mar-07, at 11:14 AM, Stephen Woodbridge wrote:
> Paul,
>
> I am somewhat neutral on this proposal, since I don't see a general
> need for it. I can see how it might have some benefits for what I
> think is a generally small audience.
>
> My basic concerns are:
>
> 1) this seems to add significant complexity for the user
> - mitigated by optional inclusion
> - mitigated partially by the additional documentation
> - technically this does not seem overly complex to add
only for those that want to use it. I would consider this a feature
that is not for the faint of heart!
> 2) limited audience (at least that is my perception)
Anyone using MapServer for OGC services could be potentially
benefited. I'm not sure how many people that is, but its a non-
trivial number :)
> 3) additional drag on the overall development/test/release process
> - mitigated partially by the added msautotest
Not sure I understand your concern on this? Do you think it would be
more difficult to build code using libxml2 rather than using
msIO_printf? I believe that it will result in less erroneous output
since the structure and formatting of the output will be handled in a
consistent and reliable manner. The current approach is a ripe
breeding ground for all kinds of problems. Once there is a
sufficient base of code built on libxml2, I think it will be quite
easy to work with for most of the core developers.
> 4) Steve L raised some issues around WFS that I would like to see
> addressed in the RFC
I am adding notes about:
* limitations of using XSLT (entire doc in memory)
* requiring incoming transformations for WFS GetFeature
* that schemas for transformed WFS GetFeature results would be
handled by the implementor (i.e. they provide the xslt or external
reference to the schema)
> 5) Would you be targeting this for 5.0 release? What if any impact
> would you see to the release if this was included?
Given that MapServer 5.0 is currently looking at a May timeframe, I
doubt this would be targetted for 5.0. This effort is to build a
reasonable case that can be used to obtain funding post March 31st.
Given the way funding and budgets go, I would suspect it would be
6-12 months before this would be brought to implementation assuming
it is technically approved.
>
> I don't think I would veto this.
>
> Something that you could add to the RFC that might get me more
> excited is something like examples/benefits that are more concrete.
> Like what this would give a mapserver user that they don't have
> today. Adding embedded XSLT to transform your output is pretty
> abstract conceptually.
>
> How might this allow a user to do something neat with OpenLayers
> and WFS that I can't easily do with the existing code? Or something
> else along the lines of showing benefits to the users.
Good points. I won't have anything like this before I hand the
results off, but I will include a recommendation that concrete
examples be provided before submitting the RFC.
>
> -Steve W
>
> Paul Spencer wrote:
>> MapServer devs ...
>> DMSG has a contract with Environment Canada/Tom Kralidis to
>> implement some changes to the Sensor Observation Service.
>> Part of this contract is to write a technical report on the
>> feasibility of introducing the ability to transform XML coming out
>> of MapServer's various OGC interfaces (specifically WMS
>> GetFeatureInfo, WFS GetFeature and SOS GetObservation) using an
>> XSLT directly in MapServer. I (with help from Assefa) have
>> prepared a draft of this report structured more-or-less like an RFC.
>> One of the requirements is to 'gauge the reaction of the TSC to
>> this proposal', which is the purpose of this email. If the general
>> feedback from you is positive, I believe Tom will then submit this
>> as an RFC for approval and, if approved, will seek funding to
>> implement the changes.
>> Please give the attached proposal a read over and let me know:
>> * would you be for or against this capability being introduced
>> into mapserver?
>> * if against, would any rewording or additional detail change your
>> mind?
>> * is there enough technical detail in the proposal?
>> * is there anything obvious missing (I know I may have missed some
>> files that would need to be modified)?
>> * any other comments?
>> I know this is last minute, but the report is due for mid next
>> week so getting your feedback ASAP is highly desirable.
>> Cheers
>> Paul
>> +-----------------------------------------------------------------+
>> |Paul Spencer pspencer at dmsolutions.ca |
>> +-----------------------------------------------------------------+
>> |Chief Technology Officer |
>> |DM Solutions Group Inc http://www.dmsolutions.ca/ |
>> +-----------------------------------------------------------------+
>> Mapserver Output Transformation Extension (MOTE)
>> Motivation
>> The purpose of MOTE is to allow for the manipulation of XML output
>> from MapServer through an XSLT transformation. Existing
>> interfaces that output XML are the WMS GetFeatureInfo (with
>> appropriate mime type), WFS server and SOS server.
>> Solutions external to MapServer are possible using other
>> technology. This can be undesirable for several reasons:
>> * portability
>> * performance
>> * additional points of failure for a service
>> * additional complexity for IT installation, setup, and maintenance
>> * potential exposure of the original XML if clients can connect
>> directly to MapServer
>> This type of conversion could also be handled in the WxS scripting
>> interface of mapscript. As with an external solution, this
>> requires additional complexity, overhead and skill sets beyond the
>> minimum required to implement this kind of transformation.
>> Introducing this capability directly in MapServer avoids these
>> problems.
>> Technical approach
>> This work will be done by adding the option to include libxslt -
>> http://xmlsoft.org/XSLT/index.html - in the configure/build stage
>> and by adding map and/or layer level metadata that specifies an
>> external XSLT file to use for processing the final output.
>> The work will require that all existing, related XML output
>> methods be converted from msIO_printf to libxml2. Note that the
>> SOS Server already uses libxml2 and would not need to be converted.
>> In addition to the mandatory changes to functions that output XML
>> that would need to be transformed, it may be desirable to convert
>> other functions that output XML to use libxml2 and to share/reuse
>> common code (such as exception output).
>> The potential to apply XSLT transformations would be introduced
>> into WMS GetFeatureInfo (for the appropriate mime types), WFS
>> GetFeature, and SOS GetObservation service capabilities. In each
>> of these components, as each layer is processed, the resulting xml
>> node may be transformed if an XSLT is specified at the layer
>> level. The final step before outputting the root node as XML
>> would be to apply a map level transform if an XSLT is specified at
>> the web metadata level.
>> An XSLT document may be referenced in the file system using either
>> a relative path (relative to the map file) or an absolute path, or
>> via a URL.
>> Error Handling
>> Suitable error handling will be specified for the following
>> conditions:
>> * the XSLT document could not be loaded because it does not exist
>> at the specified location. This should be a non-fatal error
>> condition that results in a notice or warning being emitted via
>> normal logging functions
>> * the XSLT document could not be loaded because it is invalid.
>> This is caused by a syntax error in the XSLT document that
>> prevents libxslt from loading the XSLT document. This should be a
>> non-fatal error condition that results in a notice or warning
>> being emitted via normal logging functions
>> * an exception occurred processing the XSLT document. Where
>> possible, exceptions will be trapped and converted into meaningful
>> error messages emitted via normal logging channels and processing
>> should continue with untransformed output.
>> All error messages shall include as much contextual information
>> and information from the underlying subsystem as possible to
>> assist in fixing problems with XSLT transformations.
>> Additional Libraries
>> A dependency on libxslt (with a prerequisite to have libxml2)
>> would be introduced for WMS, WFS and SOS builds.
>> Files and objects affected
>> The following files will be affected by this work.
>> configure/make
>> The mapserver build system for windows and linux will be modified
>> to include a dependency on libxslt if appropriately configured.
>> mapwfs.c
>> Only msWFSGetFeature would require conversion, although other
>> functions could be converted for consistency.
>> mapgml.c
>> All functions need to be converted. Existing functions that call
>> msIO_printf would create or add to an xmlNode object and return
>> the result.
>> mapwms.c
>> Only msWMSFeatureInfo would require conversion, although other
>> functions could be converted for consistency.
>> mapwcs.c
>> No functions in mapwcs.c require conversion, but it may be
>> desirable to convert them for consistency with other OGC services.
>> Backwards compatibility issues
>> There are no backwards compatibility issues since adding libxslt
>> is a build-time configuration option and there are no new or
>> changed keywords in the mapfile syntax.
>> Implementation issues
>> Documentation and examples will be provided as part of the
>> implementation of this work, in the form of updates to existing
>> How To documents for WMS, WFS and SOS, and a new How To dealing
>> specifically with the XSLT component.
>> There should be mapscript no implications.
>> There may be build considerations for php_mapscript versions if
>> PHP is loading libxslt as well.
>> Test suite
>> msautotest scripts will be extended to include tests for this
>> functionality, including normal processing cases (without XSLT)
>> and each combination of possibilities of XSLT processing including
>> all known sources of erroneous input.
>> Examples
>> Sample XML Output
>> <?xml version='1.0' encoding="ISO-8859-1" ?>
>> <owscat:msFeatureCollection
>> version="1.0.0"
>> xmlns:owscat="http://www.ec.gc.ca/owscat"
>> xmlns:gml="http://www.opengis.net/gml"
>> xmlns:ogc="http://www.opengis.net/ogc"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.ec.gc.ca/owscat http://
>> devgeo.cciw.ca/cgi-bin/mapserv/owscat?
>> SERVICE=WFS&VERSION=1.0.0&REQUEST=DescribeFeatureType&TYP
>> ENAME=service_resources&OUTPUTFORMAT=SFE_XMLSCHEMA">
>> <gml:boundedBy>
>> <gml:Envelope srsName="EPSG:4326">
>> <gml:pos>-180.000000 -90.000000</gml:pos>
>> <gml:pos>180.000000 90.000000</gml:pos>
>> </gml:Envelope>
>> </gml:boundedBy>
>> <gml:featureMember>
>> <owscat:service_resources>
>> <gml:boundedBy>
>> <gml:Envelope srsName="EPSG:4326">
>> <gml:pos>-180.000000 -90.000000</gml:pos>
>> <gml:pos>180.000000 90.000000</gml:pos>
>> </gml:Envelope>
>> </gml:boundedBy>
>> <owscat:msGeometry>
>> <gml:Polygon srsName="EPSG:4326">
>> <gml:exterior>
>> <gml:LinearRing>
>> <gml:posList srsDimension="2">-180.000000
>> -90.000000 -180.000000 90.000000 180.000000 90.000000 180.000000
>> -90.000000 -180.000000 -90.000000 </gml:posList>
>> </gml:LinearRing>
>> </gml:exterior>
>> </gml:Polygon>
>> </owscat:msGeometry>
>> <owscat:lang>en-CA</owscat:lang>
>> <owscat:organization>NASA</owscat:organization>
>> <owscat:logourl></owscat:logourl>
>> <owscat:endpoint_getresource>http://aes.gsfc.nasa.gov/cgi-
>> bin/wms?</owscat:endpoint_getresource>
>> <owscat:service_type>OGC:WMS</owscat:service_type>
>> <owscat:service_version>1.1.1</owscat:service_version>
>> <owscat:resource_id>3572</owscat:resource_id>
>> <owscat:service_id>111</owscat:service_id>
>> <owscat:name>2916_21230</owscat:name>
>> <owscat:title>Complete Earth (2048x1024 Image)</owscat:title>
>> <owscat:abstract>This image of Earth&#039;s city
>> lights was created with data from the Defense Meteorological
>> Satellite Program (DMSP) Operational Linescan System (OLS).
>> Originally designed to view clouds by moonlight, the OLS is also
>> used to map the locations of permanent lights on the
>> Earth&#039;s surface.
>> The brightest areas of the Earth are the most urbanized, but not
>> necessarily the most populated. (Compare western Europe with China
>> and India.) Cities tend to grow along coastlines and
>> transportation networks. Even without the underlying map, the
>> outlines of many continents would still be visible. The United
>> States interstate highway system appears as a lattice connecting
>> the brighter dots of city centers. In Russia, the Trans-Siberian
>> railroad is a thin line stretching from Moscow through the center
>> of Asia to Vladivostok. The Nile River, from the Aswan Dam to the
>> Mediterranean Sea, is another bright thread through an otherwise
>> dark region.
>> Even more than 100 years after the invention of the electric
>> light, some regions remain thinly populated and unlit. Antarctica
>> is entirely dark. The interior jungles of Africa and South America
>> are mostly dark, but lights are beginning to appear there. Deserts
>> in Africa, Arabia, Australia, Mongolia, and the United States are
>> poorly lit as well (except along the coast), along with the boreal
>> forests of Canada and Russia, and the great mountains of the
>> Himalaya.</owscat:abstract>
>> <owscat:keywords>Human geography,Lights,Urbanization,EARTH
>> SCIENCE:Human Dimensions:Population:Population Distribution</
>> owscat:keywords>
>> <owscat:dataurl></owscat:dataurl>
>> <owscat:metadataurl>http://svs.gsfc.nasa.gov/vis/a000000/
>> a002900/a002916/a002916.fgdc</owscat:metadataurl>
>> <owscat:legendurl></owscat:legendurl>
>> <owscat:scale_min>-99</owscat:scale_min>
>> <owscat:scale_max>-99</owscat:scale_max>
>> <owscat:srs>EPSG:4326</owscat:srs>
>> <owscat:format_list>image/png</owscat:format_list>
>> <owscat:style_list>opaque</owscat:style_list>
>> <owscat:time_extent></owscat:time_extent>
>> <owscat:queryable>0</owscat:queryable>
>> </owscat:service_resources>
>> </gml:featureMember>
>> </owscat:msFeatureCollection>
>> Sample XLST
>> <?xml version="1.0" encoding="iso-8859-1"?>
>> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/
>> XSL/Transform" xmlns:owscat="http://www.ec.gc.ca/owscat"
>> xmlns:gml="http://www.opengis.net/gml">
>> <xsl:output indent="yes" medhod="xml"/>
>> <xsl:template match="/">
>> <records>
>> <xsl:apply-templates/>
>> </records>
>> </xsl:template>
>> <xsl:template match="gml:featureMember/owscat:service_resources">
>> <record>
>> <name><xsl:value-of select="owscat:name"/></name>
>> <title><xsl:value-of select="owscat:title"/></title>
>> <abstract><xsl:value-of select="owscat:abstract"/></abstract>
>> </record>
>> </xsl:template>
>> </xsl:stylesheet>
>> Example Result
>> <?xml version="1.0"?>
>> <records xmlns:owscat="http://www.ec.gc.ca/owscat"
>> xmlns:gml="http://www.opengis.net/gml">
>> <record><name>2916_21230</name><title>Complete Earth
>> (2048x1024 Image)</title><abstract>This image of Earth&#039;s
>> city lights was created with data from the Defense Meteorological
>> Satellite Program (DMSP) Operational Linescan System (OLS).
>> Originally designed to view clouds by moonlight, the OLS is also
>> used to map the locations of permanent lights on the
>> Earth&#039;s surface.
>> The brightest areas of the Earth are the most urbanized, but not
>> necessarily the most populated. (Compare western Europe with China
>> and India.) Cities tend to grow along coastlines and
>> transportation networks. Even without the underlying map, the
>> outlines of many continents would still be visible. The United
>> States interstate highway system appears as a lattice connecting
>> the brighter dots of city centers. In Russia, the Trans-Siberian
>> railroad is a thin line stretching from Moscow through the center
>> of Asia to Vladivostok. The Nile River, from the Aswan Dam to the
>> Mediterranean Sea, is another bright thread through an otherwise
>> dark region.
>> Even more than 100 years after the invention of the electric
>> light, some regions remain thinly populated and unlit. Antarctica
>> is entirely dark. The interior jungles of Africa and South America
>> are mostly dark, but lights are beginning to appear there. Deserts
>> in Africa, Arabia, Australia, Mongolia, and the United States are
>> poorly lit as well (except along the coast), along with the boreal
>> forests of Canada and Russia, and the great mountains of the
>> Himalaya.</abstract></record>
>> </records>
>
+-----------------------------------------------------------------+
|Paul Spencer pspencer at dmsolutions.ca |
+-----------------------------------------------------------------+
|Chief Technology Officer |
|DM Solutions Group Inc http://www.dmsolutions.ca/ |
+-----------------------------------------------------------------+
More information about the mapserver-dev
mailing list