[mapserver-dev] Proposal for GeoJSON support in MapServer
Bob Basques
Bob.Basques at ci.stpaul.mn.us
Wed Jan 28 15:19:57 EST 2009
All, see followup to brents comments.
bobb
>>> Stephen Woodbridge <woodbri at swoodbridge.com> 01/28/09 11:53 AM >>>
Brent Fraser wrote:
> Bob,
>
> Is your suggestion to make the output drivers dynamically loadable at
> run time (via .so/.dll modules), with a documented API so developers
> could supply drivers without disturbing the Mapserver code/doc? I
> like it (except for the user-support-nightmare aspect).
>
> Or is it something else? I don't understand the adding request modes
> thing. Do you have an example?
Dido on Brent's comments.
I general I find it extremely confusing when feature requests are
presented as technical solutions. It would be much more helpful if
people requesting a feature presented the use case for the enhancement
and describe what other similar features mapserver has that may or may
not work well in support of your use case. Use cases are often presented
as a story, like:
I want to build a website where the user does this and from that I
request something from the server, then I present that back to the user
thusly. The user then interacts with it in this way, etc.
This greatly improves our understanding of the problem you are
attempting to solve and helps us focus on the best solution to that
problem. If you present a technical solution, then we tend to get
involved at picking nits with your proposal without really understand
what your real problem is.
Hope this helps,
-Steve W
> Brent Fraser
>
>
> Bob Basques wrote:
>> All,
>>
>> I would like to suggest a slightly different approach to
>> integration of this sort into MapServer. It will require some
>> extra work though.
>>
>> A couple of years ago we were trying to do some specialized
>> queries, mostly related to getCapabilties sort of requests. One
>> thought was to build into MapServer a method for adding request
>> modes. Where request parameters could be mapped to the mapserver
>> internal calls. The idea would be to allow for compiling in a
>> custom front end to mapserver.
>>
>> Our desire at the time (and its still present) was to be able to
>> build a single executable for portability, with connectors to our
>> custom clients. I know I'm not laying this out in a very well
>> defined way as an idea (yet) but I hope some others can see the
>> possible benefits of this type of approach.
>>
>> In the end, with a MODE parameter added into MapServer, it could
>> start using Request plugin(s) frameworks that could be used to
>> build all sorts of custom data transport operations. A connector
>> front end to MapServer if you will. This leaves the option of
>> compiling/binary building, still up to the implementor.
>>
>> This could easily go beyond just the GeoJson connector/outputs
>>
>> bobb
>>
>>
>>
>>>>> Jan Hartmann <j.l.h.hartmann at uva.nl> 01/28/09 8:54 AM >>>
>> I have been thinking about the discussion we had about Canvas
>> support in MapServer, and although the problem has been solved as
>> far as I am concerned, perhaps a more general solution could be
>> considered: to implement GeoJSON natively in MapServer, like it has
>> been done in GeoServer. This would not only make Canvas support
>> easier, but could have some further advantages , as you can see
>> further on. So I would like to propose:
>>
>> 1) To implement a full GeoJSON representation of the complete
>> MapServer map at the server side: layer geometries and styles. This
>> would be an ASCII string and could be constructed quite easily,
>> AFAICS, by looping throug the map elements and writing them to a
>> string sourrounded by the appropriate square and curly brackets.
>> This string could be made available to the calling client via a
>> template variable, e.g. [GeoJSON]. The only thing the client has to
>> do is to declare a JavaScript header in the template file:
>>
>> <!-- pt> var gs = eval([GeoJSON]) </scr -->
>>
>> and from that point on the complete internal representation of the
>> map would be available (loopable and searcheable) in the variable
>> gs as Javascript arrays and objects. Extra properties like bounding
>> boxes for map and elements could be added to facilitate client side
>> searching and zooming.
>>
>> Note that this GeoJSON template variable can be combined with a
>> regular map returned as an image. The information in the GeoJSON
>> template variable could be used e.g. to zoom to (parts of) the
>> returned map.
>>
>> If you don't want the complete map tree, you can always use Steve's
>> procedure as he described it below to return parts of the map via
>> the TEMPLATE output driver. It's simple enough.
>>
>> 2) To add "mode=geojson" to the mapserver CGI mode control. This
>> for people like me who prefer to work without templates (although
>> they are a brilliant invention). The Javascript header would be in
>> that case:
>>
>> <!-- pt> var gs = </scr --> <!-- pt
>> src=http://mapserver.sara.nl/cgi-bin-mapserv.map=GeoJSON.map&mode=geojson></scr
>> --> <!-- pt> gs = eval(gs) </scr -->
>>
>> 3) To add GeoJSON as output driver. This overlaps with 2), so it
>> wouldn't be strictly necessary. The only difference would be that
>> the javaScript call would have "mode=map"
>>
>> 4) To write separate generalized Javascript routines to convert
>> the returned GeoJSON to Canvas and VRM. If the GeoJSON object will
>> be implemented, I am willing to do this.
>>
>> 5) To implement the GeoJSON structures in the scripting languages.
>> JSON can be converted to about any language available, so if a
>> script language, e.g. PHP MapScript, can obtain the ASCII GeoJSON
>> representation of the complete MapServer map, it can convert its
>> members to native PHP arrays and objects. That would make the link
>> between the MapServer engine and the scripting language engines
>> much simpler and much more uniform. In the last resort, this would
>> make redundant the SWIG interface, as the complete interface would
>> now be implemented via an ASCII string.
>>
>> This could cause some performance overhead, but probably not very
>> serious, especially since I would expect really performance-hungry
>> applications to be written in MapServer CGI, while scripted
>> applications will probably be more directed to analytic cartography
>> and have other performance bottlenecks.
>>
>> 6) To implement new drivers via the GeoJSON interface. The Canvas
>> driver is a simple example, but new drivers for other formats could
>> just as well communicate with the MapServer core with GeoJSON, via
>> a JSON-C interface. Perhaps even the GDAL/OGR bridge could benefit
>> from this construction. However, in this case performance will be a
>> major issue: it is not clear whether this way of writing driver
>> will be more or less efficient than the plugin scenario discussed
>> on this list.
>>
>> 7) To implement the MapFile in GeoJSON format. If the output of a
>> MapServer map can be represented in GeoJSON, then so can the input
>> MapFile. There has been for years a discussion on this list about
>> a MapFile in SVG format. Perhaps a GeoJSON MapFile will be easier
>> to implement, certainly if it can be derived from the template
>> mechanism of 1). Once such a GeoJSON mapfile exists, it can easily
>> be converted to SVG.
>>
>> Jan
>>
>>
>> Jan Hartmann wrote:
>>> As an afterthought: wouldn't it be an idea to output this
>>> template directly via the OutputFormat, as a "GeoJSON" Driver
>>> type?. In that case, MapServer would return the GeoJSON object
>>> with all dependent properties directly as an ASCII string. If the
>>> calling HTML file had a line like:
>>>
>>> <!-- pt
>>> src=http://mapserver.sara.nl/cgi-bin/mapserv?map=testGeoJSON.map&mode=map>
>>>
>>>
>>>
>>> the GeoJSON object would be loaded into the global namespace of
>>> the web-page and could then be processed further. That way you
>>> wouldn't need a template file, you wouldn't have to write
>>> separate templates for each mapping application, and you would
>>> have a regular GeoJSON driver, like GeoServer has. It wouldn't
>>> take much programming time too, as far as I can see.
>>>
>>> Jan
>>>
>>> Jan Hartmann wrote:
>>>> Wow, brilliant, this solves the problem. Just add a few
>>>> JavaScript routines in the template to convert the GeoJSON
>>>> object into Canvas commands and you are finished. You can
>>>> process all elements after that via the GeoJSON arrays and
>>>> never need any DOM tree (which isn't there anyway, as Christian
>>>> Schmidt remarked).
>>>>
>>>> Just one question: how do you get styling information into the
>>>> GeoJSON object? GeoJSON allows additional members at any level
>>>> in a GeoJSON object, so these could be used to store colors,
>>>> linewidths etc. Is it possible to get these parameters from the
>>>> Layer's Class object into the template?
>>>>
>>>> Jan
>>>>
>>>> Steve Lime wrote:
>>>>> MapServer can output GeoJSON via query calls to the CGI...
>>>>> Here's an example template:
>>>>>
>>>>> var csgSites = { "type": "FeatureCollection", [resultset
>>>>> layer="sites"]"features": [ [feature limit=-1
>>>>> trimlast=","]{"type": "Feature", "geometry": {"type":
>>>>> "Point", "coordinates": [[shpxy]]}, "properties": {
>>>>> "station": "[station]", "long_name": "[long_name]",
>>>>> "short_name": "[short_name]", [item escape="none"
>>>>> name="storet_id" pattern="." format="'storet_id':'$value',"]
>>>>> [item escape="none" name="usgs_id" pattern="."
>>>>> format="'usgs_id':'$value',"] "has_telemetry":
>>>>> [has_telemetry], "has_archive": [has_archive],
>>>>> "has_water_chemistry": [has_water_chemistry], "has_gagings":
>>>>> [has_gagings], "has_photos": [has_photos] } },[/feature]
>>>>> ][/resultset] };
>>>>>
>>>>> You'd pair this with a new-style OUTPUTFORMAT like so:
>>>>>
>>>>> OUTPUTFORMAT NAME json'' DRIVER 'TEMPLATE' MIMETYPE
>>>>> 'aplication/json' FORMATOPTION 'FILE=sites.js' END
>>>>>
>>>>> Steve
>>>>>
>>>>>
>>>>>>>> On 1/27/2009 at 10:25 AM, in message
>>>>>>>> <497F358E.8000908 at uva.nl>, Jan Hartmann
>>>>>>>>
>>>>> <j.l.h.hartmann at uva.nl> wrote:
>>>>>
>>>>>> Another option would be to use GeoJSON. Is MapServer able
>>>>>> to output
>>>>>>
>>>>>> GeoJSON, or is it going to be? I guess it would be much
>>>>>> easier to write a conversion program from GeoJSON to Canvas
>>>>>> than from SVG, and I am sure it would be much more
>>>>>> efficient.
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> Stephen Woodbridge wrote:
>>>>>>
>>>>>>> Dan Little wrote:
>>>>>>>
>>>>>>>> I think there is some problems with the Canvas
>>>>>>>> implementations across the different browsers. I've
>>>>>>>> actually done quite a bit of work using the Canvas and
>>>>>>>> the exCanvas extension from Google. While exCanvas
>>>>>>>> SAYS it is a 100% implementation for IE there tends to
>>>>>>>> be a number of quirks when it comes to actually placing
>>>>>>>> objects in a drawing stack, animating them, or simply
>>>>>>>> clearing the canvas at a predictable point.
>>>>>>>>
>>>>>>>>
>>>>>>>> Also, I think it would be much more to the benefit to
>>>>>>>> simply have a VML output. A little bit of browser
>>>>>>>> sniffing is easy to do with any JS application (or even
>>>>>>>> some server side scripting) and you'd be outputting
>>>>>>>> either SVG or VML in that browser's native preferred
>>>>>>>> format. No goofy javascript library required.
>>>>>>>>
>>>>>>> I think converting some SVG into the proposed Canvas
>>>>>>> format would be a good starting point so we can evaluate
>>>>>>> the performance. If consuming a large Canvas file causes
>>>>>>> the browser to be slow or crash, then it might not make
>>>>>>> sense to even consider moving forward with something like
>>>>>>> this. It would seem to be a pretty straight forward task
>>>>>>> if we can find someone that knows perl, Canvas and svg or
>>>>>>> is willing to do a little research.
>>>>>>>
>>>>>>> -Steve W
>>>>>>>
>>>>>>>
>>>>>>>> ----- Original Message ----
>>>>>>>>
>>>>>>>>> From: Brent Fraser <bfraser at geoanalytic.com> To: Jan
>>>>>>>>> Hartmann <j.l.h.hartmann at uva.nl> Cc:
>>>>>>>>> mapserver-dev at lists.osgeo.org Sent: Tuesday, January
>>>>>>>>> 27, 2009 9:07:32 AM Subject: Re: [mapserver-dev]
>>>>>>>>> Canvas support for MapServer
>>>>>>>>>
>>>>>>>>> Jan,
>>>>>>>>>
>>>>>>>>> So your thought is to have MapServer generate
>>>>>>>>> something like (hacked from the example on
>>>>>>>>> https://developer.mozilla.org/en/Canvas_tutorial/Basic_usage):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sure, it could be done. One way is to write another
>>>>>>>>> output driver for Mapserver. Another is to use some
>>>>>>>>> server-side scripting to ingest Mapserver's SVG and
>>>>>>>>> change into JavaScript-canvas code like that above. A
>>>>>>>>> performance hit, but at least it would be on the
>>>>>>>>> server side, and it might be a good way to prototype
>>>>>>>>> the Mapserver driver.
>>>>>>>>>
>>>>>>>>> Brent
>>>>>>>>>
>>>>>>>>> Jan Hartmann wrote:
>>>>>>>>>
>>>>>>>>>> Hi Brent,
>>>>>>>>>>
>>>>>>>>>> First let me say that Canvas support is just an
>>>>>>>>>> idea; I think it can be done
>>>>>>>>>>
>>>>>>>>> but whether it is advisable to do so is up to you to
>>>>>>>>> decide. I liked the idea because the Canvas element
>>>>>>>>> is supported nowadays by most browsers, unlike SVG
>>>>>>>>> which needs a plugin. It is also much simpler, so
>>>>>>>>> it's easier to use for simple maps, and especially
>>>>>>>>> for simple interactive operations like digitizing, or
>>>>>>>>> combining server-side with client-side graphics.
>>>>>>>>> That's why I would prefer Canvas above SVG, let alone
>>>>>>>>> the fact that I much prefer to program in native
>>>>>>>>> Javascript above parsing XML. Most of my maps are
>>>>>>>>> simple (I would guess that goes for many people
>>>>>>>>> here), so SVG is decidedly overkill. The Canvas HTML
>>>>>>>>> element also fits nicely in the page layout and can
>>>>>>>>> be processed easily by web-editors.
>>>>>>>>>
>>>>>>>>>> Your suggestion that Canvas could be implemented at
>>>>>>>>>> the client side by
>>>>>>>>>>
>>>>>>>>> catching SVG output and transforming it into Canvas
>>>>>>>>> directives is certainly viable. That is also Paul
>>>>>>>>> Spencer's view in his reply to my email. Yet I would
>>>>>>>>> prefer a server-side approach: it's more efficient
>>>>>>>>> (no second parsing of SVG code that had to be
>>>>>>>>> generated in the first place), and I think the
>>>>>>>>> server-side code can be easily adapted from the SVG
>>>>>>>>> driver. Canvas is a very simple format, just a few
>>>>>>>>> graphic primitives, more like GD than SVG actually.
>>>>>>>>>
>>>>>>>>>> Again, I don't know how difficult it would be to
>>>>>>>>>> add a Canvas outputformat,
>>>>>>>>>>
>>>>>>>>> and whether it fits at all into the MapServer design
>>>>>>>>> philosophy. However, when it could be done, it would
>>>>>>>>> give me an a far more easier tool than SVG to
>>>>>>>>> combine server-side and client-side mapping.
>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>>
>>>>>>>>>> Brent Fraser wrote:
>>>>>>>>>>
>>>>>>>>>>> Jan,
>>>>>>>>>>>
>>>>>>>>>>> My interest is in building a (or re-purposing an
>>>>>>>>>>> existing) tool for
>>>>>>>>>>>
>>>>>>>>> Cartographic Layout, for printing or rendering a
>>>>>>>>> graphics format for easy viewing (e.g. PDF). Steve's
>>>>>>>>> comments were about using existing Web formats and
>>>>>>>>> syntax to describe the layout, and existing(?) web
>>>>>>>>> editors to do the placement/moving of the layout
>>>>>>>>> components.
>>>>>>>>>
>>>>>>>>>>> My (very limited) understanding of the HTML tag
>>>>>>>>>>> is that it's a
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> "container" recognized by the browser, and vector
>>>>>>>>> (and raster) graphics are rendered into it using
>>>>>>>>> client-side JavaScript. Sort of the JavaScript
>>>>>>>>> equivalent of the SVG format.
>>>>>>>>>
>>>>>>>>>>> It would be possible to add another output format
>>>>>>>>>>> to Mapserver to generate
>>>>>>>>>>>
>>>>>>>>> JavaScript code of canvas rendering methods, but a
>>>>>>>>> more elegant approach might be to write a small(?)
>>>>>>>>> JavaScript module to ingest Mapserver's SVG and
>>>>>>>>> render the objects to the canvas.
>>>>>>>>>
>>>>>>>>>>> Are there some specific features of the
>>>>>>>>>>> direct-canvas-output approach not
>>>>>>>>>>>
>>>>>>>>> available in SVG that you need?
>>>>>>>>>
>>>>>>>>>>> Brent Fraser
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Jan Hartmann wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>
>>>>>>>>>>>> A few weeks ago I suggested in a thread about
>>>>>>>>>>>> PDF support that Mapserver
>>>>>>>>>>>>
>>>>>>>>> could perhaps be made to support the new "Canvas"
>>>>>>>>> tag. There were no reactions to this, so I don't know
>>>>>>>>> if this suggestion is viable/advisable/practicable,
>>>>>>>>> or just downright stupid. Can anyone comment? (from
>>>>>>>>> http://lists.osgeo.org/pipermail/mapserver-dev/2009-January/008055.html)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> I am wondering if MapServer support for the
>>>>>>>>>>>>> "Canvas" tag could do what
>>>>>>>>>>>>>
>>>>>>>>> Steve suggests in a more simple way. Originally, only
>>>>>>>>> Safari, Opera and Firefox supported this tag to allow
>>>>>>>>> (simple) 2D drawing, but recently a surprisingly
>>>>>>>>> simple Javascript interface for IE has become
>>>>>>>>> available, translating Canvas command to native IE
>>>>>>>>> VRML commands. It requires just one single script tag
>>>>>>>>> in the web page.
>>>>>>>>>
>>>>>>>>>>>>> See the tutorial at the Mozilla site at:
>>>>>>>>>>>>>
>>>>>>>>> https://developer.mozilla.org/en/Canvas_tutorial
>>>>>>>>>
>>>>>>>>>>>>> and the IE interface (from the Google
>>>>>>>>>>>>> stables) at:
>>>>>>>>>>>>>
>>>>>>>>> http://excanvas.sourceforge.net/
>>>>>>>>>>>>> This is something I have long been looking
>>>>>>>>>>>>> for. The graphics are very
>>>>>>>>>>>>>
>>>>>>>>> simple, so the functionality is nothing like PDF or
>>>>>>>>> SVG, but I could imagine that a driver for MapServer
>>>>>>>>> could fulfill many needs. I can use it already by
>>>>>>>>> letting Mapserver generate raw coordinates and
>>>>>>>>> catching them with some sort of Ajax, but a separate
>>>>>>>>> driver would be very neat of course.
>>>>>>>>>
>>>>>>>>>>>>> How about it?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jan
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> mapserver-dev mailing list
>>>>>>>>>>>> mapserver-dev at lists.osgeo.org
>>>>>>>>>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> mapserver-dev mailing list
>>>>>>>>> mapserver-dev at lists.osgeo.org
>>>>>>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> mapserver-dev mailing list
>>>>>>>> mapserver-dev at lists.osgeo.org
>>>>>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> mapserver-dev mailing list mapserver-dev at lists.osgeo.org
>>>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>>>
>>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> _______________________________________________ mapserver-dev
>>>> mailing list mapserver-dev at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>>
>>
>> _______________________________________________ mapserver-dev
>> mailing list mapserver-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
> _______________________________________________ mapserver-dev mailing
> list mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
More information about the mapserver-dev
mailing list