[mapserver-dev] pre-RFC - Add an HTTP request data provider to mapserver

[Stephen Woodbridge]

Steve, Tamas,

Good questions, I guess I did not realize the all these formats were 
currently be handled by OGR and not Mapserver directly. There is no 
reason that OGR could not be used to do the parsing provided it does not 
significantly slow down the process and I have no information that it 
would, but you guys might have a better idea about that.

The requirement here is passing the parameters to the external URL so 
that is can do its thing efficiently and then get the data back into 
Mapserver.

I do not have a requirement for multiple formats, but thought that it 
made sense to support them in the global design. Any one that would be 
fast would be fine for my requirements. I'm a little concerned that 
getting too many layers in the processing path will make things slow, 
but it might be too early to worry about that. I do not have a hard 
requirements for performance at the moment other than this will likely 
be used for serving data from a consumer based site that will need to 
handle a lot of requests.

-Steve W

Lime, Steve D (DNR) wrote:
> What would process the data on the back end? OGR? If so what about a
> parameterized (e.g. URL template) that it would do the work on. E.g.:
> 
> 
> CONNECTIONTYPE OGR CONNECTION 'a url that we substitute some values
> (e.g. extent, scale, projection) into'
> 
> and take advantage of OGR's format recognition. Seems like a useful
> addition to that library if it doesn't do it already. As long as the
> URL returned data OGR could process it would work in MapServer.

Tamas Szekeres wrote:
> Do we intend to reimplement parsing each response formats in
> mapserver, or we would rather extend existing implementations (ie. in
> OGR) to support fetching data from an HTTP connection?
> 
> The are existing implementations like GeoJSON (see 
> http://www.gdal.org/ogr/drv_geojson.html) already support this use
> case. In this regard the only issue would be to customize the url
> (OGR connection) with some dynamic information, or do we have further
>  requirements?


> -----Original Message----- From:
> mapserver-dev-bounces at lists.osgeo.org
> [mailto:mapserver-dev-bounces at lists.osgeo.org] On Behalf Of Stephen
> Woodbridge Sent: Tuesday, July 27, 2010 3:14 PM To: mapserver-dev 
> Subject: [mapserver-dev] pre-RFC - Add an HTTP request data provider
> to mapserver
> 
> Hi Mapserver Devs,
> 
> I am interested in getting input and interest in doing some possibly
> or partially funded development of the following.
> 
> USE CASE:
> 
> User has spatial data stored in a proprietary system, database,
> search engine, whatever. They want to pull that data into mapserver
> for rendering, querying, etc like any other data layer. They can
> access that data via PHP and possibly other tools. They would like to
> have a generic HTTP data provider LAYER added to mapserver where
> mapserver would make URL requests based on some defined template URL
> and the URL would return data in some  well know format like GeoJSON,
> GPX, KML, others.
> 
> The LAYER definition would look something like:
> 
> LAYER NAME "mylines" CONNECTIONTYPE HTTP CONNECTIONFORMAT
> GEOJSON|GPX|KML|... DATA "http://localhost/mydata.php" TYPE LINE ... 
> END
> 
> or maybe:
> 
> LAYER NAME "mylines" CONNECTIONTYPE HTTP/(GEOJSON|GPX|KML|...) DATA
> "http://localhost/mydata.php[?optional=user_args&...]" TYPE LINE ... 
> END
> 
> Mapserver would add well defined arguments to the URL like:
> 
> MODE=render|query|etc      - might be needed to set context of
> request BBOX=xmin,ymin,xmax,ymax   - area of interest FILTER=...
> - this could be a filter string of some kind ATTRS=gid,color,name,...
> - list of expected/required? attributes TYPE=POINT|LINE|POLYGON    -
> type of spatial data expected FORMAT=GEOJSON|GPX|KML|... - format to
> respond in ...                        - others as required
> 
> 
> The idea here is to forward the data request to the URL and it would
>  validate the request and throw an error or handle the request and
> return the results as expected. Mapserver would then handle this like
> any other LAYER.
> 
> BENEFITS:
> 
> In general this opens up mapserver to easily be glued into any data 
> source by writing an small external adapter in PHP, Perl or anything
>  thing that can be queried via a URL that can conform to whatever 
> protocol we require for mapserver.
> 
> Much of the code already exists in mapserver for this, it is just 
> getting reused as a new data provider.
> 
> ISSUES:
> 
> Other than the fact this this will not be the fastest data provider 
> available, I can not think of any major one.
> 
> FUNDING:
> 
> I may be able to get some funding for this if we can get it into 6.0
> and back ported to 5.6 even if it is not released on 5.6. Please
> contact me off list if you are interested in developing this.
> Currently on one format would need to be supported.
> 
> COMMENTS:
> 
> Please toss in your 2 cents on this. Raise issues, etc. The idea is
> to make this GENERIC so it can by used by anyone. 
> _______________________________________________ mapserver-dev mailing
> list mapserver-dev at lists.osgeo.org 
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev