MS RFC: Layer Plug-in Architecture

Jani Averbach javerbach at EXTENDTHEREACH.COM
Mon Aug 22 14:54:29 EDT 2005

[[[ Resending the message, this time without line wraps ]]]


Here is my formal proposal for Layer Plugin/VTable architecture for
MapServer.  There are few questions and  unclear things in the proposal,
hence I have provided it as ''draft''.  If the MapServer Steering
Committee feels that the proposal is clear enough, I am fine if the Status is
changed to "proposed".  The proposal should be valid restructured text.

If you have any comments, recommendations or issues about this proposal,
please let me know. Thank you.

BR, Jani

MS RFC: Layer Plug-in Architecture
:Date:  2005/08/22
:Author: Jani Averbach
:Contact: javerbach at
:Last Edited: $Date: 2005-08-22 11:45:31 -0600 (Mon, 22 Aug 2005) $
:Status: draft

**Description:** Change current layer infrastructure from switch based
static solution to the virtual table based dynamic solution.  The main
objectives for this proposal are cleaning and simplifying code and
provide a robust architecture to implement dynamic plug-in API for
external layer providers.

Abstract Solution

Implement a virtual table structure for ``layerObj``. This structure will
contain function pointers for all layer specific operations.  This
structure will be populated when the layer is created or when it is
opened. After that all layer operations will happen through this
vtable which is cached in the ``layerObj`` struct.

Technical Solution

All file names and numbers are against released MapServer 4.6.0 source

1. Add new field to the ``layerObj``. It will be pointer to the vtable,
   which will contains function pointers for this layer.

2. Add the virtual table architecture

   2.1. Standard functions which are found currently from ``maplayer.c``

        2.1.1. InitItemInfo

                 int (*fpLayerInitItemInfo)(layerObj *layer);

        2.1.2. FreeItemInfo

                  void (*fpLayerFreeItemInfo)(layerObj *layer);

        2.1.3. Open

                  int (*fpLayerOpen)(layerObj *layer);

               Currently there are two layers which accept more than
               the generic layer arg for LayerOpen function:

                  int msOGRLayerOpen(layerObj *layer, 
                                     const char *pszOverrideConnection);
                  int msWFSLayerOpen(layerObj *lp, 
                                     const char *pszGMLFilename, 
                                     rectObj *defaultBBOX);
               However, these are called from ``msLayerOpen`` with
               ``NULL`` args, so I think that proposed interface for
               this virtual table function should be fine.

        2.1.4. IsOpen

                  int (*fpLayerIsOpen)(layerObj *layer);

        2.1.5. WhichShapes

                  int (*fpLayerWhichShapes)(layerObj *layer, 
                                            rectObj rect);

        2.1.6. NextShape

                  int (*fpLayerNextShape)(layerObj *layer, shapeObj *shape);

        2.1.7. GetShape

                  int (*fpLayerGetShape)(layerObj *layer, shapeObj *shape, 
                                         int tile, long record);

        2.1.8. LayerClose

                  int (*fpLayerClose)(layerObj *layer);

        2.1.9. LayerGetItems

                  int (*fpLayerGetItems)(layerObj *layer);

        2.1.10. GetExtent

                   int (*fpLayerGetExtent)(layerObj *layer, 
                                           rectObj *extent);

        2.1.11. GetAutoStyle

                   int (*fpLayerGetAutoStyle)(mapObj *map, layerObj *layer,
                                              classObj *c, int tile, 
                                              long record);

   2.2.  New functions and/or fields for vtable

         2.2.1. CloseConnection

                This function is used to actually close the
                connection, in case that layer implements some kind of
                connection pooling by its own.  If layer doesn't use
                any connection pooling, this function should be
                implemented as no-op. Caller should first call layer's
                "close" function, and finally at the very end ``CloseConnection``.

                The signature of function is

                   int (*fpLayerCloseConnection)(layerObj *layer);

                And the main place where this function will be called,
                is ``mapfile.c: 4822``
                   void msCloseConnections(mapObj *map)

                This function is needed because e.g. ``POSTGIS`` is
                implementing this usage pattern at the moment

                    void msLayerClose(layerObj *layer)
                    * Due to connection sharing, we need to close the results
                    * and free the cursor, but not close the connection.

         2.2.2. GetConnectionTypeName
                This function is used to get MAP-file name for this
                layer/connectiontype.  It is used when map file is
                written back (``mapfile.c: 2856``), actual strings will
                be: ``OGR``, ``POSTGIS`` and so on.

                The signature for this function is

                    const char * (*fpGetConnectionTypeName)();
                Another solution could be to add one extra field
                (``const char * connectionTypeName``) to the ``layerObj``,
                which is set by layer's initialization function.

         2.2.3. SetTimeFilter

                This function is used to create a time filter for
                layer. At the moment we have three special cases
                (``maplayer.c: 1635``):

                1. ``POSTGIS`` with it's own function
                2. Layers with backticks delimited expressions
                3. Layers without backticks

                The idea is provide a generic helper function,

                   int makeTimeFilter(layerObj *lp,
                                      const char *timestring,
                                      const char *timefield,
                                      const bool bBackTicks)

                And the actual layer's ``SetTimeFilter`` could use the
                above, or implement something totally different as
                ``POSTGIS`` is doing at the moment.

                The signature for layer's vtable function is

                   int (*fpLayerSetTimeFilter)(layerObj *lp, 
                                               const char *timestring, 
                                               const char *timefield);

   2.3. Things to fix or extra functions to add to the vtable

        I would like to see following items fixed, but I don't know
        which way to go.

        2.3.1. FLTApplyFilterToLayer (mapogcfilter.c: 1084)

               Could we create two helper functions, one for "SQL"
               case and the another for "non-SQL" case, and set all
               layers, except ``POSTGIS`` and ``ORACLESPATIAL`` to call
               directly this "non-SQL" version of this helper function
               (else-branch of if).  ``ORACLESPATIAL`` and ``POSTGIS`` could
               use "SQL" version of the helper function (actual
               if-branch) if the conditions for this are met. This way
               we could remove all layer specific logic from this
               function, and make this function's logic actually generic.

        2.3.2. layerObj->items allocation
               There is layer (SDE) specific code in ``msLayerWhichItems``
               (``maplayer.c: 880``) when initial items allocation is
                  /* TODO: it would be nice to move this into the SDE
                   * code itself, feels wrong here... 
                  if(layer->connectiontype == MS_SDE) {

               One solution would be to provide a layer specific

                  int (*fpCreateItems)(layerObj *) 

               which does the allocation and set numitems to correct

        2.3.3. msCheckConnection (``mapfile.c: 4779``)

               This api looks to be deprecated. It is called only from
               ``msMYGISLayerOpen``.  Is it hard to fix GIS to use
               mappool?  If this isn't feasible thing to do, we could
               remove all other cases and just leave GIS inside this

   2.4. Interface functions for internal layers

        We have to add at least some new interface functions for
        internal layers, e.g. ``msXXXInitializeLayerVirtualTable``,
        where ``XXX`` is the layer's name.

3. Remove unwanted interfaces
   Frank Warmerdam proposed [FW1]_ that we remove all layer specific
   interface functions from ``map.h``.

      I see each "built-in" module such as mapsde.c providing a
      registration function such as "msSDEInitializeLayerVirtualTable"
      so that none of the layer type specific definitions need to
      appear in map.h any more.

   .. [FW1] 
      | ``Frank Warmerdam on the mapserver-dev:``
      | ``Message-Id: <smtpd.490f.4303f2ee.d8246.1 at>``
      | ``Date:       Wed, 17 Aug 2005 22:31:09 -0400``
      | ``Subject:    Re: Mapserver Plug-in Infastructure: RFC and PATCH``

Files and objects affected

This proposal will affect at least following files and objects:

* ``map.h``

  * ``layerObj`` will contain new fields. There will be a new
    object ``vtableObj`` in the ``map.h``.

* ``maplayer.c`` 

  * Various changes, layer specific ``switch``-statements will go
    away, vtable handling and layers vtable initialization will be

* ``mapfile.c``

  * mapfile writing will change slightly, no more ``if-else``
    structure for connectiontype strings.
  * Removing/Cleaning of  ``msCheckConnection``
  * Vtable for ``msCloseConnection``

* ``mapogcfilter.c``

  * Remove layer-logic from ``FLTApplyFilterToLayer``

* ``mapXXX.c``, where ``XXX`` is the name of layer.

  * Add new initialization function
  * Add all new interface functions
  * Fix existing interface functions, if needed / add wrappers for
    ``msOGRLayerOpen`` and ``msWFSLayerOpen``. 

Backwards compatibility issues

This is binary and source code level backward incompatible change. The
proposal will remove some previously public functions, and add new
field(s) to the ``layerObj`` struct.  The proposal is ``MAP``-file
backward compatible.

Implementation Issues

* Biggest problem is probably that the author has ignored or missed
  something by oversight which will show up during implementation.
  However, there is a prototype implementation of external plug-in
  architecture which works at the moment and is based on ideas
  presented in this proposal.  So there is some real life experience
  that this architecture change is feasible thing to do.

* To find best place to initialize layer's vtable. One possible place
  could be ``msLayerOpen``. The problem with that is that if we need
  before opening the layer something layer-specific information, it
  won't be available.

* I also like to note that this proposal won't remove all layer
  specific code from MapServer e.g. ``WMF``, ``WMS`` and ``GRATICULE``
  are used as special cases from place to place.

Bug ID


Voting history


Open questions

* ``SWIG``. What is the impact of this change to the MapServer's SWIG

* How do we like to expose layer's virtual table to the layers. We
  have at least two different routes to go:

  * expose it as a struct, layers will fill vtable pointers by
    accessing directly struct's field.

  * expose it as a complete opaque type, vtable pointers will be set
    by accessing functions ``setLayerOpenFP``, ``setLayerCloseFP`` and
    so on.

  The advance of second option is that this way we could easily add
  new functions to the struct if we refactor code more or found some
  logic which is ignored by oversight in this proposal.

Working Notes

.. Local Variables: 
   mode: text

Jani Averbach

More information about the mapserver-dev mailing list