[Mapserver-dev] Optimizing for ArcSDE

[Peter Slootweg]

Brock,

> 
> We could pool the streams to save this cost.  Since we only ever use one 
> stream per connection, would it make sense to pool streams together with 
> connections?  I was thinking we could create a new structure with both 
> the sde->connection, and the stream, and we could pass this to the 
> connection pooling API.  The msSDECloseConnection() callback function 
> could be modified to close the stream, then the connection.

Are there any cases where more than one layer can be open at a time? How 
does MapServer handle multiple threads - i.e. could one thread get 
interrupted and another start processing while it is processing a layer?
If no to the above then something like the following should work

/* in mapsde.c */
typedef struct {
   SE_CONNECTION connection;
   SE_STREAM stream;
} msSDEConnPoolInfo;


static void msSDECloseConnection( void *conn_handle )
{
   long status;
   msSDEConnPoolInfo * poolinfo=conn_handle;

   if (poolinfo)
      {
      if (poolinfo->stream)
         {
         SE_stream_free(poolinfo->stream);
         }
      if (poolinfo->connection)
         {
         status= SE_connection_free_all_locks (poolinfo->connection);
         if (status == SE_SUCCESS) {
            SE_connection_free(poolinfo->connection);
         }
      free(poolinfo);
      }
   }
}

static int msSDEOpenConnection(layerObj *layer, msSDELayerInfo *sde)
{
   long status;
   msSDEConnPoolInfo * poolinfo;
   poolinfo = (msSDEConnPoolInfo *)msConnPoolRequest( layer );
   if (!poolinfo)
      {
      poolinfo = malloc(sizeof *poolinfo);
      if (!poolinfo)
         {
         return MS_FAILURE;
         }
      else
         {
         poolinfo->stream = NULL;
         poolinfo->connection = NULL;
         /* stuff to setup connection and create stream goes here */
         /* .... */

         }

      if (status == SE_SUCCESS)
         {
         msConnPoolRegister(layer, poolinfo, msSDECloseConnection);
         }
      else
         { /* cleanup if something in the SDE API failed */
         msSDECloseConnection(poolinfo);
         }
      }
   if (poolinfo != NULL)
      {
      sde->connection = poolinfo->connection;
      sde->stream = poolinfo->stream;
      }
   return MS_SUCCESS;
}

> Would this approach work to save us from the stream overhead?  
yes
> Do we even need a solution this complex
yes
> is there something simple that could alleviate this overhead?
no

> 
> Independent of streams, we are incurring about a 20ms cost in getting 
> layer info.  Would it make sense to cache the layer infos?  The layer 
> info probably shouldn't be cached with the connections.
Caching the layerinfo has the danger of the layer definition changing 
while your process is still referencing the cached version. The place 
that this will likely be an issue is where SE_layerinfo_get_envelope is 
called.
The layer envelope changes when features are added outside the exisiting 
envelope.
I am not sure if it would be faster but there is also a 
SE_layer_get_info_by_id - you could then cache just the layerid - this 
is very unlikely to change - unless someone deletes/recreates a layer.

> 
> My take is that for SDE the "connection" we pool should include the stream
> and possibly the layer info.  To make this work we will need to do some fiddling
> to ensure the identifier (normally the CONNECTION string) is unique for different
> versions, streams and/for layers.
> 
> Of course, I don't know much about SDE, so keep that in mind.
> 
> Best regards,