[Mapserver-dev] Re: mapsde.c

[Paul Ramsey]

The strangest aspect of this so far is that there is a non-trivial cost 
to *closing* a stream. I cannot help but think that is a clue to what 
we are doing wrong.  The fact that ArcIMS is beating our performance 
against SDE indicates to me that somewhere there is something we are 
doing wrong in our SDE handling.  Now that connection pooling is in, I 
think we are starting from the same infrastructural start line as 
ArcIMS, so why are we still carrying so much overhead on our setup and 
takedown?

On 18-Oct-04, at 5:34 PM, Howard Butler wrote:

> Paul,
>
> This afternoon Brock and I implemented a couple of small changes to 
> prevent this from happening.  The behavior we defined is that if the 
> user did not set a version, no version queries are done (instead of 
> defaulting to SDE.DEFAULT).  We still need to have the ability to set 
> the version to SDE.DEFAULT, however.
>
> If no version is specified, then the query returned to MapServer will 
> be the data in SDE just as they were originally loaded (for the most 
> part, users could have also compressed their edits back into the main 
> table as well).  If SDE.DEFAULT is specified, then that version will 
> be used to query, just like any other version.
>
> Brock tested these changes and saw a 8-10% increase in rendering time 
> (turning off all of the versioned query stuff).  Each layer was making 
> extra trips to the db to find out version information, and the cleanup 
> was costing a bit extra as well.
>
> Brock and I will continue to work on the changes tomorrow, and if they 
> seem to be behaving, we'll check them in.
>
> SE_DEFAULT_STATE_ID is a *special* state_id that I've only seen come 
> up if the SDE server *doesn't* support versioned queries (like SDE for 
> Coverages, for example).  It is not related to the version 
> SDE.DEFAULT, even though their names are scarily similar.  
> SDE.DEFAULT's state_id will move around as the users make and post 
> their edits to the database (hence the query for the info struct, even 
> though SDE.DEFAULT wasn't explicitly specified).  The changes that 
> Brock and I made mean that if no version was specified, nothing is 
> done with respect to versions and no default version is assumed.
>
> Howard
>
>
> At 07:03 PM 10/18/2004, Paul Ramsey wrote:
>> I have a suspicion that some of the version handling code might be 
>> making us slower than we need to be.
>>
>> In particular, here we check for a version, and use SDE.DEFAULT if 
>> there is not one provided, then we go and do SE_version_get_info() 
>> which will later provide us with a state_id. But, if our version is 
>> SDE.DEFAULT, isn't our state_id == SE_DEFAULT_STATE_ID. So we don't 
>> need to do the version_get_info(), and we also do not need to do the 
>> tests for SE_stateinfo_is_open(), or the SE_stream_set_state() and 
>> all the stuff preceding it.
>>
>> These are questions, not statements, I could be completely wrong in 
>> my interpretation.
>>
>>   if (numparams < 3){
>>     // User didn't specify a version, we'll use SDE.DEFAULT
>>     if (layer->debug)
>>       msDebug("msSDELayerOpen(): Layer %s did not have a"
>>               "specified version.  Using SDE.DEFAULT.\n", 
>> layer->name);
>>       status = SE_version_get_info(sde->connection, "SDE.DEFAULT", 
>> version);
>>   }
>>   else {
>>     if (layer->debug)
>>       msDebug("msSDELayerOpen(): Layer %s specified version %s.\n",
>>               layer->name,
>>               params[2]);
>>     status = SE_version_get_info(sde->connection, params[2], version);
>>   }
>>
>> Paul
>>
>> Brock Anderson wrote:
>>
>>> Now that FastCGI and the SDE connection pooling code seem to work,
>>> I've had a chance to do some performance tests on the new and
>>> improved MapServer. Some early results show that Mapserver is about
>>> 1-2 seconds faster per getMap request than it was with regular CGI,
>>> but it is still much slower than ArcIMS for simple getMap requests. I
>>> was testing with a map that had three fairly simple layers; ArcIMS
>>> returns in ~800ms, MapServer in about 1600ms.
>>> It looks like msSDELayerOpen() and msSDELayerClose() account for much
>>> of that difference. I added some debugging statements to the code,
>>> which revealed:
>>> In msSDELayerOpen():
>>> SE_version_get_info() takes 20 - 30ms
>>> SE_layer_get_info() takes ~100ms
>>> SE_stream_set_state() takes ~10ms
>>> the code inside the block:
>>>   if (!(sde->state_id == SE_DEFAULT_STATE_ID)){
>>>    ...
>>>   }
>>> takes 20 - 100ms
>>> In msSDELayerClose()
>>> SE_stream_free() takes 20 - 30ms
>>> These may seem like minor performance hits, but they occur for each 
>>> layer, and those numbers become quite significant.  I especially
>>> worry about the way we handle state ids, and I wonder if we are doing
>>> it efficiently.  Frank and Howard: do either of you have ideas for
>>> how mapsde.c could be streamlined?
>> _______________________________________________
>> Mapserver-dev mailing list
>> Mapserver-dev at lists.gis.umn.edu
>> http://lists.gis.umn.edu/mailman/listinfo/mapserver-dev
>