[mapserver-users] SDE performance

[Mapserver account for mailing list]

We have been running MapServer (MS) on Linux since 1998 and have been using it recently to connect to SDE.  As our repositories of information grow inside SDE, we have been using MS to hit those very same datasets as we do in ArcMap.  Connectivity to the data sets is not a problem, but performance is. 

Our MS is running on a Linux server while our SDE runs on a Win2K server.  Performance from other Win2K servers and workstations to SDE, on databases of varying sizes, is very good (largest DB is ~32GB).  When hitting the SDE from the Linux server, performance is horrible.  I began tracing the problem about a week ago and have hypothesized that the performance issue lies in how MS is connecting to SDE.  

The SDE server is 'big and fast' and is not the bottleneck (other apps hit SDE with prowess).  The network connectivity between the Linux server and the SDE server is not the issue as it is on a gig ether connection (yes, a decent backplane on the switch) and workstations with a 100Mbit connection retrieve the same data faster.

The processor on the Linux box is fast as well.  It gets pinned for a period of time initially, but processor load is very short and in sporadic bursts (mostly initially).

I have monitored the network load between the 2 boxes and the processor, and memory loads on each box as I spawn a 'map' via MS.  Even very simple layers with little data to display take a long time to display.  Processor, memory, and network hits are intermittent and not what I would call large for the infrastructure.  

My theory is that the latency between MS request and display has to do more with the number of connections/authentications (viz., for each layer) that occur between MS and SDE.  (Steve, ideas?)  The nature of the map file and a brief peek at the source seemed to point to frequent connections occurring.  If this is the case, would it not make more sense to permit a nesting of layer objects inside a 'connection object' so that one single connection could be made for multiple layer requests (I am not proposing this, simply exemplifying how the number of connection transactions could be reduced).  Persistent connections are obviously not a solution for a web-based CGI, but a persistent connection during a single MS request does make sense to me.  Perhaps I am missing something

Suggestions from Steve or anyone with similar configs would be appreciated.  MS with tiled shapefiles was blazing, but the sheer volume of data we manage makes that option defunct.  Thanks in advance for your help.


Cheers,
Jim