[mapserver-users] SDE performance and error

[Ballard,Lowell]

My general experience has been that for very large data sets SDE will smoke
a shapefile.  This is only true however if the SDE and Oracle instances are
tuned AND your requests is limited in extent (i.e., don't try to draw all
the contours for the state of Virginia when zoomed all the way out).  If
your view window is of reasonable size SDE will smoke -- assuming things are
tuned correctly.  It makes sense if you think about the process.  If you ask
a database for the ALL the records you need to go through the DB, SDE blah
blah and it would probably be easier to get a single file.  However, if you
only want a few records (i.e., spatial features) a DB can consult an index,
search it very fast and return your request.  I think the same idea can be
used for small data sets.  Do I really need to go through the DB/SDE etc.
for the county boundaries? Probably not....

I'm curious how you have SDE on a Linux Box?  Did you get a pre-release or
something?

The tuning comes in many levels.  These range from hardware (e.g., RAID
configurations, network hardware) to Oracle (about a million setting here)
and SDE (i.e., grid tiling mostly).  Each SDE layer needs to be tuned
separately.  It's a bit of a pain but it needs to be done.  Don't use the
GUI Arc Loaders---use the command line SDE tools.  On ESRI's website you can
get the SDELoader/SDEMonitor utils to help with tuning a bit.


As for the error I haven't seen that one.  I'm still having a few issues
with queries against SDE from ms but....none exactly like that.


-----Original Message-----
From: Andrea Goethals [mailto:andreag at geoplan.ufl.edu] 
Sent: Tuesday, July 10, 2001 1:23 PM
To: mapserver-users at lists.gis.umn.edu
Subject: [mapserver-users] SDE performance and error

I thought the posting the other day about the performance of PostGIS
vs shapefiles was interesting. 

>>The PostGIS performance has been about 1/2 to 1/3 the speed as using
>>shapefiles on a RAID device.

I have been looking at shapefiles vs SDE and Oracle.

In that email Dave noted that there should be places where optimization
could take place. Similarly, I have been told that SDE and Oracle both
can be "tuned" (whatever that entails) to speed up the process. Neither
our SDE nor our Oracle has been tuned yet. Having said that it appears
that shapefiles are faster than SDE.

For initial testing purposes I accessed shapefiles on a different
machine
than mapserver was running on, which is the same machine the SDE server
is on. The Oracle 8.1.x is on a third machine.

-------------------       ---------------       ---------
 mapserver          |     |  shapefiles  |     |         |
                    | --- |              | --- | Oracle  |
 apache web server  |     |  SDE server  |     |         |
-------------------       ---------------       ---------
Linux                     Linux                Solaris

At the worst case SDE took 4 times longer than shapefiles. This seems
to occur for the larger datasets (ex: a shapefile of size:
12.6 MB .shp, 4.5 MB .dbf  took on average 10.5 seconds to access for an
SDE layer; 2.6 seconds for a shapefile)

For the smaller datasets and especially at smaller scales, the
difference 
between them is less.
(ex: a shapefile of size: 294 KB .shp, 419 KB .dbf took an average of
1.7 seconds for the SDE layer, and 1.3 seconds for the shapefile.)

I don't know if the slowdown is the communication between SDE and
Oracle,
the extra hop to the Oracle machine, or just SDE itself.



Beyond the speed issue, every once in a while I would get an error
trying
to access SDE layers, but only when trying to access the larger layers
(in filesize):
msSDELayerNextShape(): SDE error. SE_stream_fetch(): Network I/O error.
(-10)

According to the SDE book, an error code of -10 means that "a
catastrophic
error occurred during a client/server round-trip request and the
connection between the client task and the server was severed"

Is there a limit set somewhere as to how much data can be sent? No
errors
like this were found accessing the same datasets from ESRI's ArcView.

Anybody else getting this error?

Thanks,
Andrea

-- 
@-------------------------@
| Andrea Goethals         |
| andreag at geoplan.ufl.edu |
| GeoPlan Center          |
| University of Florida   |
@-------------------------@