[mapserver-users] SDE performance and error

[Stephen Lime]

I wish I could take the credit. I ported Frank Warmerdam's original quadtree code and
wrote the read/write routines plus the mapserver interface, but Frank did the hard part.
Then, Carl Anderson took my crappy, non-portable indexes, and fixed them up along
with some other speed modifications.

Steve

Stephen Lime
Internet Applications Analyst

Minnesota DNR
500 Lafayette Road
St. Paul, MN 55155
651-297-2937

>>> "Ballard,Lowell" <LBallard at YesVirginia.org> 07/12/01 01:34PM >>>
Holy smokes--------------

I just reverted one of my test map files back to a shapefile (vdotrd) of all
the roads in va.  This is a file I had split into 2 distinct shapefiles for
performance reasons.  There was always too much latency for me so I made a
minrds and majrds (interstates-4-lanes etc.).

The result after shptree using default parameters was great.  I can now get
rid of the 2 files and use vdotrd.

If I get the time to benchmark against SDE I will let you know.  I'm
installing Oracle 8.1.7 and SDE 8.1 as we speak on a SUN E4500 (8 400mhz
CPUs, 8 mb ECache each CPU and 4 GB RAM). This should be a nice test.

I assume you wrote this index?  It's based on R-tree or Quad-tree type
algorithms as opposed to ESRI's indexing scheme?

Thanks for the tip......

Lowell


-----Original Message-----
From: Stephen Lime [mailto:steve.lime at dnr.state.mn.us] 
Sent: Thursday, July 12, 2001 10:50 AM
To: andreag at geoplan.ufl.edu; mapserver-users at lists.gis.umn.edu;
LBallard at YesVirginia.org 
Subject: RE: [mapserver-users] SDE performance and error

Lowell: By chance have you played at all with spatial indexes (i.e. shptree)
on
large datasets. I'm wondering how comparing indexed shapefiles with SDE
looks.

Stephen Lime
Internet Applications Analyst

Minnesota DNR
500 Lafayette Road
St. Paul, MN 55155
651-297-2937

>>> "Ballard,Lowell" <LBallard at YESVIRGINIA.ORG> 07/10/01 02:43PM >>>
My general experience has been that for very large data sets SDE will smoke
a shapefile.  This is only true however if the SDE and Oracle instances are
tuned AND your requests is limited in extent (i.e., don't try to draw all
the contours for the state of Virginia when zoomed all the way out).  If
your view window is of reasonable size SDE will smoke -- assuming things are
tuned correctly.  It makes sense if you think about the process.  If you ask
a database for the ALL the records you need to go through the DB, SDE blah
blah and it would probably be easier to get a single file.  However, if you
only want a few records (i.e., spatial features) a DB can consult an index,
search it very fast and return your request.  I think the same idea can be
used for small data sets.  Do I really need to go through the DB/SDE etc.
for the county boundaries? Probably not....

I'm curious how you have SDE on a Linux Box?  Did you get a pre-release or
something?

The tuning comes in many levels.  These range from hardware (e.g., RAID
configurations, network hardware) to Oracle (about a million setting here)
and SDE (i.e., grid tiling mostly).  Each SDE layer needs to be tuned
separately.  It's a bit of a pain but it needs to be done.  Don't use the
GUI Arc Loaders---use the command line SDE tools.  On ESRI's website you can
get the SDELoader/SDEMonitor utils to help with tuning a bit.


As for the error I haven't seen that one.  I'm still having a few issues
with queries against SDE from ms but....none exactly like that.


-----Original Message-----
From: Andrea Goethals [mailto:andreag at geoplan.ufl.edu] 
Sent: Tuesday, July 10, 2001 1:23 PM
To: mapserver-users at lists.gis.umn.edu 
Subject: [mapserver-users] SDE performance and error

I thought the posting the other day about the performance of PostGIS
vs shapefiles was interesting. 

>>The PostGIS performance has been about 1/2 to 1/3 the speed as using
>>shapefiles on a RAID device.

I have been looking at shapefiles vs SDE and Oracle.

In that email Dave noted that there should be places where optimization
could take place. Similarly, I have been told that SDE and Oracle both
can be "tuned" (whatever that entails) to speed up the process. Neither
our SDE nor our Oracle has been tuned yet. Having said that it appears
that shapefiles are faster than SDE.

For initial testing purposes I accessed shapefiles on a different
machine
than mapserver was running on, which is the same machine the SDE server
is on. The Oracle 8.1.x is on a third machine.

-------------------       ---------------       ---------
 mapserver          |     |  shapefiles  |     |         |
                    | --- |              | --- | Oracle  |
 apache web server  |     |  SDE server  |     |         |
-------------------       ---------------       ---------
Linux                     Linux                Solaris

At the worst case SDE took 4 times longer than shapefiles. This seems
to occur for the larger datasets (ex: a shapefile of size:
12.6 MB .shp, 4.5 MB .dbf  took on average 10.5 seconds to access for an
SDE layer; 2.6 seconds for a shapefile)

For the smaller datasets and especially at smaller scales, the
difference 
between them is less.
(ex: a shapefile of size: 294 KB .shp, 419 KB .dbf took an average of
1.7 seconds for the SDE layer, and 1.3 seconds for the shapefile.)

I don't know if the slowdown is the communication between SDE and
Oracle,
the extra hop to the Oracle machine, or just SDE itself.



Beyond the speed issue, every once in a while I would get an error
trying
to access SDE layers, but only when trying to access the larger layers
(in filesize):
msSDELayerNextShape(): SDE error. SE_stream_fetch(): Network I/O error.
(-10)

According to the SDE book, an error code of -10 means that "a
catastrophic
error occurred during a client/server round-trip request and the
connection between the client task and the server was severed"

Is there a limit set somewhere as to how much data can be sent? No
errors
like this were found accessing the same datasets from ESRI's ArcView.

Anybody else getting this error?

Thanks,
Andrea

-- 
@-------------------------@
| Andrea Goethals         |
| andreag at geoplan.ufl.edu |
| GeoPlan Center          |
| University of Florida   |
@-------------------------@