[Mapserver-users] SDE Direct Connect

Thu Dec 11 11:30:51 PST 2003

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C3C01D.49858EA0
Content-Type: text/plain

After looking through the API, I think you're right Steve. There are no
specific functions related to direct connect. Giver that the C API does
support direct connect, it may just a simple matter of changing the
connection string processing to allow the : in the connect string.

Mike

-----Original Message-----
From: Steve Lime [mailto:steve.lime at dnr.state.mn.us] 
Sent: Thursday, December 11, 2003 2:23 PM
To: godwinl at agr.gc.ca; Jean-Francois.Doyon at ccrs.nrcan.gc.ca;
Michael.Smith at erdc.usace.army.mil; mapserver-users at lists.gis.umn.edu;
CGREER at sonoma-county.org
Subject: RE: [Mapserver-users] SDE Direct Connect

Regarding 4, which functions are those? I wasn't able to  find any
references to direct connect in my admittedly outdated SDE C API
documentation.

Steve

>>> <Michael.Smith at erdc.usace.army.mil> 12/11/2003 12:45:52 PM >>>
There is some errors is here that I have to correct.

1. Oracle does indeed have direct spatial support but that is not
necessarily incompatitble with SDE. SDE can store data in sdebinary format,
longraw format (which is being deprecated I belive) and native Oracle
Spatial format. 

2. Direct connect actually has nothing to do with how SDE stores
information. SDE can operate in two modes. Standard or 3 tier mode, in which
the client connects to an SDE server which will then communicate with the
database. Or Direct Connect or 2 tier mode, in which some of abilities of
the sde server are done in the client. ArcGIS can operate this way. The part
of SDE that communicates with the database in built in to ArcGIS to do this.
Mapserver still doesn't have to know anything about Oracle Spatial as that
is all part of the direct connect code built into the client. However, the
client abilities are limited in that it is a read only connection, even if
you have an sde license. 

3. I wouldn't say that SDE 3 tier is slower than 2 tier. As with so many
things, it depends.

4. The C Api that mapserver links to does support direct connect however I
don't believe that the direct connection functions have been implemented

Here what Esri says about Direct Connect

Direct Connect is a method for ArcSDE client applications to connect
directly to the spatial database (hosted on either Oracle 8i or Microsoft
SQL Server) without having to use the ArcSDE application server. 

The classic ArcSDE connection architecture is three-tiered, it includes an
application server process (giomgr), a per-connection server process
(gsrvr), and a client application. ESRI client applications include ArcGIS
Desktop and ArcView. Within this architecture, the ArcSDE giomgr process
manages connections from client applications. When a connection to Oracle8i
is granted through the giomgr process, a gsrvr process is spawned to service
the new connection. The gsrvr process manages all subsequent query
operations between the client application and the RDBMS for the duration of
the connection. Both the giomgr process and gsrvr process exist on the
server machine that hosts ArcSDE and (usually) Oracle. 

Direct-connect is a two-tiered architecture, wherein ArcSDE client
applications can connect directly to an Oracle8i instance. The functionality
of the gsrvr process has been added to a set of libraries used directly by
ArcSDE clients. Typically, ArcSDE clients run on remote desktop machines,
while the Oracle8i instance runs on a server-class machine. The functions of
the gsrvr process become part of the client application running on the
client machine. The ArcSDE giomgr process (part of the three-tiered
architecture) is not used to manage direct-connect connections, though it
can continue to be used for three-tiered client connections simultaneously
with two-tiered direct-connect client connections. 

-----Original Message-----
From: Charles Greer [mailto:CGREER at sonoma-county.org] 
Sent: Thursday, December 11, 2003 12:16 PM
To: godwinl at agr.gc.ca; Jean-Francois.Doyon at CCRS.NRCan.gc.ca;
mapserver-users at lists.gis.umn.edu 
Subject: RE: [Mapserver-users] SDE Direct Connect

I've been following this list for some time now, eager to do more MapServer
development and to contribute to the discussion, but for the time being all
my time is taken up by things like SDE and ArcIMS, 
so now I've got the opportunity to clarify the issues a little bit. 
Caveat
- I'm a DB2 spatial guy 
and don't know Oracle I've become more familiar over time with the bizarre
issues surrounding Oracle and spatial stuff, so...

Oracle has inherent spatial support that is in many ways incompatible with
the way that SDE handles things.  In fact, spatial types in "ArcSDE with
Oracle" are simply BLOBS, whereas the "direct connect" method supports
Oracle's native geometric/geographic types.  The creation of a "direct
connect" for Oracle was ESRI's response to the inherent incompatability
between how Oracle Spatial and ArcSDE handle things.

In other words, direct connect is a way to bypass SDE.  Thus as J.F. says
below MapServer would have to know a lot more about native Oracle Spatial
support in order to support the sde:oracle method.  That's also why, for
Oracle, SDE is 
slower than direct connect in many applications.

ArcSDE is a "lowest common denominator" API, which supports geographic types
even when the underlying RDBMS doesn't (like SQL Server, or the base Oracle
product).  
So its a good API to deliver geography to MapServer, but will not take
advantage of all the 
full range of spatial features in any of the spatial databases.

I hope that helps with the discussion somewhat.  Hope to participate more
fully in the future,

Charles Greer

>>> <Jean-Francois.Doyon at CCRS.NRCan.gc.ca> 12/11/03 08:53AM >>>
Liz,

The Arc clients do provide that flexibility, though I don't think you can do
that right now with MapServer.

The main problem is that the underlying organization of data (i.e. SDE table
structure) doesn't lend itself to that kind of access without the client
explicitly understanding it, which mapserver natively wouldn't ...

If you really wanted you could do something like use PHP/MapScript to read
the data directly out of Oracle, and then put it into a format MapScript
understands and so on, but that's probably much slower than just using the
SDE service !!

As a general rule MapServer's SDE support is fairly basic ... It only reads
basic tables and so on ... If for example you use more advanced features
such as geodatabases, you can no longer really use MapServer with SDE
(That's what I was told by someone who looked at that at least ...)

As a general idea it might be nice to look at expanding MapServer support
for SDE to be more aware of some SDE specific features, such as GeoDatabases
and the direct connect feature ... We'd have to find out whether the latest
Client SDK provides the necessary functionality and look at adding mapserver
support for it.  What you've found about "sde:oracle" suggests that at least
part of the funcitonality is probably there, but mapserver doesn't work with
it nicely ...

But I'm pretty sure the short answer right now is that you can't do it
:)

J.F.

-----Original Message-----
From: mapserver-users-admin at lists.gis.umn.edu 
[mailto:mapserver-users-admin at lists.gis.umn.edu]On Behalf Of Liz Godwin
Sent: Thursday, December 11, 2003 11:24 AM
To: mapserver-users at lists.gis.umn.edu 
Subject: [Mapserver-users] SDE Direct Connect

Hello,

Is it possible to connect directly to SDE through mapserver, or only to the
the sde service?  I have tried playing with the connection string in my map
file, but to no avail.  

I have heard that a direct connection is faster than a service connection,
that is why I would like to try this.

In the SDE config/tuning guide for Oracle, it suggests I use "sde:oracle" as
my service, instead of "esri_sde " for a direct connection. My connection
string in my map file: 
"<hostname>,sde:oracle,<database>,<user>,<password>" (all <parameters>
filled in) My error: msSDELayerOpen(): SDE error.
SE_connection_create():
The file path does not exist, or is too long (> 512 chars), or has invalid
characters.. (-269)

This error suggests to me that the SDE handler in Mapserver doesn't like the
colon.  Is this a bug, or is it just not possible for a direct connection?

Thanks!!

Liz Godwin

_______________________________________________
Mapserver-users mailing list
Mapserver-users at lists.gis.umn.edu 
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users 
_______________________________________________
Mapserver-users mailing list
Mapserver-users at lists.gis.umn.edu 
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users 

_______________________________________________
Mapserver-users mailing list
Mapserver-users at lists.gis.umn.edu 
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users

------_=_NextPart_001_01C3C01D.49858EA0
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3DUS-ASCII">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
5.5.2654.45">
<TITLE>RE: [Mapserver-users] SDE Direct Connect</TITLE>
</HEAD>
<BODY>

After looking through the API, I think you're right =
Steve. There are no specific functions related to direct connect. Giver =
that the C API does support direct connect, it may just a simple matter =
of changing the connection string processing to allow the : in the =
connect string.

<P><FONT SIZE=3D2>Mike</FONT>
</P>

<P><FONT SIZE=3D2>-----Original Message-----</FONT>
<BR><FONT SIZE=3D2>From: Steve Lime [<A =
HREF=3D"mailto:steve.lime at dnr.state.mn.us">mailto:steve.lime at dnr.state.m=
n.us</A>] </FONT>
<BR><FONT SIZE=3D2>Sent: Thursday, December 11, 2003 2:23 PM</FONT>
<BR><FONT SIZE=3D2>To: godwinl at agr.gc.ca; =
Jean-Francois.Doyon at ccrs.nrcan.gc.ca; =
Michael.Smith at erdc.usace.army.mil; mapserver-users at lists.gis.umn.edu; =
CGREER at sonoma-county.org</FONT></P>

<P><FONT SIZE=3D2>Subject: RE: [Mapserver-users] SDE Direct =
Connect</FONT>
</P>
<BR>

Regarding 4, which functions are those? I wasn't able =
to  find any references to direct connect in my admittedly =
outdated SDE C API documentation.

<P><FONT SIZE=3D2>Steve</FONT>
</P>

<P><FONT SIZE=3D2>>>> =
<Michael.Smith at erdc.usace.army.mil> 12/11/2003 12:45:52 PM =
>>></FONT>
<BR><FONT SIZE=3D2>There is some errors is here that I have to =
correct.</FONT>
</P>

1. Oracle does indeed have direct spatial support but =
that is not necessarily incompatitble with SDE. SDE can store data in =
sdebinary format, longraw format (which is being deprecated I belive) =
and native Oracle Spatial format.

2. Direct connect actually has nothing to do with how =
SDE stores information. SDE can operate in two modes. Standard or 3 =
tier mode, in which the client connects to an SDE server which will =
then communicate with the database. Or Direct Connect or 2 tier mode, =
in which some of abilities of the sde server are done in the client. =
ArcGIS can operate this way. The part of SDE that communicates with the =
database in built in to ArcGIS to do this. Mapserver still doesn't have =
to know anything about Oracle Spatial as that is all part of the direct =
connect code built into the client. However, the client abilities are =
limited in that it is a read only connection, even if you have an sde =
license.

3. I wouldn't say that SDE 3 tier is slower than 2 =
tier. As with so many things, it depends.

4. The C Api that mapserver links to does support =
direct connect however I don't believe that the direct connection =
functions have been implemented

<P><FONT SIZE=3D2>Here what Esri says about Direct Connect</FONT>
</P>

Direct Connect is a method for ArcSDE client =
applications to connect directly to the spatial database (hosted on =
either Oracle 8i or Microsoft SQL Server) without having to use the =
ArcSDE application server.

The classic ArcSDE connection architecture is =
three-tiered, it includes an application server process (giomgr), a =
per-connection server process (gsrvr), and a client application. ESRI =
client applications include ArcGIS Desktop and ArcView. Within this =
architecture, the ArcSDE giomgr process manages connections from client =
applications. When a connection to Oracle8i is granted through the =
giomgr process, a gsrvr process is spawned to service the new =
connection. The gsrvr process manages all subsequent query operations =
between the client application and the RDBMS for the duration of the =
connection. Both the giomgr process and gsrvr process exist on the =
server machine that hosts ArcSDE and (usually) Oracle.

Direct-connect is a two-tiered architecture, wherein =
ArcSDE client applications can connect directly to an Oracle8i =
instance. The functionality of the gsrvr process has been added to a =
set of libraries used directly by ArcSDE clients. Typically, ArcSDE =
clients run on remote desktop machines, while the Oracle8i instance =
runs on a server-class machine. The functions of the gsrvr process =
become part of the client application running on the client machine. =
The ArcSDE giomgr process (part of the three-tiered

architecture) is not used to manage direct-connect =
connections, though it can continue to be used for three-tiered client =
connections simultaneously with two-tiered direct-connect client =
connections.

<P><FONT SIZE=3D2>-----Original Message-----</FONT>
<BR><FONT SIZE=3D2>From: Charles Greer [<A =
HREF=3D"mailto:CGREER at sonoma-county.org">mailto:CGREER at sonoma-county.org=
</A>] </FONT>
<BR><FONT SIZE=3D2>Sent: Thursday, December 11, 2003 12:16 PM</FONT>
<BR><FONT SIZE=3D2>To: godwinl at agr.gc.ca; =
Jean-Francois.Doyon at CCRS.NRCan.gc.ca;</FONT>
<BR><FONT SIZE=3D2>mapserver-users at lists.gis.umn.edu </FONT>
<BR><FONT SIZE=3D2>Subject: RE: [Mapserver-users] SDE Direct =
Connect</FONT>
</P>
<BR>

<P><FONT SIZE=3D2>I've been following this list for some time now, =
eager to do more MapServer development and to contribute to the =
discussion, but for the time being all my time is taken up by things =
like SDE and ArcIMS, </FONT></P>

so now I've got the opportunity to clarify the issues =
a little bit. 
 Caveat
 - I'm a DB2 spatial guy 
 and don't know Oracle I've become more familiar over =
time with the bizarre issues surrounding Oracle and spatial stuff, =
so...

<P><FONT SIZE=3D2>Oracle has inherent spatial support that is in many =
ways incompatible with the way that SDE handles things.  In fact, =
spatial types in "ArcSDE with Oracle" are simply BLOBS, =
whereas the "direct connect" method supports Oracle's native =
geometric/geographic types.  The creation of a "direct =
connect" for Oracle was ESRI's response to the inherent =
incompatability between how Oracle Spatial and ArcSDE handle =
things.</FONT></P>

In other words, direct connect is a way to bypass =
SDE.  Thus as J.F. says below MapServer would have to know a lot =
more about native Oracle Spatial support in order to support the =
sde:oracle method.  That's also why, for Oracle, SDE is =

<P><FONT SIZE=3D2>slower than direct connect in many =
applications.</FONT>
</P>

<P><FONT SIZE=3D2>ArcSDE is a "lowest common denominator" =
API, which supports geographic types even when the underlying RDBMS =
doesn't (like SQL Server, or the base Oracle product).  =
</FONT></P>

<P><FONT SIZE=3D2>So its a good API to deliver geography to MapServer, =
but will not take advantage of all the </FONT>
<BR><FONT SIZE=3D2>full range of spatial features in any of the spatial =
databases.</FONT>
</P>

<P><FONT SIZE=3D2>I hope that helps with the discussion somewhat.  =
Hope to participate more fully in the future,</FONT>
</P>

<P><FONT SIZE=3D2>Charles Greer</FONT>
</P>
<BR>

<P><FONT SIZE=3D2>>>> =
<Jean-Francois.Doyon at CCRS.NRCan.gc.ca> 12/11/03 08:53AM =
>>></FONT>
<BR><FONT SIZE=3D2>Liz,</FONT>
</P>

<P><FONT SIZE=3D2>The Arc clients do provide that flexibility, though I =
don't think you can do that right now with MapServer.</FONT>
</P>

The main problem is that the underlying organization =
of data (i.e. SDE table
 structure) doesn't lend itself to that kind of =
access without the client explicitly understanding it, which mapserver =
natively wouldn't ...

<P><FONT SIZE=3D2>If you really wanted you could do something like use =
PHP/MapScript to read the data directly out of Oracle, and then put it =
into a format MapScript understands and so on, but that's probably much =
slower than just using the SDE service !!</FONT></P>

As a general rule MapServer's SDE support is fairly =
basic ... It only reads basic tables and so on ... If for example you =
use more advanced features such as geodatabases, you can no longer =
really use MapServer with SDE (That's what I was told by someone who =
looked at that at least ...)

As a general idea it might be nice to look at =
expanding MapServer support for SDE to be more aware of some SDE =
specific features, such as GeoDatabases and the direct connect feature =
... We'd have to find out whether the latest Client SDK provides the =
necessary functionality and look at adding mapserver support for =
it.  What you've found about "sde:oracle" suggests that =
at least part of the funcitonality is probably there, but mapserver =
doesn't work with it nicely ...

<P><FONT SIZE=3D2>But I'm pretty sure the short answer right now is =
that you can't do it</FONT>
<BR><FONT SIZE=3D2>:)</FONT>
</P>

<P><FONT SIZE=3D2>J.F.</FONT>
</P>

<P><FONT SIZE=3D2>-----Original Message-----</FONT>
<BR><FONT SIZE=3D2>From: mapserver-users-admin at lists.gis.umn.edu =
</FONT>
<BR><FONT SIZE=3D2>[<A =
HREF=3D"mailto:mapserver-users-admin at lists.gis.umn.edu">mailto:mapserver=
-users-admin at lists.gis.umn.edu</A>]On Behalf Of Liz Godwin</FONT>
<BR><FONT SIZE=3D2>Sent: Thursday, December 11, 2003 11:24 AM</FONT>
<BR><FONT SIZE=3D2>To: mapserver-users at lists.gis.umn.edu </FONT>
<BR><FONT SIZE=3D2>Subject: [Mapserver-users] SDE Direct Connect</FONT>
</P>
<BR>

<P><FONT SIZE=3D2>Hello,</FONT>
</P>

<P><FONT SIZE=3D2>Is it possible to connect directly to SDE through =
mapserver, or only to the the sde service?  I have tried playing =
with the connection string in my map file, but to no avail.  =
</FONT></P>

<P><FONT SIZE=3D2>I have heard that a direct connection is faster than =
a service connection, that is why I would like to try this.</FONT>
</P>

In the SDE config/tuning guide for Oracle, it =
suggests I use "sde:oracle" as my service, instead of =
"esri_sde " for a direct connection. My connection string in =
my map file:

<P><FONT =
SIZE=3D2>"<hostname>,sde:oracle,<database>,<user>=
,<password>"  (all <parameters> filled in) My =
error:  msSDELayerOpen(): SDE error.</FONT></P>

SE_connection_create():
 The file path does not exist, or is too long (> =
512 chars), or has invalid characters.. (-269)

<P><FONT SIZE=3D2>This error suggests to me that the SDE handler in =
Mapserver doesn't like the colon.  Is this a bug, or is it just =
not possible for a direct connection?</FONT></P>

<P><FONT SIZE=3D2>Thanks!!</FONT>
</P>

<P><FONT SIZE=3D2>Liz Godwin</FONT>
</P>
<BR>
<BR>

<P><FONT =
SIZE=3D2>_______________________________________________</FONT>
<BR><FONT SIZE=3D2>Mapserver-users mailing list</FONT>
<BR><FONT SIZE=3D2>Mapserver-users at lists.gis.umn.edu </FONT>
<BR><FONT SIZE=3D2><A =
HREF=3D"http://lists.gis.umn.edu/mailman/listinfo/mapserver-users" =
TARGET=3D"_blank">http://lists.gis.umn.edu/mailman/listinfo/mapserver-us=
ers</A> </FONT>
<BR><FONT =
SIZE=3D2>_______________________________________________</FONT>
<BR><FONT SIZE=3D2>Mapserver-users mailing list</FONT>
<BR><FONT SIZE=3D2>Mapserver-users at lists.gis.umn.edu </FONT>
<BR><FONT SIZE=3D2><A =
HREF=3D"http://lists.gis.umn.edu/mailman/listinfo/mapserver-users" =
TARGET=3D"_blank">http://lists.gis.umn.edu/mailman/listinfo/mapserver-us=
ers</A> </FONT>
</P>

<P><FONT =
SIZE=3D2>_______________________________________________</FONT>
<BR><FONT SIZE=3D2>Mapserver-users mailing list</FONT>
<BR><FONT SIZE=3D2>Mapserver-users at lists.gis.umn.edu </FONT>
<BR><FONT SIZE=3D2><A =
HREF=3D"http://lists.gis.umn.edu/mailman/listinfo/mapserver-users" =
TARGET=3D"_blank">http://lists.gis.umn.edu/mailman/listinfo/mapserver-us=
ers</A></FONT>
</P>

</BODY>
</HTML>
------_=_NextPart_001_01C3C01D.49858EA0--