[Mapserver-users] Dusting off the ArcIMS wrapper project

[Paul Ramsey]

This is a multi-part message in MIME format.
--------------000705030302040800050208
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Ed,
Start with the beta perl version, at the postgis.refractions.net site, 
it will give you an idea of some of the operating modes, and the "right" 
way to talk XML to ArcMap and other ESRI client (you can crash ArcMap by 
just giving it a slightly different (but legal) "<?xml ?>" header than 
it expects). I sniffed the network using 'ngrep', an exceptional tool. 
If you aim to implement full support (ArcMap as a client) and want to 
support feature identification, you'll also have to reverse engineer the 
binary stream format (hexedit was useful for that). I got a good start 
on that, and am attaching my draft specification.
TTYL,
Paul

Ed McNierney wrote:
> Folks -
> 
> I'd like to spend some time on getting a real ArcIMS wrapper for
> MapServer going.  Waiting for ESRI to actually implement open server
> support in their client products is like waiting for the next glacier
> to show up.
> 
> There were a few conversations a while back, but things have gone
> quiet.  I've been looking through the ArcXML docs and things look
> pretty doable, but I'm not at all familiar with the standard
> client/server "conversation" between a client and an ArcIMS server.
> Anyone with that experience would be very helpful.
> 
> In addition, if anyone knows good "sniffer" tools to log the
> conversation (I've got ArcGIS 8.2 and can talk to ArcIMS servers) I'd
> appreciate pointers.  Thanks.
> 
> - Ed
> 
> Ed McNierney President and Chief Mapmaker TopoZone.com / Maps a la
> carte, Inc. 73 Princeton Street, Suite 305 North Chelmsford, MA
> 01863 ed at topozone.com (978) 251-4242
> 
> _______________________________________________ Mapserver-users
> mailing list Mapserver-users at lists.gis.umn.edu 
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users


-- 
       __
      /
      | Paul Ramsey
      | Refractions Research
      | Email: pramsey at refractions.net
      | Phone: (250) 885-0632
      \_

--------------000705030302040800050208
Content-Type: text/plain;
 name="ims_binary_spec.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="ims_binary_spec.txt"

ArcIMS GET_FEATURES "Compressed Binary Format"

GENERAL PRINCIPLES

When a client calls GET_FEATURES on ArcIMS without specifying an outputmode,
the default mode is to return "Compressed Binary Format" (CBF). CBF appears to
be completely undocumented. I have attempted to understand what makes up a CBF
file.

CBF is not OpenGIS Well-Known Binary, it is not the ESRI Shape File format, and
it is not the "compressed binary" format used by ArcSDE. It is an odd hybrid all
its own, combining storage for attribute information alongside coordinate
information.

This specification still has some holes, where "mystery bytes" live whose
purpose has not been determined yet.

The CBF file has the following general format:

  FILE HEADER
  [ATTRIBUTE FIELD DESCRIPTION 1]
  [ATTRIBUTE FIELD DESCRIPTION 2]
  [ATTRIBUTE FIELD DESCRIPTION 3]
  ...
  [ATTRIBUTE FIELD DESCRIPTION N]
  ATTRIBUTE FOOTER
  [SHAPE HEADER 1]
  [SHAPE COORDINATES 1]
  [SHAPE ATTRIBUTE VALUES 1]
  [SHAPE HEADER 2]
  [SHAPE COORDINATES 2]
  [SHAPE ATTRIBUTE VALUES 2]
  ...
  [SHAPE HEADER X]
  [SHAPE COORDINATES X]
  [SHAPE ATTRIBUTE VALUES X]
  FILE END


FILE HEADER

Bytes 1-4

  Standard header values. The first four bytes appear to be the same all
  the time. The standard values for the header (in hex) are: 05 18 01 00

Bytes 5-8

  Unknown header bytes. These four bytes vary for every sample I have, but I
  have not figured out what they mean yet.

ATTRIBUTE FIELD DESCRIPTION

This gives information about the attribute fields. The type is important,
because type is necessary in order to read the attribute values farther below.
Integer types will always have values stored in 4-byte integers. Floating point
types will always have values stored in 8-byte doubles. Strings will be stored
as variable length, with null-termination to indicate the end of the value.

Shape fields (-98) always have 0 precision and size.
Object ID fields always have 0 precision and size 10.

Bytes 1-2

  Field type number. Signed two-byte integer.

   91 - Date
   12 - Variable Length String
    8 - Floating Point 8 Bytes
    6 - Floating Point 4 Bytes
    5 - Integer 2 Bytes
    4 - Integer 4 Bytes
   -5 - Integer 8 Bytes
   -7 - Boolean
  -98 - Shape
  -99 - Object ID

Byte 3

  Field precision. Unsigned, one byte. Corresponds to usual DBF file description
  of numeric precision. e.g. number(12,5) where 5 is the precision.

Bytes 4-5

  Field size. Unsigned, two bytes. Corresponds to usual DBF file description
  of numeric size. e.g. number(12,5) where 12 is the size.

Bytes 6-N

  Field name. Null-terminated character array. Read until you hit the null
  value (00).


ATTRIBUTE FOOTER

Bytes 1-9|1

In an object with both shapes and attributes, after the attribute field information, there always 
seems to be a constant set of nine bytes: 08 43 41 C3 79 37 E0 80 00
In an object with only shapes: 08 40 44 94 23 DD 91 43 82
In an object with only attributes: 0A (And 0A repeats between each set of attribute values.)


SHAPE HEADER

I do not competely understand the shape header right now. In addition to the
information given below about the basic structure (with types 1, 2 and 3)
there is another type, 9, which occurs only sometimes, and is followed by a
variable number of bytes of mostly null values. I am guessing for now that it is
possible to write valid CBF files completely ignoring the 9 type and just using
the 1, 2 and 3 types, which are at least self-consistent.

Byte 1

  Shape type. One-byte value.

  1 - MultiPoint
  2 - MultiLine
  3 - MultiPolygon

Bytes 2-5

  Number of parts in the shape (points in multipoint, lines in multiline, rings
  plus holes in multipolygon). Signed four-byte integer.

SHAPE COORDINATES

Each part of the shape consists of a length, and then the double-value
vertices. The vertices are stored as offsets relative to the previous vertex.
So, the first vertex is a spatial coordinate, and all the rest are relative
offsets from the previous vertex.

Bytes 1-4

  Number of vertices in the part. Signed four-byte integer.

Bytes 5-N

  Vertices as eight-byte doubles, calculated in the following manner:

  Coordinate 1: X1, Y1
  Coordinate 2: (X2-X1), (Y2-Y1)
  Coordinate 3: (X3-X2), (Y3-Y2)
  ...
  Coordinate N: (XN - X(N-1)), (YN - Y(N-1))

Example:

  Spatial coordinates:

  (2,3) (2,4) (5,6) (8,3)

  Corresponding ESRI CBF coordinates:

  (2,3) (0,1) (3,2) (3,-3)

SHAPE ATTRIBUTE VALUES

Attribute values corresponding to the attribute fields defined in the ATTRIBUTE
FIELD DESCRIPTIONGS. Integers are encoded as 4-byte signed integers. Floats are
encoded as 8-byte doubles. Strings are null-terminated variable-length arrays of
characters (you know you're at the end when you hit the null).

The "SHAPE" attribute field is *ignored* in the attribute values. No value is
entered for it at all, just skip right over.


FILE END

The CBF file always ends with a single end-of-file hex byte: FF




--------------000705030302040800050208--