[Mapserver-users] Sensor Web Enablement (SWE), YSI and MapServer

[Gerry Creager n5jxs]

I'll try not to sound too pedantic, but it may be a little difficult in 
a couple of places...  In fact, I'll get that out of the way first.

<rant>Let's consider that 'Metadata' are data describing the data.  Thus 
we have a method of describing how data are collected: methods, 
instruments, calibration specifics, units, sensitivities, etc.  Your 
catalog of such, below, is a good start.

What often gets missed in the early generation of metadata is a 
consistent method of data characterization... standardization of 
collection, characterization, storage, units, necessary elements, etc. 
The standards piece needs to be considered as either a precursor or an 
equal partner to the creation of metadata and metadata descriptors.
</rant>

OK.  That's out of my system.  Metadata's a pain to populate, but vital. 
  It's also sexier than creating and adhering to data standards, hence 
my frustration...

More inline:

Yves Moisan wrote:
> Hi Gerry,
> 
> I sent an email to YSI people mentioning I saw the letters "YSI" in XML tag
> names in the SWE working documents on the OGC site, so I thought maybe YSI
> were going to make their data format align towards a standard in the near
> future (but then, the documents I saw date back to 2002).  No answer yet.
> Their DAT format is proprietary and they won't let you know what the data
> structure is.

Yellow Springs Instruments?  The thermocouple folks?  I'd be very 
surprised if the DAT format isn't explained or reverse-engineerable with 
a little bit of work...

> What I would like to do in the short term is sort out what technologies are
> fit for my problem (a problem you have already tackled very well, looking at
> your site).  I am a newbie to all this so bear with me a little ;).  I
> thought I needed SWE for what it is, that is web enablement of sensor data.
> To me, that means an open data format that people can freely play with.  A
> format that not only contains data, but also sensor sensitivity, sensor
> problems (e.g. saturation), settings, value domain for each measured
> parameter and all sorts of metadata that clients consuming the data might
> want to know if they are doing any serious work.  Of course, showing up a
> temperature value over the internet does not necessarily require all that
> metadata, but I want to have it for power users.  Meaningful data
> interoperability requires that such metadata be available.  Since a standard
> format for the YSI sonde data is not coming any time soon, I guess I'll have
> to resort to their CSV interchange format.  I would have preferred to gain
> access to the binary data and serialize it myself into some storable XML-ish
> format, in which I could have for example added some important metadata
> before serailization, but again CSV is all there is to people that don't
> want to use YSI software.

Barring either a reverse engineered solution, or a key to their format, 
using their CVS interchange will allow a pretty quick is only slightly 
more complicated, approach.

The key is having a set of (standardized) metadata available that allows 
you to determine the data parsing, precision and relative rank 
importance for storage.  A lot of instruments send a lot of data, some 
of which may or may not be necessary, and may actually detract from your 
work by adding confusion.

The approach we use is to acquire the data, stuff it immediately into a 
database (PostGres/PostGIS) then perform operations on the data.  Once 
it hits the database/archive, its manipulations are not re-stored.  We 
operate on the original data for any transformations, and then report it 
that way.  Thus, we have the original data to fall back on, and a 
separate archive of the metadata associated with it.

> I guess once I get the sonde data in CSV, I can pump it into PostGIS and use
> MapServer WFS.  Right ?  I still want to make a sonde data simulator, though
> ...

I see no reason why not.  The idea, by the way, of a sonde simulator is 
a good one.

> Thanx for your input

Good luck,  and feel free to ask more questions.

Regards,
Gerry
> ----- Original Message ----- 
> From: "Gerry Creager N5JXS" <gerry.creager at tamu.edu>
> To: "Kralidis,Tom [Burlington]" <Tom.Kralidis at ec.gc.ca>;
> <Mapserver-users at lists.gis.umn.edu>
> Cc: <ymoisan at groupesm.com>
> Sent: Wednesday, July 21, 2004 10:02 AM
> Subject: Re: [Mapserver-users] Sensor Web Enablement (SWE), YSI and
> MapServer
> 
> 
> 
>>I'm starting to take an interest in SWE at OGC.  And since I've got a
>>fairly extensive site (http://mesonet.tamu.edu/currentwx.html) for
>>sensor data it'd make sense:-)
>>
>>We're currently doing WMS and if I can get time to upgrade to 4.2, we'll
>>be doing WFS.  That's the approach I recommend, too.
>>
>>I'd be willing to start writing up something for a wiki addition on
>>sensor data acquisition, but I suspect it's really low-level stuff to
>>most of the folks here.
>>
>>Data structure determination is a big player.  It's also often something
>>that's not done by design, but initiated by accident and improved
>>iteratively ina crisis management manner.  This from experience,
>>although we did start with a design.  Really.  I mean it!
>>
>>If there's interest, I'll expand further.  There are others here doing
>>the same sorts of things...
>>
>>gerry
>>
>>Kralidis,Tom [Burlington] wrote:
>>
>>>Hi Yves,
>>>
>>>Quite simply, UMN MapServer currently does not support the OGC SWE
>>>approaches.  Most OGC SWE approaches are in discussion paper form and
>>>not adopted OGC specifications.  I would love to see MapServer support
>>>OGC SWE, once OGC SWE specs are publically adopted.
>>>
>>>If you want to do this with UMN MapServer, my initial suggestion is
>>>setup as OGC:WMS and OGC:WFS.  This is what we've done with some of our
>>>water quality monitoring data with UMN MapServer.  We do have OGC:SCS of
>>>some of our sensor stuff, deployed with a commercial solution.
>>>
>>>Hope this helps.
>>>
>>>..Tom
>>>
>>>=========================
>>>Tom Kralidis
>>>Systems Scientist
>>>Environment Canada
>>>Tel: +01-905-336-4409
>>>http://www.ec.gc.ca/
>>>
>>>
>>>-----Original Message-----
>>>From: mapserver-users-admin at lists.gis.umn.edu
>>>[mailto:mapserver-users-admin at lists.gis.umn.edu] On Behalf Of Yves
>>>Moisan
>>>Sent: Tuesday, July 20, 2004 10:49
>>>To: Mapserver-users at lists.gis.umn.edu
>>>Subject: [Mapserver-users] Sensor Web Enablement (SWE), YSI and
>>>MapServer
>>>
>>>
>>>Hi All,
>>>
>>>One of the objectives I had attending the recent OGIS conference,
>>>besides getting to know you all, was to understand how to
>>>store/retrieve/process in situ sensor data, in my particular case water
>>>quality data coming from YSI 6600 sondes, and deliver it in a map over
>>>the web.  I wanted to be able to draw the envelope of the "data service
>>>tier" vs any client app, because I want to build my own client without
>>>bundling the data tier into it.  Data must remain accessible to other
>>>parties and of course MapServer/Server is where the data tier ends.  At
>>>the time, I figured MapServer could spew out maps with little dots
>>>showing sensor locations, people click on the dots, get the data in
>>>tabular format.  Standard stuff.  I hadn't thought of sensor data
>>>*standard storage* per se, though.
>>>
>>>I am currently developing a small Python utility to gobble up YSI 6600
>>>sonde data.  For now, I want to be able to simulate sonde data using a
>>>"data generator" that would write such binary files for the sake of
>>>having a system pseudo-working.  We are not getting a sonde any time
>>>soos, but I want to get going.  When I get a sensor working, then I want
>>>to use such a utility to QA sensor data and potentially to view it in
>>>environments such as SciPy.  I don't want to use YSI's Ecowatch, at
>>>least not in a production environment.  The data format is binary and as
>>>of yet I have no clue as to how to decypher it, so I am playing around
>>>with the file.  BTW, anyone knows of a "binary file explorer" type of
>>>thing that would try and find patterns in arbitrary binary files and
>>>suggest data extraction schemes, e.g. firts 4 bytes are chars, next 600
>>>are floats ... ?
>>>
>>>Anyhow, the jist of this message is about the SWE initiative -- which I
>>>have just found out about today -- and its relation to MapServer.  I was
>>>hoping to store sensor data in a sort of staging environment.  At the
>>>first level, the original .DAT files.  At the second level, the data
>>>would be stored in more pallatable formats for HTML/SVG output and it
>>>would have been QA'd.  For that last level, I am not sure I would want
>>>to serialize the data in XML (but then, maybe yes!) before stuffing it
>>>in a DB.  Since I am using Python, I thought maybe of pickling it to a
>>>flat file.  That would allow me to write a QA module (checking for NaN's
>>>etc.) in Python and that would also let me use Python XML serialization
>>>modules to pipe it through (using mod_python ?) before sending out to
>>>the Web.  That's the flatfile/ObjectDB option.  Another would be to use
>>>the flat file for QA, then spew the results out in tabular format into
>>>PostGIS.  That'a probably closest to what would be amenable to MapSever.
>>>What is the status of SWE in the context of MapServer ?  I gather SWE it
>>>is not a spec yet, but is it coming strong?
>>>
>>>The point is simple : I have sensor data that I want to show up on a map
>>>and above all store it so it can be "OGC Serviced" when SWE clients
>>>start to appear.  I'll probably have to write a SWE client.  If I do, I
>>>want to do it with Python (thanx Sean for that point on your slide, but
>>>I was already convinced!) and I'll want an adapter for Zope and maybe
>>>Twisted (also my problem).  I just want to do it in the most "OGC
>>>compliant", Pythonic way.
>>>
>>>Any pointers ideas out there ?
>>>
>>>Thanx
>>>
>>>Yves Moisan
>>>
>>>_______________________________________________
>>>Mapserver-users mailing list
>>>Mapserver-users at lists.gis.umn.edu
>>>http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
>>
>>-- 
>>Gerry Creager -- gerry.creager at tamu.edu
>>Texas Mesonet -- AATLT, Texas A&M University
>>Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
>>Page: 979.228.0173
>>Office: 903A Eller Bldg, TAMU, College Station, TX 77843
>>
> 
> 
> _______________________________________________
> Mapserver-users mailing list
> Mapserver-users at lists.gis.umn.edu
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users

-- 
Gerry Creager -- gerry.creager at tamu.edu
Network Engineering -- AATLT, Texas A&M University	
Cell: 979.229.5301 Office: 979.458.4020
FAX:  979.847.8578 Pager:  979.228.0173
Office: 903A Eller Bldg, TAMU, College Station, TX 77843