[Mapserver-users] Sensor Web Enablement (SWE), YSI and MapServer

Wed Jul 21 12:11:00 EDT 2004

Hi Gerry,

I sent an email to YSI people mentioning I saw the letters "YSI" in XML tag
names in the SWE working documents on the OGC site, so I thought maybe YSI
were going to make their data format align towards a standard in the near
future (but then, the documents I saw date back to 2002).  No answer yet.
Their DAT format is proprietary and they won't let you know what the data
structure is.

What I would like to do in the short term is sort out what technologies are
fit for my problem (a problem you have already tackled very well, looking at
your site).  I am a newbie to all this so bear with me a little ;).  I
thought I needed SWE for what it is, that is web enablement of sensor data.
To me, that means an open data format that people can freely play with.  A
format that not only contains data, but also sensor sensitivity, sensor
problems (e.g. saturation), settings, value domain for each measured
parameter and all sorts of metadata that clients consuming the data might
want to know if they are doing any serious work.  Of course, showing up a
temperature value over the internet does not necessarily require all that
metadata, but I want to have it for power users.  Meaningful data
interoperability requires that such metadata be available.  Since a standard
format for the YSI sonde data is not coming any time soon, I guess I'll have
to resort to their CSV interchange format.  I would have preferred to gain
access to the binary data and serialize it myself into some storable XML-ish
format, in which I could have for example added some important metadata
before serailization, but again CSV is all there is to people that don't
want to use YSI software.

I guess once I get the sonde data in CSV, I can pump it into PostGIS and use
MapServer WFS.  Right ?  I still want to make a sonde data simulator, though
...

Thanx for your input

----- Original Message ----- 
From: "Gerry Creager N5JXS" <gerry.creager at tamu.edu>
To: "Kralidis,Tom [Burlington]" <Tom.Kralidis at ec.gc.ca>;
<Mapserver-users at lists.gis.umn.edu>
Cc: <ymoisan at groupesm.com>
Sent: Wednesday, July 21, 2004 10:02 AM
Subject: Re: [Mapserver-users] Sensor Web Enablement (SWE), YSI and
MapServer

> I'm starting to take an interest in SWE at OGC.  And since I've got a
> fairly extensive site (http://mesonet.tamu.edu/currentwx.html) for
> sensor data it'd make sense:-)
>
> We're currently doing WMS and if I can get time to upgrade to 4.2, we'll
> be doing WFS.  That's the approach I recommend, too.
>
> I'd be willing to start writing up something for a wiki addition on
> sensor data acquisition, but I suspect it's really low-level stuff to
> most of the folks here.
>
> Data structure determination is a big player.  It's also often something
> that's not done by design, but initiated by accident and improved
> iteratively ina crisis management manner.  This from experience,
> although we did start with a design.  Really.  I mean it!
>
> If there's interest, I'll expand further.  There are others here doing
> the same sorts of things...
>
> gerry
>
> Kralidis,Tom [Burlington] wrote:
> > Hi Yves,
> >
> > Quite simply, UMN MapServer currently does not support the OGC SWE
> > approaches.  Most OGC SWE approaches are in discussion paper form and
> > not adopted OGC specifications.  I would love to see MapServer support
> > OGC SWE, once OGC SWE specs are publically adopted.
> >
> > If you want to do this with UMN MapServer, my initial suggestion is
> > setup as OGC:WMS and OGC:WFS.  This is what we've done with some of our
> > water quality monitoring data with UMN MapServer.  We do have OGC:SCS of
> > some of our sensor stuff, deployed with a commercial solution.
> >
> > Hope this helps.
> >
> > ..Tom
> >
> > =========================
> > Tom Kralidis
> > Systems Scientist
> > Environment Canada
> > Tel: +01-905-336-4409
> > http://www.ec.gc.ca/
> >
> >
> > -----Original Message-----
> > From: mapserver-users-admin at lists.gis.umn.edu
> > [mailto:mapserver-users-admin at lists.gis.umn.edu] On Behalf Of Yves
> > Moisan
> > Sent: Tuesday, July 20, 2004 10:49
> > To: Mapserver-users at lists.gis.umn.edu
> > Subject: [Mapserver-users] Sensor Web Enablement (SWE), YSI and
> > MapServer
> >
> >
> > Hi All,
> >
> > One of the objectives I had attending the recent OGIS conference,
> > besides getting to know you all, was to understand how to
> > store/retrieve/process in situ sensor data, in my particular case water
> > quality data coming from YSI 6600 sondes, and deliver it in a map over
> > the web.  I wanted to be able to draw the envelope of the "data service
> > tier" vs any client app, because I want to build my own client without
> > bundling the data tier into it.  Data must remain accessible to other
> > parties and of course MapServer/Server is where the data tier ends.  At
> > the time, I figured MapServer could spew out maps with little dots
> > showing sensor locations, people click on the dots, get the data in
> > tabular format.  Standard stuff.  I hadn't thought of sensor data
> > *standard storage* per se, though.
> >
> > I am currently developing a small Python utility to gobble up YSI 6600
> > sonde data.  For now, I want to be able to simulate sonde data using a
> > "data generator" that would write such binary files for the sake of
> > having a system pseudo-working.  We are not getting a sonde any time
> > soos, but I want to get going.  When I get a sensor working, then I want
> > to use such a utility to QA sensor data and potentially to view it in
> > environments such as SciPy.  I don't want to use YSI's Ecowatch, at
> > least not in a production environment.  The data format is binary and as
> > of yet I have no clue as to how to decypher it, so I am playing around
> > with the file.  BTW, anyone knows of a "binary file explorer" type of
> > thing that would try and find patterns in arbitrary binary files and
> > suggest data extraction schemes, e.g. firts 4 bytes are chars, next 600
> > are floats ... ?
> >
> > Anyhow, the jist of this message is about the SWE initiative -- which I
> > have just found out about today -- and its relation to MapServer.  I was
> > hoping to store sensor data in a sort of staging environment.  At the
> > first level, the original .DAT files.  At the second level, the data
> > would be stored in more pallatable formats for HTML/SVG output and it
> > would have been QA'd.  For that last level, I am not sure I would want
> > to serialize the data in XML (but then, maybe yes!) before stuffing it
> > in a DB.  Since I am using Python, I thought maybe of pickling it to a
> > flat file.  That would allow me to write a QA module (checking for NaN's
> > etc.) in Python and that would also let me use Python XML serialization
> > modules to pipe it through (using mod_python ?) before sending out to
> > the Web.  That's the flatfile/ObjectDB option.  Another would be to use
> > the flat file for QA, then spew the results out in tabular format into
> > PostGIS.  That'a probably closest to what would be amenable to MapSever.
> > What is the status of SWE in the context of MapServer ?  I gather SWE it
> > is not a spec yet, but is it coming strong?
> >
> > The point is simple : I have sensor data that I want to show up on a map
> > and above all store it so it can be "OGC Serviced" when SWE clients
> > start to appear.  I'll probably have to write a SWE client.  If I do, I
> > want to do it with Python (thanx Sean for that point on your slide, but
> > I was already convinced!) and I'll want an adapter for Zope and maybe
> > Twisted (also my problem).  I just want to do it in the most "OGC
> > compliant", Pythonic way.
> >
> > Any pointers ideas out there ?
> >
> > Thanx
> >
> > Yves Moisan
> >
> > _______________________________________________
> > Mapserver-users mailing list
> > Mapserver-users at lists.gis.umn.edu
> > http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
>
> -- 
> Gerry Creager -- gerry.creager at tamu.edu
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
> Page: 979.228.0173
> Office: 903A Eller Bldg, TAMU, College Station, TX 77843
>