[Mapserver-users] Sensor Web Enablement (SWE), YSI and MapServer

Tue Jul 20 07:48:38 PDT 2004

This is a multi-part message in MIME format.

------=_NextPart_000_01C2_01C46E47.1D053370
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi All,

One of the objectives I had attending the recent OGIS conference, =
besides getting to know you all, was to understand how to =
store/retrieve/process in situ sensor data, in my particular case water =
quality data coming from YSI 6600 sondes, and deliver it in a map over =
the web.  I wanted to be able to draw the envelope of the "data service =
tier" vs any client app, because I want to build my own client without =
bundling the data tier into it.  Data must remain accessible to other =
parties and of course MapServer/Server is where the data tier ends.  At =
the time, I figured MapServer could spew out maps with little dots =
showing sensor locations, people click on the dots, get the data in =
tabular format.  Standard stuff.  I hadn't thought of sensor data =
*standard storage* per se, though.

I am currently developing a small Python utility to gobble up YSI 6600 =
sonde data.  For now, I want to be able to simulate sonde data using a =
"data generator" that would write such binary files for the sake of =
having a system pseudo-working.  We are not getting a sonde any time =
soos, but I want to get going.  When I get a sensor working, then I want =
to use such a utility to QA sensor data and potentially to view it in =
environments such as SciPy.  I don't want to use YSI's Ecowatch, at =
least not in a production environment.  The data format is binary and as =
of yet I have no clue as to how to decypher it, so I am playing around =
with the file.  BTW, anyone knows of a "binary file explorer" type of =
thing that would try and find patterns in arbitrary binary files and =
suggest data extraction schemes, e.g. firts 4 bytes are chars, next 600 =
are floats ... ?

Anyhow, the jist of this message is about the SWE initiative -- which I =
have just found out about today -- and its relation to MapServer.  I was =
hoping to store sensor data in a sort of staging environment.  At the =
first level, the original .DAT files.  At the second level, the data =
would be stored in more pallatable formats for HTML/SVG output and it =
would have been QA'd.  For that last level, I am not sure I would want =
to serialize the data in XML (but then, maybe yes!) before stuffing it =
in a DB.  Since I am using Python, I thought maybe of pickling it to a =
flat file.  That would allow me to write a QA module (checking for NaN's =
etc.) in Python and that would also let me use Python XML serialization =
modules to pipe it through (using mod_python ?) before sending out to =
the Web.  That's the flatfile/ObjectDB option.  Another would be to use =
the flat file for QA, then spew the results out in tabular format into =
PostGIS.  That'a probably closest to what would be amenable to MapSever. =
 What is the status of SWE in the context of MapServer ?  I gather SWE =
it is not a spec yet, but is it coming strong?

The point is simple : I have sensor data that I want to show up on a map =
and above all store it so it can be "OGC Serviced" when SWE clients =
start to appear.  I'll probably have to write a SWE client.  If I do, I =
want to do it with Python (thanx Sean for that point on your slide, but =
I was already convinced!) and I'll want an adapter for Zope and maybe =
Twisted (also my problem).  I just want to do it in the most "OGC =
compliant", Pythonic way.

Any pointers ideas out there ?

Thanx

Yves Moisan
------=_NextPart_000_01C2_01C46E47.1D053370
Content-Type: text/html;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2800.1400" name=3DGENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>
<DIV><FONT face=3DArial size=3D2>Hi All,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>One of the objectives I had attending =
the recent=20
OGIS conference, besides getting to know you all, was to understand how =
to=20
store/retrieve/process in situ sensor data, in my particular case water =
quality=20
data coming from YSI 6600 sondes, and deliver it in a map over the =
web.  I=20
wanted to be able to draw the envelope of the "data service tier" vs any =
client=20
app, because I want to build my own client without bundling the data =
tier into=20
it.  Data must remain accessible to other parties and of course=20
MapServer/Server is where the data tier ends.  At the time, I =
figured=20
MapServer could spew out maps with little dots showing sensor locations, =
people=20
click on the dots, get the data in tabular format.  Standard =
stuff.  I=20
hadn't thought of sensor data *standard storage* per se, =
though.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>I am currently developing a small =
Python utility to=20
gobble up YSI 6600 sonde data.  For now, I want to be able to =
simulate=20
sonde data using a "data generator" that would write such binary files =
for the=20
sake of having a system pseudo-working.  We are not getting a sonde =
any=20
time soos, but I want to get going.  When I get a sensor working, =
then I=20
want to use such a utility to QA sensor data and potentially to view it =
in=20
environments such as SciPy.  I don't want to use YSI's Ecowatch, at =
least=20
not in a production environment.  The data format is binary and as =
of yet I=20
have no clue as to how to decypher it, so I am playing around with the=20
file.  BTW, anyone knows of a "binary file explorer" type of thing =
that=20
would try and find patterns in arbitrary binary files and suggest data=20
extraction schemes, e.g. firts 4 bytes are chars, next 600 are floats =
...=20
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>Anyhow, the jist of this message is =
about the SWE=20
initiative -- which I have just found out about today -- and its =
relation to=20
MapServer.  I was hoping to store sensor data in a sort of staging=20
environment.  At the first level, the original .DAT files.  At =
the=20
second level, the data would be stored in more pallatable formats for =
HTML/SVG=20
output and it would have been QA'd.  For that last level, I am not =
sure I=20
would want to serialize the data in XML (but then, maybe yes!) before =
stuffing=20
it in a DB.  Since I am using Python, I thought maybe of pickling =
it to a=20
flat file.  That would allow me to write a QA module (checking for=20
NaN's etc.) in Python and that would also let me use Python XML=20
serialization modules to pipe it through (using mod_python ?) before =
sending out=20
to the Web.  That's the flatfile/ObjectDB option.  Another =
would be to=20
use the flat file for QA, then spew the results out in tabular format =
into=20
PostGIS.  That'a probably closest to what would be amenable to=20
MapSever.  What is the status of SWE in the context of MapServer =
?  I=20
gather SWE it is not a spec yet, but is it coming strong?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>The point is simple : I have sensor =
data that I=20
want to show up on a map and above all store it so it can be "OGC =
Serviced" when=20
SWE clients start to appear.  I'll probably have to write a SWE=20
client.  If I do, I want to do it with Python (thanx Sean for that =
point on=20
your slide, but I was already convinced!) and I'll want an adapter for =
Zope and=20
maybe Twisted (also my problem).  I just want to do it in the most =
"OGC=20
compliant", Pythonic way.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>Any pointers ideas out there =
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>Thanx</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2>Yves =
Moisan</FONT></DIV></FONT></DIV></BODY></HTML>

------=_NextPart_000_01C2_01C46E47.1D053370--