[OSGeo-Discuss] Mobile GPS data collection, summary of sorts.

Just van den Broecke just at justobjects.nl
Thu Aug 3 03:27:22 PDT 2006


Bob Basques wrote:

> 
> 
> Just van den Broecke wrote:
> 
>> (Back from holiday...just went through the MGPSDC threads...wew..)
>>
>> What I can offer is the GeoTracing (www.geotracing.com) platform, not 
>> as much not as the solution but as a way to try out, share ideas, 
>> design issues and experience. If the FOSS4G submission is accepted I 
>> hope to present and/or demo in Lausanne (and/or have interesting 
>> discussions with you).
> 
> I actually looked at this when you posted (just before you went on 
> Vacation??).
yes, I thought to expand some more and give out the (hidden) URLs to try 
out things
> 
>>
>> I plan to move the code to geotracing.codehaus.org somewhere this 
>> month (time!).  Documentation (time!) is lacking, but several working 
>> applications have been delivered that do personalized GPS tracking 
>> from a mobile phone through a server finally into a (MySQL 
>> non-spatial) DB.
> 
> Geez, somebody else that think like me.  Non-Spatial database, oh no!!  :c)
the data/queries were simple (see your comment below)
> 
> So it this set up as a standalone operation entirely on the phone (as an 
> optional mode of operation) or is the data sent directly to a Mainserver 
> for Storage?
> 
> MYSQL work great for simple datasets, even simple to moderately complex 
> Spatial databases, as long as the data are indexed approriately.
ok, still need to figure out if one can set indexes on REAL (lat/lon)
> 
>> Application-requirements were the main driver for the project. The 
>> underlying Media/CMS platform KeyWorx (www.keyworx.org) that e.g. 
>> describes protocol concepts is reasonably documented: 
>> http://kwx.dev.waag.org/docs
>>
>> If you have a smartphone with bluetooth (JSR-82) plus bt gps you may 
>> try out the MobiTracer client by signing up and downloading OTA: 
>> http://www.geotracing.com/gt/my
>> The related TraceLand webapp runs at http://www.geotracing.com/gt
>> For MacOSX a _very_ experimental browser-AJAX-based client is available:
>> http://www.geotracing.com/gt/webtracer and 
>> http://www.geotracing.com/gt/gpsdoor (small CLI Python program similar 
>> to gpsd but HTTP-service-based)
> 
> Is there a TREO 650/Socket (BT) GPS package?
no(t yet)
> 
>>
>> Other:
>> - client/server comms uses lightweight XML over HTTP or TCP/IP
>> - hence client technology is independent of server
> 
> As it should be  :c)
> 
>> - authorization: users logon before any other action like submitting data
> 
> How is this mnanaged on the server side, just an on/off proposiition, or 
> can things be controlled at a particular layer of information?
(not entirely clear what you mean) the server manages 
users/roles/permissions; users and data items are first class objects 
(records), protocol requests are intercepted for authorization
> 
>> - personalization: all submitted data (GPS, media, POIs) is linked to 
>> user (in DB).
>> - basically client does remote track recording through the XML 
>> protocol (create-track, write-track, suspend/resume (creates 
>> segments), etc)
> 
> Very similar to how we set up out Listener some years back.
> 
>> - media files (photo/video) can be uploaded (HTTP) or emailed
>> - geotagging media is attempted by matching EXIF timestamp data with 
>> GPS points or else submission time
>> - raw GPS (NMEA) samples are sent to server, who parses, checks for 
>> quality and spikes (GPS data can be "filthy") before appending to track
> 
> A little different from our version, we did the cleaning at the server 
> the same way, but all points were time indexed and IDed seperately and 
> simply recorded in the database, The AVL hardware seems to have been a 
> bit more stable than GPS receivers I think.
> 
>> - client does some GPS-prefiltering: taking best sample in an 
>> configurable interval
> 
> We grabbed everything, got a point every 1.5 seconds or so by polling 
> the remote capture device (IP polling), this was plenty of data to build 
> a track for following vehicles.
track-data is file-oriented with meta-data in DB records (a KeyWorx 
feature to manage large data blobs like media files and appendable 
tracks with transactional integrity), since 99% of the use is about 
managing individual tracks not points in tracks.
> 
>> - time is a nasty issue: though a GPS has exact time, some phones do 
>> not provide correct UTC, photos do not provide timezone info in EXIF etc
>> - no local storage/later syncing yet (a much wanted option since 
>> connectivity may not be present in e.g. rural areas)
> 
> Almost a must if you need to do any sort of trail analysis.  Make the 
> dispatching system more effective for example.
yes,
> 
>> - for desktop and possibly/hopefully some PDA-type clients 
>> (http://weblogs.mozillazine.org/dougt/archives/016719.html) I see 
>> browser-AJAX-based solutions as a way to go over dedicated apps. You 
>> will then only need small components to handle GPS and possibly local 
>> storage.
>> - Opera Mobile on Symbian is not yet there, Minimo for CE is hopeful 
>> with AJAX support.
>> - data organisation: server stores tracks per user divided into 
>> segments (like GPX)
> 
> Can you explain this a bit further, segment of time, Distance, other ???
I think you hit an interesting point: how to manage/organize/store GPS 
data. The GeoTracing approach is to store GPS-data as Tracks. Each Track 
is related to a User. Any media/POIs are both related to the User and 
the Track (KeyWorx supports a generic N-M DB relationship).

A Track (in GT) is not a contiguous number of points but divided in 
Segments. At least two reasons: (1) statistics: e.g. sports events where 
you e.g. skate a trail but pause inbetween (suspend/resume) (2) within a 
single Track for various reasons (crashes/network unavailability, or 
simply not having switched the system on) one may resume at some 
distance further along. When rendering the Track you don't want to draw 
a straight line between two too distant points (but rather have a hole).

Segments can be created explicitly with the track-suspend/resume service 
or implicitly when one restarts the client-app using the same track. 
There is always an active track. Creating a new Track archives the 
previous. (and we may add a third reason to implicitly create segments 
based on minimal distance between consecutive points or even minimal 
movement).

It depends on the intended application, but wherever one needs to 
collect GPS-data and treat the data as polylines you have to consider 
these issues and consider using segments. I think this is like most 
handheld GPS-es manage track data ? At least I got the idea from the 
GPX-standard.

At least when you start implementing GPS-data collection especially in a 
constrained (in connectivity, data cost) network environment like GPRS 
there's among others these issues to consider:

1) filter out faulty GPS data (xDOP, spikes, drifts). Strangely some 
Sirf III GPS-es (like the Holux GPSlim 236) appear to work indoor/under 
foliage but give large spikes.
2) cope with GPS availability
3) cope with network availability (ok need local store/later sync)
4) cope Bluetooth connection errors
5) bandwidth/data storage optimization: e.g.
a) only send/store a point if it has some minimal distance from its 
previous point (e.g. if a user stays at a single point)
b) e.g. driving a straight line, no need to send/store intermediate points
6) also store raw GPS-NMEA samples (tracklogs), not just extracted 
lon/lat. You may regret that later (e.g. if your cleanup algoritm 
appeared incorrect or you forgot to store e.g. elevation or quality)
7) don't treat a Track as a single polyline but use segments (e.g. N 
polylines). For example DB each record could have lat/lon/time/ele etc 
plus a trackid + segmentid.
8) Consider criteria for creating tracks (explicit by user or by day) 
and segments (suspend/resume, minimal distance etc).
9) use configurable parameters for the above GPS-filtering (e.g. min 
xDOP), minimal distance, sample rate, max speed etc.

These were at least some of the issues I encountered.

> 
>> - REST service to acquire data as XML (e.g. GPX export)
>> - realtime tracking in browser with AJAX-based Pushlets 
>> (www.pushlets.com)
>> - yesyes, GeoTracing does not comply with OpenGIS stacks yet
>>
>> I realize the above is somewhat long/unordered and not a direct reply 
>> but I am enthousiastic about this topic and many issues discussed are 
>> familiar to those that I have been working on the past year.
> 
> Same here, don't worry about it, all good stuff from what I can tell.
> 
> How would you guess this would fit into what has been described so far?
Good/difficult question. What I found missing (or maybe overlooked) in 
the discussions are the intended applications. With GeoTracing 
development was application-driven (rather than starting a platform) 
more from a User/Multimedia/CMS viewpoint than a GIS-viewpoint. I would 
like to see a list of domains/applications for which Mobile Data 
Collection is intended. From there we may extract common requirements 
and come up with an architecture.

Back to fitting in:
- like said: I don't see GeoTracing as the solution, what is mainly lacking:
a) openGIS-compliance, WFS etc (I sort of feel many of you require this)
b) broad client-support (currently J2ME Mobile and experimental 
Desktop-browser)
c) local storage/remote sync
d) developers (now mainly a solo-project) and thus time

Just


> 
> bobb
> 
>>
>> best,
>>
>> --Just
>>
>> Just van den Broecke  just at justobjects.nl
>> Just Objects B.V.     tel +31 65 4268627 Skype: justb4
>> The Netherlands       http://www.justobjects.nl
>>
>>
>>
>>
>>
>>
>>
>> Bob Basques wrote:
>>
>>> First off, I'm about ready to put some of this into a WIKI, just 
>>> figured it needs to be hashed out here first.  :c)
>>>
>>> Mateusz Loskot wrote:
>>>
>>>> Bob Basques wrote:
>>>>   
>>>
>>>
>>> Development of a BETA package for Mobile Data capture:
>>>
>>>>> The CLI approach seems to most sane to me.  This will most like 
>>>>> change over time with some of the seperate utilities being combined 
>>>>> into bigger and better things later on.  One reason I like this 
>>>>> approach is that it's very easy to segment the utilities into smaller
>>>>>  work area/groups.  As long as a good Comm directive is followed, ann
>>>>>  utilities should work in harmony together.  This "Utility" 
>>>>> approach also affords us the opportunity to work precisely on the 
>>>>> things that need to be worked on, while at the same time using the 
>>>>> tools already developed for other things.
>>>>>     
>>>>
>>>>
>>>>
>>>> What about the idea of modularized package + UI, for example command
>>>> line interface at the first stage?
>>>>   
>>>
>>>
>>> I like thinking about this as a CGI sort of request handler.  If this 
>>> method were followed, then it's just that much easier to use 
>>> components via a Webserver when the time comes, or even to combine 
>>> utilities into a piped structure as handlers.
>>>
>>>>>    Related to the above is the question of what build.
>>>>>    I keep pointing
>>>>>    to LINUX as the first version of things because it will require the
>>>>>    least amount of NEW work to get a prototype operational.
>>>>> Basic requirements (as I see them):
>>>>>
>>>>>    Every Device that is considered for porting, should have:
>>>>>
>>>>>        * Have a Web Server of some sort (in the near term, nothing
>>>>>          exotic should be attempted on the LINUX side, just keep it
>>>>>          short and sweet so that the capabilities are easier to port
>>>>>          to other OS's.
>>>>>     
>>>>
>>>>
>>>>
>>>> What is the idea of communication, could you elaborate?
>>>>   
>>>
>>>
>>> Since my notion is that the Main server (mothership) and the portable 
>>> device (figher ship) will each have about the same capabilites 
>>> (ideally THE same), the only real problem is with syncing the data 
>>> between them.  The next biggest piece will be in updating tracking 
>>> layers at each remote device, but the DB sysning should take care of 
>>> this easily.
>>>
>>>>  
>>>>
>>>>>        * Have a Web Browser that has at least some Javascript
>>>>>          capabilities, (most do, BTW, although most are also
>>>>>          incomplete when talking about what features they support,
>>>>>          don't support)
>>>>>     
>>>>
>>>>
>>>>
>>>> Yes, though it may be a problem with portability.
>>>> What Web browsers do you consider?
>>>>   
>>>
>>>
>>> Since I wouldn't advocate anything to wierd in the Client GUI, it 
>>> could be just about anything with Javascript support, Blazer (PALM), 
>>> Opera(lot's of portable OS's), WinCE, etc. Even some of the Phone 
>>> browser should support basic Javascript operations. I say we shoot 
>>> for most, but not all up front.
>>>
>>>>>        * These might be optional items
>>>>>           * Have a way of generating a Map, even in a simplified
>>>>>             form as a standalone operation.
>>>>>     
>>>>
>>>>
>>>>
>>>> What does "generate a map" means?
>>>> Do you mean it should be able to display map on a device screen?
>>>>   
>>>
>>>
>>> Yup, the portable device acts just like a desktop version of thing, 
>>> even when disconnected (it has some basic mapping data resident on 
>>> the machine.  Some smaller form factors will have limited space and 
>>> the Syncing process could be made a bit more robust in these 
>>> situations, where only a localized version of the data is resident on 
>>> the portable device based on dispatching etc.
>>>
>>>>>            * Have a way of connecting to a AVL hardware stack ??(etc.)
>>>>>     
>>>>
>>>>
>>>>
>>>> I'm not experienced with technical specification of AVL but OK :-)
>>>>   
>>>
>>>
>>> I explain some of this in the earlier Email, but the important piece, 
>>> is that they too also capture GPS data as a built in Capability.
>>>
>>>>>  If we can agree to the above, then what we'll need to build (short
>>>>>    list):
>>>>>
>>>>>        * Collection format for the data (Points only to start with?)
>>>>>     
>>>>
>>>>
>>>>
>>>> What does "collection format" means?
>>>> Is this about log GPS locations or also from manual map editing?
>>>>   
>>>
>>>
>>> Yes, and yes, with my description in the previous Email, I think you 
>>> can see how this would work usig a database storage technique.
>>>
>>>>>            * Database, Flat File, etc.
>>>>>     
>>>>
>>>>
>>>>
>>>> IMHO one of the most important requirement is insert/update/query
>>>> performance with good indexing.
>>>> So, I'd stick to SQLite.
>>>>   
>>>
>>>
>>> Any SQL database, I'm most familiar with MYSQL.  But since I'm 
>>> suggesting that no special data formatting is needed, that the data 
>>> could reside in anything with a SQL retrieval mechanism.
>>>
>>>>  
>>>>
>>>>>        * Server comm LIB, Main (MotherShip) to Secondary (Attack
>>>>>          Fighter) web servers.
>>>>>            * Syncing of captured data points
>>>>>            * Syncing of Mapping data between servers fore
>>>>>                re-distribution.
>>>>>        * Daemon (listener) to monitor GPS (could be a wrapper around
>>>>>          any of the options already available for reading
>>>>>          GPS devices.) coordinate locations, and other digital
>>>>>          devices, Compass, etc.
>>>>>     
>>>>
>>>>
>>>>
>>>> As Paulo has suggested, gpsd should work well.
>>>>   
>>>
>>>
>>> See my notes on this in other Emails.
>>>
>>>>>        * Some sort of GUI (web Based, even when running standalone)
>>>>> Do you mean it will be a JavaScript/AJAX application>
>>>>>     
>>>
>>>
>>> Yes.
>>>
>>>>>        * MapService, for providing Automated location services.
>>>>>     
>>>>
>>>>
>>>>
>>>> I imagine it can be one of the biggest problem to find/create services
>>>> infrastructure.
>>>>   
>>>
>>>
>>> I'm not sure I understand this statement.   If by it you mean a 
>>> Mapping system with this type data already present, I have access to 
>>> this type of thing.  It's a big reason why I'm pushing hard to make 
>>> this component based, as the CLIENT end of anything will end up in 
>>> our Main Spatial service, not as a seperate application, but as an 
>>> enhancement to the existing one.
>>>
>>>>> could be a start for pulling a project proposal out of.
>>>>>     
>>>>
>>>>
>>>> There are a bunch of details I'm glossing over here, but I think this
>>>>
>>>> Although it's hard for me to comment a Web-based interface for
>>>> GPS-driven application for data collecting, I think the plan is good.
>>>>
>>>> Here are my additional questions:
>>>> 1. What's the suggested development platform for the prototype?
>>>> 2. Are we still going to create and use underlying core components for
>>>> data accessing, processing, analysing, computing, etc?
>>>> 3. What about services infrastructure?
>>>>   
>>>
>>>
>>> I think these questions should start to become clearer for all those 
>>> reading so far, let me know if anyone is still needing some aspects 
>>> (that I have described) explained further.  I'm very interested in 
>>> hearing about why my ideas are hair brained, etc.  Don't worry about 
>>> offending me either, I have this same problem with every application 
>>> team I get on it seems.  :c)
>>>
>>> bobb
>>>
>>>> Best regards
>>>>   
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: discuss-unsubscribe at mail.osgeo.org
>>> For additional commands, e-mail: discuss-help at mail.osgeo.org
>>>
>>>
>>>
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: discuss-unsubscribe at mail.osgeo.org
> For additional commands, e-mail: discuss-help at mail.osgeo.org
> 
> 
> 


-- 

--Just

Just van den Broecke  just at justobjects.nl
Just Objects B.V.     tel +31 65 4268627 Skype: justb4
The Netherlands       http://www.justobjects.nl









More information about the Discuss mailing list