[OSGeo-Discuss] Mobile GPS data collection, summary of sorts.

[Bob Basques]

Just van den Broecke wrote:
> Bob Basques wrote:
>
>>
>>> - authorization: users logon before any other action like submitting 
>>> data
>>
>> How is this mnanaged on the server side, just an on/off proposiition, 
>> or can things be controlled at a particular layer of information?
> (not entirely clear what you mean) the server manages 
> users/roles/permissions; users and data items are first class objects 
> (records), protocol requests are intercepted for authorization
>>
I need to control mapping output at the layer level as far as 
authentication is all, is a fairly specialized need, so I don't holler 
about it much on these lists.
>>> - personalization: all submitted data (GPS, media, POIs) is linked 
>>> to user (in DB).
>>> - basically client does remote track recording through the XML 
>>> protocol (create-track, write-track, suspend/resume (creates 
>>> segments), etc)
>>
>>
>>> - client does some GPS-prefiltering: taking best sample in an 
>>> configurable interval
>>
>> We grabbed everything, got a point every 1.5 seconds or so by polling 
>> the remote capture device (IP polling), this was plenty of data to 
>> build a track for following vehicles.
> track-data is file-oriented with meta-data in DB records (a KeyWorx 
> feature to manage large data blobs like media files and appendable 
> tracks with transactional integrity), since 99% of the use is about 
> managing individual tracks not points in tracks.
We wanted to be able to replay data track in the view from database 
stored points, we started out like you with seperate files, but had 
problems smoothly replaying tracks across many files.  The replayer 
worked just like a VHS, Plat, FF, RW, 2X, 4X, etc.  We needed to pull 
the chunk in a single stream for playback, made everyones life easier in 
the end.  Agin this was just our need for playback that required this 
type of storage, but it worked out well in the end as a standalong data 
capture pocess.
>>
>>> - time is a nasty issue: though a GPS has exact time, some phones do 
>>> not provide correct UTC, photos do not provide timezone info in EXIF 
>>> etc
>>> - no local storage/later syncing yet (a much wanted option since 
>>> connectivity may not be present in e.g. rural areas)
I think worried about it at the server, at least is was a single source 
for the time.  Easy to reset via a global variable too.  Yeah, I know, 
it was cheating.  :c)
>>
>> Almost a must if you need to do any sort of trail analysis.  Make the 
>> dispatching system more effective for example.
> yes,
>>
>>> - for desktop and possibly/hopefully some PDA-type clients 
>>> (http://weblogs.mozillazine.org/dougt/archives/016719.html) I see 
>>> browser-AJAX-based solutions as a way to go over dedicated apps. You 
>>> will then only need small components to handle GPS and possibly 
>>> local storage.
>>> - Opera Mobile on Symbian is not yet there, Minimo for CE is hopeful 
>>> with AJAX support.
>>> - data organisation: server stores tracks per user divided into 
>>> segments (like GPX)
>>
>> Can you explain this a bit further, segment of time, Distance, other ???
> I think you hit an interesting point: how to manage/organize/store GPS 
> data. The GeoTracing approach is to store GPS-data as Tracks. Each 
> Track is related to a User. Any media/POIs are both related to the 
> User and the Track (KeyWorx supports a generic N-M DB relationship).
>
> A Track (in GT) is not a contiguous number of points but divided in 
> Segments. At least two reasons: (1) statistics: e.g. sports events 
> where you e.g. skate a trail but pause inbetween (suspend/resume) (2) 
> within a single Track for various reasons (crashes/network 
> unavailability, or simply not having switched the system on) one may 
> resume at some distance further along. When rendering the Track you 
> don't want to draw a straight line between two too distant points (but 
> rather have a hole).
The method of collecting in our setup was to do everything, this seems 
to work well for detection of network failures as well, just query the 
database to see what was hapopening, took very little time to develop 
Reports about system workings for example.  The server was alwasy 
polling the devices 24/7.
>
> Segments can be created explicitly with the track-suspend/resume 
> service or implicitly when one restarts the client-app using the same 
> track. There is always an active track. Creating a new Track archives 
> the previous. (and we may add a third reason to implicitly create 
> segments based on minimal distance between consecutive points or even 
> minimal movement).
You'll find that this is a bit problematical in the end, hard to predict 
thing like this across different types of hard foe example.  We should 
talk more about this I think seperately.
>
> It depends on the intended application, but wherever one needs to 
> collect GPS-data and treat the data as polylines you have to consider 
> these issues and consider using segments. I think this is like most 
> handheld GPS-es manage track data ? At least I got the idea from the 
> GPX-standard.
Oooh, another big conversation, seperate table of point, seperated by 
Day, and Devices.  Always a quick return from a database query, even 
across a number of days if needed.  Most users are only looking at 
replaying an hour or two.  But we had thought about users who might want 
to do analysis on the tracking data as well.
>
> At least when you start implementing GPS-data collection especially in 
> a constrained (in connectivity, data cost) network environment like 
> GPRS there's among others these issues to consider:
>
> 1) filter out faulty GPS data (xDOP, spikes, drifts). Strangely some 
> Sirf III GPS-es (like the Holux GPSlim 236) appear to work 
> indoor/under foliage but give large spikes.
> 2) cope with GPS availability
> 3) cope with network availability (ok need local store/later sync)
> 4) cope Bluetooth connection errors
Yeah, I had thought about all these type of things as we've been running 
along with this thread.
> 5) bandwidth/data storage optimization: e.g.
> a) only send/store a point if it has some minimal distance from its 
> previous point (e.g. if a user stays at a single point)
> b) e.g. driving a straight line, no need to send/store intermediate 
> points
Disagree, capture everything, Although my need is more URBAN than RURAL, 
so even grabbing points in a straight line mean something to us. With 
the newer remote devices that data collection should be possible still 
in a standalone mode for a few data without filling the device up.
> 6) also store raw GPS-NMEA samples (tracklogs), not just extracted 
> lon/lat. You may regret that later (e.g. if your cleanup algoritm 
> appeared incorrect or you forgot to store e.g. elevation or quality)
I'm more inclined to have a fairly neutral format for storage, but I see 
your point.
> 7) don't treat a Track as a single polyline but use segments (e.g. N 
> polylines). For example DB each record could have lat/lon/time/ele etc 
> plus a trackid + segmentid.
I've got some pretty good argument for Points(with time), by Deveice ID, 
by day (or some other division, hourly for example) that I would be 
interest in discussing some more.
> 8) Consider criteria for creating tracks (explicit by user or by day) 
> and segments (suspend/resume, minimal distance etc).
All a reporting function in my preference list.
> 9) use configurable parameters for the above GPS-filtering (e.g. min 
> xDOP), minimal distance, sample rate, max speed etc.
>
> These were at least some of the issues I encountered.
>
>>
>>> I realize the above is somewhat long/unordered and not a direct 
>>> reply but I am enthousiastic about this topic and many issues 
>>> discussed are familiar to those that I have been working on the past 
>>> year.
>>
>> Same here, don't worry about it, all good stuff from what I can tell.
>>
>> How would you guess this would fit into what has been described so far?
> Good/difficult question. What I found missing (or maybe overlooked) in 
> the discussions are the intended applications. With GeoTracing 
> development was application-driven (rather than starting a platform) 
> more from a User/Multimedia/CMS viewpoint than a GIS-viewpoint. 
The same with our setup, although the idea we had from the start was to 
have a completely standalong system for the Storage/collection process 
that was all DB driven.  pretty simple overall.
> I would like to see a list of domains/applications for which Mobile 
> Data Collection is intended. From there we may extract common 
> requirements and come up with an architecture.
Agreed.
>
> Back to fitting in:
> - like said: I don't see GeoTracing as the solution, what is mainly 
> lacking:
> a) openGIS-compliance, WFS etc (I sort of feel many of you require this)
I'm not really worried about this stuff earlir on anymore.  That's all 
for the enhancement pahses as I see it.
> b) broad client-support (currently J2ME Mobile and experimental 
> Desktop-browser)
I'm not a fan of Java either, but . . .  :c)
> c) local storage/remote sync
With a littl re-organization I think this could be built into it.
> d) developers (now mainly a solo-project) and thus time
We only had a couple of folks on it too.  But I have access to more now 
if my business needs become a priority within the project all of a 
sudden.  :c)
>
>
bobb