[gdal-dev] Difficulty opening Safecast GeoJSON data

Mon May 5 14:22:39 PDT 2014

Le lundi 05 mai 2014 22:57:12, David Fawcett a écrit :
> On second thought, I see that the .csv file is 2.1 gb!  I would suggest not
> following the .csv driver approach.

Note: Download time might be the problem, but except that, the OGR CSV (and 
VRT) driver should be able to deal with multi giga byte files with minimum RAM 
usage. Basically it reads one line at a time in streaming mode.

> 
> I would follow Sean's suggestion and just use some simple Python (or
> whatever language that you want to use to consume the data) to grab the  XY
> values and write out geometries.
> 
> David.
> 
> Note to self:  When Sean suggests something, just hit +1 and hold tongue...
> 
> On Mon, May 5, 2014 at 3:43 PM, David Fawcett 
<david.fawcett at gmail.com>wrote:
> > Mark,
> > 
> > You could use the GDAL/OGR .csv driver with a VRT to access the data and
> > spit it out into a different, filtered file.
> > 
> > http://www.gdal.org/ogr/drv_csv.html
> > 
> > You could also use the VSI to pull from a .zip'd file via URL.
> > 
> > You could also likely use args with ogr2ogr to filter by attribute or
> > bbox, and even reproject the data on the way out.
> > 
> > http://www.gdal.org/ogr2ogr.html
> > 
> > David.
> > 
> > On Mon, May 5, 2014 at 3:29 PM, Mark Coletti <mcoletti at gmail.com> wrote:
> >> Thank you Even and Sean for your very quick and salient replies!  I'll
> >> notify the Safecast folks that their data might be better served using
> >> GeoJSON; meanwhile, they also make their open source data available as a
> >> single downloadable file in CSV format.  Since qgis can import files of
> >> that type, I've been using that; it's just that I have to do a lot more
> >> work on my end to pare the data down to the region of interest.  (And
> >> deal with the inevitable data corruption.  E.g., I suspect the
> >> observations from 2020 are bogus unless they have a time traveler
> >> volunteering radiation readings. =) )
> >> 
> >> Again, thank you for your help!
> >> 
> >> Cheers,
> >> 
> >> Mark
> >> 
> >> On Mon, May 5, 2014 at 4:12 PM, Sean Gillies <sean at mapbox.com> wrote:
> >>> Hi Mark,
> >>> 
> >>> Safecast's JSON isn't GeoJSON, but wouldn't be too hard to crosswalk
> >>> over to GeoJSON... make a geometry from its latitude/longitude, stuff
> >>> all the other attributes in properties.
> >>> 
> >>> On 5/5/14, 1:56 PM, Mark Coletti wrote:
> >>>> Safecast is user sourced radiation data, and has a public API serving
> >>>> data in GeoJSON format (https://api.safecast.org/en-US/home). 
> >>>> However, I'm having difficulty in accessing this data via OGR:
> >>>> 
> >>>> % ogrinfo -so
> >>>> "https://api.safecast.org/en-US/measurements?distance=10&
> >>>> latitude=37.42&longitude=141.033"
> >>>> ERROR 4: Failed to read GeoJSON data
> >>>> ERROR 4: Failed to read GeoJSON data
> >>>> FAILURE:
> >>>> Unable to open datasource
> >>>> `https://api.safecast.org/measurements.json?distance=10&
> >>>> latitude=37.42&longitude=141.033'
> >>>> with the following drivers.
> >>>> 
> >>>> So OGR sees that the data is formatted as GeoJSON, but is unable to
> >>>> read it for some reason.
> >>>> 
> >>>> Using curl to manually make the query I get:
> >>>> 
> >>>> % curl  -i -H "Accept: application/json"
> >>>> "https://api.safecast.org/en-US/measurements?distance=10&
> >>>> latitude=37.42&longitude=141.033"
> >>>> HTTP/1.1 200 OK
> >>>> Server: nginx
> >>>> Date: Mon, 05 May 2014 19:43:55 GMT
> >>>> Content-Type: application/json; charset=utf-8
> >>>> Transfer-Encoding: chunked
> >>>> Connection: keep-alive
> >>>> Status: 200 OK
> >>>> Strict-Transport-Security: max-age=31536000
> >>>> Access-Control-Allow-Origin: safecast.org <http://safecast.org>
> >>>> 
> >>>> Access-Control-Allow-Methods: POST, GET, OPTIONS
> >>>> Access-Control-Allow-Headers: *, X-Requested-With
> >>>> Access-Control-Max-Age: 100000
> >>>> X-UA-Compatible: IE=Edge,chrome=1
> >>>> ETag: "e4650539a53d3698d8deb02e912b9ae3"
> >>>> Cache-Control: max-age=0, private, must-revalidate
> >>>> X-Request-Id: a30a38247c16374a39ada0a8d4d2fcde
> >>>> X-Runtime: 0.022855
> >>>> X-Rack-Cache: miss
> >>>> X-Powered-By: cloud66
> >>>> 
> >>>> [{"captured_at":"2014-01-30T05:03:25Z","device_id":
> >>>> null,"id":30981293,"location_name":null,"original_id":null,
> >>>> "unit":"cpm","user_id":3,"value":4675.0,"latitude":37.
> >>>> 41997666666666,"longitude":141.03297166666667},{"
> >>>> captured_at":"2014-01-30T05:25:26Z","device_id":null,"id":
> >>>> 30981556,"location_name":null,"original_id":null,"unit":"
> >>>> cpm","user_id":3,"value":2470.0,"latitude":37.
> >>>> 42001666666667,"longitude":141.03303666666667},{"
> >>>> captured_at":"2014-01-30T05:00:54Z","device_id":null,"id":
> >>>> 30981263,"location_name":null,"original_id":null,"unit":"
> >>>> cpm","user_id":3,"value":5428.0,"latitude":37.
> >>>> 41996666666667,"longitude":141.03301666666667},{"
> >>>> captured_at":"2014-01-30T05:21:20Z","device_id":null,"id":
> >>>> 30981507,"location_name":null,"original_id":null,"unit":"
> >>>> cpm","user_id":3,"value":12191.0,"latitude":37.
> >>>> 420003333333334,"longitude":141.03305333333333},{"
> >>>> captured_at":"2014-01-30T05:00:49Z","device_id":null,"id":
> >>>> 30981262,"location_name":null,"original_id":null,"unit":"
> >>>> cpm","user_id":3,"value":5199.0,"lat
> >>> 
> >>> i
> >>> tude":37.41999666666667,"longitude":141.03306166666667}
> >>> ,{"captured_at":"2014-01-30T05:25:21Z","device_id":
> >>> null,"id":30981555,"location_name":null,"original_id":null,
> >>> "unit":"cpm","user_id":3,"value":2423.0,"latitude":37.
> >>> 42006833333333,"longitude":141.03301333333334},{"
> >>> captured_at":"2014-01-30T05:03:30Z","device_id":null,"id":
> >>> 30981294,"location_name":null,"original_id":null,"unit":"
> >>> cpm","user_id":3,"value":4456.0,"latitude":37.
> >>> 41992833333333,"longitude":141.03297333333333},{"
> >>> captured_at":"2014-01-30T05:00:59Z","device_id":null,"id":
> >>> 30981264,"location_name":null,"original_id":null,"unit":"
> >>> cpm","user_id":3,"value":5672.0,"latitude":37.
> >>> 419963333333335,"longitude":141.03291666666667},{"
> >>> captured_at":"2014-01-30T05:00:44Z","device_id":null,"id":
> >>> 30981261,"location_name":null,"original_id":null,"unit":"
> >>> cpm","user_id":3,"value":5135.0,"latitude":37.420035,"
> >>> longitude":141.03309},{"captured_at":"2014-01-30T05:
> >>> 21:14Z","device_id":null,"id":30981506,"location_name":null,
> >>> "original_id":null,"unit":"c
> >>> p
> >>> m","user_id":3,"value":12164.0,"latitude":37.42006333333333,"longitude"
> >>> : 141.03293666666667}]
> >>> 
> >>>> Those do look like legitimate observations, but I don't know if they
> >>>> comply to the GeoJSON format since I'm new to JSON.
> >>>> 
> >>>> I'm using this version of GDAL/OGR, which was part of the pre-packaged
> >>>> binaries for qgis for my Mac:
> >>>> % ogrinfo --version
> >>>> GDAL 1.10.1, released 2013/08/26
> >>>> 
> >>>> Eventually I'd like to use GDAL/OGR to conveniently create a local
> >>>> spatial database from such a query.  However, I need to get over this
> >>>> hurdle first.
> >>>> 
> >>>> Cheers,
> >>>> 
> >>>> Mark
> >>>> --
> >>>> mcoletti at gmail.com <mailto:mcoletti at gmail.com>
> >>>> 
> >>>> 
> >>>> 
> >>>> _______________________________________________
> >>>> gdal-dev mailing list
> >>>> gdal-dev at lists.osgeo.org
> >>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >>> 
> >>> --
> >>> Sean Gillies
> >>> sean at mapbox.com
> >>> _______________________________________________
> >>> gdal-dev mailing list
> >>> gdal-dev at lists.osgeo.org
> >>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >> 
> >> --
> >> mcoletti at gmail.com
> >> 
> >> 
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html