<div dir="ltr"><div><div><div>On second thought, I see that the .csv file is 2.1 gb! I would suggest not following the .csv driver approach.<br><br></div>I would follow Sean's suggestion and just use some simple Python (or whatever language that you want to use to consume the data) to grab the XY values and write out geometries. <br>
<br></div>David.<br><br></div>Note to self: When Sean suggests something, just hit +1 and hold tongue...<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 5, 2014 at 3:43 PM, David Fawcett <span dir="ltr"><<a href="mailto:david.fawcett@gmail.com" target="_blank">david.fawcett@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Mark, <br><br></div>You could use the GDAL/OGR .csv driver with a VRT to access the data and spit it out into a different, filtered file. <br>
<br><a href="http://www.gdal.org/ogr/drv_csv.html" target="_blank">http://www.gdal.org/ogr/drv_csv.html</a><br>
<br></div>You could also use the VSI to pull from a .zip'd file via URL. <br><br></div>You could also likely use args with ogr2ogr to filter by attribute or bbox, and even reproject the data on the way out.<br><br><a href="http://www.gdal.org/ogr2ogr.html" target="_blank">http://www.gdal.org/ogr2ogr.html</a><span class="HOEnZb"><font color="#888888"><br>
<br></font></span></div><span class="HOEnZb"><font color="#888888">David.<br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 5, 2014 at 3:29 PM, Mark Coletti <span dir="ltr"><<a href="mailto:mcoletti@gmail.com" target="_blank">mcoletti@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thank you Even and Sean for your very quick and salient replies! I'll notify the Safecast folks that their data might be better served using GeoJSON; meanwhile, they also make their open source data available as a single downloadable file in CSV format. Since qgis can import files of that type, I've been using that; it's just that I have to do a lot more work on my end to pare the data down to the region of interest. (And deal with the inevitable data corruption. E.g., I suspect the observations from 2020 are bogus unless they have a time traveler volunteering radiation readings. =) )<div>
<br></div><div>Again, thank you for your help!</div><div><br></div><div>Cheers,</div><div><br></div><div>Mark</div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Mon, May 5, 2014 at 4:12 PM, Sean Gillies <span dir="ltr"><<a href="mailto:sean@mapbox.com" target="_blank">sean@mapbox.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Mark,<br>
<br>
Safecast's JSON isn't GeoJSON, but wouldn't be too hard to crosswalk over to GeoJSON... make a geometry from its latitude/longitude, stuff all the other attributes in properties.<div><div><br>
<br>
On 5/5/14, 1:56 PM, Mark Coletti wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<br>
Safecast is user sourced radiation data, and has a public API serving<br>
data in GeoJSON format (<a href="https://api.safecast.org/en-US/home" target="_blank">https://api.safecast.org/en-<u></u>US/home</a>). However,<br>
I'm having difficulty in accessing this data via OGR:<br>
<br>
% ogrinfo -so<br>
"<a href="https://api.safecast.org/en-US/measurements?distance=10&latitude=37.42&longitude=141.033" target="_blank">https://api.safecast.org/en-<u></u>US/measurements?distance=10&<u></u>latitude=37.42&longitude=141.<u></u>033</a>"<br>
ERROR 4: Failed to read GeoJSON data<br>
ERROR 4: Failed to read GeoJSON data<br>
FAILURE:<br>
Unable to open datasource<br>
`<a href="https://api.safecast.org/measurements.json?distance=10&latitude=37.42&longitude=141.033" target="_blank">https://api.safecast.org/<u></u>measurements.json?distance=10&<u></u>latitude=37.42&longitude=141.<u></u>033</a>'<br>
with the following drivers.<br>
<br>
So OGR sees that the data is formatted as GeoJSON, but is unable to read<br>
it for some reason.<br>
<br>
Using curl to manually make the query I get:<br>
<br>
% curl -i -H "Accept: application/json"<br>
"<a href="https://api.safecast.org/en-US/measurements?distance=10&latitude=37.42&longitude=141.033" target="_blank">https://api.safecast.org/en-<u></u>US/measurements?distance=10&<u></u>latitude=37.42&longitude=141.<u></u>033</a>"<br>
HTTP/1.1 200 OK<br>
Server: nginx<br>
Date: Mon, 05 May 2014 19:43:55 GMT<br>
Content-Type: application/json; charset=utf-8<br>
Transfer-Encoding: chunked<br>
Connection: keep-alive<br>
Status: 200 OK<br>
Strict-Transport-Security: max-age=31536000<br></div></div>
Access-Control-Allow-Origin: <a href="http://safecast.org" target="_blank">safecast.org</a> <<a href="http://safecast.org" target="_blank">http://safecast.org</a>><div><div><br>
Access-Control-Allow-Methods: POST, GET, OPTIONS<br>
Access-Control-Allow-Headers: *, X-Requested-With<br>
Access-Control-Max-Age: 100000<br>
X-UA-Compatible: IE=Edge,chrome=1<br>
ETag: "<u></u>e4650539a53d3698d8deb02e912b9a<u></u>e3"<br>
Cache-Control: max-age=0, private, must-revalidate<br>
X-Request-Id: a30a38247c16374a39ada0a8d4d2fc<u></u>de<br>
X-Runtime: 0.022855<br>
X-Rack-Cache: miss<br>
X-Powered-By: cloud66<br>
<br>
[{"captured_at":"2014-01-<u></u>30T05:03:25Z","device_id":<u></u>null,"id":30981293,"location_<u></u>name":null,"original_id":null,<u></u>"unit":"cpm","user_id":3,"<u></u>value":4675.0,"latitude":37.<u></u>41997666666666,"longitude":<u></u>141.03297166666667},{"<u></u>captured_at":"2014-01-30T05:<u></u>25:26Z","device_id":null,"id":<u></u>30981556,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":2470.<u></u>0,"latitude":37.<u></u>42001666666667,"longitude":<u></u>141.03303666666667},{"<u></u>captured_at":"2014-01-30T05:<u></u>00:54Z","device_id":null,"id":<u></u>30981263,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":5428.<u></u>0,"latitude":37.<u></u>41996666666667,"longitude":<u></u>141.03301666666667},{"<u></u>captured_at":"2014-01-30T05:<u></u>21:20Z","device_id":null,"id":<u></u>30981507,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":<u></u>12191.0,"latitude":37.<u></u>420003333333334,"longitude":<u></u>141.03305333333333},{"<u></u>captured_at":"2014-01-30T05:<u></u>00:49Z","device_id":null,"id":<u></u>30981262,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":5199.<u></u>0,"lat<br>
</div></div></blockquote><div><div>
i<br>
tude":37.41999666666667,"<u></u>longitude":141.03306166666667}<u></u>,{"captured_at":"2014-01-<u></u>30T05:25:21Z","device_id":<u></u>null,"id":30981555,"location_<u></u>name":null,"original_id":null,<u></u>"unit":"cpm","user_id":3,"<u></u>value":2423.0,"latitude":37.<u></u>42006833333333,"longitude":<u></u>141.03301333333334},{"<u></u>captured_at":"2014-01-30T05:<u></u>03:30Z","device_id":null,"id":<u></u>30981294,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":4456.<u></u>0,"latitude":37.<u></u>41992833333333,"longitude":<u></u>141.03297333333333},{"<u></u>captured_at":"2014-01-30T05:<u></u>00:59Z","device_id":null,"id":<u></u>30981264,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":5672.<u></u>0,"latitude":37.<u></u>419963333333335,"longitude":<u></u>141.03291666666667},{"<u></u>captured_at":"2014-01-30T05:<u></u>00:44Z","device_id":null,"id":<u></u>30981261,"location_name":null,<u></u>"original_id":null,"unit":"<u></u>cpm","user_id":3,"value":5135.<u></u>0,"latitude":37.420035,"<u></u>longitude":141.03309},{"<u></u>captured_at":"2014-01-30T05:<u></u>21:14Z","device_id":null,"id":<u></u>30981506,"location_name":null,<u></u>"original_id":null,"unit":"c<br>
p<br>
m","user_id":3,"value":12164.<u></u>0,"latitude":37.<u></u>42006333333333,"longitude":<u></u>141.03293666666667}]<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<br>
Those do look like legitimate observations, but I don't know if they<br>
comply to the GeoJSON format since I'm new to JSON.<br>
<br>
I'm using this version of GDAL/OGR, which was part of the pre-packaged<br>
binaries for qgis for my Mac:<br>
% ogrinfo --version<br>
GDAL 1.10.1, released 2013/08/26<br>
<br>
Eventually I'd like to use GDAL/OGR to conveniently create a local<br>
spatial database from such a query. However, I need to get over this<br>
hurdle first.<br>
<br>
Cheers,<br>
<br>
Mark<br>
--<br>
</div></div><a href="mailto:mcoletti@gmail.com" target="_blank">mcoletti@gmail.com</a> <mailto:<a href="mailto:mcoletti@gmail.com" target="_blank">mcoletti@gmail.com</a>><br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/<u></u>mailman/listinfo/gdal-dev</a><br>
<br><span><font color="#888888">
</font></span></blockquote><span><font color="#888888">
<br>
-- <br>
Sean Gillies<br>
<a href="mailto:sean@mapbox.com" target="_blank">sean@mapbox.com</a><br>
______________________________<u></u>_________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/<u></u>mailman/listinfo/gdal-dev</a><br>
</font></span></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><a href="mailto:mcoletti@gmail.com" target="_blank">mcoletti@gmail.com</a><br><div><br></div>
</font></span></div>
<br>_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>