fastest way to convert lon-lat to points

[P Kishor]

On 7/20/06, Pagurek,Debbie [NCR] <Debbie.Pagurek at ec.gc.ca> wrote:
..
> Funny you should mention all this point in polygon search stuff, because
> that is exactly what I'm working on right now, but I'm using a .csv
> file, WFS requests with a DWITHIN filter, and adding the results to the
> original .csv as new fields. I don't think the performance is going to
> be very good though ;)

"Algorithms with Perl" aka the wolf book
<http://www.oreilly.com/catalog/maperl/reviews.html> has a
point-in-poly algorigthm that I have used in the past successfully
with shapefiles. I used Geo::ShapeFile
<http://search.cpan.org/~jasonk/Geo-ShapeFile-2.51/ShapeFile.pm> to
work with the shapefiles. I know that will work, but I have to work on
optimizing the speed part because doing a 5 mill points in 200k polys
is a cool trillion intersections.

By the way, the task is not one-off. Hence the need for a program.

>
> -----Original Message-----
> From: punk.kish at gmail.com [mailto:punk.kish at gmail.com] On Behalf Of P
> Kishor
> Sent: Thursday, July 20, 2006 10:37 AM
> To: Pagurek,Debbie [NCR]
> Cc: MAPSERVER-USERS at lists.umn.edu
> Subject: Re: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to
> points
>
> Hey Debbie, Thanks for your reply. A few clarifications...
>
> On 7/20/06, Pagurek,Debbie [NCR] <Debbie.Pagurek at ec.gc.ca> wrote:
>
>
> > I am working on some similar work with .csv files and OGR right now.
> > I don't know why your field is being changed like that...
> >
>
> Yup, this is strange, and unacceptable, so I need a workaround for this.
>
> > As for debugging, aside from the ogrinfo test, you might want to embed
>
> > your <OGRVRTDataSource> into a mapfile and then try the shp2img tool.
> > If this produces an image for you, all the mapserver stuff is fine.
> > But you might be having permissions problems on your server, which is
> > where MY problem lies right now. I'm using IIS5, not apache... So I
> > have some work ahead of me!
>
> I am not doing anything with the web at all here. This is pure command
> line stuff. The complete process I want to accomplish is as follows --
>
> 1. convert lon-lat into shapefile points.
> 2. overlay (point-in-poly) #1 shapefile with a shapefile polys to
> determine, for each point, what poly it falls in... so, if #1 shapefile
> has cities, and the polys shapefile has states, I want to create an
> additional attribute for each city and fill it with the states value.
>
> This is all command line, and for quite a bit of data (upward of 5
> million), so speed is an issue, although CPU and memory are not an issue
> (good hardware). I am using Perl for all the automation.
>
> I would like to keep everything in the realm of text files and dbf files
> as much as possible, but I am not averse to using SQLite or Berkeley DB.
> If push-comes-to-shove, I might be open to experimenting with PostGIS,
> but only if push really comes to shove.
>
>
> Many thanks, (and hope to see you again soon somewhere... maybe I will
> have to make a trip up North).
>
>
> >
> > You might also want to write a little PHP script to make sure that
> > your ODBC DSN (a system DSN, right?) is working ok as well. It's when
> > I did this that I realized that my problem has nothing to do with
> > mapserver, ogrinfo etc.
> >
> > Good luck and let us know what you discover, Debbie Pagurek
> >
> > -----Original Message-----
> > From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
> > On Behalf Of P Kishor
> > Sent: Thursday, July 20, 2006 8:19 AM
> > To: MAPSERVER-USERS at LISTS.UMN.EDU
> > Subject: [UMN_MAPSERVER-USERS] fastest way to convert lon-lat to
> > points
> >
> > Folks,
> >
> > I have seen this question asked in the past, and have a similar
> problem.
> > I want to convert many lon-lat to a shapefile of points quickly. The
> > lon-lat are in a csv text file (values.txt) like so
> >
> > foo,bar,baz,long,lat,qux
> > 1, bar1, baz1, -87.796341, 41.907504, s10 ...
> > ...
> >
> > So, I installed FW_Tools 1.05, created a DSN out of the text file
> > (this is and has to be on a Windows box), and ogrinfo happily gave me
> > information that I wanted, but with a twist...
> >
> > OGRFeature(values.txt):1056
> >   foo (Integer) = 1
> >   bar(String) = ba1
> >   baz(String) = baz1
> >   long (Real) = -87.796341
> >   lat (Real) = 41.907504
> >   qux(String) = 10.0000
> >
> > So, what's with interpreting qux? It converted my "s10" into a string
> > of value "10.0000".
> >
> > Anyway, then I created an OVF file (values.ovf) like so
> >
> > <OGRVRTDataSource>
> >   <OGRVRTLayer  name="values">
> >   <SrcDataSource>ODBC:values</SrcDataSource>
> >   <SrcLayer>values.txt</SrcLayer>
> >   <GeometryField encoding="PointFromColumns" x="Long" y="Lat"/>
> >   <GeometryType>wkbPoint</GeometryType>
> >   </OGRVRTLayer>
> > </OGRVRTDataSource>
> >
> > well,
> >
> > ogr2ogr -f "ESRI Shapefile" . values.ovf values
> >
> > but while that seemed to do something without any error, I got no
> > output.
> >
> > So, is what I am doing the right way of going about this? Is there a
> > better or another way to try out and benchmark? Oh, did I mention? --
> > The values.txt file has about 5.25 million rows, so speed is
> important.
> >
>
>
>
> --
> Puneet Kishor http://punkish.eidesis.org/ Nelson Inst. for Env. Studies,
> UW-Madison http://www.ies.wisc.edu/ Open Source Geospatial Foundation
> https://edu.osgeo.org/
>


-- 
Puneet Kishor http://punkish.eidesis.org/
Nelson Inst. for Env. Studies, UW-Madison http://www.ies.wisc.edu/
Open Source Geospatial Foundation https://edu.osgeo.org/