<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks Jonathan for the quick reply.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am doing the similar thing as you did except doing it locally. I got the address divided into 4 columns, address, city, state and zip. I was able to query for each state and run the geocode. I was never successful running the complete state. The query takes forever to finish even though the query plan shows it is using index scan etc. I even wrote a psql function to pass one address at a time to get the geocode, this one also could not complete because I could not commit each address as I get the gecode and ended up running out of resources.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>How you were able to geocode by state, did you write any scripts or psql functions? If yes, do you mind sharing? I am having hard time geocoding these addresses.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Appreciate any help is extended.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi Ada<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> postgis-users-bounces@postgis.refractions.net [mailto:postgis-users-bounces@postgis.refractions.net] <b>On Behalf Of </b>Johnathan Leppert<br><b>Sent:</b> Tuesday, November 29, 2011 11:38 PM<br><b>To:</b> PostGIS Users Discussion<br><b>Subject:</b> Re: [postgis-users] Batch geocoding (~2 mil addresses)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi Ravi,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>There is a thread about the tiger geocoder performance from awhile back you might want to look into. I had to do this before with about 1M addresses and ended up creating an ec2 instance for each state and sending addresses with that state to that specific host (using a simple regex to match state in an address line). Somewhat expensive, but got the job done quickly.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Johnathan<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Tue, Nov 29, 2011 at 9:18 PM, Ravi ada <<a href="mailto:raviada@dobeyond.com">raviada@dobeyond.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello All,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have been fighting with PostGIS for the last week or two to geocode about 2 million addresses. Here are the challenges that I am facing.<o:p></o:p></p><p>1)<span style='font-size:7.0pt'> </span>I sorted the addresses by zip, address, city to group the similar addresses together.<o:p></o:p></p><p>2)<span style='font-size:7.0pt'> </span>So far I have seen only 50% of rating zero, that means perfect match.<o:p></o:p></p><p>3)<span style='font-size:7.0pt'> </span>Some addresses have Suite# or Apt# in address line 1. Geocode function takes longer time for such addresses<o:p></o:p></p><p>4)<span style='font-size:7.0pt'> </span>Addresses that are on the interstate, may be on the service road. These are taking longer to code.<o:p></o:p></p><p>5)<span style='font-size:7.0pt'> </span>Getting syntax errors on line_segment function when processing some addresses that have additional data in parenthesis and a period. For example "5727 FM 3097(HORIZON RD.),ROCKWALL,TX,750327786" <o:p></o:p></p><p>6)<span style='font-size:7.0pt'> </span>I am using batch updates selecting the addresses per state to reduce the size of the batch. I used straight SQL update as mentioned in the documentation, and wrote a function to call geo code for each address. In both cases the query ran for more than 6 hours and never been able finish successfully. If I kill the query, nothing gets updated to the address table because the whole transaction is killed and not committed.<o:p></o:p></p><p>7)<span style='font-size:7.0pt'> </span>I am running this on Ubuntu 11.10 and postgresql 9.1 on a virtual machine on 8GB i5 machine. I know the resources are very less for such a task.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>My questions are<o:p></o:p></p><p>1)<span style='font-size:7.0pt'> </span>How other people are geocoding so many addresses, what is the best way to do it, knowing that we may not have fully normalized addresses.<o:p></o:p></p><p>2)<span style='font-size:7.0pt'> </span>How long it should take to geocode about 2mil addresses.<o:p></o:p></p><p>3)<span style='font-size:7.0pt'> </span>What does it mean if the rating is non-zero, and more than 50. Do we get the distance wring for such addresses?<o:p></o:p></p><p>4)<span style='font-size:7.0pt'> </span>What is the recommended hardware configuration? We need to geocode these addresses only once, and periodically to the delta that we get every month. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thank you so much for all the efforts that you guys are contributing to the project. Any help with my questions would greatly be appreciated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Ravi Ada<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>postgis-users mailing list<br><a href="mailto:postgis-users@postgis.refractions.net">postgis-users@postgis.refractions.net</a><br><a href="http://postgis.refractions.net/mailman/listinfo/postgis-users" target="_blank">http://postgis.refractions.net/mailman/listinfo/postgis-users</a><o:p></o:p></p></div><p class=MsoNormal><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <br><i><span style='color:#CCCCCC'>Johnathan</span></i><o:p></o:p></p><div><p class=MsoNormal><span style='color:#CCCCCC'>Software Engineer</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#CCCCCC'>San Francisco, California</span><o:p></o:p></p></div><div><p class=MsoNormal><i><span style='color:#CCCCCC'><a href="http://twitter.com/#!/iamleppert" target="_blank">Follow me on Twitter: @iamleppert</a></span></i><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>