<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hello Regina,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I was able to load the data for all states, now I have to gocode about 2 mil addresses . I followed the link below and used the pgScript option to batch 100 addresses. I am having a terrible performance, query return time is 10 to 100 times more than the bench marks. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoPlainText><u><span style='color:#1F497D'>This query is supposed to take only (61ms) but on my machine is was (734ms).<o:p></o:p></span></u></p><p class=MsoPlainText><span style='color:#1F497D'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:#1F497D'>SELECT g.rating, ST_X(g.geomout) As lon, ST_Y(g.geomout) As lat, <o:p></o:p></span></p><p class=MsoPlainText><span style='color:#1F497D'> (addy).address As stno, (addy).streetname As street, <o:p></o:p></span></p><p class=MsoPlainText><span style='color:#1F497D'> (addy).streettypeabbrev As styp, (addy).location As city, (addy).stateabbrev As st,(addy).zip <o:p></o:p></span></p><p class=MsoPlainText><span style='color:#1F497D'> FROM geocode('75 State Street, Boston MA 02109') As g;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am running this on a machine which has 16GB RAM and 6 cores AMD processor. Something is missing to cause this hang up. please direct me to right direction. I am looking for help.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi Ada<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> postgis-users-bounces@postgis.refractions.net [mailto:postgis-users-bounces@postgis.refractions.net] <b>On Behalf Of </b>Paragon Corporation<br><b>Sent:</b> Wednesday, November 30, 2011 8:39 AM<br><b>To:</b> 'PostGIS Users Discussion'<br><b>Subject:</b> Re: [postgis-users] Batch geocoding (~2 mil addresses)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Ravi,</span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>If you have pgAdmin, you might want to try something like pgScript. Unlike plpgsql it commits on each update so doesn't suffer from the same issue as a stored proc.</span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><a href="http://www.postgresonline.com/journal/archives/181-pgscript_intro.html">http://www.postgresonline.com/journal/archives/181-pgscript_intro.html</a></span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Hope that helps,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Regina</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'><a href="http://www.postgis.us">http://www.postgis.us</a></span><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal><o:p> </o:p></p><div class=MsoNormal align=center style='text-align:center'><hr size=2 width="100%" align=center></div><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:postgis-users-bounces@postgis.refractions.net">postgis-users-bounces@postgis.refractions.net</a> <a href="mailto:[mailto:postgis-users-bounces@postgis.refractions.net]">[mailto:postgis-users-bounces@postgis.refractions.net]</a> <b>On Behalf Of </b>Ravi ada<br><b>Sent:</b> Wednesday, November 30, 2011 7:06 AM<br><b>To:</b> 'PostGIS Users Discussion'<br><b>Subject:</b> Re: [postgis-users] Batch geocoding (~2 mil addresses)</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks Jonathan for the quick reply.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am doing the similar thing as you did except doing it locally. I got the address divided into 4 columns, address, city, state and zip. I was able to query for each state and run the geocode. I was never successful running the complete state. The query takes forever to finish even though the query plan shows it is using index scan etc. I even wrote a psql function to pass one address at a time to get the geocode, this one also could not complete because I could not commit each address as I get the gecode and ended up running out of resources.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>How you were able to geocode by state, did you write any scripts or psql functions? If yes, do you mind sharing? I am having hard time geocoding these addresses.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Appreciate any help is extended.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ravi Ada<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:postgis-users-bounces@postgis.refractions.net">postgis-users-bounces@postgis.refractions.net</a> <a href="mailto:[mailto:postgis-users-bounces@postgis.refractions.net]">[mailto:postgis-users-bounces@postgis.refractions.net]</a> <b>On Behalf Of </b>Johnathan Leppert<br><b>Sent:</b> Tuesday, November 29, 2011 11:38 PM<br><b>To:</b> PostGIS Users Discussion<br><b>Subject:</b> Re: [postgis-users] Batch geocoding (~2 mil addresses)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi Ravi,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>There is a thread about the tiger geocoder performance from awhile back you might want to look into. I had to do this before with about 1M addresses and ended up creating an ec2 instance for each state and sending addresses with that state to that specific host (using a simple regex to match state in an address line). Somewhat expensive, but got the job done quickly.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Johnathan<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Tue, Nov 29, 2011 at 9:18 PM, Ravi ada <<a href="mailto:raviada@dobeyond.com">raviada@dobeyond.com</a>> wrote:<o:p></o:p></p><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello All,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have been fighting with PostGIS for the last week or two to geocode about 2 million addresses. Here are the challenges that I am facing.<o:p></o:p></p><p>1)<span style='font-size:7.0pt'> </span>I sorted the addresses by zip, address, city to group the similar addresses together.<o:p></o:p></p><p>2)<span style='font-size:7.0pt'> </span>So far I have seen only 50% of rating zero, that means perfect match.<o:p></o:p></p><p>3)<span style='font-size:7.0pt'> </span>Some addresses have Suite# or Apt# in address line 1. Geocode function takes longer time for such addresses<o:p></o:p></p><p>4)<span style='font-size:7.0pt'> </span>Addresses that are on the interstate, may be on the service road. These are taking longer to code.<o:p></o:p></p><p>5)<span style='font-size:7.0pt'> </span>Getting syntax errors on line_segment function when processing some addresses that have additional data in parenthesis and a period. For example "5727 FM 3097(HORIZON RD.),ROCKWALL,TX,750327786" <o:p></o:p></p><p>6)<span style='font-size:7.0pt'> </span>I am using batch updates selecting the addresses per state to reduce the size of the batch. I used straight SQL update as mentioned in the documentation, and wrote a function to call geo code for each address. In both cases the query ran for more than 6 hours and never been able finish successfully. If I kill the query, nothing gets updated to the address table because the whole transaction is killed and not committed.<o:p></o:p></p><p>7)<span style='font-size:7.0pt'> </span>I am running this on Ubuntu 11.10 and postgresql 9.1 on a virtual machine on 8GB i5 machine. I know the resources are very less for such a task.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>My questions are<o:p></o:p></p><p>1)<span style='font-size:7.0pt'> </span>How other people are geocoding so many addresses, what is the best way to do it, knowing that we may not have fully normalized addresses.<o:p></o:p></p><p>2)<span style='font-size:7.0pt'> </span>How long it should take to geocode about 2mil addresses.<o:p></o:p></p><p>3)<span style='font-size:7.0pt'> </span>What does it mean if the rating is non-zero, and more than 50. Do we get the distance wring for such addresses?<o:p></o:p></p><p>4)<span style='font-size:7.0pt'> </span>What is the recommended hardware configuration? We need to geocode these addresses only once, and periodically to the delta that we get every month. <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thank you so much for all the efforts that you guys are contributing to the project. Any help with my questions would greatly be appreciated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Thanks<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Ravi Ada<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>postgis-users mailing list<br><a href="mailto:postgis-users@postgis.refractions.net">postgis-users@postgis.refractions.net</a><br><a href="http://postgis.refractions.net/mailman/listinfo/postgis-users" target="_blank">http://postgis.refractions.net/mailman/listinfo/postgis-users</a><o:p></o:p></p></div><p class=MsoNormal><br><br clear=all><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>-- <br><i><span style='color:#CCCCCC'>Johnathan</span></i><o:p></o:p></p><div><p class=MsoNormal><span style='color:#CCCCCC'>Software Engineer</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='color:#CCCCCC'>San Francisco, California</span><o:p></o:p></p></div><div><p class=MsoNormal><i><span style='color:#CCCCCC'><a href="http://twitter.com/#!/iamleppert" target="_blank">Follow me on Twitter: @iamleppert</a></span></i><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></blockquote></div></body></html>