[postgis-users] Loading Large Datasets

[Randy Page]

Hi All,

 

I'm relatively new to PostGIS, but have a lot of experience with ArcSDE.
I'm getting ready to load the entire national set of Tiger data (roads,
tracts, block groups, etc.) into PostGIS.   I have a tool that allows me to
generate shape files either for the entire country or an entire state, so I
can either build a single large shape file for each feature, or start out
with smaller ones and append them to create a seamless coverage in PostGIS.
My goal is to have a single dataset for each feature type.   So, I have a
few questions for those of you who have experience loading large datasets.

 

1)  Are there any issues with loading a single large dataset such as
national coverage of roads (the SQL file is nearly 3GB?)   Is it preferable
to load the data in smaller batches maybe by state?

2)  I have been running a few tests, and have had no problems loading a few
individual counties.   However, I did try loading the entire national set of
streets using shp2pgsql and ran across a few problems while executing the
sql statement.   I found that about 1 in 3 blocks produced an error similar
to this:

 

NOTICE:  current transaction is aborted, queries ignored until end of
transaction block

 

I checked the sql statement, and it's using varchar for each data field, so
I don't think there is an issue with data types.   Any ideas?

 

Thanks in advance,

 

Randy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20040709/f196a687/attachment.html>