[postgis-users] Loading Large Datasets

[Tyler Mitchell]

I'm not sure if this applies to your situation or not but I recommend 
using the ogr2ogr utility from: 
http://gdal.maptools.org/ogr/
which can basically import numerous types directly into PostGIS.  I use 
this instead of creating intermediate SQL files from other utilities 
(though even with shp2pgsql you can pipe | it into psql directly and 
forego creating 3GB text files :)

Tyler

postgis-users-bounces at postgis.refractions.net wrote on 07/09/2004 06:09:34 
AM:

> Hi All,
> 
> I’m relatively new to PostGIS, but have a lot of experience with 
> ArcSDE.   I’m getting ready to load the entire national set of Tiger
> data (roads, tracts, block groups, etc.) into PostGIS.   I have a 
> tool that allows me to generate shape files either for the entire 
> country or an entire state, so I can either build a single large 
> shape file for each feature, or start out with smaller ones and 
> append them to create a seamless coverage in PostGIS.   My goal is 
> to have a single dataset for each feature type.   So, I have a few 
> questions for those of you who have experience loading large datasets.
> 
> 1)  Are there any issues with loading a single large dataset such as
> national coverage of roads (the SQL file is nearly 3GB?)   Is it 
> preferable to load the data in smaller batches maybe by state?
> 2)  I have been running a few tests, and have had no problems 
> loading a few individual counties.   However, I did try loading the 
> entire national set of streets using shp2pgsql and ran across a few 
> problems while executing the sql statement.   I found that about 1 
> in 3 blocks produced an error similar to this:
> 
> NOTICE:  current transaction is aborted, queries ignored until end 
> of transaction block
> 
> I checked the sql statement, and it’s using varchar for each data 
> field, so I don’t think there is an issue with data types.   Any ideas?
> 
> Thanks in advance,
> 
> Randy_______________________________________________
> postgis-users mailing list
> postgis-users at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-users