[postgis-users] Loading Large Datasets
Tyler Mitchell
tjmitchell at riverside.bc.ca
Fri Jul 9 09:02:51 PDT 2004
I'm not sure if this applies to your situation or not but I recommend
using the ogr2ogr utility from:
http://gdal.maptools.org/ogr/
which can basically import numerous types directly into PostGIS. I use
this instead of creating intermediate SQL files from other utilities
(though even with shp2pgsql you can pipe | it into psql directly and
forego creating 3GB text files :)
Tyler
postgis-users-bounces at postgis.refractions.net wrote on 07/09/2004 06:09:34
AM:
> Hi All,
>
> I’m relatively new to PostGIS, but have a lot of experience with
> ArcSDE. I’m getting ready to load the entire national set of Tiger
> data (roads, tracts, block groups, etc.) into PostGIS. I have a
> tool that allows me to generate shape files either for the entire
> country or an entire state, so I can either build a single large
> shape file for each feature, or start out with smaller ones and
> append them to create a seamless coverage in PostGIS. My goal is
> to have a single dataset for each feature type. So, I have a few
> questions for those of you who have experience loading large datasets.
>
> 1) Are there any issues with loading a single large dataset such as
> national coverage of roads (the SQL file is nearly 3GB?) Is it
> preferable to load the data in smaller batches maybe by state?
> 2) I have been running a few tests, and have had no problems
> loading a few individual counties. However, I did try loading the
> entire national set of streets using shp2pgsql and ran across a few
> problems while executing the sql statement. I found that about 1
> in 3 blocks produced an error similar to this:
>
> NOTICE: current transaction is aborted, queries ignored until end
> of transaction block
>
> I checked the sql statement, and it’s using varchar for each data
> field, so I don’t think there is an issue with data types. Any ideas?
>
> Thanks in advance,
>
> Randy_______________________________________________
> postgis-users mailing list
> postgis-users at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-users
More information about the postgis-users
mailing list