[postgis-users] identify rows in a shapefile that have illegal characters

Mark Volz MarkVolz at co.lyon.mn.us
Wed Oct 24 08:08:03 PDT 2012


Hello,

I am getting closer to a solution for loading PostGIS data into ArcGIS.  But I am still having some trouble.  Here are some things that I found thus far:
1.  I had a bad character "ó" in my legal description on a single row that stopped the data from loading into postgis using either UTF-8 or LATIN1 encoding.  That was corrected and I was able to load data into PostGIS with either encoding.
2.   Next when I tried to load PostGIS data into ArcGIS, ArcGIS complained about the spatial extent.  I noticed that the shapefile geometry is POLYGON_ZM.  I used feature class to shapefile to remove the Z and M values.  Once I did that I was able to load data into PostGIS again, and then into ArcGIS.  The PostGIS Data appears to be in the correct spot.  However, If I try to open the attribute table ArcGIS reports "Error reading OID from table.  Reading rows has been sopped.  Check that the datasource is valid.  OID mapped column has null value.  The operation is not supported by this implementation." 
3.  I removed the parcels with short and very long legal descriptions, and I changed the length from 254 characters to 250 characters.  Like step 2, I could load the geography into ArcGIS, however the attributes did not display.
4.  I removed the legal description from the parcels.  Once I did that I was able to view both the geography and the database within a PostGIS query layer in ArcGIS. 

So I still need to find out what is causing ArcGIS to not read the legal description from my PostGIS database.


Paul,
To answer your questions I am using PostGIS 2.0 Shapefile and DBF Loader and Exporter.  On a failed load the program advised me to change from UTF-8 to LATIN1 in the Import Options dialog.  I also think that the real issue is not because of the encoding.  It turns out that PostGIS did not like the "ó" character.

Hugues,
I made sure that my database does not have any null values in the legal description.  I also made sure that the legal description and column name lgl were all shorter than 254 characters.   

Thanks for your Ideas

Mark Volz
GIS Specialist

 
> On Mon, Oct 22, 2012 at 1:01 PM, Mark Volz <MarkVolz at co.lyon.mn.us>
> wrote:
> > I am trying to load my parcels into PostGIS, which will eventually be
> consumed by MapServer, and ArcGIS.  Initially when loaded my data I
> received a warning that I should change my encoding from UTF-8 to LATIN1.
> 
> How did you "change your encoding"? In your database, or in your data load?
> If you just ran
> 
> shp2pgsql -W LATIN1 shpfile.shp tblename
> 
> Then the non-ASCII characters in your dbf file would have been transcoded
> to UTF8 during the load and landed nicely in the database with the right UTF
> code points.
> 
> 
> > Doing so allowed me to load data into PostGIS however, I could not
> consume the data in ArcGIS.
> 
> This seems fishy. If your database is UTF and you load using the -W flag as
> above, everything is pretty bog standard and ArcGIS should be able to read it
> fine (particularly since the libpq library does all the transcoding for client
> apps! ArcGIS doesn't even have to think about transcoding, just declare the
> encoding it desires!)
> 
> > How can I find out which rows in my shapefile have illegal characters for
> UTF-8 encoding?
> 
> There are no illegal character for UTF, UTF can represent any and all
> characters (and does). There's something else going on
> 
> P.
> 
> 
> Hello,
> 
> I often play with postgis data in arcgis, using original shapefiles encoded using
> LATIN1 and I have no characters problem.
> 
> My major problem comes from null values which prevent arcgis to display
> data attributes but, usually, all my geometries are here and I can even play
> with symbology without the possibility to display the data I'm working on.
> 
> Did you check null values or did you try to import them in arcgis as a request
> layer using the where clause to exclude null data ?
> 
> Hugues.
> 
> 
> 



More information about the postgis-users mailing list