[GRASSLIST:5778] Re: m.in.e00, missing areas

[cheg01 at attbi.com]

This may help if the errors are caused by stray arcs in the areas: OK in
ARCINFO, not OK in Grass.

----- Original Message -----
From: "cheg"
Newsgroups: comp.infosystems.gis
Sent: Saturday, March 30, 2002 12:47 PM
Subject: Partial fix for GRASS " PNT_TO_AREA failed " e00 import errors

I have written a couple of awk programs to diagnose and fix many of the
"PNT_TO_AREA failed" error that occur when importing ARC/INFO E00 transfer
files into GRASS. The programs modify the E00 file before importation. The
problem is that GRASS will not accept the stray arcs that ARC/INFO defines
from the edge of a polygon to an island within the polygon or from the edge
to a random point within the polygon. Most of the polygons that fail have
these arc listed in their PAL definition. They can be identified in the ARC
section by the fact that their left and right hand polygon are the same.
The awk program "fix_e00.awk" listed below removes the bad arcs and
converts all PAL references to them to "0 0 0". GRASS seems to ignore the
"virtual" (0 0 0) arcs, though I can't say what effect the change will have
on any other transfers with the altered E00 files. The diagnostic programs
"test_e00.awk" and "scan_e00.awk" listed below find and list problems to
see if a fix is required before importing an e00 file into GRASS.

There is also an example script file listed below that walks through a
directory tree and does dos2unix conversion and runs the fix
program on selected E00 files. The selection criterion is that the file has
to contain a PAL section. THE FIX_E00.AWK PROGRAM WILL TRASH E00 FILES THAT
ARE NOT POLYGON COVERAGES because all arcs will disappear. If the coverage
does not include a PAL section, don't use the fix program. Caveat emptor.
Don't do anything you cannot recover from.

The programs require that the E00 files be uncompressed. If you have
compressed files, look for the "e00conv" program available on the internet
that will compress and decompress the files. It can be invoked from a
script before calling the awk routines. (Or feel free to rewrite the awk
programs to work with compressed files... )

fix_e00.awk will not eliminate all PNT_TO_AREA errors. The residual errors
seem to come in groups and are caused by digitizing errors in the E00 file.
One polygon fails because it has an arc with a badly defined series of
points (very large changes of slope over a very short distance) and the
adjacent polygons fail because they share the boundary. The discontinuity
in slope has to be fixed manually and the polygons relabeled by hand in
v.digit.

The awk programs will give errors if used with the original "old awk". Make
sure that the command awk (usually in /usr/bin/awk) points to "new awk" or
nawk. This is my first attempt at awk programming, so suggestions on
maximizing efficiency are welcome.

The syntax of the repair program is:
"awk -f fix_e00.awk [filename].e00 > [new filename]"

For the diagnostics:
"awk -f scan_e00.awk [filename].e00"
"awk -f test_e00.awk [filename].e00"

__________BEGIN CUT SCRIPT EXAMPLE __________
#!/bin/csh
cd /usr2/grass/data/100k
foreach directory ( q* )
    cd $directory
    foreach e00_file ( *.e00 )
          dos2unix $e00_file $e00_file
          if ( { grep -c PAL $e00_file } ) then
               echo "Fixing /$directory/$e00_file"
               awk -f ../../test.awk $e00_file
               awk -f ../../fix_e00.awk $e00_file > $e00_file.new
#               mv $e00_file.new $e00_file
          endif
     end
     cd ..
end
____________END CUT SCRIPT EXAMPLE __________

_____________ BEGIN CUT FIX_E00.AWK ____________

####################################################
## fix_e00.awk: Removes bad arc definitions from uncompressed  ##
## e00 files before importing into GRASS gis
##
####################################################

function abs(x) {return x<0?-x:x}

BEGIN { poly=0; found=0; j=0}

# Find section headers
/^[A-Z]..  [2-3]?/ { section = $1; precision = $2}

# Save all unchanged sections
section!="ARC" && section!="PAL" { print $0; next }

# Save first and last line of ARC section
section=="ARC" && $1=="ARC" { print $0; next }
section=="ARC" && $1==-1 { print $0; next }

# Save all good arcs
section == "ARC" && ($5!=$6 || $5==0) {
    nnodes=$7
    print $0
    if (precision==2) nlines=(nnodes%2)?nnodes/2+0.5:nnodes/2
    if (precision==3) nlines=nnodes
    for (i=1;i<=nlines;i++) {getline; print $0}
    next
    }

# Skip (delete) arcs with the same poly on both sides
# Deletes Arc definition and all nodes of the arc
section=="ARC" && $5==$6 && $5!=0 {
    bad_line[j]=$1
    bad_poly[j]=$5
    nnodes=$7
    if (precision==2) nlines=(nnodes%2)?nnodes/2+0.5:nnodes/2
    if (precision==3) nlines=nnodes
    for (i=0;i<nlines;i++) getline
    j++; next
    }

# Save first and last line of PAL section
section=="PAL" && $1=="PAL" { print $0; next }
section=="PAL" && NF==7 { print $0; next }

# Locate polygons that reference deleted arcs and determine how
# many lines are used in the polygon definition
section=="PAL" && ((precision==2 && NF==5) || \
                                        (precision==3 && NF==3)) {
    poly++; print $0
    if (found==1) found=0
    n_arc=$1; nlines=(n_arc%2)?n_arc/2+0.5:n_arc/2
    for (k in bad_poly) { if (poly==bad_poly[k]) found=1 }

# Preserve all arc lines in good polygons
    if (found==0) {
        for (i=1;i<=nlines;i++) { getline; print $0 }
        next
        }
    }

# For all bad polygons, convert references to bad arcs to (0 0 0)
# This preserves the total number of arcs in the polygon definition
found==1  {
    # Collect all arc definitions in an array
    for (i=0;i<nlines;i++) { getline; for (j=1;j<7;j++) a[i,j]=$j }
    # Inspect array for references to bad arcs
    for (i=1;i<=nlines;i++) {
        for (j in bad_lines) {
            if (abs(a[i,1])==bad_lines[j]) fix[i,1]=-1
            else if (abs(a[i,1])==0) fix[i,1]=0
            else fix[i,1]=1

            if (abs(a[i,4])==bad_lines[j]) fix[i,2]=-1
            else if (abs(a[i,4])==0) fix[i,2]=0
            else fix[i,2]=1
            }
        }
    # Print arc list with bad arcs replaced
    for (i=1;i<=nlines;i++) {
        # Print first arc
        if (fix[i,1]<0) printf(" %9d %9d %9d",0,0,0)
        else printf(" %9d %9d %9d",a[i,1],a[i,2],a[i,3])
        # Print second arc unless last line has single arc definition
        if (i<nlines) {
            if (fix[i,2]<0) printf(" %9d %9d %9d",0,0,0)
            else printf(" %9d %9d %9d",a[i,4],a[i,5],a[i,6])
            }
        else {
            # Skip last arc for polygons with odd number of arcs
            if (a[i,4]!=0) {
                 if (fix[i,2]<0) printf(" %9d %9d %9d",0,0,0)
                 else printf(" %9d %9d %9d",a[i,4],a[i,5],a[i,6])
                 }
             }
        printf("\n")
        # Free memory from arcs after use
        for (j=1;j<7;j++) delete a[i,j]
        }
    }

__________ END CUT FIX_E00.AWK ____________


__________BEGIN CUT TEST_E00.AWK __________
####################################################
## test_e00.awk: Locates and lists bad arc and poly definitions in ##
## uncompressed e00 files before importing into GRASS gis        ##
####################################################

function abs(x) {return x<0?-x:x}

BEGIN {poly=0}

/^[A-Z]..  [2-3]?/ {section=$1}

section=="ARC" && NF==7 && $5==$6 && $5!=0 {
    bad_line[j]=$1
    bad_poly[j]=$5
    print "Arc ",bad_line[j]," is bad"
    j++
    }

section=="PAL" && NF==5 {
    poly++;
    if (found) found=0
    for (k in bad_poly) if (poly==bad_poly[k]) found=1
    if (found==1) print "Poly",poly,"is bad"
    }

found==1 {
    for (l in bad_line) if (abs($1)==bad_line[l] ||
abs($4)==bad_line[l] )
    print "Bad Arc",bad_line[l]," referenced on line",NR,"in
poly",poly
    }
____________END CUT TEST_E00.AWK_______________

____________BEGIN CUT SCAN_E00.AWK_______________

BEGIN {poly=0;bad=0;arcs=0}

/^[A-Z]..  [2-3]?/ { section=$1; precision=$2 }

section!="ARC" && section!="PAL" { next }

section=="ARC" && NF==7 && $5==$6 && $5!=0 {
    bad_line[j]=$1
    bad_poly[j]=$5
    j++
    }

section=="ARC" && NF==7 && $1!=-1  { arcs++
    nnodes=$7
    if (precision==2) nlines=(nnodes%2)?nnodes/2+0.5:nnodes/2
    if (precision==3) nlines=nnodes
    for (i=0;i<nlines;i++) getline
    next
    }

section=="PAL" && ((precision==3 && NF==3) || \
                                        (precision==2 && NF=5)) {
    poly++;
    n_arc=$1;
    if (precision==2) nlines=(n_arc%2)?n_arc/2+0.5:n_arc/2
    if (precision==3) nlines=n_arc
    for (k in bad_poly) if (poly==bad_poly[k]) { bad++; break }
    for (i=1;i<=nlines;i++) {getline}
    next
    }

END {
    print j,"/",arcs,"arcs bad"
    print bad,"/",poly,"polygons bad"
    }
____________END CUT SCAN_E00.AWK_______________