[mapserver-dev] Regression test issue: NULLs in shapefiles...

[Lime, Steve D (MNIT)]

Just a FYI that this was included as part of the RFC 91 change set - it only affects regex comparisons. Probably something to make note of in the release notes.

Steve

-----Original Message-----
From: mapserver-dev-bounces at lists.osgeo.org [mailto:mapserver-dev-bounces at lists.osgeo.org] On Behalf Of Lime, Steve D (MNIT)
Sent: Monday, August 25, 2014 2:25 PM
To: Even Rouault; mapserver-dev at lists.osgeo.org
Subject: Re: [mapserver-dev] Regression test issue: NULLs in shapefiles...

I think your last suggestion is a good one - it's a pretty localized change to msEvalExpression() for regex expressions and the logical expression parser (mapparser.y).

Steve

-----Original Message-----
From: Even Rouault [mailto:even.rouault at spatialys.com]
Sent: Monday, August 25, 2014 1:01 PM
To: mapserver-dev at lists.osgeo.org
Cc: Lime, Steve D (MNIT)
Subject: Re: [mapserver-dev] Regression test issue: NULLs in shapefiles...

Le lundi 25 août 2014 17:42:26, Lime, Steve D (MNIT) a écrit :
> Hi all: I'm trying to debug a failing CITE test
> (wfs_200_cite_filter_like.xml) for RFC 91 that uses PropertyIsLike. In 
> master that resolves to "item like '%'"  for OGR and other databases.
> For a shapefile it resolves to "item ~ '.*'". In the test shapefile 
> data there are a bunch of null values for the item being checked. The 
> test in question uses the OGR driver. Here's a quick summary:
> 
> Master:
> 
> -          Using OGR driver gets expression WHERE island_e like "%" and
> returns 3 features, ignoring nulls
> 
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
> 
> RFC 91:
> 
> -          Using OGR driver gets expression  "[island_e]" ~ ".*" and
> returns all features
> 
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
> 
> This change is because RFC 91 translation runs through the MapServer 
> expression syntax and the OGR driver can handle both cases. However, 
> that's not the problem since you shouldn't see the difference in 
> master either - a straight shapefile fails. I'm guessing that way down 
> in the shapefile driver nulls in the DBF are handled as empty string 
> which technically match the "0 or more character" pattern. Any ideas 
> on how to deal with something like this?

In OGR, empty strings in .DBF are consider as NULL values, and not empty strings. I'm not sure this is a universal choice, because DBF has no proper way of specifying the NULL concept.
And I don't think there's this concept of NULL currently in MapServer attributes (the lack of typing is also quite limiting sometimes). The start would be to change msDBFReadAttribute() to return NULL if DBFIsValueNULL(). 
But that would likely cause many problems in the rest of the code base.... 
Or a more restricted change when a LIKE is evaluated to consider that empty string is NULL and then evaluate it to FALSE.

> Changing the regex to ".+" makes the test pass but I don't think 
> that's the correct solution.
> 
> Steve
> 
> StEVE LIME  | DATA & Applications MANAGER MN.IT Services @ MnDNR
> 651-259-5473 (w)  |  651-297-4946 (f) | 
> steve.lime at state.mn.us<mailto:Your.name at state.mn.us>
> 
> [cid:image003.jpg at 01CFC051.43750460]<http://www.mn.gov/oet>
> 
> Information Technology for Minnesota Government   |  
> mn.gov/oet<http://www.mn.gov/oet>

--
Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________
mapserver-dev mailing list
mapserver-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-dev