[postgis-devel] Point Selectivity

[Paul Ramsey]

Sandro,
Could you confirm for me:
When a search box is a point, the search box size is very small, so
when the box is compared to the histogram it finds one cell, and then
pro-rates it by the overlap of the search box on the cell, which is
very very small. This is used to pro-rate the value count, which
causes the selectivity number to be a very very small portion of the
cell count. It seems like this will generate pretty small selectivity
estimates, smaller than are warranted.

AOI = intersect_x*intersect_y;
gain = AOI/cell_area;

The comments say

* If the search_box is a point, it will
* overlap a single cell and thus get
* it's value, which is the fraction of
* samples (we can presume of row set also)
* which bumped to that cell.

but I think, contra the comment, the search box does not get the value
of a cell, but a very small fraction of that cell value.

If you can confirm, that would help me move forward without making
some subtle error in interpreting your prior work.

Thanks,

P.