[GeoNode-devel] Table Join - Field Format Conversion

[Jeffrey Johnson]

Hi Gents,

As mentioned in a previous post and in GNIP #1915, I've been working
on adding a new 'datatables' contrib module for GeoNode that allows
for uploading of CSV or other delimited files and loading these into
PostgreSQL and joining a existing geometry to them. You can see the
implementation here
https://github.com/jj0hns0n/geonode/compare/table-join?expand=1 ...
Everything is mostly working and functional at an API level. You can
see the test setup we are using in tabular-api/code/tabular_test.py
(this will get converted into a more complete integration test suite).

The largest issue we have run into in our testing is field type
mismatches. The simplest case is an int field in either the table or
the layer and a string field in the other. There are also issues with
the field being specified with leading zeroes or other characters in
one of the 2 join components and not in the other. The folks at
harvard have found other instances where they need to actually join
the contents of 2 fields together before joining.

Some tickets in my own fork for background/reference.

https://github.com/jj0hns0n/geonode/issues/34
https://github.com/jj0hns0n/geonode/issues/30
https://github.com/jj0hns0n/geonode/issues/40
https://github.com/jj0hns0n/geonode/issues/50
https://github.com/jj0hns0n/geonode/issues/49 (the ticket Im dealing
with _right now_)

Raman (@raprasad) and I have discussed several options for dealing
with this kind of problem and we decided to add a class
JoinTargetFormatType that could store information necessary to do
these kinds of conversions. You can see that class here
https://github.com/jj0hns0n/geonode/blob/table-join/geonode/contrib/datatables/models.py#L51
which is used in the JoinTarget class here
https://github.com/jj0hns0n/geonode/blob/table-join/geonode/contrib/datatables/models.py#L71
... The basic idea is that you could have a set of pre-defined
transformation functions or include some regex or a python function to
do the conversion. None of this is hooked up yet, but thats the basic
idea.

So, we wanted to consult with all of you to see if anyone had any
ideas about how best to hook this kind of thing up ... or perhaps some
words of caution about how NOT to do this.

Happy to discuss further if anyone has any questions or is unclear on
what we are trying to do.

Thanks in advance,

Jeff