[PROJ] Advice for debugging network callback implementation

[Stephan Hügel]

> On 13 Jul 2020, at 19:05, Even Rouault <even.rouault at spatialys.com> wrote:
> 
> Hi,
>  
> > I’m currently implementing PROJ's network callback functionality for the
> > proj-rs Rust crate. I’ve got a skeleton of the functionality, but the docs
> > are a little vague on some points, which is making debugging tricky.
> > Specifically, I’ve implemented the four callbacks (open_cbk, close_cbk,
> > get_header_value_cbk, read_range_cbk), and I’ve set up my test as follows:
> > 
> > - chosen EPSG codes that will trigger a download: from EPSG4326, to
> > EPSG:4326+3855 - create new proj context with proj_context_create
> > - disabled the grid cache
> > - enabled the network callbacks using set_network_callbacks
> > - used the context to create a transform between known EPSG codes using
> > proj_create_crs_to_crs - A call to open_cbk is triggered:
> > 
> > The url is https://cdn.proj.org/us_nga_egm08_25.tif, with a “Range:
> > bytes=0-16384” header.
>  
> If offset == 0 and size_to_read == 16384, you should issue a 
> "Range: bytes=0-16383" header, since this is the HTTP convention is to use inclusive start and end offset. See
> https://github.com/OSGeo/PROJ/blob/master/src/networkfilemanager.cpp#L1693
>  
> > I can confirm that I get 1639 bytes of content back,
> > along with a response header: “content-range: bytes 0-1638/93581”
>  
> The values you report are not consistent with your above range
>  
> > I cast
> > the response bytes to a void pointer, and assign it to the buffer that is
> > passed into open_cbk by PROJ. I assign the headers and a client instance
> > (this is an implementation detail for now) to a struct which I cast to a
> > void pointer and then to a *PROJ_NETWORK_HANDLE, which I return.
>  
> Sounds fine
>  
> > I’m now not sure where to proceed next: Is there any special format or
> > structure I should be assigning to the bytes from the GET call before I
> > cast them to a void pointer and put them into the buffer?
>  
> No. Up to you to create your own structure.

>  
> > Is there any way
> > I can verify that I’m getting back what I should? I can certainly manually
> > read the first 16385 bytes from the tif, and compare them to what I get
> > back over the wire. What is PROJ trying to do as a next step after it
> > checks the header value, as this is the step which is currently failing?
>  
> You could look at the default curl implementation. See entry points at
> https://github.com/OSGeo/PROJ/blob/master/src/networkfilemanager.cpp#L1907
>  
> When using that default implementation, you can set the PROJ_CURL_VERBOSE environment variable (to any value), and you'll get curl verbose traces for requests issued.
>  
> Even
>  
> -- 
> Spatialys - Geospatial professional services
> http://www.spatialys.com

Hi Even,

This was really useful, thanks. It looks like I have something with minimal functionality working (network calls, header lookups, range reads, and network close calls are happening, and cache.db is being populated with valid data from the network, and being correctly read for subsequent calls using the same transform)

— 
s