[QGIS-Developer] SIP is insane

[Julien Cabieces]

Hi all, 

> Am I the only QGIS dev highly frustrated by the bottleneck of
> sip-build when building QGIS? 

No, you're not alone. I also experience crashes or freezes when building
and I have considerably reduced the number of processes to be used to
avoid these...

> Can we do something (funding upstream?)
> to improve that?

We can still reach out Phil Thompson on the Python-SIP issue tracker [0]

I have done it several times and things have been improved or fixed
(unique_ptr management for instance lately).

Problem is that we depend on SIP versions shipped in linux distributions,
so we would have to wait quite a long time before using those improvments. 

> - we bind to Python a very large part of our header files (more than
>   half). I've always found that the amount of things exposed to Python
>   was excessive. IMHO we should be much more conservative, and wait
>   for plugin authors to ask for an API to be exposed, rather than
>  expose it prematurely.
>

I completely agree. And I'm wondering what part of the API is really
used by the existing plugins, I'm suspecting that it's a very small part.

A while ago, I take a look about what Blender was exposing in its Python
API. It's so small in comparison.

If other contributors agree with this statement, maybe we could try to
work on a Policy to avoid adding any thing new to the Python API that
it's not worth it.

Regards,
Julien


[0]: https://github.com/Python-SIP/sip


> Hi,
>
> sorry for the  eye-catching email title, couldn't resist.
>
> Am I the only QGIS dev highly frustrated by the bottleneck of
> sip-build when building QGIS? Can we do something (funding upstream?)
> to improve that?
>
> Also some mind bogging stats:
>
> - size of all QGIS headers: find ../src -name "*.h" -exec cat {} \;  |
>   wc -l :  570 288
> - size of QGIS .cpp + .ui: find ../src \( -name "*.cpp" -o -name
>   "*.ui" \) -exec cat {} \;  | wc -l : 1 497 216
>
> So QGIS is just a 2 million LoC (excluding tests) code base
>
> - size of .sip generated files: find . -name "*.sip" -exec cat {} \; |
>   wc -l : 335 425
> - size of SIP generated .cpp files:   find python/ -name "*.cpp" -exec
>   cat {} \;  | wc -l:  7 532 645  .  7.5 millions LoC !!!
> - size of MOC generated .cpp files: find src/ -name "*.cpp" -exec cat
>   {} \;  | wc -l :  2 051 769
>
> So 2 observations:
>
> - we bind to Python a very large part of our header files (more than
>   half). I've always found that the amount of things exposed to Python
>   was excessive. IMHO we should be much more conservative, and wait
>   for plugin authors to ask for an API to be exposed, rather than
>  expose it prematurely.
>
> - SIP generates > 20 times more lines of .cpp than .sip files!!!  Or
>   3.75 times more lines of code than QGIS actual logic.
>
> That shows in (stripped) .so files:
>
> -  output/python/qgis/*.so : 130 MB
> - output/lib/*.so.4.1.0 : 170 MB.
>
> So not as crazy as the 7.5 vs 2 MLOC of ratio, but still.
>
> Comparison with GDAL and SWIG:
> - size of public .h headers: 50 162 .
> - size of Python SWIG source .i files:  ~43 000  (hard to count
>   exactly as there's some bits of Java and C# in this count)
> - size of SWIG Python generated .cpp files: 164 151 .
> So a 4x only ratio of .cpp vs .i
>
> I don't recommend SWIG, but something looks fishy in the kingdom of SIP
>
> Even

-- 

Julien Cabieces
Senior Developer at Oslandia
julien.cabieces at oslandia.com