Back to home page

EIC code displayed by LXR

 
 

    


Warning, /jana2/src/python/externals/pybind11-2.10.3/docs/faq.rst is written in an unsupported language. File is not indexed.

0001 Frequently asked questions
0002 ##########################
0003 
0004 "ImportError: dynamic module does not define init function"
0005 ===========================================================
0006 
0007 1. Make sure that the name specified in PYBIND11_MODULE is identical to the
0008 filename of the extension library (without suffixes such as ``.so``).
0009 
0010 2. If the above did not fix the issue, you are likely using an incompatible
0011 version of Python that does not match what you compiled with.
0012 
0013 "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
0014 ========================================================================
0015 
0016 See the first answer.
0017 
0018 "SystemError: dynamic module not initialized properly"
0019 ======================================================
0020 
0021 See the first answer.
0022 
0023 The Python interpreter immediately crashes when importing my module
0024 ===================================================================
0025 
0026 See the first answer.
0027 
0028 .. _faq_reference_arguments:
0029 
0030 Limitations involving reference arguments
0031 =========================================
0032 
0033 In C++, it's fairly common to pass arguments using mutable references or
0034 mutable pointers, which allows both read and write access to the value
0035 supplied by the caller. This is sometimes done for efficiency reasons, or to
0036 realize functions that have multiple return values. Here are two very basic
0037 examples:
0038 
0039 .. code-block:: cpp
0040 
0041     void increment(int &i) { i++; }
0042     void increment_ptr(int *i) { (*i)++; }
0043 
0044 In Python, all arguments are passed by reference, so there is no general
0045 issue in binding such code from Python.
0046 
0047 However, certain basic Python types (like ``str``, ``int``, ``bool``,
0048 ``float``, etc.) are **immutable**. This means that the following attempt
0049 to port the function to Python doesn't have the same effect on the value
0050 provided by the caller -- in fact, it does nothing at all.
0051 
0052 .. code-block:: python
0053 
0054     def increment(i):
0055         i += 1  # nope..
0056 
0057 pybind11 is also affected by such language-level conventions, which means that
0058 binding ``increment`` or ``increment_ptr`` will also create Python functions
0059 that don't modify their arguments.
0060 
0061 Although inconvenient, one workaround is to encapsulate the immutable types in
0062 a custom type that does allow modifications.
0063 
0064 An other alternative involves binding a small wrapper lambda function that
0065 returns a tuple with all output arguments (see the remainder of the
0066 documentation for examples on binding lambda functions). An example:
0067 
0068 .. code-block:: cpp
0069 
0070     int foo(int &i) { i++; return 123; }
0071 
0072 and the binding code
0073 
0074 .. code-block:: cpp
0075 
0076    m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
0077 
0078 
0079 How can I reduce the build time?
0080 ================================
0081 
0082 It's good practice to split binding code over multiple files, as in the
0083 following example:
0084 
0085 :file:`example.cpp`:
0086 
0087 .. code-block:: cpp
0088 
0089     void init_ex1(py::module_ &);
0090     void init_ex2(py::module_ &);
0091     /* ... */
0092 
0093     PYBIND11_MODULE(example, m) {
0094         init_ex1(m);
0095         init_ex2(m);
0096         /* ... */
0097     }
0098 
0099 :file:`ex1.cpp`:
0100 
0101 .. code-block:: cpp
0102 
0103     void init_ex1(py::module_ &m) {
0104         m.def("add", [](int a, int b) { return a + b; });
0105     }
0106 
0107 :file:`ex2.cpp`:
0108 
0109 .. code-block:: cpp
0110 
0111     void init_ex2(py::module_ &m) {
0112         m.def("sub", [](int a, int b) { return a - b; });
0113     }
0114 
0115 :command:`python`:
0116 
0117 .. code-block:: pycon
0118 
0119     >>> import example
0120     >>> example.add(1, 2)
0121     3
0122     >>> example.sub(1, 1)
0123     0
0124 
0125 As shown above, the various ``init_ex`` functions should be contained in
0126 separate files that can be compiled independently from one another, and then
0127 linked together into the same final shared object.  Following this approach
0128 will:
0129 
0130 1. reduce memory requirements per compilation unit.
0131 
0132 2. enable parallel builds (if desired).
0133 
0134 3. allow for faster incremental builds. For instance, when a single class
0135    definition is changed, only a subset of the binding code will generally need
0136    to be recompiled.
0137 
0138 "recursive template instantiation exceeded maximum depth of 256"
0139 ================================================================
0140 
0141 If you receive an error about excessive recursive template evaluation, try
0142 specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
0143 culprit is generally the generation of function signatures at compile time
0144 using C++14 template metaprogramming.
0145 
0146 .. _`faq:hidden_visibility`:
0147 
0148 "'SomeClass' declared with greater visibility than the type of its field 'SomeClass::member' [-Wattributes]"
0149 ============================================================================================================
0150 
0151 This error typically indicates that you are compiling without the required
0152 ``-fvisibility`` flag.  pybind11 code internally forces hidden visibility on
0153 all internal code, but if non-hidden (and thus *exported*) code attempts to
0154 include a pybind type (for example, ``py::object`` or ``py::list``) you can run
0155 into this warning.
0156 
0157 To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
0158 compiling pybind code.
0159 
0160 As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
0161 have been compiled under different versions of pybind itself, it is also
0162 important that the symbols defined in one module do not clash with the
0163 potentially-incompatible symbols defined in another.  While Python extension
0164 modules are usually loaded with localized symbols (under POSIX systems
0165 typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
0166 can be changed, but even if it isn't it is not always enough to guarantee
0167 complete independence of the symbols involved when not using
0168 ``-fvisibility=hidden``.
0169 
0170 Additionally, ``-fvisibility=hidden`` can deliver considerably binary size
0171 savings. (See the following section for more details.)
0172 
0173 
0174 .. _`faq:symhidden`:
0175 
0176 How can I create smaller binaries?
0177 ==================================
0178 
0179 To do its job, pybind11 extensively relies on a programming technique known as
0180 *template metaprogramming*, which is a way of performing computation at compile
0181 time using type information. Template metaprogramming usually instantiates code
0182 involving significant numbers of deeply nested types that are either completely
0183 removed or reduced to just a few instructions during the compiler's optimization
0184 phase. However, due to the nested nature of these types, the resulting symbol
0185 names in the compiled extension library can be extremely long. For instance,
0186 the included test suite contains the following symbol:
0187 
0188 .. only:: html
0189 
0190     .. code-block:: none
0191 
0192         _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
0193 
0194 .. only:: not html
0195 
0196     .. code-block:: cpp
0197 
0198         __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
0199 
0200 which is the mangled form of the following function type:
0201 
0202 .. code-block:: cpp
0203 
0204     pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
0205 
0206 The memory needed to store just the mangled name of this function (196 bytes)
0207 is larger than the actual piece of code (111 bytes) it represents! On the other
0208 hand, it's silly to even give this function a name -- after all, it's just a
0209 tiny cog in a bigger piece of machinery that is not exposed to the outside
0210 world. So we'll generally only want to export symbols for those functions which
0211 are actually called from the outside.
0212 
0213 This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
0214 and Clang, which sets the default symbol visibility to *hidden*, which has a
0215 tremendous impact on the final binary size of the resulting extension library.
0216 (On Visual Studio, symbols are already hidden by default, so nothing needs to
0217 be done there.)
0218 
0219 In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
0220 potential serious issues when loading multiple modules and is required for
0221 proper pybind operation.  See the previous FAQ entry for more details.
0222 
0223 How can I properly handle Ctrl-C in long-running functions?
0224 ===========================================================
0225 
0226 Ctrl-C is received by the Python interpreter, and holds it until the GIL
0227 is released, so a long-running function won't be interrupted.
0228 
0229 To interrupt from inside your function, you can use the ``PyErr_CheckSignals()``
0230 function, that will tell if a signal has been raised on the Python side.  This
0231 function merely checks a flag, so its impact is negligible. When a signal has
0232 been received, you must either explicitly interrupt execution by throwing
0233 ``py::error_already_set`` (which will propagate the existing
0234 ``KeyboardInterrupt``), or clear the error (which you usually will not want):
0235 
0236 .. code-block:: cpp
0237 
0238     PYBIND11_MODULE(example, m)
0239     {
0240         m.def("long running_func", []()
0241         {
0242             for (;;) {
0243                 if (PyErr_CheckSignals() != 0)
0244                     throw py::error_already_set();
0245                 // Long running iteration
0246             }
0247         });
0248     }
0249 
0250 CMake doesn't detect the right Python version
0251 =============================================
0252 
0253 The CMake-based build system will try to automatically detect the installed
0254 version of Python and link against that. When this fails, or when there are
0255 multiple versions of Python and it finds the wrong one, delete
0256 ``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your
0257 CMake configure line. (Replace ``$(which python)`` with a path to python if
0258 your prefer.)
0259 
0260 You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the
0261 new CMake FindPython support instead of pybind11's custom search. Requires
0262 CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your
0263 ``CMakeLists.txt`` before adding or finding pybind11, as well.
0264 
0265 Inconsistent detection of Python version in CMake and pybind11
0266 ==============================================================
0267 
0268 The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
0269 provided by CMake for Python version detection are modified by pybind11 due to
0270 unreliability and limitations that make them unsuitable for pybind11's needs.
0271 Instead pybind11 provides its own, more reliable Python detection CMake code.
0272 Conflicts can arise, however, when using pybind11 in a project that *also* uses
0273 the CMake Python detection in a system with several Python versions installed.
0274 
0275 This difference may cause inconsistencies and errors if *both* mechanisms are
0276 used in the same project.
0277 
0278 There are three possible solutions:
0279 
0280 1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
0281    from CMake and rely on pybind11 in detecting Python version. If this is not
0282    possible, the CMake machinery should be called *before* including pybind11.
0283 2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python
0284    COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better,
0285    3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead
0286    of the old, deprecated search tools, and these modules are much better at
0287    finding the correct Python.
0288 3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python.
0289    However, you will have to use the target-based system, and do more setup
0290    yourself, because it does not know about or include things that depend on
0291    Python, like ``pybind11_add_module``. This might be ideal for integrating
0292    into an existing system, like scikit-build's Python helpers.
0293 
0294 How to cite this project?
0295 =========================
0296 
0297 We suggest the following BibTeX template to cite pybind11 in scientific
0298 discourse:
0299 
0300 .. code-block:: bash
0301 
0302     @misc{pybind11,
0303        author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
0304        year = {2017},
0305        note = {https://github.com/pybind/pybind11},
0306        title = {pybind11 -- Seamless operability between C++11 and Python}
0307     }