Warning, /jana2/src/python/externals/pybind11-2.10.3/docs/faq.rst is written in an unsupported language. File is not indexed.
0001 Frequently asked questions
0002 ##########################
0003
0004 "ImportError: dynamic module does not define init function"
0005 ===========================================================
0006
0007 1. Make sure that the name specified in PYBIND11_MODULE is identical to the
0008 filename of the extension library (without suffixes such as ``.so``).
0009
0010 2. If the above did not fix the issue, you are likely using an incompatible
0011 version of Python that does not match what you compiled with.
0012
0013 "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
0014 ========================================================================
0015
0016 See the first answer.
0017
0018 "SystemError: dynamic module not initialized properly"
0019 ======================================================
0020
0021 See the first answer.
0022
0023 The Python interpreter immediately crashes when importing my module
0024 ===================================================================
0025
0026 See the first answer.
0027
0028 .. _faq_reference_arguments:
0029
0030 Limitations involving reference arguments
0031 =========================================
0032
0033 In C++, it's fairly common to pass arguments using mutable references or
0034 mutable pointers, which allows both read and write access to the value
0035 supplied by the caller. This is sometimes done for efficiency reasons, or to
0036 realize functions that have multiple return values. Here are two very basic
0037 examples:
0038
0039 .. code-block:: cpp
0040
0041 void increment(int &i) { i++; }
0042 void increment_ptr(int *i) { (*i)++; }
0043
0044 In Python, all arguments are passed by reference, so there is no general
0045 issue in binding such code from Python.
0046
0047 However, certain basic Python types (like ``str``, ``int``, ``bool``,
0048 ``float``, etc.) are **immutable**. This means that the following attempt
0049 to port the function to Python doesn't have the same effect on the value
0050 provided by the caller -- in fact, it does nothing at all.
0051
0052 .. code-block:: python
0053
0054 def increment(i):
0055 i += 1 # nope..
0056
0057 pybind11 is also affected by such language-level conventions, which means that
0058 binding ``increment`` or ``increment_ptr`` will also create Python functions
0059 that don't modify their arguments.
0060
0061 Although inconvenient, one workaround is to encapsulate the immutable types in
0062 a custom type that does allow modifications.
0063
0064 An other alternative involves binding a small wrapper lambda function that
0065 returns a tuple with all output arguments (see the remainder of the
0066 documentation for examples on binding lambda functions). An example:
0067
0068 .. code-block:: cpp
0069
0070 int foo(int &i) { i++; return 123; }
0071
0072 and the binding code
0073
0074 .. code-block:: cpp
0075
0076 m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
0077
0078
0079 How can I reduce the build time?
0080 ================================
0081
0082 It's good practice to split binding code over multiple files, as in the
0083 following example:
0084
0085 :file:`example.cpp`:
0086
0087 .. code-block:: cpp
0088
0089 void init_ex1(py::module_ &);
0090 void init_ex2(py::module_ &);
0091 /* ... */
0092
0093 PYBIND11_MODULE(example, m) {
0094 init_ex1(m);
0095 init_ex2(m);
0096 /* ... */
0097 }
0098
0099 :file:`ex1.cpp`:
0100
0101 .. code-block:: cpp
0102
0103 void init_ex1(py::module_ &m) {
0104 m.def("add", [](int a, int b) { return a + b; });
0105 }
0106
0107 :file:`ex2.cpp`:
0108
0109 .. code-block:: cpp
0110
0111 void init_ex2(py::module_ &m) {
0112 m.def("sub", [](int a, int b) { return a - b; });
0113 }
0114
0115 :command:`python`:
0116
0117 .. code-block:: pycon
0118
0119 >>> import example
0120 >>> example.add(1, 2)
0121 3
0122 >>> example.sub(1, 1)
0123 0
0124
0125 As shown above, the various ``init_ex`` functions should be contained in
0126 separate files that can be compiled independently from one another, and then
0127 linked together into the same final shared object. Following this approach
0128 will:
0129
0130 1. reduce memory requirements per compilation unit.
0131
0132 2. enable parallel builds (if desired).
0133
0134 3. allow for faster incremental builds. For instance, when a single class
0135 definition is changed, only a subset of the binding code will generally need
0136 to be recompiled.
0137
0138 "recursive template instantiation exceeded maximum depth of 256"
0139 ================================================================
0140
0141 If you receive an error about excessive recursive template evaluation, try
0142 specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
0143 culprit is generally the generation of function signatures at compile time
0144 using C++14 template metaprogramming.
0145
0146 .. _`faq:hidden_visibility`:
0147
0148 "'SomeClass' declared with greater visibility than the type of its field 'SomeClass::member' [-Wattributes]"
0149 ============================================================================================================
0150
0151 This error typically indicates that you are compiling without the required
0152 ``-fvisibility`` flag. pybind11 code internally forces hidden visibility on
0153 all internal code, but if non-hidden (and thus *exported*) code attempts to
0154 include a pybind type (for example, ``py::object`` or ``py::list``) you can run
0155 into this warning.
0156
0157 To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
0158 compiling pybind code.
0159
0160 As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
0161 have been compiled under different versions of pybind itself, it is also
0162 important that the symbols defined in one module do not clash with the
0163 potentially-incompatible symbols defined in another. While Python extension
0164 modules are usually loaded with localized symbols (under POSIX systems
0165 typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
0166 can be changed, but even if it isn't it is not always enough to guarantee
0167 complete independence of the symbols involved when not using
0168 ``-fvisibility=hidden``.
0169
0170 Additionally, ``-fvisibility=hidden`` can deliver considerably binary size
0171 savings. (See the following section for more details.)
0172
0173
0174 .. _`faq:symhidden`:
0175
0176 How can I create smaller binaries?
0177 ==================================
0178
0179 To do its job, pybind11 extensively relies on a programming technique known as
0180 *template metaprogramming*, which is a way of performing computation at compile
0181 time using type information. Template metaprogramming usually instantiates code
0182 involving significant numbers of deeply nested types that are either completely
0183 removed or reduced to just a few instructions during the compiler's optimization
0184 phase. However, due to the nested nature of these types, the resulting symbol
0185 names in the compiled extension library can be extremely long. For instance,
0186 the included test suite contains the following symbol:
0187
0188 .. only:: html
0189
0190 .. code-block:: none
0191
0192 __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
0193
0194 .. only:: not html
0195
0196 .. code-block:: cpp
0197
0198 __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
0199
0200 which is the mangled form of the following function type:
0201
0202 .. code-block:: cpp
0203
0204 pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
0205
0206 The memory needed to store just the mangled name of this function (196 bytes)
0207 is larger than the actual piece of code (111 bytes) it represents! On the other
0208 hand, it's silly to even give this function a name -- after all, it's just a
0209 tiny cog in a bigger piece of machinery that is not exposed to the outside
0210 world. So we'll generally only want to export symbols for those functions which
0211 are actually called from the outside.
0212
0213 This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
0214 and Clang, which sets the default symbol visibility to *hidden*, which has a
0215 tremendous impact on the final binary size of the resulting extension library.
0216 (On Visual Studio, symbols are already hidden by default, so nothing needs to
0217 be done there.)
0218
0219 In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
0220 potential serious issues when loading multiple modules and is required for
0221 proper pybind operation. See the previous FAQ entry for more details.
0222
0223 How can I properly handle Ctrl-C in long-running functions?
0224 ===========================================================
0225
0226 Ctrl-C is received by the Python interpreter, and holds it until the GIL
0227 is released, so a long-running function won't be interrupted.
0228
0229 To interrupt from inside your function, you can use the ``PyErr_CheckSignals()``
0230 function, that will tell if a signal has been raised on the Python side. This
0231 function merely checks a flag, so its impact is negligible. When a signal has
0232 been received, you must either explicitly interrupt execution by throwing
0233 ``py::error_already_set`` (which will propagate the existing
0234 ``KeyboardInterrupt``), or clear the error (which you usually will not want):
0235
0236 .. code-block:: cpp
0237
0238 PYBIND11_MODULE(example, m)
0239 {
0240 m.def("long running_func", []()
0241 {
0242 for (;;) {
0243 if (PyErr_CheckSignals() != 0)
0244 throw py::error_already_set();
0245 // Long running iteration
0246 }
0247 });
0248 }
0249
0250 CMake doesn't detect the right Python version
0251 =============================================
0252
0253 The CMake-based build system will try to automatically detect the installed
0254 version of Python and link against that. When this fails, or when there are
0255 multiple versions of Python and it finds the wrong one, delete
0256 ``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your
0257 CMake configure line. (Replace ``$(which python)`` with a path to python if
0258 your prefer.)
0259
0260 You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the
0261 new CMake FindPython support instead of pybind11's custom search. Requires
0262 CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your
0263 ``CMakeLists.txt`` before adding or finding pybind11, as well.
0264
0265 Inconsistent detection of Python version in CMake and pybind11
0266 ==============================================================
0267
0268 The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
0269 provided by CMake for Python version detection are modified by pybind11 due to
0270 unreliability and limitations that make them unsuitable for pybind11's needs.
0271 Instead pybind11 provides its own, more reliable Python detection CMake code.
0272 Conflicts can arise, however, when using pybind11 in a project that *also* uses
0273 the CMake Python detection in a system with several Python versions installed.
0274
0275 This difference may cause inconsistencies and errors if *both* mechanisms are
0276 used in the same project.
0277
0278 There are three possible solutions:
0279
0280 1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
0281 from CMake and rely on pybind11 in detecting Python version. If this is not
0282 possible, the CMake machinery should be called *before* including pybind11.
0283 2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python
0284 COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better,
0285 3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead
0286 of the old, deprecated search tools, and these modules are much better at
0287 finding the correct Python.
0288 3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python.
0289 However, you will have to use the target-based system, and do more setup
0290 yourself, because it does not know about or include things that depend on
0291 Python, like ``pybind11_add_module``. This might be ideal for integrating
0292 into an existing system, like scikit-build's Python helpers.
0293
0294 How to cite this project?
0295 =========================
0296
0297 We suggest the following BibTeX template to cite pybind11 in scientific
0298 discourse:
0299
0300 .. code-block:: bash
0301
0302 @misc{pybind11,
0303 author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
0304 year = {2017},
0305 note = {https://github.com/pybind/pybind11},
0306 title = {pybind11 -- Seamless operability between C++11 and Python}
0307 }