Add and document py::error_already_set::discard_as_unraisable()

To deal with exceptions that hit destructors or other noexcept functions.

Includes fixes to support Python 2.7 and extends documentation on
error handling.

@virtuald and @YannickJadoul both contributed to this PR.

docs/advanced/classes.rst

@@ -559,6 +559,44 @@ crucial that instances are deallocated on the C++ side to avoid memory leaks.
     py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass")
     py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass")
         .def(py::init<>())
         .def(py::init<>())
 
 
+.. _destructors_that_call_python:
+
+Destructors that call Python
+============================
+
+If a Python function is invoked from a C++ destructor, an exception may be thrown
+of type :class:`error_already_set`. If this error is thrown out of a class destructor,
+``std::terminate()`` will be called, terminating the process. Class destructors
+must catch all exceptions of type :class:`error_already_set` to discard the Python
+exception using :func:`error_already_set::discard_as_unraisable`.
+
+Every Python function should be treated as *possibly throwing*. When a Python generator
+stops yielding items, Python will throw a ``StopIteration`` exception, which can pass
+though C++ destructors if the generator's stack frame holds the last reference to C++
+objects.
+
+For more information, see :ref:`the documentation on exceptions <unraisable_exceptions>`.
+
+.. code-block:: cpp
+
+    class MyClass {
+    public:
+        ~MyClass() {
+            try {
+                py::print("Even printing is dangerous in a destructor");
+                py::exec("raise ValueError('This is an unraisable exception')");
+            } catch (py::error_already_set &e) {
+                // error_context should be information about where/why the occurred,
+                // e.g. use __func__ to get the name of the current function
+                e.discard_as_unraisable(__func__);
+            }
+        }
+    };
+
+.. note::
+
+    pybind11 does not support C++ destructors marked ``noexcept(false)``.
+
 .. _implicit_conversions:
 .. _implicit_conversions:
 
 
 Implicit conversions
 Implicit conversions

docs/advanced/exceptions.rst

@@ -53,9 +53,15 @@ exceptions:
 |                                      | a Python exception back to Python.   |
 |                                      | a Python exception back to Python.   |
 +--------------------------------------+--------------------------------------+
 +--------------------------------------+--------------------------------------+
 
 
-When a Python function invoked from C++ throws an exception, it is converted
-into a C++ exception of type :class:`error_already_set` whose string payload
-contains a textual summary.
+When a Python function invoked from C++ throws an exception, pybind11 will convert
+it into a C++ exception of type :class:`error_already_set` whose string payload
+contains a textual summary. If you call the Python C-API directly, and it
+returns an error, you should ``throw py::error_already_set();``, which allows
+pybind11 to deal with the exception and pass it back to the Python interpreter.
+(Another option is to call ``PyErr_Clear`` in the
+`Python C-API <https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_
+to clear the error. The Python error must be thrown or cleared, or Python/pybind11
+will be left in an invalid state.)
 
 
 There is also a special exception :class:`cast_error` that is thrown by
 There is also a special exception :class:`cast_error` that is thrown by
 :func:`handle::call` when the input arguments cannot be converted to Python
 :func:`handle::call` when the input arguments cannot be converted to Python
@@ -142,3 +148,43 @@ section.
     Exceptions that you do not plan to handle should simply not be caught, or
     Exceptions that you do not plan to handle should simply not be caught, or
     may be explicitly (re-)thrown to delegate it to the other,
     may be explicitly (re-)thrown to delegate it to the other,
     previously-declared existing exception translators.
     previously-declared existing exception translators.
+
+.. _unraisable_exceptions:
+
+Handling unraisable exceptions
+==============================
+
+If a Python function invoked from a C++ destructor or any function marked
+``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
+is no way to propagate the exception, as such functions may not throw at
+run-time.
+
+Neither Python nor C++ allow exceptions raised in a noexcept function to propagate. In
+Python, an exception raised in a class's ``__del__`` method is logged as an
+unraisable error. In Python 3.8+, a system hook is triggered and an auditing
+event is logged. In C++, ``std::terminate()`` is called to abort immediately.
+
+Any noexcept function should have a try-catch block that traps
+class:`error_already_set` (or any other exception that can occur). Note that pybind11
+wrappers around Python exceptions such as :class:`pybind11::value_error` are *not*
+Python exceptions; they are C++ exceptions that pybind11 catches and converts to
+Python exceptions. Noexcept functions cannot propagate these exceptions either.
+You can convert them to Python exceptions and then discard as unraisable.
+
+.. code-block:: cpp
+
+    void nonthrowing_func() noexcept(true) {
+        try {
+            // ...
+        } catch (py::error_already_set &eas) {
+            // Discard the Python error using Python APIs, using the C++ magic
+            // variable __func__. Python already knows the type and value and of the
+            // exception object.
+            eas.discard_as_unraisable(__func__);
+        } catch (const std::exception &e) {
+            // Log and discard C++ exceptions.
+            // (We cannot use discard_as_unraisable, since we have a generic C++
+            // exception, not an exception that originated from Python.)
+            third_party::log(e);
+        }
+    }

include/pybind11/pytypes.h

@@ -337,6 +337,20 @@ public:
     /// error variables (but the `.what()` string is still available).
     /// error variables (but the `.what()` string is still available).
     void restore() { PyErr_Restore(m_type.release().ptr(), m_value.release().ptr(), m_trace.release().ptr()); }
     void restore() { PyErr_Restore(m_type.release().ptr(), m_value.release().ptr(), m_trace.release().ptr()); }
 
 
+    /// If it is impossible to raise the currently-held error, such as in destructor, we can write
+    /// it out using Python's unraisable hook (sys.unraisablehook). The error context should be
+    /// some object whose repr() helps identify the location of the error. Python already knows the
+    /// type and value of the error, so there is no need to repeat that. For example, __func__ could
+    /// be helpful. After this call, the current object no longer stores the error variables,
+    /// and neither does Python.
+    void discard_as_unraisable(object err_context) {
+        restore();
+        PyErr_WriteUnraisable(err_context.ptr());
+    }
+    void discard_as_unraisable(const char *err_context) {
+        discard_as_unraisable(reinterpret_steal<object>(PYBIND11_FROM_STRING(err_context)));
+    }
+
     // Does nothing; provided for backwards compatibility.
     // Does nothing; provided for backwards compatibility.
     PYBIND11_DEPRECATED("Use of error_already_set.clear() is deprecated")
     PYBIND11_DEPRECATED("Use of error_already_set.clear() is deprecated")
     void clear() {}
     void clear() {}

tests/test_exceptions.cpp

@@ -65,6 +65,25 @@ struct PythonCallInDestructor {
     py::dict d;
     py::dict d;
 };
 };
 
 
+
+
+struct PythonAlreadySetInDestructor {
+    PythonAlreadySetInDestructor(const py::str &s) : s(s) {}
+    ~PythonAlreadySetInDestructor() {
+        py::dict foo;
+        try {
+            // Assign to a py::object to force read access of nonexistent dict entry
+            py::object o = foo["bar"];
+        }
+        catch (py::error_already_set& ex) {
+            ex.discard_as_unraisable(s);
+        }
+    }
+
+    py::str s;
+};
+
+
 TEST_SUBMODULE(exceptions, m) {
 TEST_SUBMODULE(exceptions, m) {
     m.def("throw_std_exception", []() {
     m.def("throw_std_exception", []() {
         throw std::runtime_error("This exception was intentionally thrown.");
         throw std::runtime_error("This exception was intentionally thrown.");
@@ -183,6 +202,11 @@ TEST_SUBMODULE(exceptions, m) {
         return false;
         return false;
     });
     });
 
 
+    m.def("python_alreadyset_in_destructor", [](py::str s) {
+        PythonAlreadySetInDestructor alreadyset_in_destructor(s);
+        return true;
+    });
+
     // test_nested_throws
     // test_nested_throws
     m.def("try_catch", [m](py::object exc_type, py::function f, py::args args) {
     m.def("try_catch", [m](py::object exc_type, py::function f, py::args args) {
         try { f(*args); }
         try { f(*args); }

tests/test_exceptions.py

 # -*- coding: utf-8 -*-
 # -*- coding: utf-8 -*-
+import sys
+
 import pytest
 import pytest
 
 
 from pybind11_tests import exceptions as m
 from pybind11_tests import exceptions as m
@@ -48,6 +50,33 @@ def test_python_call_in_catch():
     assert d["good"] is True
     assert d["good"] is True
 
 
 
 
+def test_python_alreadyset_in_destructor(monkeypatch, capsys):
+    hooked = False
+    triggered = [False]  # mutable, so Python 2.7 closure can modify it
+
+    if hasattr(sys, 'unraisablehook'):  # Python 3.8+
+        hooked = True
+        default_hook = sys.unraisablehook
+
+        def hook(unraisable_hook_args):
+            exc_type, exc_value, exc_tb, err_msg, obj = unraisable_hook_args
+            if obj == 'already_set demo':
+                triggered[0] = True
+            default_hook(unraisable_hook_args)
+            return
+
+        # Use monkeypatch so pytest can apply and remove the patch as appropriate
+        monkeypatch.setattr(sys, 'unraisablehook', hook)
+
+    assert m.python_alreadyset_in_destructor('already_set demo') is True
+    if hooked:
+        assert triggered[0] is True
+
+    _, captured_stderr = capsys.readouterr()
+    # Error message is different in Python 2 and 3, check for words that appear in both
+    assert 'ignored' in captured_stderr and 'already_set demo' in captured_stderr
+
+
 def test_exception_matches():
 def test_exception_matches():
     assert m.exception_matches()
     assert m.exception_matches()
     assert m.exception_matches_base()
     assert m.exception_matches_base()