Allow binding factory functions as constructors

This allows you to use:

    cls.def(py::init(&factory_function));

where `factory_function` returns a pointer, holder, or value of the
class type (or a derived type).  Various compile-time checks
(static_asserts) are performed to ensure the function is valid, and
various run-time type checks where necessary.

Some other details of this feature:
- The `py::init` name doesn't conflict with the templated no-argument
  `py::init<...>()`, but keeps the naming consistent: the existing
  templated, no-argument one wraps constructors, the no-template,
  function-argument one wraps factory functions.
- If returning a CppClass (whether by value or pointer) when an CppAlias
  is required (i.e. python-side inheritance and a declared alias), a
  dynamic_cast to the alias is attempted (for the pointer version); if
  it fails, or if returned by value, an Alias(Class &&) constructor
  is invoked.  If this constructor doesn't exist, a runtime error occurs.
- for holder returns when an alias is required, we try a dynamic_cast of
  the wrapped pointer to the alias to see if it is already an alias
  instance; if it isn't, we raise an error.
- `py::init(class_factory, alias_factory)` is also available that takes
  two factories: the first is called when an alias is not needed, the
  second when it is.
- Reimplement factory instance clearing.  The previous implementation
  failed under python-side multiple inheritance: *each* inherited
  type's factory init would clear the instance instead of only setting
  its own type value.  The new implementation here clears just the
  relevant value pointer.
- dealloc is updated to explicitly set the leftover value pointer to
  nullptr and the `holder_constructed` flag to false so that it can be
  used to clear preallocated value without needing to rebuild the
  instance internals data.
- Added various tests to test out new allocation/deallocation code.
- With preallocation now done lazily, init factory holders can
  completely avoid the extra overhead of needing an extra
  allocation/deallocation.
- Updated documentation to make factory constructors the default
  advanced constructor style.
- If an `__init__` is called a second time, we have two choices: we can
  throw away the first instance, replacing it with the second; or we can
  ignore the second call.  The latter is slightly easier, so do that.

CMakeLists.txt

@@ -39,6 +39,7 @@ set(PYBIND11_HEADERS
   include/pybind11/detail/class.h
   include/pybind11/detail/common.h
   include/pybind11/detail/descr.h
+  include/pybind11/detail/init.h
   include/pybind11/detail/typeid.h
   include/pybind11/attr.h
   include/pybind11/buffer_info.h

docs/advanced/classes.rst

@@ -322,6 +322,8 @@ can now create a python class that inherits from ``Dog``:
     See the file :file:`tests/test_virtual_functions.cpp` for complete examples
     using both the duplication and templated trampoline approaches.
 
+.. _extended_aliases:
+
 Extended trampoline class functionality
 =======================================
 
@@ -358,16 +360,124 @@ Custom constructors
 ===================
 
 The syntax for binding constructors was previously introduced, but it only
-works when a constructor with the given parameters actually exists on the C++
-side. To extend this to more general cases, let's take a look at what actually
-happens under the hood: the following statement
+works when a constructor of the appropriate arguments actually exists on the
+C++ side.  To extend this to more general cases, pybind11 offers two different
+approaches: binding factory functions, and placement-new creation.
+
+Factory function constructors
+-----------------------------
+
+It is possible to expose a Python-side constructor from a C++ function that
+returns a new object by value or pointer.  For example, suppose you have a
+class like this:
+
+.. code-block:: cpp
+
+    class Example {
+    private:
+        Example(int); // private constructor
+    public:
+        // Factory function:
+        static Example create(int a) { return Example(a); }
+    };
+
+While it is possible to expose the ``create`` method to Python, it is often
+preferrable to expose it on the Python side as a constructor rather than a
+named static method.  You can do this by calling ``.def(py::init(...))`` with
+the function reference returning the new instance passed as an argument.  It is
+also possible to use this approach to bind a function returning a new instance
+by raw pointer or by the holder (e.g. ``std::unique_ptr``).
+
+The following example shows the different approaches:
+
+.. code-block:: cpp
+
+    class Example {
+    private:
+        Example(int); // private constructor
+    public:
+        // Factory function - returned by value:
+        static Example create(int a) { return Example(a); }
+
+        // These constructors are publicly callable:
+        Example(double);
+        Example(int, int);
+        Example(std::string);
+    };
+
+    py::class_<Example>(m, "Example")
+        // Bind the factory function as a constructor:
+        .def(py::init(&Example::create))
+        // Bind a lambda function returning a pointer wrapped in a holder:
+        .def(py::init([](std::string arg) {
+            return std::unique_ptr<Example>(new Example(arg));
+        }))
+        // Return a raw pointer:
+        .def(py::init([](int a, int b) { return new Example(a, b); }))
+        // You can mix the above with regular C++ constructor bindings as well:
+        .def(py::init<double>())
+        ;
+
+When the constructor is invoked from Python, pybind11 will call the factory
+function and store the resulting C++ instance in the Python instance.
+
+When combining factory functions constructors with :ref:`overriding_virtuals`
+there are two approaches.  The first is to add a constructor to the alias class
+that takes a base value by rvalue-reference.  If such a constructor is
+available, it will be used to construct an alias instance from the value
+returned by the factory function.  The second option is to provide two factory
+functions to ``py::init()``: the first will be invoked when no alias class is
+required (i.e. when the class is being used but not inherited from in Python),
+and the second will be invoked when an alias is required.
+
+You can also specify a single factory function that always returns an alias
+instance: this will result in behaviour similar to ``py::init_alias<...>()``,
+as described in :ref:`extended_aliases`.
+
+The following example shows the different factory approaches for a class with
+an alias:
+
+.. code-block:: cpp
+
+    #include <pybind11/factory.h>
+    class Example {
+    public:
+        // ...
+        virtual ~Example() = default;
+    };
+    class PyExample : public Example {
+    public:
+        using Example::Example;
+        PyExample(Example &&base) : Example(std::move(base)) {}
+    };
+    py::class_<Example, PyExample>(m, "Example")
+        // Returns an Example pointer.  If a PyExample is needed, the Example
+        // instance will be moved via the extra constructor in PyExample, above.
+        .def(py::init([]() { return new Example(); }))
+        // Two callbacks:
+        .def(py::init([]() { return new Example(); } /* no alias needed */,
+                      []() { return new PyExample(); } /* alias needed */))
+        // *Always* returns an alias instance (like py::init_alias<>())
+        .def(py::init([]() { return new PyExample(); }))
+        ;
+
+Low-level placement-new construction
+------------------------------------
+
+A second approach for creating new instances use C++ placement new to construct
+an object in-place in preallocated memory.  To do this, you simply bind a
+method name ``__init__`` that takes the class instance as the first argument by
+pointer or reference, then uses a placement-new constructor to construct the
+object in the pre-allocated (but uninitialized) memory.
+
+For example, instead of:
 
 .. code-block:: cpp
 
     py::class_<Example>(m, "Example")
         .def(py::init<int>());
 
-is short hand notation for
+you could equivalently write:
 
 .. code-block:: cpp
 
@@ -378,9 +488,7 @@ is short hand notation for
             }
         );
 
-In other words, :func:`init` creates an anonymous function that invokes an
-in-place constructor. Memory allocation etc. is already take care of beforehand
-within pybind11.
+which will invoke the constructor in-place at the pre-allocated memory.
 
 .. _classes_with_non_public_destructors:
 

include/pybind11/attr.h

@@ -223,7 +223,7 @@ struct type_record {
     void (*init_instance)(instance *, const void *) = nullptr;
 
     /// Function pointer to class_<..>::dealloc
-    void (*dealloc)(const detail::value_and_holder &) = nullptr;
+    void (*dealloc)(detail::value_and_holder &) = nullptr;
 
     /// List of base classes of the newly created type
     list bases;

include/pybind11/cast.h

@@ -45,7 +45,7 @@ struct type_info {
     size_t type_size, holder_size_in_ptrs;
     void *(*operator_new)(size_t);
     void (*init_instance)(instance *, const void *);
-    void (*dealloc)(const value_and_holder &v_h);
+    void (*dealloc)(value_and_holder &v_h);
     std::vector<PyObject *(*)(PyObject *, PyTypeObject *)> implicit_conversions;
     std::vector<std::pair<const std::type_info *, void *(*)(void *)>> implicit_casts;
     std::vector<bool (*)(PyObject *, void *&)> *direct_conversions;
@@ -301,11 +301,15 @@ struct value_and_holder {
     const detail::type_info *type;
     void **vh;
 
+    // Main constructor for a found value/holder:
     value_and_holder(instance *i, const detail::type_info *type, size_t vpos, size_t index) :
         inst{i}, index{index}, type{type},
         vh{inst->simple_layout ? inst->simple_value_holder : &inst->nonsimple.values_and_holders[vpos]}
     {}
 
+    // Default constructor (used to signal a value-and-holder not found by get_value_and_holder())
+    value_and_holder() : inst{nullptr} {}
+
     // Used for past-the-end iterator
     value_and_holder(size_t index) : index{index} {}
 
@@ -323,22 +327,26 @@ struct value_and_holder {
             ? inst->simple_holder_constructed
             : inst->nonsimple.status[index] & instance::status_holder_constructed;
     }
-    void set_holder_constructed() {
+    void set_holder_constructed(bool v = true) {
         if (inst->simple_layout)
-            inst->simple_holder_constructed = true;
-        else
+            inst->simple_holder_constructed = v;
+        else if (v)
             inst->nonsimple.status[index] |= instance::status_holder_constructed;
+        else
+            inst->nonsimple.status[index] &= (uint8_t) ~instance::status_holder_constructed;
     }
     bool instance_registered() const {
         return inst->simple_layout
             ? inst->simple_instance_registered
             : inst->nonsimple.status[index] & instance::status_instance_registered;
     }
-    void set_instance_registered() {
+    void set_instance_registered(bool v = true) {
         if (inst->simple_layout)
-            inst->simple_instance_registered = true;
-        else
+            inst->simple_instance_registered = v;
+        else if (v)
             inst->nonsimple.status[index] |= instance::status_instance_registered;
+        else
+            inst->nonsimple.status[index] &= (uint8_t) ~instance::status_instance_registered;
     }
 };
 
@@ -403,7 +411,7 @@ public:
  * The returned object should be short-lived: in particular, it must not outlive the called-upon
  * instance.
  */
-PYBIND11_NOINLINE inline value_and_holder instance::get_value_and_holder(const type_info *find_type /*= nullptr default in common.h*/) {
+PYBIND11_NOINLINE inline value_and_holder instance::get_value_and_holder(const type_info *find_type /*= nullptr default in common.h*/, bool throw_if_missing /*= true in common.h*/) {
     // Optimize common case:
     if (!find_type || Py_TYPE(this) == find_type->type)
         return value_and_holder(this, find_type, 0, 0);
@@ -413,6 +421,9 @@ PYBIND11_NOINLINE inline value_and_holder instance::get_value_and_holder(const t
     if (it != vhs.end())
         return *it;
 
+    if (!throw_if_missing)
+        return value_and_holder();
+
 #if defined(NDEBUG)
     pybind11_fail("pybind11::detail::instance::get_value_and_holder: "
             "type is not a pybind11 base of the given instance "
@@ -1454,7 +1465,7 @@ protected:
             throw cast_error("Unable to load a custom holder type from a default-holder instance");
     }
 
-    bool load_value(const value_and_holder &v_h) {
+    bool load_value(value_and_holder &&v_h) {
         if (v_h.holder_constructed()) {
             value = v_h.value_ptr();
             holder = v_h.holder<holder_type>();

include/pybind11/detail/common.h

@@ -441,8 +441,9 @@ struct instance {
     void deallocate_layout();
 
     /// Returns the value_and_holder wrapper for the given type (or the first, if `find_type`
-    /// omitted)
-    value_and_holder get_value_and_holder(const type_info *find_type = nullptr);
+    /// omitted).  Returns a default-constructed (with `.inst = nullptr`) object on failure if
+    /// `throw_if_missing` is false.
+    value_and_holder get_value_and_holder(const type_info *find_type = nullptr, bool throw_if_missing = true);
 
     /// Bit values for the non-simple status flags
     static constexpr uint8_t status_holder_constructed  = 1;

include/pybind11/pybind11.h

@@ -43,6 +43,7 @@
 #include "attr.h"
 #include "options.h"
 #include "detail/class.h"
+#include "detail/init.h"
 
 NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
 
@@ -198,6 +199,8 @@ protected:
                 a.descr = strdup(a.value.attr("__repr__")().cast<std::string>().c_str());
         }
 
+        rec->is_constructor = !strcmp(rec->name, "__init__") || !strcmp(rec->name, "__setstate__");
+
         /* Generate a proper function signature */
         std::string signature;
         size_t type_depth = 0, char_index = 0, type_index = 0, arg_index = 0;
@@ -239,6 +242,12 @@ protected:
                                      .cast<std::string>() + ".";
 #endif
                     signature += tinfo->type->tp_name;
+                } else if (rec->is_constructor && arg_index == 0 && detail::same_type(typeid(handle), *t) && rec->scope) {
+                    // A py::init(...) constructor takes `self` as a `handle`; rewrite it to the type
+#if defined(PYPY_VERSION)
+                    signature += rec->scope.attr("__module__").cast<std::string>() + ".";
+#endif
+                    signature += ((PyTypeObject *) rec->scope.ptr())->tp_name;
                 } else {
                     std::string tname(t->name());
                     detail::clean_type_id(tname);
@@ -267,7 +276,6 @@ protected:
 #endif
         rec->signature = strdup(signature.c_str());
         rec->args.shrink_to_fit();
-        rec->is_constructor = !strcmp(rec->name, "__init__") || !strcmp(rec->name, "__setstate__");
         rec->nargs = (std::uint16_t) args;
 
         if (rec->sibling && PYBIND11_INSTANCE_METHOD_CHECK(rec->sibling.ptr()))
@@ -709,7 +717,11 @@ protected:
         } else {
             if (overloads->is_constructor) {
                 auto tinfo = get_type_info((PyTypeObject *) overloads->scope.ptr());
-                tinfo->init_instance(reinterpret_cast<instance *>(parent.ptr()), nullptr);
+                auto *pi = reinterpret_cast<instance *>(parent.ptr());
+                auto v_h = pi->get_value_and_holder(tinfo);
+                if (!v_h.holder_constructed()) {
+                    tinfo->init_instance(pi, nullptr);
+                }
             }
             return result.ptr();
         }
@@ -1045,6 +1057,12 @@ public:
         return *this;
     }
 
+    template <typename... Args, typename... Extra>
+    class_ &def(detail::initimpl::factory<Args...> &&init, const Extra&... extra) {
+        std::move(init).execute(*this, extra...);
+        return *this;
+    }
+
     template <typename Func> class_& def_buffer(Func &&func) {
         struct capture { Func func; };
         capture *ptr = new capture { std::forward<Func>(func) };
@@ -1225,11 +1243,15 @@ private:
     }
 
     /// Deallocates an instance; via holder, if constructed; otherwise via operator delete.
-    static void dealloc(const detail::value_and_holder &v_h) {
-        if (v_h.holder_constructed())
+    static void dealloc(detail::value_and_holder &v_h) {
+        if (v_h.holder_constructed()) {
             v_h.holder<holder_type>().~holder_type();
-        else
+            v_h.set_holder_constructed(false);
+        }
+        else {
             detail::call_operator_delete(v_h.value_ptr<type>(), v_h.type->type_size);
+        }
+        v_h.value_ptr() = nullptr;
     }
 
     static detail::function_record *get_function_record(handle h) {
@@ -1328,6 +1350,23 @@ private:
     handle m_parent;
 };
 
+/// Binds an existing constructor taking arguments Args...
+template <typename... Args> detail::init<Args...> init() { return detail::init<Args...>(); }
+/// Like `init<Args...>()`, but the instance is always constructed through the alias class (even
+/// when not inheriting on the Python side).
+template <typename... Args> detail::init_alias<Args...> init_alias() { return detail::init_alias<Args...>(); }
+
+/// Binds a factory function as a constructor
+template <typename Func, typename Ret = detail::initimpl::factory_t<Func>>
+Ret init(Func &&f) { return {std::forward<Func>(f)}; }
+
+/// Dual-argument factory function: the first function is called when no alias is needed, the second
+/// when an alias is needed (i.e. due to python-side inheritance).  Arguments must be identical.
+template <typename CFunc, typename AFunc, typename Ret = detail::initimpl::factory_t<CFunc, AFunc>>
+Ret init(CFunc &&c, AFunc &&a) {
+    return {std::forward<CFunc>(c), std::forward<AFunc>(a)};
+}
+
 NAMESPACE_BEGIN(detail)
 template <typename... Args> struct init {
     template <typename Class, typename... Extra, enable_if_t<!Class::has_alias, int> = 0>
@@ -1431,9 +1470,6 @@ struct iterator_state {
 
 NAMESPACE_END(detail)
 
-template <typename... Args> detail::init<Args...> init() { return detail::init<Args...>(); }
-template <typename... Args> detail::init_alias<Args...> init_alias() { return detail::init_alias<Args...>(); }
-
 /// Makes a python iterator from a first and past-the-end C++ InputIterator.
 template <return_value_policy Policy = return_value_policy::reference_internal,
           typename Iterator,

setup.py

@@ -15,6 +15,7 @@ else:
         'include/pybind11/detail/class.h',
         'include/pybind11/detail/common.h',
         'include/pybind11/detail/descr.h',
+        'include/pybind11/detail/init.h',
         'include/pybind11/detail/typeid.h'
         'include/pybind11/attr.h',
         'include/pybind11/buffer_info.h',

tests/CMakeLists.txt

@@ -39,6 +39,7 @@ set(PYBIND11_TEST_FILES
   test_enum.cpp
   test_eval.cpp
   test_exceptions.cpp
+  test_factory_constructors.cpp
   test_kwargs_and_defaults.cpp
   test_local_bindings.cpp
   test_methods_and_attributes.cpp

tests/conftest.py

@@ -211,6 +211,8 @@ def pytest_namespace():
         'requires_eigen_and_scipy': skipif(not have_eigen or not scipy,
                                            reason="eigen and/or scipy are not installed"),
         'unsupported_on_pypy': skipif(pypy, reason="unsupported on PyPy"),
+        'unsupported_on_py2': skipif(sys.version_info.major < 3,
+                                     reason="unsupported on Python 2.x"),
         'gc_collect': gc_collect
     }
 

tests/test_multiple_inheritance.py

@@ -71,9 +71,9 @@ def test_multiple_inheritance_python():
             MI2.__init__(self, i, j)
 
     class MI4(MI3, m.Base2):
-        def __init__(self, i, j, k):
-            MI3.__init__(self, j, k)
-            m.Base2.__init__(self, i)
+        def __init__(self, i, j):
+            MI3.__init__(self, i, j)
+            # m.Base2 is already initialized (via MI2)
 
     class MI5(m.Base2, B1, m.Base1):
         def __init__(self, i, j):
@@ -127,10 +127,10 @@ def test_multiple_inheritance_python():
     assert mi3.foo() == 5
     assert mi3.bar() == 6
 
-    mi4 = MI4(7, 8, 9)
+    mi4 = MI4(7, 8)
     assert mi4.v() == 1
-    assert mi4.foo() == 8
-    assert mi4.bar() == 7
+    assert mi4.foo() == 7
+    assert mi4.bar() == 8
 
     mi5 = MI5(10, 11)
     assert mi5.v() == 1

tests/test_virtual_functions.cpp

@@ -234,6 +234,7 @@ TEST_SUBMODULE(virtual_functions, m) {
 
     py::class_<A2, PyA2>(m, "A2")
         .def(py::init_alias<>())
+        .def(py::init([](int) { return new PyA2(); }))
         .def("f", &A2::f);
 
     m.def("call_f", [](A2 *a2) { a2->f(); });
@@ -444,4 +445,3 @@ void initialize_inherited_virtuals(py::module &m) {
         .def(py::init<>());
 
 };
-

tests/test_virtual_functions.py

@@ -128,11 +128,19 @@ def test_alias_delay_initialization2(capture):
         m.call_f(a2)
         del a2
         pytest.gc_collect()
+        a3 = m.A2(1)
+        m.call_f(a3)
+        del a3
+        pytest.gc_collect()
     assert capture == """
         PyA2.PyA2()
         PyA2.f()
         A2.f()
         PyA2.~PyA2()
+        PyA2.PyA2()
+        PyA2.f()
+        A2.f()
+        PyA2.~PyA2()
     """
 
     # Python subclass version