Automate generation of reference docs with doxygen and breathe (#598)

* Make 'any' the default markup role for Sphinx docs

* Automate generation of reference docs with doxygen and breathe

* Improve reference docs coverage

.readthedocs.yml

+conda:
+  file: docs/environment.yml

.travis.yml

@@ -51,7 +51,10 @@ matrix:
     env: DOCS STYLE LINT
     install:
     - pip install --upgrade sphinx sphinx_rtd_theme flake8 pep8-naming
-    - pip install docutils==0.12
+    - |
+      curl -fsSL ftp://ftp.stack.nl/pub/users/dimitri/doxygen-1.8.12.linux.bin.tar.gz | tar xz
+      export PATH="$PWD/doxygen-1.8.12/bin:$PATH"
+      pip install https://github.com/michaeljones/breathe/archive/master.zip
     script:
     - make -C docs html SPHINX_OPTIONS=-W
     - tools/check-style.sh

docs/Doxyfile

+PROJECT_NAME           = pybind11
+INPUT                  = ../include/pybind11/
+
+GENERATE_HTML          = NO
+GENERATE_LATEX         = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = .build/doxygenxml
+XML_PROGRAMLISTING     = YES
+
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = YES
+EXPAND_AS_DEFINED      = PYBIND11_RUNTIME_EXCEPTION
+
+ALIASES                = "rst=\verbatim embed:rst"
+ALIASES               += "endrst=\endverbatim"
+
+QUIET                  = YES
+WARNINGS               = YES
+WARN_IF_UNDOCUMENTED   = NO

docs/advanced/cast/chrono.rst

@@ -4,8 +4,8 @@ Chrono
 When including the additional header file :file:`pybind11/chrono.h` conversions
 from C++11 chrono datatypes to python datetime objects are automatically enabled.
 This header also enables conversions of python floats (often from sources such
-as `time.monotonic()`, `time.perf_counter()` and `time.process_time()`) into
-durations.
+as ``time.monotonic()``, ``time.perf_counter()`` and ``time.process_time()``)
+into durations.
 
 An overview of clocks in C++11
 ------------------------------

docs/advanced/functions.rst

@@ -14,7 +14,7 @@ lifetime of objects managed by them. This can lead to issues when creating
 bindings for functions that return a non-trivial type. Just by looking at the
 type information, it is not clear whether Python should take charge of the
 returned value and eventually free its resources, or if this is handled on the
-C++ side. For this reason, pybind11 provides a several `return value policy`
+C++ side. For this reason, pybind11 provides a several *return value policy*
 annotations that can be passed to the :func:`module::def` and
 :func:`class_::def` functions. The default policy is
 :enum:`return_value_policy::automatic`.
@@ -24,11 +24,11 @@ Just to illustrate what can go wrong, consider the following simple example:
 
 .. code-block:: cpp
 
-    /* Function declaration */ 
+    /* Function declaration */
     Data *get_data() { return _data; /* (pointer to a static data structure) */ }
     ...
 
-    /* Binding code */ 
+    /* Binding code */
     m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python
 
 What's going on here? When ``get_data()`` is called from Python, the return
@@ -44,7 +44,7 @@ silent data corruption.
 
 In the above example, the policy :enum:`return_value_policy::reference` should have
 been specified so that the global data instance is only *referenced* without any
-implied transfer of ownership, i.e.: 
+implied transfer of ownership, i.e.:
 
 .. code-block:: cpp
 
@@ -158,9 +158,9 @@ targeted arguments can be passed through the :class:`cpp_function` constructor:
 Additional call policies
 ========================
 
-In addition to the above return value policies, further `call policies` can be
-specified to indicate dependencies between parameters. In general, call policies 
-are required when the C++ object is any kind of container and another object is being 
+In addition to the above return value policies, further *call policies* can be
+specified to indicate dependencies between parameters. In general, call policies
+are required when the C++ object is any kind of container and another object is being
 added to the container.
 
 There is currently just

docs/advanced/pycpp/object.rst

@@ -33,6 +33,8 @@ The reverse direction uses the following syntax:
 
 When conversion fails, both directions throw the exception :class:`cast_error`.
 
+.. _calling_python_functions:
+
 Calling Python functions
 ========================
 

docs/classes.rst

@@ -38,7 +38,7 @@ The binding code for ``Pet`` looks as follows:
         return m.ptr();
     }
 
-:class:`class_` creates bindings for a C++ `class` or `struct`-style data
+:class:`class_` creates bindings for a C++ *class* or *struct*-style data
 structure. :func:`init` is a convenience function that takes the types of a
 constructor's parameters as template arguments and wraps the corresponding
 constructor (see the :ref:`custom_constructors` section for details). An

docs/conf.py

@@ -16,6 +16,7 @@
 import sys
 import os
 import shlex
+import subprocess
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
@@ -30,7 +31,11 @@ import shlex
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ['breathe']
+
+breathe_projects = {'pybind11': '.build/doxygenxml/'}
+breathe_default_project = 'pybind11'
+breathe_domain_by_extension = {'h': 'cpp'}
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['.templates']
@@ -79,7 +84,7 @@ exclude_patterns = ['.build', 'release.rst']
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
-#default_role = None
+default_role = 'any'
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 #add_function_parentheses = True
@@ -306,3 +311,22 @@ texinfo_documents = [
 
 primary_domain = 'cpp'
 highlight_language = 'cpp'
+
+
+def generate_doxygen_xml(app):
+    build_dir = '.build'
+    if not os.path.exists(build_dir):
+        os.mkdir(build_dir)
+
+    try:
+        subprocess.call(['doxygen', '--version'])
+        retcode = subprocess.call(['doxygen'])
+        if retcode < 0:
+            sys.stderr.write("doxygen error code: {}\n".format(-retcode))
+    except OSError as e:
+        sys.stderr.write("doxygen execution failed: {}\n".format(e))
+
+
+def setup(app):
+    """Add hook for building doxygen xml when needed"""
+    app.connect("builder-inited", generate_doxygen_xml)

docs/environment.yml

+name: rtd
+channels:
+- dean0x7d
+- defaults
+dependencies:
+- doxygen
+- pip
+- pip:
+  - https://github.com/michaeljones/breathe/archive/master.zip

docs/reference.rst

@@ -12,236 +12,69 @@ Reference
 Macros
 ======
 
-.. function:: PYBIND11_PLUGIN(const char *name)
-
-    This macro creates the entry point that will be invoked when the Python
-    interpreter imports a plugin library. Please create a
-    :class:`module` in the function body and return the pointer to its
-    underlying Python object at the end.
-
-    .. code-block:: cpp
-
-        PYBIND11_PLUGIN(example) {
-            pybind11::module m("example", "pybind11 example plugin");
-            /// Set up bindings here
-            return m.ptr();
-        }
+.. doxygendefine:: PYBIND11_PLUGIN
 
 .. _core_types:
 
 Convenience classes for arbitrary Python types
 ==============================================
 
-Without reference counting
---------------------------
-
-.. class:: handle
-
-    The :class:`handle` class is a thin wrapper around an arbitrary Python
-    object (i.e. a ``PyObject *`` in Python's C API). It does not perform any
-    automatic reference counting and merely provides a basic C++ interface to
-    various Python API functions.
-
-.. seealso::
-
-    The :class:`object` class inherits from :class:`handle` and adds automatic
-    reference counting features.
-
-.. function:: handle::handle()
-
-    The default constructor creates a handle with a ``nullptr``-valued pointer.
-
-.. function:: handle::handle(const handle&)
-
-    Copy constructor
-
-.. function:: handle::handle(PyObject *)
-
-    Creates a :class:`handle` from the given raw Python object pointer.
-
-.. function:: PyObject * handle::ptr() const
-
-    Return the ``PyObject *`` underlying a :class:`handle`.
-
-.. function:: const handle& handle::inc_ref() const
-
-    Manually increase the reference count of the Python object. Usually, it is
-    preferable to use the :class:`object` class which derives from
-    :class:`handle` and calls this function automatically. Returns a reference
-    to itself.
-
-.. function:: const handle& handle::dec_ref() const
-
-    Manually decrease the reference count of the Python object. Usually, it is
-    preferable to use the :class:`object` class which derives from
-    :class:`handle` and calls this function automatically. Returns a reference
-    to itself.
-
-.. function:: void handle::ref_count() const
-
-    Return the object's current reference count
-
-.. function:: handle handle::get_type() const
-
-    Return a handle to the Python type object underlying the instance
-
-.. function detail::accessor handle::operator[](handle key) const
-
-    Return an internal functor to invoke the object's sequence protocol.
-    Casting the returned ``detail::accessor`` instance to a :class:`handle` or
-    :class:`object` subclass causes a corresponding call to ``__getitem__``.
-    Assigning a :class:`handle` or :class:`object` subclass causes a call to
-    ``__setitem__``.
-
-.. function detail::accessor handle::operator[](const char *key) const
-
-    See the above function (the only difference is that they key is provided as
-    a string literal).
-
-.. function detail::accessor handle::attr(handle key) const
-
-    Return an internal functor to access the object's attributes.
-    Casting the returned ``detail::accessor`` instance to a :class:`handle` or
-    :class:`object` subclass causes a corresponding call to ``__getattr``.
-    Assigning a :class:`handle` or :class:`object` subclass causes a call to
-    ``__setattr``.
-
-.. function detail::accessor handle::attr(const char *key) const
-
-    See the above function (the only difference is that they key is provided as
-    a string literal).
-
-.. function operator handle::bool() const
-
-    Return ``true`` when the :class:`handle` wraps a valid Python object.
-
-.. function str handle::str() const
-
-    Return a string representation of the object. This is analogous to
-    the ``str()`` function in Python.
-
-.. function:: template <typename T> T handle::cast() const
-
-    Attempt to cast the Python object into the given C++ type. A
-    :class:`cast_error` will be throw upon failure.
+Common member functions
+-----------------------
 
-.. function:: template <typename ... Args> object handle::call(Args&&... args) const
+.. doxygenclass:: object_api
+    :members:
 
-    Assuming the Python object is a function or implements the ``__call__``
-    protocol, ``call()`` invokes the underlying function, passing an arbitrary
-    set of parameters. The result is returned as a :class:`object` and may need
-    to be converted back into a Python object using :func:`handle::cast`.
+Without reference counting
+--------------------------
 
-    When some of the arguments cannot be converted to Python objects, the
-    function will throw a :class:`cast_error` exception. When the Python
-    function call fails, a :class:`error_already_set` exception is thrown.
+.. doxygenclass:: handle
+    :members:
 
 With reference counting
 -----------------------
 
-.. class:: object : public handle
-
-    Like :class:`handle`, the object class is a thin wrapper around an
-    arbitrary Python object (i.e. a ``PyObject *`` in Python's C API). In
-    contrast to :class:`handle`, it optionally increases the object's reference
-    count upon construction, and it *always* decreases the reference count when
-    the :class:`object` instance goes out of scope and is destructed. When
-    using :class:`object` instances consistently, it is much easier to get
-    reference counting right at the first attempt.
-
-.. function:: object::object(const object &o)
-
-    Copy constructor; always increases the reference count
-
-.. function:: object::object(const handle &h, bool borrowed)
-
-    Creates a :class:`object` from the given :class:`handle`. The reference
-    count is only increased if the ``borrowed`` parameter is set to ``true``.
-
-.. function:: object::object(PyObject *ptr, bool borrowed)
-
-    Creates a :class:`object` from the given raw Python object pointer. The
-    reference  count is only increased if the ``borrowed`` parameter is set to
-    ``true``.
-
-.. function:: object::object(object &&other)
-
-    Move constructor; steals the object from ``other`` and preserves its
-    reference count.
-
-.. function:: handle object::release()
+.. doxygenclass:: object
+    :members:
 
-    Resets the internal pointer to ``nullptr`` without without decreasing the
-    object's reference count. The function returns a raw handle to the original
-    Python object.
+.. doxygenfunction:: reinterpret_borrow
 
-.. function:: object::~object()
-
-    Destructor, which automatically calls :func:`handle::dec_ref()`.
+.. doxygenfunction:: reinterpret_steal
 
 Convenience classes for specific Python types
 =============================================
 
+.. doxygenclass:: module
+    :members:
 
-.. class:: module : public object
-
-.. function:: module::module(const char *name, const char *doc = nullptr)
-
-    Create a new top-level Python module with the given name and docstring
-
-.. function:: module module::def_submodule(const char *name, const char *doc = nullptr)
-
-    Create and return a new Python submodule with the given name and docstring.
-    This also works recursively, i.e.
-
-    .. code-block:: cpp
-
-        pybind11::module m("example", "pybind11 example plugin");
-        pybind11::module m2 = m.def_submodule("sub", "A submodule of 'example'");
-        pybind11::module m3 = m2.def_submodule("subsub", "A submodule of 'example.sub'");
-
-.. cpp:function:: template <typename Func, typename ... Extra> module& module::def(const char *name, Func && f, Extra && ... extra)
-
-    Create Python binding for a new function within the module scope. ``Func``
-    can be a plain C++ function, a function pointer, or a lambda function. For
-    details on the ``Extra&& ... extra`` argument, see section :ref:`extras`.
+.. doxygengroup:: pytypes
+    :members:
 
 .. _extras:
 
-Passing extra arguments to the def function
-===========================================
-
-.. class:: arg
-
-.. function:: arg::arg(const char *name)
-
-.. function:: template <typename T> arg_v arg::operator=(T &&value)
-
-.. class:: arg_v : public arg
-
-    Represents a named argument with a default value
-
-.. class:: sibling
-
-    Used to specify a handle to an existing sibling function; used internally
-    to implement function overloading in :func:`module::def` and
-    :func:`class_::def`.
-
-.. function:: sibling::sibling(handle handle)
+Passing extra arguments to ``def`` or ``class_``
+================================================
 
-.. class doc
+.. doxygengroup:: annotations
+    :members:
 
-    This is class is internally used by pybind11.
+Python build-in functions
+=========================
 
-.. function:: doc::doc(const char *value)
+.. doxygengroup:: python_builtins
+    :members:
 
-    Create a new docstring with the specified value
+Exceptions
+==========
 
-.. class name
+.. doxygenclass:: error_already_set
+    :members:
 
-    This is class is internally used by pybind11.
+.. doxygenclass:: builtin_exception
+    :members:
 
-.. function:: name::name(const char *value)
 
-    Used to specify the function name
+Literals
+========
 
+.. doxygennamespace:: literals

include/pybind11/attr.h

@@ -14,6 +14,9 @@
 
 NAMESPACE_BEGIN(pybind11)
 
+/// \addtogroup annotations
+/// @{
+
 /// Annotation for methods
 struct is_method { handle class_; is_method(const handle &c) : class_(c) { } };
 
@@ -56,6 +59,8 @@ struct metaclass { };
 /// Annotation to mark enums as an arithmetic type
 struct arithmetic { };
 
+/// @} annotations
+
 NAMESPACE_BEGIN(detail)
 /* Forward declarations */
 enum op_id : int;

include/pybind11/cast.h

@@ -1177,14 +1177,18 @@ template <return_value_policy policy = return_value_policy::automatic_reference,
     return result;
 }
 
+/// \ingroup annotations
 /// Annotation for keyword arguments
 struct arg {
+    /// Set the name of the argument
     constexpr explicit arg(const char *name) : name(name) { }
+    /// Assign a value to this argument
     template <typename T> arg_v operator=(T &&value) const;
 
     const char *name;
 };
 
+/// \ingroup annotations
 /// Annotation for keyword arguments with values
 struct arg_v : arg {
     template <typename T>
@@ -1213,7 +1217,9 @@ arg_v arg::operator=(T &&value) const { return {name, std::forward<T>(value)}; }
 template <typename /*unused*/> using arg_t = arg_v;
 
 inline namespace literals {
-/// String literal version of arg
+/** \rst
+    String literal version of `arg`
+ \endrst */
 constexpr arg operator"" _a(const char *name, size_t) { return arg(name); }
 }
 

include/pybind11/common.h

@@ -159,6 +159,19 @@ extern "C" {
 #define PYBIND11_INTERNALS_ID "__pybind11_" \
     PYBIND11_TOSTRING(PYBIND11_VERSION_MAJOR) "_" PYBIND11_TOSTRING(PYBIND11_VERSION_MINOR) "__"
 
+/** \rst
+    This macro creates the entry point that will be invoked when the Python interpreter
+    imports a plugin library. Please create a `module` in the function body and return
+    the pointer to its underlying Python object at the end.
+
+    .. code-block:: cpp
+
+        PYBIND11_PLUGIN(example) {
+            pybind11::module m("example", "pybind11 example plugin");
+            /// Set up bindings here
+            return m.ptr();
+        }
+\endrst */
 #define PYBIND11_PLUGIN(name)                                                  \
     static PyObject *pybind11_init();                                          \
     PYBIND11_PLUGIN_IMPL(name) {                                               \
@@ -388,7 +401,7 @@ template <bool B> using bool_constant = std::integral_constant<bool, B>;
 template <class T> using negation = bool_constant<!T::value>;
 #endif
 
-/// Compile-time all/any/none of that check the ::value of all template types
+/// Compile-time all/any/none of that check the boolean value of all template types
 #ifdef PYBIND11_CPP17
 template <class... Ts> using all_of = bool_constant<(Ts::value && ...)>;
 template <class... Ts> using any_of = bool_constant<(Ts::value || ...)>;
@@ -532,7 +545,8 @@ private:
 class builtin_exception : public std::runtime_error {
 public:
     using std::runtime_error::runtime_error;
-    virtual void set_error() const = 0; /// Set the error using the Python C API
+    /// Set the error using the Python C API
+    virtual void set_error() const = 0;
 };
 
 #define PYBIND11_RUNTIME_EXCEPTION(name, type) \

include/pybind11/pybind11.h

@@ -544,6 +544,7 @@ class module : public object {
 public:
     PYBIND11_OBJECT_DEFAULT(module, object, PyModule_Check)
 
+    /// Create a new top-level Python module with the given name and docstring
     explicit module(const char *name, const char *doc = nullptr) {
         if (!options::show_user_defined_docstrings()) doc = nullptr;
 #if PY_MAJOR_VERSION >= 3
@@ -562,6 +563,11 @@ public:
         inc_ref();
     }
 
+    /** \rst
+        Create Python binding for a new function within the module scope. ``Func``
+        can be a plain C++ function, a function pointer, or a lambda function. For
+        details on the ``Extra&& ... extra`` argument, see section :ref:`extras`.
+    \endrst */
     template <typename Func, typename... Extra>
     module &def(const char *name_, Func &&f, const Extra& ... extra) {
         cpp_function func(std::forward<Func>(f), name(name_), scope(*this),
@@ -572,6 +578,16 @@ public:
         return *this;
     }
 
+    /** \rst
+        Create and return a new Python submodule with the given name and docstring.
+        This also works recursively, i.e.
+
+        .. code-block:: cpp
+
+            py::module m("example", "pybind11 example plugin");
+            py::module m2 = m.def_submodule("sub", "A submodule of 'example'");
+            py::module m3 = m2.def_submodule("subsub", "A submodule of 'example.sub'");
+    \endrst */
     module def_submodule(const char *name, const char *doc = nullptr) {
         std::string full_name = std::string(PyModule_GetName(m_ptr))
             + std::string(".") + std::string(name);
@@ -582,6 +598,7 @@ public:
         return result;
     }
 
+    /// Import and return a module or throws `error_already_set`.
     static module import(const char *name) {
         PyObject *obj = PyImport_ImportModule(name);
         if (!obj)