Commit 28f46a10 by Ken Oslund Committed by Copybara-Service

Internal change

PiperOrigin-RevId: 417834077
parent 8e9b6c5a
......@@ -17,13 +17,6 @@ in the .cc file with your bindings:
#include "pybind11_abseil/absl_casters.h"
```
Support for non-const `absl::Span` for numeric types is also available by
including a separated header file:
```
#include "pybind11_abseil/absl_numpy_span_caster.h"
```
## Installation
You will need to depend on `pybind11`, `pybind11_bazel`(see
......@@ -96,58 +89,62 @@ less granular C++ types, and time zone information is ignored.
## absl::Span
For non-const `absl::Span` and conversion from `numpy` arrays, see
[non-const absl::Span](#non-const-abslspan) later.
When `absl::Span<const T>` (i.e. the `const` version) is considered, there is
full support to mapping into Python sequences.
Currently, this will always result in the list being copied, so you lose the
efficiency gains of spans in native C++, but you still get the API versatility.
The value type in the span can be any type that pybind knows about. However, it
must be immutable (i.e., `absl::Span<const ValueType>`). Theoretically mutable
ValueTypes could be supported, but with some subtle limitations, and this is
not needed right now, so the implementation has been deferred.
The `convert` and `return_value_policy` parameters will apply to the *elements*.
The list containing those elements will aways be converted/copied.
### non-const absl::Span
Support for non-cost `absl::Span`, for numeric types only, is provided for
`numpy` arrays. Support is only for output function parameters and not for
returned value. The rationale behind this decision is that, if a `absl::Span`
were to be returned, the C++ object would have needed to outlive the mapped
Python object. Given the complexity of memory management across languages, we
did not add support of returned `absl::Span`.
That is the following is supported:
```
void Foo(absl::Span<double> some_span);
```
while the following is not (it will generate a compile error):
```
absl::Span<double> Bar();
```
Note: It is possible to use the non-const `absl::Span` bindings to wrap a
function with `absl::Span<const T>` argument if you are using `numpy` arrays
and you do not want a copy to be performed. This can be done by defining a
lambda function in the `pybind11` wrapper, as in the following example. See
b/155596364 for more details.
```
void MyConstSpanFunction(absl::Span<const double> a_span);
...
PYBIND11_MODULE(bindings, m) {
m.def(
"wrap_span",
[](absl::Span<double> span) {
MyConstSpanFunction(span);
});
}
```
### Loading
Some python types can be loaded (Python->C++) without copying or converting the
list, while some require copying/ converting the list. The non-converting load
methods will be tried first, and, if the span elements are const, the converting
load methods will be tried next.
Arguments cast to a span with *non-const* elements can never be copied/converted.
To prevent an argument cast to a span with *const* elements from being copied or
converted, mark it as `noconvert()` (see go/pybind11-non-converting-arguments).
The following python types can be loaded *without* copying or converting:
- Numpy array (or anything else that supports [buffer protocol](
https://docs.python.org/3/c-api/buffer.htm)) => `Span<{const or non-const} T>`
if *all* of the following conditions are satisfied:
- The buffer is 1-D.
- T is a numeric type.
- The array dtype matches T exactly.
- If T is not const, the buffer allows writing.
- The stride does not indicate to skip elements or go in reverse order.
- [Opaque](go/pybind11-opaque-types) `std::vector<T>` => `Span<{const or non-const} T>`.
- T can be any type, including converted or pointer types, but must
match exactly between C++ and python.
- Opaque vectors are *not* currently compatible with the smart holder.
The following python types must be copied/converted to be loaded:
- Python sequence of elements that require conversion (numbers, strings,
datetimes, etc) => `Span<const T>`.
- The elements will be copied/ converted, so that conversion must be legal.
- T *cannot* be a pointer.
- Python sequence of elements that do *not* require conversion (ie, classes
wrapped with py::class_) => `Span<const T>` (elements *will* be copied) or
`Span<{const or non-const} T* const>` (elements will *not* be copied).
Specifically, this conversion will *fail* if any of the following are true:
- `noconvert()` was specified (see go/pybind11-non-converting-arguments).
- The element conversion is not allowed (eg, floating point to integer).
- The sequence is being loaded into a `Span<{non-const} T>` or
`Span<{const or non-const} T* {non-const}>`.
- The elements require conversion *and* the sequence is being loaded into a
`Span<T*>` (regardless of any `const`s; the element caster which owns the
converted value would be destroyed before `load` is complete, resulting in
dangling references).
- The span is nested (ie, `absl::Span<absl::Span<T>>`, regardless of any `const`s).
Note: These failure conditions only apply to *converted* python types.
### Casting
Spans are cast (C++->Python) with the standard list caster, which always
converts the list. This could be changed in the future (eg, using [buffer protocol](
https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#buffer-protocol))
but generally using spans as return values is not recommended.
## absl::string_view
......
......@@ -12,6 +12,7 @@ pybind_library(
name = "absl_casters",
hdrs = ["absl_casters.h"],
deps = [
"@com_google_absl//absl/cleanup",
"@com_google_absl//absl/container:flat_hash_map",
"@com_google_absl//absl/container:flat_hash_set",
"@com_google_absl//absl/strings",
......@@ -24,7 +25,8 @@ pybind_library(
pybind_library(
name = "absl_numpy_span_caster",
hdrs = ["absl_numpy_span_caster.h"],
deps = ["@com_google_absl//absl/types:span"],
deprecation = "Please use //pybind11_abseil:absl_casters.",
deps = [":absl_casters"],
)
cc_library(
......
......@@ -13,7 +13,8 @@
// - absl::Duration- converted to/from python datetime.timedelta
// - absl::CivilTime- converted to/from python datetime.datetime and from date.
// - absl::Time- converted to/from python datetime.datetime and from date.
// - absl::Span- const value types only.
// - absl::Span- converted to python sequences and from python buffers,
// opaque std::vectors and/or sequences.
// - absl::string_view
// - absl::optional- converts absl::nullopt to/from python None, otherwise
// converts the contained value.
......@@ -33,13 +34,11 @@
#include <cmath>
#include <cstdint>
#include <exception>
#include <memory>
#include <stdexcept>
#include <tuple>
#include <type_traits>
#include <typeinfo>
#include <vector>
#include "absl/cleanup/cleanup.h"
#include "absl/container/flat_hash_map.h"
#include "absl/container/flat_hash_set.h"
#include "absl/strings/string_view.h"
......@@ -228,56 +227,70 @@ template <>
struct type_caster<absl::CivilYear>
: public absl_civil_time_caster<absl::CivilYear> {};
// Convert between absl::Span and python sequence types.
//
// TODO(kenoslund): It may be possible to avoid copies in some cases:
// Python to C++: Numpy arrays are contiguous, so we could overlay without copy.
// C++ to Python: Python buffer.
// https://pybind11.readthedocs.io/en/stable/advanced/pycpp/numpy.html#buffer-protocol
// Returns {true, a span referencing the data contained by src} without copying
// or converting the data if possible. Otherwise returns {false, an empty span}.
template <typename T, typename std::enable_if<std::is_arithmetic<T>::value,
bool>::type = true>
std::tuple<bool, absl::Span<T>> LoadSpanFromBuffer(handle src) {
Py_buffer view;
int flags = PyBUF_STRIDES | PyBUF_FORMAT;
if (!std::is_const<T>::value) flags |= PyBUF_WRITABLE;
if (PyObject_GetBuffer(src.ptr(), &view, flags) == 0) {
auto cleanup = absl::MakeCleanup([&view] { PyBuffer_Release(&view); });
if (view.ndim == 1 && view.strides[0] == sizeof(T) &&
view.format[0] == format_descriptor<T>::c) {
return {true, absl::MakeSpan(static_cast<T*>(view.buf), view.shape[0])};
}
} else {
// Clear the buffer error (failure is reported in the return value).
PyErr_Clear();
}
return {false, absl::Span<T>()};
}
// If T is not a numeric type, the buffer interface cannot be used.
template <typename T, typename std::enable_if<!std::is_arithmetic<T>::value,
bool>::type = true>
constexpr std::tuple<bool, absl::Span<T>> LoadSpanFromBuffer(handle src) {
return {false, absl::Span<T>()};
}
// Helper to determine whether T is a span.
template <typename T>
struct is_absl_span : std::false_type {};
template <typename T>
struct type_caster<absl::Span<const T>> {
// NOTE: In general, it's *unsafe* to have a pointer type for T. It's
// ok for unmutable sequences of pybind11-wrapped objects.
//
// In general if `T` is a pointer or other non-owning reference type, the
// resultant `Span` may immediately contain dangling references to temporary
// objects owned by temporary `type_caster<T>` objects, and cannot be used.
// In particular, passing a list that is mutated before the C++ call
// returns, or a lazy sequence is unsafe. Use `std::vector<pybind11::object>`
// instead and convert the element in C++.
//
// If the Python calling code ensures that the sequence of elements used to
// initialize the Span are kept alive until the pybind11-bound function
// returns, then the Span can safely be used, e.g. using a tuple or a
// not-mutated list of pybind11-wrapped objects is ok.
//
// The static_assert below prevents spans of pointers to converted types from
// being used, since the caster (which owns the converted value) would be
// destroyed before the function that uses it executes, resulting in a
// dangling reference.
static_assert(
!std::is_pointer<T>::value ||
std::is_base_of<type_caster_base<pybind11::detail::intrinsic_t<T>>,
pybind11::detail::make_caster<T>>::value,
"Spans of pointers are not supported for converted types.");
type_caster() : vector_converter_(), value_(get_vector()) {}
// Copy and Move constructors need to ensure the span points to the copied
// or moved vector, not the original one.
type_caster(const type_caster<absl::Span<const T>>& other)
: vector_converter_(other.vector_converter_), value_(get_vector()) {}
type_caster(type_caster<absl::Span<const T>>&& other)
: vector_converter_(std::move(other.vector_converter_)),
value_(get_vector()) {}
type_caster& operator=(const type_caster<absl::Span<const T>>& other) {
vector_converter_ = other.vector_converter_;
value_ = get_vector();
struct is_absl_span<absl::Span<T>> : std::true_type {};
// Convert between absl::Span and sequence types.
// See http://g3doc/pybind11_abseil/README.md#abslspan
template <typename T>
struct type_caster<absl::Span<T>> {
public:
// The type referenced by the span, with const removed.
using value_type = typename std::remove_cv<T>::type;
static_assert(!is_absl_span<value_type>::value,
"Nested absl spans are not supported.");
type_caster() = default;
// Copy and Move operations must ensure the span points to the copied or
// moved vector (if any), not the original one. Allows implicit conversions.
template <typename U>
type_caster(const type_caster<absl::Span<U>>& other) {
*this = other;
}
template <typename U>
type_caster(type_caster<absl::Span<U>>&& other) {
*this = std::move(other);
}
template <typename U>
type_caster& operator=(const type_caster<absl::Span<U>>& other) {
list_caster_ = other.list_caster_;
value_ = list_caster_ ? get_value(*list_caster_) : other.value_;
return *this;
}
type_caster& operator=(type_caster<absl::Span<const T>>&& other) {
vector_converter_ = std::move(other.vector_converter_);
value_ = get_vector();
template <typename U>
type_caster& operator=(type_caster<absl::Span<U>>&& other) {
list_caster_ = std::move(other.list_caster_);
value_ = list_caster_ ? get_value(*list_caster_) : other.value_;
return *this;
}
......@@ -287,30 +300,58 @@ struct type_caster<absl::Span<const T>> {
// no advantage to moving and 2) the span cannot exist without the caster,
// so moving leaves an implicit dependency (while a reference or pointer
// make that dependency explicit).
operator absl::Span<const T>*() { return &value_; }
operator absl::Span<const T>&() { return value_; }
operator absl::Span<T>*() { return &value_; }
operator absl::Span<T>&() { return value_; }
template <typename T_>
using cast_op_type = cast_op_type<T_>;
bool load(handle src, bool convert) {
if (!vector_converter_.load(src, convert)) return false;
// std::vector implicitly converted to absl::Span.
value_ = get_vector();
return true;
// Attempt to reference a buffer, including np.ndarray and array.arrays.
bool loaded;
std::tie(loaded, value_) = LoadSpanFromBuffer<T>(src);
if (loaded) return true;
// Attempt to unwrap an opaque std::vector.
type_caster_base<std::vector<value_type>> caster;
if (caster.load(src, false)) {
value_ = get_value(caster);
return true;
}
// Attempt to convert a native sequence. If the is_base_of_v check passes,
// the elements do not require converting and pointers do not reference a
// temporary object owned by the element caster. Pointers to converted
// types are not allowed because they would result a dangling reference
// when the element caster is destroyed. TODO(b/169068487): improve this.
if (convert && std::is_const<T>::value &&
(!std::is_pointer<T>::value ||
std::is_base_of<type_caster_generic, make_caster<T>>::value)) {
list_caster_.emplace();
if (list_caster_->load(src, convert)) {
value_ = get_value(*list_caster_);
return true;
} else {
list_caster_.reset();
}
}
return false; // Python type cannot be loaded into a span.
}
template <typename CType>
static handle cast(CType&& src, return_value_policy policy, handle parent) {
return VectorConverter::cast(src, policy, parent);
return ListCaster::cast(src, policy, parent);
}
private:
std::vector<T>& get_vector() {
return static_cast<std::vector<T>&>(vector_converter_);
template <typename Caster>
absl::Span<T> get_value(Caster& caster) {
return absl::MakeSpan(static_cast<std::vector<value_type>&>(caster));
}
using VectorConverter = make_caster<std::vector<T>>;
VectorConverter vector_converter_;
absl::Span<const T> value_;
using ListCaster = list_caster<std::vector<value_type>, value_type>;
absl::optional<ListCaster> list_caster_;
absl::Span<T> value_;
};
// Convert between absl::flat_hash_map and python dict.
......
#ifndef PYBIND11_ABSEIL_ABSL_NUMPY_SPAN_CASTER_H_
#define PYBIND11_ABSEIL_ABSL_NUMPY_SPAN_CASTER_H_
#include <type_traits>
#include "absl/types/span.h"
#include "pybind11/cast.h"
#include "pybind11/numpy.h"
// Support for conversion from NumPy array to template specializations of
// absl::Span. Note that no data is copied, the `Span`ned memory is owned by the
// NumPy array.
//
// Only one direction of conversion is provided: a Span argument can be exposed
// to Python as a NumPy array argument. No conversion is provided for a Span
// return value.
//
// Lifetime management will become complicated if a `Span` returned by a C++
// object is converted into a NumPy array which outlives the C++ object. Such
// memory management concerns are normal for C++ but undesirable to introduce to
// Python.
//
// A `Span` argument can be used to expose an output argument to Python so that
// Python is wholly responsible for memory management of the output object.
// Using an output argument rather than a return value means that memory can be
// reused.
//
// C++:
// Simulation::Simulation(int buffer_size);
// void Simulation::RenderFrame(int frame_index, Span<uint8> buffer);
//
// Python:
// buffer = np.zeroes(1024*768, dtype='uint8')
// simulation = Simulation(1024*768)
// simulation.renderFrame(0, buffer)
// # RGB data can now be read from the buffer.
namespace pybind11::detail {
// Conversion of non-const Span.
// Note that there are two template specialisations for `absl::Span`: the one in
// this file and the one for `const T` in `absl_caster.h`.
// The use of `std::enable_if_t`, and in particular of the `std::is_const_v`
// condition check, is really important and prevents unpleasant ODR (one
// definition rule) violations when linking together different object files with
// different includes.
// TODO(b/155596364) merge this implementation with the absl::Span<const T>
// caster.
template <typename T>
class type_caster<absl::Span<T>,
// C++11 compatibility.
typename std::enable_if<std::is_arithmetic<T>::value &&
!std::is_const<T>::value>::type> {
public:
// Instead of using the macro PYBIND11_TYPE_CASTER we explicitly define value
// and functions. This is because, the macro has default implementations for
// move constructors and cast method.
// Note: we also disable the linter on these methods and variables.
static constexpr auto name = // NOLINT
_("Span[") + make_caster<T>::name + _("]");
// We do not allow moving because 1) spans are super lightweight, so there is
// no advantage to moving and 2) the span cannot exist without the caster,
// so moving leaves an implicit dependency (while a reference or pointer
// make that dependency explicit).
operator absl::Span<T>*() { // NOLINT(google-explicit-constructor)
return &value_;
}
operator absl::Span<T>&() { // NOLINT(google-explicit-constructor)
return value_;
}
template <typename T_>
using cast_op_type = cast_op_type<T_>;
// Conversion Python->C++: convert a NumPy array into a Span.
bool load(handle src, bool /* convert */) {
// Extract PyObject from handle.
if (!pybind11::isinstance<pybind11::array_t<T>>(src)) {
return false;
}
auto array = src.cast<pybind11::array_t<T>>();
if (!array || array.ndim() != 1 || array.strides()[0] != sizeof(T) ||
!array.writeable()) {
return false;
}
value_ = absl::Span<T>(static_cast<T*>(array.mutable_data()), array.size());
return true;
}
protected:
absl::Span<T> value_;
};
} // namespace pybind11::detail
#include "pybind11_abseil/absl_casters.h"
#endif // PYBIND11_ABSEIL_ABSL_NUMPY_SPAN_CASTER_H_
......@@ -10,7 +10,6 @@ pybind_extension(
srcs = ["absl_example.cc"],
deps = [
"//pybind11_abseil:absl_casters",
"//pybind11_abseil:absl_numpy_span_caster",
"@com_google_absl//absl/container:flat_hash_set",
"@com_google_absl//absl/strings",
"@com_google_absl//absl/time",
......
......@@ -4,7 +4,10 @@
// BSD-style license that can be found in the LICENSE file.
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/stl_bind.h>
#include <cstddef>
#include <vector>
#include "absl/container/flat_hash_set.h"
......@@ -15,7 +18,6 @@
#include "absl/types/optional.h"
#include "absl/types/span.h"
#include "pybind11_abseil/absl_casters.h"
#include "pybind11_abseil/absl_numpy_span_caster.h"
namespace pybind11 {
namespace test {
......@@ -35,21 +37,6 @@ bool CheckDatetime(const absl::Time& datetime, double secs) {
return datetime == MakeTime(secs);
}
bool CheckSpan(absl::Span<const int> span, const std::vector<int>& values) {
if (span.size() != values.size()) return false;
for (int i = 0; i < span.size(); ++i) {
if (span[i] != values[i]) return false;
}
return true;
}
bool CheckSpanCasterCopy(const handle& span, const std::vector<int>& values) {
pybind11::detail::make_caster<absl::Span<const int>> caster;
caster = pybind11::detail::load_type<absl::Span<const int>>(span);
return CheckSpan(pybind11::detail::cast_op<absl::Span<const int>>(caster),
values);
}
absl::CivilSecond MakeCivilSecond(double secs) {
return absl::ToCivilSecond(absl::FromUnixSeconds(static_cast<int64_t>(secs)),
absl::UTCTimeZone());
......@@ -176,12 +163,24 @@ bool CheckSet(const absl::flat_hash_set<int>& set,
return set == check;
}
// Non-const Span.
template <typename T>
void FillNonConstSpan(T value, absl::Span<T> output_span) {
for (auto& i : output_span) {
i = value;
// Span
bool CheckSpan(absl::Span<const int> span, const std::vector<int>& values) {
if (span.size() != values.size()) return false;
for (size_t i = 0; i < span.size(); ++i) {
if (span[i] != values[i]) return false;
}
return true;
}
bool CheckSpanCasterCopy(const handle& span, const std::vector<int>& values) {
pybind11::detail::make_caster<absl::Span<const int>> caster;
caster = pybind11::detail::load_type<absl::Span<const int>>(span);
return CheckSpan(pybind11::detail::cast_op<absl::Span<const int>>(caster),
values);
}
void FillSpan(int value, absl::Span<int> output_span) {
for (auto& i : output_span) i = value;
}
struct ObjectForSpan {
......@@ -189,18 +188,25 @@ struct ObjectForSpan {
int value;
};
int SumObjectPointersSpan(absl::Span<ObjectForSpan* const> inputs) {
void FillObjectPointersSpan(int value,
absl::Span<ObjectForSpan* const> output_span) {
for (ObjectForSpan* item : output_span) item->value = value;
}
void FillObjectSpan(int value, absl::Span<ObjectForSpan> output_span) {
for (auto& item : output_span) item.value = value;
}
int SumObjectPointersSpan(absl::Span<const ObjectForSpan* const> span) {
int result = 0;
for (ObjectForSpan* item : inputs) {
result += item->value;
}
for (const ObjectForSpan* item : span) result += item->value;
return result;
}
template <typename T>
void DefineNonConstSpan(module* py_m, absl::string_view type_name) {
py_m->def(absl::StrCat("fill_non_const_span_", type_name).c_str(),
&FillNonConstSpan<T>, arg("value"), arg("output_span").noconvert());
int SumObjectSpan(absl::Span<const ObjectForSpan> span) {
int result = 0;
for (auto& item : span) result += item.value;
return result;
}
// absl::variant
......@@ -230,6 +236,24 @@ std::vector<absl::variant<A*, B*>> Identity(
return value;
}
} // namespace test
} // namespace pybind11
PYBIND11_MAKE_OPAQUE(std::vector<pybind11::test::ObjectForSpan>);
namespace pybind11 {
namespace test {
// Demonstration of constness check for span template parameters.
static_assert(std::is_const<const int>::value);
static_assert(
std::is_const<int* const>::value); // pointer is const, int is not.
static_assert(std::is_const<const int* const>::value);
static_assert(!std::is_const<int>::value);
static_assert(!std::is_const<int*>::value);
static_assert(
!std::is_const<const int*>::value); // int is const, pointer is not.
PYBIND11_MODULE(absl_example, m) {
// absl::Time/Duration bindings.
m.def("make_duration", &MakeDuration, arg("secs"));
......@@ -253,25 +277,28 @@ PYBIND11_MODULE(absl_example, m) {
// absl::Span bindings.
m.def("check_span", &CheckSpan, arg("span"), arg("values"));
m.def("check_span_no_convert", &CheckSpan, arg("span").noconvert(),
arg("values"));
m.def("check_span_caster_copy", &CheckSpanCasterCopy, arg("span"),
arg("values"));
class_<VectorContainer>(m, "VectorContainer")
.def(init())
.def("make_span", &VectorContainer::MakeSpan, arg("values"));
// non-const absl::Span bindings.
DefineNonConstSpan<double>(&m, "double");
DefineNonConstSpan<int>(&m, "int");
// Wrap a const Span with a non-const Span lambda to avoid copying data.
m.def(
"check_span_no_copy",
[](absl::Span<int> span, const std::vector<int>& values) -> bool {
return CheckSpan(span, values);
},
arg("span"), arg("values"));
class_<ObjectForSpan>(m, "ObjectForSpan").def(init<int>());
m.def("SumObjectPointersSpan", &SumObjectPointersSpan);
// Non-const spans can never be converted, so `output_span` could be marked as
// `noconvert`, but that would be redundant (so test that it is not needed).
m.def("fill_span", &FillSpan, arg("value"), arg("output_span"));
// Span of objects.
class_<ObjectForSpan>(m, "ObjectForSpan")
.def(init<int>())
.def_readwrite("value", &ObjectForSpan::value);
bind_vector<std::vector<ObjectForSpan>>(m, "ObjectVector");
m.def("sum_object_pointers_span", &SumObjectPointersSpan, arg("span"));
m.def("sum_object_span", &SumObjectSpan, arg("span"));
m.def("sum_object_span_no_convert", &SumObjectSpan, arg("span").noconvert());
m.def("fill_object_pointers_span", &FillObjectPointersSpan, arg("value"),
arg("output_span"));
m.def("fill_object_span", &FillObjectSpan, arg("value"), arg("output_span"));
// absl::string_view bindings.
m.def("check_string_view", &CheckStringView, arg("view"), arg("values"));
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment