Wrapping a single function

Say we're implementing imgproc module bindings for Torch. So let's see what we have in it by inspecting opencv2/imgproc.hpp as it contains function signatures and their doc descriptions.

It looks like there's cv::GaussianBlur function:

CV_EXPORTS_W void GaussianBlur(InputArray src, OutputArray dst, Size ksize,
                               double sigmaX, double sigmaY = 0,
                               int borderType = BORDER_DEFAULT);

Note the CV_EXPORTS macro -- from it, we infer the need to export this function into foreign language bindings.

We start off by defining a Lua function in ./cv/imgproc.lua:

function cv.GaussianBlur(t)
    -- function body
end

Note that this function belongs to cv table.

As can be seen from the signature and the docs, this filter is intended for application in 3 ways:

(1) output to an empty cv::Mat:

cv::Mat image = imread('img.png');
cv::Mat blurred;
cv::GaussianBlur(image, blurred, cv::Size(3, 3), 1.0, 1.0);

(2) output to a cv::Mat of the same size and type as src:

cv::Mat image = imread('img.png');
cv::Mat blurred = image.clone() * 0;
cv::GaussianBlur(image, blurred, cv::Size(3, 3), 1.0, 1.0);

(3) filter in-place:

cv::Mat image = imread('img.png');
cv::GaussianBlur(image, image, cv::Size(3, 3), 1.0, 1.0);

Let's make these use cases possible for Torch users!

We want cv.GaussianBlur to accept a table of arguments t, so the calls for those use cases should look like this:

-- output to retval, as dst is not provided!
local image_B = cv.GaussianBlur{src=image, ksize={7, 7}, sigmaX=3.5, sigmaY=3.5}
-- output to another Tensor of same size & type
local image_A = image * 0
cv.GaussianBlur{src=image, dst=image_A, ksize={7, 7}, sigmaX=3.5, sigmaY=3.5}

-- or filter in-place
cv.GaussianBlur{src=image, dst=image, ksize={width=7, height=7}, sigmaX=3.5, sigmaY=3.5}

So let's convert t's fields to local variables representing arguments, with regard to the original function signature:

• If var argument is required, we write assert(t.var) to ensure it's present
• If var is optional and has a default value, we write t.var or <default-value>

./cv/imgproc.lua

function cv.GaussianBlur(t)
    local src =        assert(t.src)
    local dst =        t.dst
    local ksize =      assert(t.ksize) -- how to convert this?
    local sigmaX =     assert(t.sigmaX)
    local sigmaY =     t.sigmaY or 0
    local borderType = t.borderType or cv.BORDER_DEFAULT

    -- to be continued
end

Some things to note here:

• src and dst are assumed to be torch.Tensors
• dst is optional because of the first use case
• Almost all (more than 1000) OpenCV constants and enums like cv.BORDER_DEFAULT are defined in ./cv/constants.lua

And another thing: how to cope with that argument of type cv::Size? A great deal of OpenCV functions require argument(s) of some OpenCV class type, like cv::Size, cv::Point, cv::TermCriteria, cv::RotatedRect etc. We have to somehow create their instances in Lua, that's why for each such class cv::<class-name> we create a C wrapper struct <class-name>Wrapper in common code (cv.lua and Common.[hpp|cpp]). So let's wrap our cv::Size:

./include/Common.hpp

struct SizeWrapper {
    int width, height;

    // an operator for implicit automatic conversion to cv::Size
    inline operator cv::Size() { return cv::Size(width, height); }
};

./cv.lua

function cv.Size(data)
    return ffi.new('struct SizeWrapper', data)
end

Here, we exploit the intelligence of this ffi.new function: if you want to make a struct SizeWrapper from cv::Size(150, 280), you may pass either of the following to cv.Size:

• 150, 280
• {150, 280}
• {width=150, height=280}

Cool. Now we can directly convert t.ksize to struct SizeWrapper:

./cv/imgproc.lua

function cv.GaussianBlur(t)
    local src =        assert(t.src)
    local dst =        t.dst
    local ksize =      cv.Size(assert(t.ksize))
    local sigmaX =     assert(t.sigmaX)
    local sigmaY =     t.sigmaY or 0
    local borderType = t.borderType or cv.BORDER_DEFAULT

    -- a call to a C function that invokes cv::GaussianBlur goes here
end

You may notice that it's not very pleasant to call all the functions this way, as you'll have to either memorize all the argument names or to consult OpenCV docs right along:

cv.GaussianBlur{
    src = image, 
    dst = image, 
    ksize = {7,7}, 
    sigmaX = 1.5,
    sigmaY = 1.5
}

cv.Canny{
    image = image,
    edges = edges,
    threshold1 = 0,
    threshold2 = 30,
    apertureSize = 3
}
cv.demosaicing{_src=edges _dst=edges, code=2}
cv.imshow{winname="an image", image=edges}
cv.waitKey{delay=30}

That's why it may be a good idea to support positional arguments as well:

function cv.GaussianBlur(t)
    local src =        assert(t[1] or t.src)
    local dst =        t[2] or t.dst
    local ksize =      cv.Size(assert(t[3] or t.ksize))
    local sigmaX =     assert(t[4] or t.sigmaX)
    local sigmaY =     t[5] or t.sigmaY or 0
    local borderType = t[6] or t.borderType or cv.BORDER_DEFAULT

    -- a call to a C function that invokes cv::GaussianBlur goes here
end

Now let's turn to our C function that will be called through FFI. According to how we've prepared our args, the signature has to be as follows:

./include/imgproc.hpp

extern "C" struct TensorWrapper GaussianBlur(
    struct TensorWrapper src, struct TensorWrapper dst,
    struct SizeWrapper ksize, double sigmaX,
    double sigmaY, int borderType
);

Remember the 3 use cases of this filter? That's what we have to implement in ./src/imgproc.cpp. The code should be self-explanatory:

./src/imgproc.cpp

extern "C"
struct TensorWrapper GaussianBlur(struct TensorWrapper src, struct TensorWrapper dst,
                                  struct SizeWrapper ksize, double sigmaX,
                                  double sigmaY, int borderType)
{
    // first, check if dst is provided
    if (dst.isNull()) {
        // [use case 1]: output to return value
        cv::Mat retval;
        cv::GaussianBlur(
                src.toMat(), retval, ksize, sigmaX, sigmaY, borderType);
        // create a NEW Tensor in C and pass it to Lua
        return TensorWrapper(retval);
    } else if (dst.tensorPtr == src.tensorPtr) {
        // [use case 3]: filter in-place
        cv::Mat source = src.toMat();
        cv::GaussianBlur(
                source, source, ksize, sigmaX, sigmaY, borderType);
    } else {
        // [use case 2]: try to output to a Tensor dst of same size & type as src
        cv::GaussianBlur(
                src.toMat(), dst.toMat(), ksize, sigmaX, sigmaY, borderType);
    }
    return dst;
}

Note that:

• ksize is passed to cv::GaussianBlur directly as SizeWrapper::operator cv::Size() is defined. It would be also correct to write cv::Size(ksize).
• In contrast, struct TensorWrapper can't be converted to cv::Mat implicitly. Instead, you have to call .toMat().
• There's also a TensorWrapper::operator cv::Mat, so a return retval could be fine, but I construct TensorWrapper explicitly for code readability.
• Some OpenCV functions provide a cv::noArray() default value for InputArray/OutputArray arguments. Unfortunately, this can't be constructed (in an elegant way) in Lua, so you should use TO_MAR_OR_NOARRAY(tensor) macro.

Next, compile the libs:

mkdir build
cd build
cmake ..
# or, optionally: `cmake .. -DOpenCV_DIR=<path-to-your-opencv-3.x.x>`
make

Our function now resides in ./lib/libimgproc.so (for Linux and the like) so we can call it through FFI:

./cv/imgproc.lua

function cv.GaussianBlur(t)
    local src =        assert(t[1] or t.src)
    local dst =        t[2] or t.dst
    local ksize =      cv.Size(assert(t[3] or t.ksize))
    local sigmaX =     assert(t[4] or t.sigmaX)
    local sigmaY =     t[5] or t.sigmaY or 0
    local borderType = t[6] or t.borderType or cv.BORDER_DEFAULT

    return cv.unwrap_tensors(
        C.GaussianBlur(
            cv.wrap_tensors(src), cv.wrap_tensors(dst), ksize, sigmaX, sigmaY, borderType))
end

cv.wrap_tensors(tensors) is a special function that takes one or more Tensors (either in a table or not) and returns a TensorWrapper (or a TensorArray) over them. It should be applied to every torch.Tensor that is going to be passed to C++ via FFI.

cv.unwrap_tensors(wrapper) converts a TensorWrapper or a TensorArray to a torch.Tensor (or to a list of them).

That's it! See also ./demo/filtering.lua.