15 Aug Cross-language shared libraries using SWIG
In a nutshell
-
- Making software libraries available on multiple platforms and for multiple languages can be complicated and expensive.
- SWIG makes this very simple to create and maintain, making it a practicable option for more programmers.
- Using SWIG to generate code for your project does not modify your project’s licensing position.
- Bottom Line: If you have a project that needs to be cross-platform &/or cross-language, SWIG is an excellent tool to accelerate development and reduce maintenance costs.
Have you ever wanted to have software libraries available on multiple platforms, for a range of languages, but were put off by the complexity and expense? I ran into this several years ago, when I was developing tools for the Internet of Things (IoT). I wanted to create a messaging and communications stack that could be used across a range of different platforms and languages, needed to be able to deploy it as a shared library on Windows, Linux, and Mac, as well as iOS and Android, and wanted the maximum flexibility for what programming language my users could use.
One approach would have been to manually create different libraries for each language – and potentially for each operating system, too – but that would rapidly become costly and complicated to maintain (especially having to fix the same bug in multiple versions of the library!).
That’s when I came across the Open Source development tool SWIG (https://www.swig.org), which was the answer to all my problems, though not without its own challenges.
With SWIG, I was able to create a shared library in C++ and then generate language-specific wrappers for each language I wanted, including Java, Ruby, Python, C# & VB.NET, etc. This allowed me to use a common code base across these different platforms and provide seamless interoperability and transparency, making developers’ lives easier too. The nice thing about this is that it is pretty easy to create and maintain, and does not require any great expertise beyond basic C++ skills.
Providing shared libraries that work in their preferred language removes barriers for new developers. It enables them to start using your code or services more rapidly, and gives them a predictable user experience in their preferred language – all of which makes for happier developers.
Obviously, these libraries still needed to be compiled against each platform and language variation, but managing a CI/CD (continuous integration/continuous deployment) pipeline that tests and builds artifacts for each one, and managing a common code base, is much easier than handling different code bases and all the issues with maintaining feature parity across libraries.
If you haven’t played with this technology before, it’s actually simpler than you might think. First, you need to define the interfaces you want to make available, in a file or set of files that SWIG can understand (which can be as simple as including the public header files from your project). Then just run a command to get SWIG to read the interface file and generate the source files that act as translators between your source language and target language.
For example, if you wanted to take a C++ library you’d written, and make it available to Python, you could simply create an interface file that points to the public header files for your library, then run the relevant command, and SWIG will generate the C++ code required to expose your library to Python, as well as a Python wrapper around it. Once you have compiled this code, you can simply import the module, as you would any other Python module, and use it the way you would expect.
Creating an example shared library
Let’s use a simple example library to demonstrate this. The files swigutils.cpp and swigutils.h are the source code and header respectively for a very simple function that takes a positive integer number and returns true if the number is prime, and false if it is not. I have also included a sample C++ program swigutils-test.cpp that consumes this library, so you can test the library to prove it works.
To keep things simple, I am only going to focus on the Linux platform in this article, but will cover cross-platform compilation in a later post. I use Linux Mint, which is an Ubuntu derivative, so my examples will be geared towards that, but the concepts here are broadly applicable, with the main differences being the package managers and file-paths.
Source Code
// declare a function to calculate whether a supplied number is a prime number
bool IsPrimeNumber(const int numberToCheck) {
//default to prime number being true
bool isPrime = true;
//loop through all numbers up to 1/2 of numberToCheck
for (int i = 2; i <= numberToCheck / 2; ++i) {
//If this number is cleanly divisible by I, i.e. there is no remainder
if (numberToCheck % i == 0) {
isPrime = false;
//No need to continue trying other options, so break out of the loop
break;
}
}
//Will return true if it is a prime number, and false if not.
return isPrime;
}
#include <stdbool.h> // declare a function to calculate whether a supplied number is a prime number bool IsPrimeNumber(const int numberToCheck);
#include "swigutils.h"
#include <iostream>
// Main code that will be run when we execute this program
int main() {
// Define a variable for the number that the user will provide
int numberToCheck;
// Prompt the user to enter a number
std::cout << "Please enter a positive integer number ";
// Get a number from the user - note that this is not checking the data type, which you would want to do in a real program,
std::cin >> numberToCheck;
// Call the IsPrimeNumber using the number provided - this is where we are using a function from the shared library
if (isPrimeNumber(numberToCheck))
// Let the user know that it was a prime number
std::cout << "that is a prime number";
else
// Let the user know that it was a prime number
std::cout << "that is not a prime number";
// return a value of 0 to let the shell know that it executed successfully.
return 0;
}
# If you do not already have build tools installed, you should install them.
# On Ubuntu or equivalent, you would do this using the following
sudo apt-get install build-essential
# Command to compile the library
g++ -shared -Wall swigutils.cpp -o libswigutils.so
# Command to compile the sample code, with reference to the swigutils library in current dir.
g++ -Wall swigutils-test.cpp -o ./test -I . -L. libswigutils.so
# Tell Linux to append the current directory to the path of places to search for library files. This is not required if you install libswigutils in the default library folder.
export LD_LIBRARY_PATH=./${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
# Execute the test file.
./test
Creating a SWIG interface file
Based on this, we can simply create an interface file that references the header file. The interface code looks a little like a C/C++ header file, but has the options for some SWIG-specific commands. In this case, we are simply defining that there will be a module created that will be calledswigutils and we are specifying that we should include and wrap the swigutils.h file.
Note that SWIG allows for a lot of sophistication in terms of object inheritance, events, and other object-oriented concepts, some of which requires special handling in the interface file, but that is beyond the scope of this article.
Source Code
%module swigutils
%{
/* Includes the header */
#include "swigutils.h"
%}
/* generate the wrappers */
%include "swigutils.h"
In order to make this shared library compatible with another language, we simply need to:
- Generate the language-specific code using the SWIG command line tool
- Compile and link the library
- Test using some sample code
If you have not already done so, you will need to download and install SWIG. Note that for most Linux distros, it is simply a matter of using the package manager to install SWIG, but Windows requires a download from the above page.
Creating language-specific wrappers
Below, I am providing examples for several languages. You can pick which target language you want to compile for by simply clicking on the appropriate tab.
To make our shared library compatible with Python, we first need to generate the language-specific code using the command line tool.
The command line is pretty simple. First we call the swig executable, and pass it the -c++ argument, since we are working with C++ source code rather than C. We then pass it the -python argument to tell it we want to target the Python language for output. In this case, I am specifying the output filename for the C++ code, but that is optional. The last thing is to reference the interface file we created, swigutils.i. It should look like this:
swig -c++ -python -o swigutils_python.cpp swigutils.i
Running this command will not show any response unless there is an error, but you will find that two new files have been created, swigutils_python.cpp and swigutils.py. These are the wrapper files that SWIG creates for Python.
Compiling the Python library
Having generated the code, we now need to compile the shared library but, before we can do that, we need to have the header files for the version of Python we are using. In most cases this should be as simple as installing the python-dev package.
For compiling the library, there are a couple of ways we can do this:
Manually
The manual method is fairly simple, using g++ command line options, but requires different settings for each platform we are compiling for. Since we are only using Linux in this example, it is fairly simple to demonstrate this approach, but there are a few points to note here:
- We are using the
-fPICoption to make the code suitable for inclusion in a library - We need to reference the header files for the version of Python we are using, in this case, 2.7
- We are specifying which version of C++ to support. Note that it doesn’t have to be C++11
- Because of the way the Python wrapper works, we are naming the shared library the same as the module name, but with an underscore at the start, and the shared library extension of .so, i.e.
_swigutils.so
This means that the overall command line to compile the shared library is:
g++ -shared -Wall -fPIC -I/usr/include/python2.7 -std=c++11 swigutils.cpp swigutils_python.cpp -o _swigutils.so
Assuming this compiles correctly, you should now have a file called _swigutils.so.
Using Python Distutils
The other way to compile this library is to use Python’s distutils method. This is the preferred approach, as it works cross platform and takes care of all the compilation parameters. It is also easy to make this into a distributable module using this method (though you would be distributing the raw source, which would be compiled upon install).
To use distutils, you need to create a file, typically called setup.py, as shown in the example.
Note that this configuration file is referencing the two C++ source files in the sources array, and describing the required output name of _swigutilsfor the compiled library, as well as the swigutils module name for the Python code that was generated by SWIG.
To use this, simply create a setup.py file containing the above in the same directory, and run the following command:
python setup.py build_ext --inplace
This will compile the C++ code into a shared library on Linux, but can also create a shared DLL in Windows, or a dylib on Mac, which is part of what makes this approach appealing.
Source Code
#!/usr/bin/env python
"""
Distutils example for swigutils
"""
from distutils.core import setup, Extension
swigutils_module = Extension('_swigutils', sources=['swigutils.cpp', 'swigutils_python.cpp'],
)
setup (
name = 'swigutils',
version = '1.0',
author = "Example Author",
description = "simple utility to check prime numbers",
ext_modules = [swigutils_module],
py_modules = ["swigutils"],
)
Testing the library
Now to use that in Python, we can simply import the library much like any other Python library file, and use it how you would expect.
As shown in the source code example, test.py is a sample Python test file that imports the methods from the swigutils library (our shared library in C++), and then calls the isPrimeNumber method twice to determine if 15 or 17 are prime numbers. It then outputs that to the screen.
To run this Python file, simply execute the following command:
python test.py
It should output the following:
15 is NOT prime number
17 is a prime number
Source Code
from swigutils import *
if isPrimeNumber(15):
print "15 is a prime number"
else:
print "15 is NOT a prime number"
if isPrimeNumber(17):
print "17 is a prime number"
else:
print "17 is NOT a prime number"
To make our shared library compatible with Ruby, we first need to generate the language-specific code using the command line tool.
The command line is pretty simple. First we call the swig executable, and pass it the -c++ argument, since we are working with C++ source code rather than C. We then pass it the -ruby argument to tell it we want to target the Ruby language for output. In this case, I am specifying the output filename for the C++ code, but that is optional. The last thing is to reference the interface file we created, swigutils.i. It should look like this:
swig -c++ -ruby -o swigutils_ruby.cpp swigutils.i
Running this command will not show any response unless there is an error, but you will find that one new file has been created, swigutils_ruby.cpp. This is the wrapper file that SWIG creates for Ruby.
Compiling the Ruby library
Having generated the code, we now need to compile the shared library, but before we can do that, we need to have the header files for the version of Ruby we are using. In most cases this should be as simple as installing the ruby-dev package.
For compiling the library, there are a couple of ways we can do this:
Manually
The manual method is fairly simple, using g++ command line options, but requires different settings for each platform we are compiling for. Since we are only using Linux in this example, it is fairly simple to demonstrate this approach, but there are a few points to note here:
- We are using the
-fPICoption to make the code suitable for inclusion in a library. - We need to include the folders that have the header files for Ruby. Note, this requires the hardware-specific folder, too.
- We are specifying which version of C++ to support. Note that it doesn’t have to be C++11.
- Because of the way the Ruby wrapper works, we are naming the shared library the same as the module name, but with the shared library extension of .so, i.e.
swigutils.so.
This means that the overall command line to compile the shared library is:
g++ -shared -Wall -fPIC -I/usr/include/ruby-2.5.0/ -I/usr/include/x86_64-linux-gnu/ruby-2.5.0/ -std=c++11 swigutils.cpp swigutils_ruby.cpp -o swigutils.so
Assuming this compiles correctly, you should now have a file called swigutils.so.
Makefile Compilation using extconf.rb
The other way to compile this library is to use Ruby’s extconf.rb method. This is the preferred approach, as it works cross platform, and takes care of all the compilation parameters.
To use this approach, you need to create a file called extconf.rb as shown in the example.
Note that this configuration file does not specify any source files. Instead, it includes ALL C/C++ files in the current directory. Therefore, for this to be successful, the only files that should be in this compilation directory in addition to extconf.rbare swigutils.cpp, swigutils.h, and swigutils_ruby.cpp.
To use this, I recommend that you create a folder to put those files in, then run the following commands from that directory:
ruby extconf.rb
make
This will compile the C++ code into a shared library on Linux, but can also create a shared DLL in Windows, or a dylib on Mac, which is part of what makes this approach appealing.
Source Code
require 'mkmf'
create_makefile('swigutils')
Testing the library
Now to use that in Ruby, we can simply require the library much like any other Ruby library file, and use it how you would expect. Test.rb is a sample Ruby test file that imports the methods from the swigutils library (our shared library in C++), and then calls the isPrimeNumber method twice to determine if 15 or 17 are prime numbers. It then outputs that to the screen.
Note that Ruby capitalizes the module name, so swigutils becomes Swigutils.
To run this Ruby file, simply execute the following command:
ruby test.rb
It should output the following:
15 is NOT a prime number
17 is a prime number
Source Code
require "./swigutils"
if Swigutils.isPrimeNumber(15)
puts "15 is a prime number"
else
puts "15 is NOT a prime number"
end
if Swigutils.isPrimeNumber(17)
puts "17 is a prime number"
else
puts "17 is NOT a prime number"
end
To make our shared library compatible with Java, we first need to generate the language-specific code using the command line tool.
The command line is pretty simple. First we call the swig executable, and pass it the -c++ argument, since we are working with C++ source code rather than C. We then pass it the -java argument to tell it we want to target the Java language for output. In this case, I am specifying the output filename for the C++ code, but that is optional. The last thing is to reference the interface file we created, swigutils.i. It should look like this:
swig -c++ -java -o swigutils_java.cpp swigutils.i
Running this command will not show any response unless there is an error, but you will find that three new files have been created, swigutils_java.cpp, swigutils.java, and swigutilsJNI.java. These are the wrapper files that SWIG creates for Java.
Compiling the library for Java
Having generated the code, we now need to compile the shared library, but before we can do that, we need to have the Java Development Kit for the version of Java we are using. In most cases this should be as simple as installing the default-jdk package.
For Java we actually need to compile both the C++ code and the Java wrapper code.
Manually
Compiling the library is fairly simple, using g++ command line options, but requires different settings for each platform we are compiling for. Since we are only using Linux in this example, it is fairly simple to demonstrate this approach, but there are a few points to note here:
- We are using the
-fPICoption to make the code suitable for inclusion in a library. - We need to include the folders that have the header files for Java. Note, this requires the JNI headers too.
- We are specifying which version of C++ to support. Note that it doesn’t have to be C++11.
- The Java wrapper uses the default naming convention for libraries on Linux, so the library name will be
libswigutils.so.
This means that the overall command line to compile the C++ shared library is:
g++ -shared -Wall -fPIC -I/usr/lib/jvm/default-java/include -I/usr/lib/jvm/default-java/include/linux -std=c++11 swigutils.cpp swigutils_java.cpp -o libswigutils.so
Assuming this compiles correctly, you should now have a file called libswigutils.so.
Now, to compile the Java wrapper, we can simply call the Java compiler to compile all files with the .java extension.
javac *.java
Testing the library
Now to use that in Java, we can simply use the LoadLibrary method, and then use it how you would expect. Test.java is a sample Java test file that loads the swigutils library, and then calls the isPrimeNumber method twice to determine if 15 or 17 are prime numbers. It then outputs that to the screen.
To run this Java file, first we must compile it
javac test.java
Then call it using the java runtime by executing the following command:
java test
It should output the following:
15 is NOT a prime number
17 is a prime number
Source Code
// test.java
public class test {
static {
System.loadLibrary("swigutils");
}
public static void main(String argv[]) {
if(swigutils.isPrimeNumber(15))
{
System.out.println("15 is a prime number");
}
else
{
System.out.println("15 is NOT a prime number");
}
if(swigutils.isPrimeNumber(17))
{
System.out.println("17 is a prime number");
}
else
{
System.out.println("17 is NOT a prime number");
}
}
}
To make our shared library compatible with C#, we first need to generate the language-specific code using the command line tool.
The command line is pretty simple. First we call the swig executable, and pass it the -c++ argument, since we are working with C++ source code rather than C. We then pass it the -csharp argument to tell it we want to target the C# language for output. In this case, I am specifying the output filename for the C++ code, but that is optional. The last thing is to reference the interface file we created, swigutils.i. It should look like this:
swig -c++ -csharp -o swigutils_csharp.cpp swigutils.i
Running this command will not show any response unless there is an error, but you will find that three new files have been created, swigutils_csharp.cpp, swigutils.cs, and swigutilsPINVOKE.cs. These are the wrapper files that SWIG creates for C#.
Compiling the library for C#
Having generated the code, we now need to compile the shared library, but before we can do that, we need to make sure we have Mono installed. In most cases this should be as simple as installing the mono-complete package.
For C# we actually need to compile both the C++ code and the C# wrapper code.
Compiling the library is fairly simple, using g++ command line options, but requires different settings for each platform we are compiling for. Since we are only using Linux in this example, it is fairly simple to demonstrate this approach, but there are a few points to note here:
- We are using the
-fPICoption to make the code suitable for inclusion in a library. - We are specifying which version of C++ to support. Note that it doesn’t have to be C++11.
- The C# wrapper uses the default naming convention for libraries on Linux, so the library name will be
libswigutils.so.
This means that the overall command line to compile the C++ shared library is:
g++ -shared -Wall -fPIC -std=c++11 swigutils.cpp swigutils_csharp.cpp -o libswigutils.so
Assuming this compiles correctly, you should now have a file called libswigutils.so.
Now, to compile the C# library, we just call the C# compiler, specifying that we are compiling a library, and list the two C# files.
mono-csc -target:library swigutils.cs swigutilsPINVOKE.cs
This will create a file called swigutils.dll. Note that this does not replace the need for libswigutils.so, but instead acts as a connector to it.
Testing the library
To keep things simple for this example, we will not create an assembly and register this DLL. Instead, we can simply reference swigutils.dll at compilation time, and then use it how you would expect . Test.cs is a sample C# test file that uses the swigutils.dll library, which in turn calls out to libswigutils.soin calls the isPrimeNumber method twice to determine if 15 or 17 are prime numbers. It then outputs that to the screen.
To run this C# file, first we must compile it using mono-csc. We need to reference the DLL using the the -r option.
mono-csc -r:swigutils.dll -out:test.exe test.cs
This will create a file called test.exe that can be called using the mono runtime by executing the following command:
mono test.exe
It should output the following:
15 is NOT a prime number
17 is a prime number
Source Code
using System;
public class test {
static void Main() {
if(swigutils.isPrimeNumber(15))
{
Console.WriteLine("15 is a prime number");
}
else
{
Console.WriteLine("15 is NOT a prime number");
}
if(swigutils.isPrimeNumber(17))
{
Console.WriteLine("17 is a prime number");
}
else
{
Console.WriteLine("17 is NOT a prime number");
}
}
}
Click on a different tab above to see this example worked through for another language.
Wrapping it all up
As you can see, SWIG makes it easy to create common library objects that can be used in a variety of platforms, giving you a common single place to write and manage the code, and automating the interface creation for other languages.
This demonstration scenario is necessarily very simplistic, and it is quite likely that you will need to modify the interface file to fine tune it for different languages, especially as your libraries get more complex, but the flexibility here is worth it. I was able to use this technology to create a common library that could be compiled on Linux, Windows, Mac, Android, and IOS, using Intel 32/64bit and ARM, and supporting these languages where possible: Java, Python, Ruby, C#.NET, and C/C++/Objective C.
Some brief pointers if you want to go down this path:
- Make sure your library exposes its code version and wrapper (interface) version somehow, e.g. as properties that can be called. This gets really important when you are working across different compiled versions of the library.
- Use a robust structure to manage the different build command lines etc. I used Maven and the NAR plugin for Maven, which allows you to compile C++ code in much the same way you would use Maven to compile Java code.
- Unit testing in each language is REALLY important. I cannot stress this enough.
- Consider using a CI solution like Jenkins to automate this process. I used TeamCity, and was able to have agents running on each of the platforms to execute the build commands and aggregate back the binaries.
- Read the excellent SWIG tutorial & documentation in detail. There’s tons of good info there.
- And most importantly, remember to support SWIG and the Open Source community wherever possible.
One other point…
Since open source licensing can be a challenging area, I thought it would be helpful to include the following open source licensing info from SWIG’s website here:
When SWIG is used as it is distributed by the SWIG developers, its output is not governed by SWIG's license (including the GPL). SWIG's output contains code from three sources:
* code generated by SWIG, which is not governed by copyright;
* code copied from the SWIG library which is permissively licensed to be redistributed without restriction;
* code derived from the user's input, which may be governed by the license of the code supplied by the user.
So, while the input supplied to SWIG may affect the license of SWIG's output (e.g. if the input code is licensed under a copyleft or proprietary license), SWIG's license does not affect the license of the output. This is consistent with the FSF's FAQ entries on this subject (GPLOutput and WhatCaseIsOutputGPL), because the SWIG code copied into the output by SWIG is not GPL-licensed.
No Comments