Implementing a Meta System in C++

Hi everybody!

I am relatively new to game development, however I would like to share my experience in developing a meta system for C++. I faced the problem of embedding a scripting language when I was developing my own 3D game engine. There can be many solutions for embedding a specific language (for example, Luabind for Lua and boost.Python for Python). Having such a variety of tools, one obviously should not reinvent the wheel.

I started by embedding the simple and fast Lua programming language with the Luabind library. I think it is very good, you may wish to see yourself:

class_<BaseScript, ScriptComponentWrapper>("BaseComponent")
    .def(constructor<>())
    .def("start", &BaseScript::start,
    &ScriptComponentWrapper::default_start)
	.def("update", &BaseScript::update,
    &ScriptComponentWrapper::default_update)
	.def("stop", &BaseScript::stop,
    &ScriptComponentWrapper::default_stop)
	.property("camera", &BaseScript::getCamera)
	.property("light", &BaseScript::getLight)
	.property("material", &BaseScript::getMaterial)
	.property("meshFilter", &BaseScript::getMeshFilter)
	.property("renderer", &BaseScript::getRenderer)
	.property("transform", &BaseScript::getTransform)

This piece of code looks highly readable to me. Class registration is simple, at least I see no obstacles. However, this solution is for Lua only.

Inspired by the Unity script system, I decided to add support for several languages into the engine, as well as a platform for interaction between them. Yet such tools as Luabind are not quite suitable for these: most of them are built on C++ templates and generate code only for a pre-specified language. Each class must be registered in each of the systems. Any user of a system has to manually define template instantiations of every class for every scripting language.

It would be great to have just one database for all script engines. Moreover, it would be nice to have the ability to load a type's specifications from plugins within runtime. Binding libraries are not good for this – it must be a real metasystem! I could see no way for adopting an existing solution. Existing libraries turned out to be huge and awkward. Some seemingly smart solutions have additional dependencies or require special tools such as Qt moc and gccxml. Of course, one could find good alternatives, such as the Camp reflection library. It looks like Luabind:

camp::Class::declare<MyClass>("MyClass")
    // ***** constant value *****
    .function("f0", &MyClass::f).callable(false)
    .function("f1", &MyClass::f).callable(true)

    // ***** function *****
    .function("f2", &MyClass::f).callable(&MyClass::b1)
    .function("f3", &MyClass::f).callable(&MyClass::b2)
    .function("f4", &MyClass::f).callable(boost::bind(&MyClass::b1, _1))
    .function("f5", &MyClass::f).callable(&MyClass::m_b)
    .function("f6", &MyClass::f).callable(boost::function<bool (MyClass&)>(&MyClass::m_b));

However, the performances of such solutions leave much to be desired. Therefore, I decided to develop my own metasystem, as any normal programmer would, I think. This is why the uMOF library has been developed.

Meet the uMOF

uMOF is a cross-platform open source library for meta programming. It resembles Qt, but, unlike Qt, it is developed by using templates. Qt declined the use of templates due to syntax matters. Although their approach brings high speed and safe memory use, it requires the use of an external tool (MOC compiler) which is not always convenient.

Now let’s get down to business. To make meta information available to users in objects inherited from Object class, you should write OBJECT macro in the class definition. Now you can write EXPOSE and PROPERTIES macros to define functions and properties.

Take a look at this example:

class Test : public Object
{
    OBJECT(Test, Object)
    EXPOSE(Test, 
        METHOD(func),
        METHOD(null),
        METHOD(test)
    )

public:
    Test() = default;

    float func(float a, float b)
    {
        return a + b;
    }

    int null()
    {
        return 0;
    }

    void test()
    {
        std::cout << "test" << std::endl;
    }
};

Test t;

Method m = t.api()->method("func(int,int)");
int i = any_cast<int>(m.invoke(&t, args));

Any res = Api::invoke(&t, "func", {5.0f, "6.0"});

In the current version, insertion of meta information is invasive; yet development of an external description is in progress.

Due to the use of advanced templates, uMOF is very fast and compact. A downside is that not all compilers are supported because of new C++11 features utilized (for example, to compile on Windows, you would need the latest version of Visual C++, the November CTP). Since usage of templates may not be too pleasant for some developers, they are wrapped up in macros. This is why the public API looks rather neat.

To prove my point, here are benchmark test results.

Test results

I compared meta-systems over three parameters: (a) compilation and link time, (b) executable size and © function call time. I took a native function call as the reference. All systems were tested on a Windows platform with Visual C++ compiler.

These results visualized:

I also considered testing other libraries:

• Boost.Mirror
• XcppRefl
• Reflex
• XRtti

However, currently this appears impossible because of various reasons. The Boost.Mirror and XcppRefl look promising, but they are not yet in an active development stage. Reflex needs GCCXML tool, but I failed to find any adequate substitution of that for Windows. Xrtti does not support Windows either in the current release.

What is in the pipeline?

So, how does it work? Variadic templates and templates with functions as arguments give speed and a compact binary. All meta information is organized as a set of static tables. No additional actions are required at runtime. A simple structure of pointer tables keeps binary tight.

Find an example of function description below:

template<typename Class, typename Return, typename... Args>
struct Invoker<Return(Class::*)(Args...)>
{
	typedef Return(Class::*Fun)(Args...);

	inline static int argCount()
	{
		return sizeof...(Args);
	}

	inline static const TypeTable **types()
	{
		static const TypeTable *staticTypes[] =
		{
			Table<Return>::get(),
			getTable<Args>()...
		};
		return staticTypes;
	}

	template<typename F, unsigned... Is>
	inline static Any invoke(Object *obj, F f, const Any *args, unpack::indices<Is...>)
	{
		return (static_cast<Class *>(obj)->*f)(any_cast<Args>(args[Is])...);
	}

	template<Fun fun>
	static Any invoke(Object *obj, int argc, const Any *args)
	{
		if (argc != sizeof...(Args))
			throw std::runtime_error("Bad argument count");
		return invoke(obj, fun, args, unpack::indices_gen<sizeof...(Args)>());
	}
};

The Any class plays an important role in the library performance. It allows allocating memory for instances and stores the associated type information efficiently. I used hold_any class from the boost.spirit library as a reference. Boost also uses templates to wrap types. Types, which are smaller than a pointer, are stored in void* directly. For a bigger type, the pointer refers to an instance of the type.

template<typename T>
struct AnyHelper<T, True>
{
	typedef Bool<std::is_pointer<T>::value> is_pointer;
	typedef typename CheckType<T, is_pointer>::type T_no_cv;

	inline static void clone(const T **src, void **dest)
	{
		new (dest)T(*reinterpret_cast<T const*>(src));
	}
};

template<typename T>
struct AnyHelper<T, False>
{
	typedef Bool<std::is_pointer<T>::value> is_pointer;
	typedef typename CheckType<T, is_pointer>::type T_no_cv;

	inline static void clone(const T **src, void **dest)
	{
		*dest = new T(**src);
	}
};

template<typename T>
Any::Any(T const& x) :
	_table(Table<T>::get()),
	_object(nullptr)
{
	const T *src = &x;
	AnyHelper<T, Table<T>::is_small>::clone(&src, &_object);
}

I had to reject using RTTI – it is too slow. Types are checked only by comparison of pointers to the static tables. All type modifiers are omitted, so that, for example, int and const int are treated as the same type.

template <typename T>
inline T* any_cast(Any* operand)
{
	if (operand && operand->_table == Table<T>::get())
		return AnyHelper<T, Table<T>::is_small>::cast(&operand->_object);

	return nullptr;
}

How to use the library?

Script engine building becomes simple and nice. For example, it is enough to define an generic call function for Lua. It will check the number of arguments and their types and, of course, call the function itself. Binding is also not difficult: just save MetaMethod in upvalue for each function in Lua. All objects in uMof are “thin”, that is to say they only wrap around pointers referring to records in the static table. Therefore, you can copy them without worrying about the performance.

Find an example of Lua binding below:

#include <lua/lua.hpp>
#include <object.h>
#include <cassert>
#include <iostream>

class Test : public Object
{
	OBJECT(Test, Object)
	EXPOSE(
		METHOD(sum),
		METHOD(mul)
	)

public:
	static double sum(double a, double b)
	{
		return a + b;
	}

	static double mul(double a, double b)
	{
		return a * b;
	}
};

int genericCall(lua_State *L)
{
	Method *m = (Method *)lua_touserdata(L, lua_upvalueindex(1));
	assert(m);

	// Retrieve the argument count from Lua
	int argCount = lua_gettop(L);
	if (m->parameterCount() != argCount)
	{
		lua_pushstring(L, "Wrong number of args!");
		lua_error(L);
	}

	Any *args = new Any[argCount];
	for (int i = 0; i < argCount; ++i)
	{
		int ltype = lua_type(L, i + 1);
		switch (ltype)
		{
		case LUA_TNUMBER:
			args[i].reset(luaL_checknumber(L, i + 1));
			break;
		case LUA_TUSERDATA:
			args[i] = *(Any*)luaL_checkudata(L, i + 1, "Any");
			break;
		default:
			break;
		}
	}

	Any res = m->invoke(nullptr, argCount, args);
	double d = any_cast<double>(res);
	if (!m->returnType().valid())
		return 0;

	return 0;
}

void bindMethod(lua_State *L, const Api *api, int index)
{
	Method m = api->method(index);
	luaL_getmetatable(L, api->name()); // 1
	lua_pushstring(L, m.name()); // 2
	Method *luam = (Method *)lua_newuserdata(L, sizeof(Method)); // 3
	*luam = m;
	lua_pushcclosure(L, genericCall, 1);
	lua_settable(L, -3); // 1[2] = 3
	lua_settop(L, 0);
}

void bindApi(lua_State *L, const Api *api)
{
	luaL_newmetatable(L, api->name()); // 1

	// Set the "__index" metamethod of the table
	lua_pushstring(L, "__index"); // 2
	lua_pushvalue(L, -2); // 3
	lua_settable(L, -3); // 1[2] = 3
	lua_setglobal(L, api->name());
	lua_settop(L, 0);

	for (int i = 0; i < api->methodCount(); i++)
		bindMethod(L, api, i);
}

int main(int argc, char *argv[])
{
	lua_State *L = luaL_newstate();
	luaL_openlibs(L);
	bindApi(L, Test::classApi());

	int erred = luaL_dofile(L, "test.lua");
	if (erred)
		std::cout << "Lua error: " << luaL_checkstring(L, -1) << std::endl;

	lua_close(L);

	return 0;
}

Conclusion

Let us summarize what we have got.

uMOF advantages:

• Compact
• Fast
• No external tools, just a modern compiler needed

uMOF disadvantages:

• Supported by only modern compilers
• Auxiliary macros are not quite polished

The library is in a rather raw stage yet. However, the approach leads to good results. So I'm going to implement a few useful features, such as variable arity functions (default parameters), external description of meta types and property change signals.

Thank you for your interest.

You can find the project here - https://bitbucket.org/occash/umof. Comments and suggestions are welcome, as “comments” I suppose.