Added runtime compile of source dependencies, and improved compiler support.

The latest update adds support for the runtime compile of source dependencies, significantly enhancing the amount and type of code which can be runtime compiled. Additionally, I've improved compiler support amongst other changes. Full details below.

Runtime compile of source dependencies

If you have a header which adds a source dependency, and you also mark that header as being runtime modifiable, the runtime object system will now track both the header and the source file for changes and recompile them along with any runtime compiled code which includes the header.

This allows pretty much any code to be runtime compiled so long as the source and header names are in the format name.h and name.cpp, and you can add the following to the header:

#include "RuntimeSourceDependency.h"

RUNTIME_COMPILER_SOURCEDEPENDENCY;

#include "RuntimeInclude.h"

RUNTIME_MODIFIABLE_INCLUDE;

Note that in order for this to work properly, the header must be included from a source file which uses the REGISTERCLASS macro on an IObject based class  (either directly or via another header). This is because the program needs to preserve state somehow across compiles, and currently the IObject class handles this.

Take care using this feature, as it can be easy to introduce changes which break the code - though in practice I've found this very useful.

Improved compiler support

• Added support for Visual Studio 2013 - however this is as yet untested.
• Added support for cross compile to 32bit on 64bit Linux/MacOSX systems, thanks to Gaoren Xu for discovering this issue and helping with the solution.

Singleton support

At the request of a project using RCC++ I've improved support for singleton type classes. This is especially useful as a good practice in using RCC++ in a performant way is to have singleton system classes which operate on arrays of data.

The REGISTERSINGLETON macro takes a class name and a boolean as arguments, where the boolean controls auto construction of the class. This macro causes the constructor to only ever make one instance of the class. For an example look at the MainObject.cpp file in SimpleTest where we use the macro as:

REGISTERSINGLETON(MainObject, true);

to auto construct an instance of the MainObject class during the Initialise call of RuntimeObjectSystem.

Cleaning build temporaries

Build temporaries can now be cleaned up using the RuntimeObjectSystem::CleanObjectFiles() call, which uses the new FileSystemUtils::PathIterator class. The demos currently do this when the program ends.

Spelling fix, improved module support and Windows fast compile option.

We've made a few changes, one of which alters the interface for the recently introduced runtime source dependency. The main changes are:

1. The macro for the runtime source dependency has had it's spelling corrected, and is now RUNTIME_COMPILER_SOURCEDEPENDENCY (no longer missing the second D). We've also cleaned up the internals as several different wrong spellings of 'dependency' were in use!
2. On initialization the runtime object system now directly uses the PerModuleInterface::GetInstance(), rather than querying the executable. This way RCC++ can be used in a DLL in the project, and multiple DLL's are supported though you will need to explicitly call RuntimeObjectSystem::SetupObjectConstructors() for each one.
3. On Windows, the compiler now kills the compile cmd process by default when complete. This adds about 0.3s to the startup of following compiles due to running the batch file for Visual Studio to set up the environment variables, however it resolves an issue where the cmd process was holding onto handles (for example sockets) the main process wanted to close. This is especially a problem if the main process terminated abnormally, as a zombie cmd process is then left behind. To get faster compiles by keeping the cmd process alive (with the risks this entails), call the RuntimeObjectSystem::SetFastCompileMode() function with true as the parameter.

The full change list can be found here.

Runtime Source Dependencies and Linux Runtime Include bug fix

The latest code on Github includes a new feature - Runtime Source Dependencies - along with a bugfix for Linux builds which resolves an issue with Runtime Includes.

A runtime source dependency is a source file which you need to have compiled when a given header is included by a runtime modifiable source file. Whilst having the source in a library and using the Runtime Link Library feature solves this problem to an extent, sometimes you don't want to create a whole library and thus the ability to compile in dependencies is a really useful feature. Using this feature simply requires the following lines in a header:

#include "RuntimeSourceDependency.h"
RUNTIME_COMPILER_SOURCEDEPENENCY;

If the header is called SomeFeature.h then the source file SomeFeature.cpp is compiled when any runtime modifiable code is changed which includes this header. Using the same filename as the header is required at the moment, in part due to issues with getting full paths from builds on Linux with GCC.

This brings us to the bug fix on Linux - although we'd implemented a system to get the full path for runtime modified source files from GCC, this hadn't been implemented for runtime includes and making the above changes caught this bug, so it's now been fixed. GCC only embeds the path passed in for __FILE__, which can be a relative path from the compile location, so we have to embed the compile path using the pre-processor define COMPILE_PATH="$(PWD)/"; Note that this isn't required if your build system uses full paths, which the cmake builds do, as does the internal GCC runtime compiler.

Note that I'm experiencing an odd issue on my cmake builds on Linux with makefiles - the Eclipse generated files work well however the makefile ones compile but the SimpleTest program fails to create a glfw window.

Added cmake files

Thanks to Anders Buus we now have a set of cmake files for the project.

Linux port of SimpleTest game example

The Pulse demo from project SimpleTest running on Ubuntu Linux compiling changes at runtime.

The SimpleTest example project, otherwise known as the Pulse game demo, now works on Linux with the Eclipse IDE. There's a known bug in that the app shut down appears to crash, which I'll investigate and fix as soon as possible. Grab the latest from Github to see it in action.

[Edit 2013/03/12:] The crash on exit bug for the SimpleTest example has now been fixed.

Linux port with Eclipse progress update

Yesterday I started porting RCC++ to Linux, using the Eclipse IDE as this seemed the favourite option, or at least that's what's been specifically asked for. It's been a while since I've done any serious Linux/Unix coding, and that was in the days before decent IDEs using vi and simple makefiles, so it took a fair few hours to get done. There's now a working version of the runtime compiler and ConsoleExample in my personal fork of RCC++, which I'll pull onto the main fork after some further testing.

[EDIT 2013/11/03]: The code (with a few fixes) has now been pulled into the main fork at Github.

The base RCC++ libaries and the ConsoleExample have no dependencies, so the most difficult problem was getting __FILE__ to reliably generate full paths, since Eclipse builds each project from the path of the output. This required embedding the compile path using COMPILE_PATH=\"$(PWD)/\" as a preprocessor define option (-DCOMPILE_PATH=\"$(PWD)/\" on the gcc command line), which uncovered some bugs in my FileSystemUtils header. Note that the executable needs to be build with the flag -export-dynamic in order to allow symbol resolution by name.

I've not yet ported the rather complex SimpleTest (which needs renaming) due to the dependencies, but this shouldn't stop anyone from using the base libs. Additionally, I'm a little at a loss as yet as to the best way to distribute multiple eclipse projects which are part of a hierarchy. For now you'll need to manually add them to your workspace, which eclipse makes easy by defaulting project files to being hidden. I feel that I've not yet spent enough time with the IDE to complain, but it seems like a good part of a programmers life is fighting the build systems rather than using the language.

Boost removal, improved library support and VS 2012 builds

We've developed a cut-down implementation of the boost::filesystem features we required in our FileSystemUtils.h header, allowing us to remove our dependency on Boost. As ever this is permissively licensed so feel free to use that file if it meets your needs outside of RCC++, and extend / modify as you need.

Additionally, we've added the capability to link libraries at runtime, in both Windows and Mac OS X builds. A simple example for how to use this with the glewfw library is to create a header which you include instead of the glewfw.h header as follows:

#pragma once

#include <GL/glewfw.h>
#include "RuntimeLinkLibrary.h"

#ifdef _WIN32
RUNTIME_COMPILER_LINKLIBRARY( "glewfw.lib");
RUNTIME_COMPILER_LINKLIBRARY( "opengl32.lib");
#endif
#ifdef __APPLE__
    RUNTIME_COMPILER_LINKLIBRARY( "-framework glewfw");
    RUNTIME_COMPILER_LINKLIBRARY( "-framework OpenGL");
#endif

When included by runtime compiled code, the library will be linked during compilation as and when required. Note that for OS X, we need to specify frameworks with "-framework [name]" and "-l[name]" where [name] is the framework or library (see ld documentation on Apple's developer site). You can link to both static and dynamic libraries this way, but the use of dynamic libraries should be preferred. With dynamic libraries only one instance will be loaded by the executable, and resources (such as graphics data) can be preserved without serialization between runtime compiles.

Being able to use libraries is a significant improvement in functionality. For example, core rendering functionality can now be runtime compiled.

Furthermore, we've fixed up the Visual Studio property sheets so the solution will now build with Visual Studio 2012 (including the express version), though you'll need to upgrade the solution so that it uses the default VS 2012 toolset (called v110) if you want or need to. Visual Studio should prompt you to upgrade the solution when opened.

There's quite a lot more, For full list check out the commits..

Mac OS X and Simultaneous Multi Platform Runtime Compiling

With the help of the lightweight cross platform GL windowing library GLFW and the Simple File Watcher, we've added Mac OS X support to Runtime Compiled C++. As an added benefit, you can now run multiple platforms from a shared directory, using one of them to code on the fly. Turn around times on Mac OS X with XCode 4 and Clang++ are excellent, better than windows on a similar system.

Mac OS X 10.8 with XCode 4 running the SimpleTest example natively top right along with a Windows 7 Parallels Desktop Coherence mode version of the example running bottom right. The apps in each OS have just compiled changes made from XCode. Turn-around times are significantly faster when not running the VM.

For the moment you'll need OS X 10.8 and XCode 4 to run the code on the Mac (we've not tested with OS X 10.7 but it might work). If you change the debugger to GDB the exception handling mechanism doesn't work as smoothly as with the default LLDB debugger, since GDB doesn't pass the exception signal on to the app.

We'll have more information on the port, the changes and our plans for the future soon.

Please do give us your feedback on the OS X version, as we've had limited ability to test this on other systems as yet.

Building a Console

Building a Console from RuntimeCompiledC++ on Vimeo.

Many game engines include a command console that allows a user to issue text commands to adjust settings, get debugging information, change the state of the world and so on. The feature opens up a great deal of potential for rapid development.

However, the implementation will usually be based on a scripting language of some kind: either giving access to one already integrated or using simple custom syntax. Both of these represent significant barriers and maintenance costs. To allow you to access all the features of your engine a scripting language must be pervasively integrated, which is a lot of work - and if you've been reading this blog, you know we see a lot of disadvantages in this approach. A simple custom scripting language will also take work and will always be much less powerful.

RCC++ offers a new option: allow the user to give commands directly in C++.

We can do this by taking the text the user has entered and dropping it into a simple .cpp file, forming the content of a standard execute() method. We can then compile this file at runtime and immediately execute it, just once, as a "snippet" of disposable code. By passing in virtual interfaces to our major subsystems we can have access to our existing functionality with very little work. In essence, anything you could do in the C++ code, you can now do on the command console.

The same crash protection discussed before allows us to catch basic crash errors, so we can use it with confidence. We can also go further: if we restrict the symbols and methods we expose, we can make it safer to use; if we write some nicer wrapper methods we can make it easier to use. This opens the way for a console friendly enough for non-programmers.

We've got some other ideas for applications of code "snippets" and making them easier for designers to use - you can expect this topic to return in future posts.

RCC++ at the Develop Conference

Last week we gave a talk on RCC++ at Develop in Brighton. 45 minutes long, it included live demos of everything you've seen on this blog and some features we so far haven't covered. We featured videos of recent integrations with the Dynamic Resolution Rendering demo Doug developed with Intel, and with the popular Recast navigation toolkit that features in my own work. We also received permission to give an overview of Crytek's Softcode implementation, which was based on our work.
We've really appreciated the great feedback we had after the talk and on Twitter. We'll be blogging about those other features and integrations in the coming weeks. In the meantime, you can download the slides.