Source directory discovery and turning off RCC++

I've just added support for discovering the source directories when the paths compiled into the modules do not match that of the machine the code is being run from. This is especially useful when using pre-compiled libraries. Once again I'd like to thank the team at Intelligent Artefacts for highlighting this issue and helping to develop the solution. For more information on this see the documentation.

This source code discovery doesn't noticeably increase start up times, but it brought forwards my priorities on the issue of improvements to turning off RCC++. I've introduced two new methods, one which can be used at runtime and another which is a compile flag. Documentation is again available on the wiki.

Details of the changes for both these fixes are found in the pull request here and bug fix here.

Márton Tamás once again has helped finding and fixing a bug with the alternative file watcher API on Windows, and whilst testing this fix I found that the watched paths were not being cleaned of ..'s so I also fixed that. The fixes for these are in the pull request here.

Build Tests, Improved Documentation and more

Build Tests

The team over at Intelligent Artefacts recently asked if I could help them add build tests for their continuous integration testing. These tests are needed to ensure that they know if they'd added code which would breaks runtime compilation. This is a really good addition to the library, and I'm thankful to them for sponsoring the work and allowing me to include the changes under the standard RCC++ zlib license so that anyone can use and modify the code.


There are two new test functions added to IRuntimeObjectSystem and implemented in RuntimeObjectSystem:


struct IRuntimeObjectSystem
{
    // other functions...

    // tests one by one touching each runtime modifiable source file
    // returns the number of errors - 0 if all passed.
   virtual int TestBuildAllRuntimeSourceFiles(
                 ITestBuildNotifier* callback, bool bTestFileTracking ) = 0;

    // tests touching each header which has RUNTIME_MODIFIABLE_INCLUDE.
    // returns the number of errors - 0 if all passed.
    virtual int TestBuildAllRuntimeHeaders(
                 ITestBuildNotifier* callback, bool bTestFileTracking ) = 0;

};

For the most simple use case, the following should be sufficient to perform the required tests:

pRuntimeObjectSystem->CleanObjectFiles();
pRuntimeObjectSystem->TestBuildAllRuntimeSourceFiles( NULL, false );

Have a look at the SimpleTest application example in the source file Game.cpp for a more complete example. This updates the GUI and permits cancelling mid test by closing the application normally. The tests are performed using the Options menu.

Note that the bTestFileTracking  parameter should be false for most circumstances. When true this also tests the file change notification system in RCC++ by changing the modification times on the files. This will then require a full rebuild of these when you next come to normally compiling the code. The TestBuildAllRuntimeHeaders function will test the header file tracking code in RCC++. Both of these tests are mainly for those modifying The RuntimeObjectSystem or RuntimeCompiler libraries.

Running these alongside any normal compilation checks on a build server after check-ins is a great way to ensure you don't break RCC++ by accident without realizing.

The changeset for these changes is here.

Improved Documentation

I've been making some effort to increase the documentation. If you're curious head over to the wiki and take a look. As always feedback appreciated. In future I plan to move to using Doxygen so everything is part of the git repository.

An alternative Windows file watcher

Márton Tamás spotted an issue with the current Windows file watcher, in that it's incompatible with an alternate API, making integration with libraries using that API difficult. Márton has contributed an alternative solution using this API, which can be enabled with the define WIN32_FW_USE_FINDFIRST_API.

And more...

• Matthew Jack at Intelligent Artefacts (and a founder of RCC++) discovered an issue when using code on machine with no development environment. I've submitted the fix here.
• Ethan Trịnh has contributed a compile fix for VS2013.
• Some further fixes, added missing license files for included dependencies and the alternate Windows file watcher API are all listed here.

Added runtime compile of source dependencies, and improved compiler support.

The latest update adds support for the runtime compile of source dependencies, significantly enhancing the amount and type of code which can be runtime compiled. Additionally, I've improved compiler support amongst other changes. Full details below.

Runtime compile of source dependencies

If you have a header which adds a source dependency, and you also mark that header as being runtime modifiable, the runtime object system will now track both the header and the source file for changes and recompile them along with any runtime compiled code which includes the header.

This allows pretty much any code to be runtime compiled so long as the source and header names are in the format name.h and name.cpp, and you can add the following to the header:

#include "RuntimeSourceDependency.h"

RUNTIME_COMPILER_SOURCEDEPENDENCY;

#include "RuntimeInclude.h"

RUNTIME_MODIFIABLE_INCLUDE;

Note that in order for this to work properly, the header must be included from a source file which uses the REGISTERCLASS macro on an IObject based class  (either directly or via another header). This is because the program needs to preserve state somehow across compiles, and currently the IObject class handles this.

Take care using this feature, as it can be easy to introduce changes which break the code - though in practice I've found this very useful.

Improved compiler support

• Added support for Visual Studio 2013 - however this is as yet untested.
• Added support for cross compile to 32bit on 64bit Linux/MacOSX systems, thanks to Gaoren Xu for discovering this issue and helping with the solution.

Singleton support

At the request of a project using RCC++ I've improved support for singleton type classes. This is especially useful as a good practice in using RCC++ in a performant way is to have singleton system classes which operate on arrays of data.

The REGISTERSINGLETON macro takes a class name and a boolean as arguments, where the boolean controls auto construction of the class. This macro causes the constructor to only ever make one instance of the class. For an example look at the MainObject.cpp file in SimpleTest where we use the macro as:

REGISTERSINGLETON(MainObject, true);

to auto construct an instance of the MainObject class during the Initialise call of RuntimeObjectSystem.

Cleaning build temporaries

Build temporaries can now be cleaned up using the RuntimeObjectSystem::CleanObjectFiles() call, which uses the new FileSystemUtils::PathIterator class. The demos currently do this when the program ends.

Spelling fix, improved module support and Windows fast compile option.

We've made a few changes, one of which alters the interface for the recently introduced runtime source dependency. The main changes are:

1. The macro for the runtime source dependency has had it's spelling corrected, and is now RUNTIME_COMPILER_SOURCEDEPENDENCY (no longer missing the second D). We've also cleaned up the internals as several different wrong spellings of 'dependency' were in use!
2. On initialization the runtime object system now directly uses the PerModuleInterface::GetInstance(), rather than querying the executable. This way RCC++ can be used in a DLL in the project, and multiple DLL's are supported though you will need to explicitly call RuntimeObjectSystem::SetupObjectConstructors() for each one.
3. On Windows, the compiler now kills the compile cmd process by default when complete. This adds about 0.3s to the startup of following compiles due to running the batch file for Visual Studio to set up the environment variables, however it resolves an issue where the cmd process was holding onto handles (for example sockets) the main process wanted to close. This is especially a problem if the main process terminated abnormally, as a zombie cmd process is then left behind. To get faster compiles by keeping the cmd process alive (with the risks this entails), call the RuntimeObjectSystem::SetFastCompileMode() function with true as the parameter.

The full change list can be found here.

Runtime Source Dependencies and Linux Runtime Include bug fix

The latest code on Github includes a new feature - Runtime Source Dependencies - along with a bugfix for Linux builds which resolves an issue with Runtime Includes.

A runtime source dependency is a source file which you need to have compiled when a given header is included by a runtime modifiable source file. Whilst having the source in a library and using the Runtime Link Library feature solves this problem to an extent, sometimes you don't want to create a whole library and thus the ability to compile in dependencies is a really useful feature. Using this feature simply requires the following lines in a header:

#include "RuntimeSourceDependency.h"
RUNTIME_COMPILER_SOURCEDEPENENCY;

If the header is called SomeFeature.h then the source file SomeFeature.cpp is compiled when any runtime modifiable code is changed which includes this header. Using the same filename as the header is required at the moment, in part due to issues with getting full paths from builds on Linux with GCC.

This brings us to the bug fix on Linux - although we'd implemented a system to get the full path for runtime modified source files from GCC, this hadn't been implemented for runtime includes and making the above changes caught this bug, so it's now been fixed. GCC only embeds the path passed in for __FILE__, which can be a relative path from the compile location, so we have to embed the compile path using the pre-processor define COMPILE_PATH="$(PWD)/"; Note that this isn't required if your build system uses full paths, which the cmake builds do, as does the internal GCC runtime compiler.

Note that I'm experiencing an odd issue on my cmake builds on Linux with makefiles - the Eclipse generated files work well however the makefile ones compile but the SimpleTest program fails to create a glfw window.

Added cmake files

Thanks to Anders Buus we now have a set of cmake files for the project.

Linux port of SimpleTest game example

The Pulse demo from project SimpleTest running on Ubuntu Linux compiling changes at runtime.

The SimpleTest example project, otherwise known as the Pulse game demo, now works on Linux with the Eclipse IDE. There's a known bug in that the app shut down appears to crash, which I'll investigate and fix as soon as possible. Grab the latest from Github to see it in action.

[Edit 2013/03/12:] The crash on exit bug for the SimpleTest example has now been fixed.

Linux port with Eclipse progress update

Yesterday I started porting RCC++ to Linux, using the Eclipse IDE as this seemed the favourite option, or at least that's what's been specifically asked for. It's been a while since I've done any serious Linux/Unix coding, and that was in the days before decent IDEs using vi and simple makefiles, so it took a fair few hours to get done. There's now a working version of the runtime compiler and ConsoleExample in my personal fork of RCC++, which I'll pull onto the main fork after some further testing.

[EDIT 2013/11/03]: The code (with a few fixes) has now been pulled into the main fork at Github.

The base RCC++ libaries and the ConsoleExample have no dependencies, so the most difficult problem was getting __FILE__ to reliably generate full paths, since Eclipse builds each project from the path of the output. This required embedding the compile path using COMPILE_PATH=\"$(PWD)/\" as a preprocessor define option (-DCOMPILE_PATH=\"$(PWD)/\" on the gcc command line), which uncovered some bugs in my FileSystemUtils header. Note that the executable needs to be build with the flag -export-dynamic in order to allow symbol resolution by name.

I've not yet ported the rather complex SimpleTest (which needs renaming) due to the dependencies, but this shouldn't stop anyone from using the base libs. Additionally, I'm a little at a loss as yet as to the best way to distribute multiple eclipse projects which are part of a hierarchy. For now you'll need to manually add them to your workspace, which eclipse makes easy by defaulting project files to being hidden. I feel that I've not yet spent enough time with the IDE to complain, but it seems like a good part of a programmers life is fighting the build systems rather than using the language.

Boost removal, improved library support and VS 2012 builds

We've developed a cut-down implementation of the boost::filesystem features we required in our FileSystemUtils.h header, allowing us to remove our dependency on Boost. As ever this is permissively licensed so feel free to use that file if it meets your needs outside of RCC++, and extend / modify as you need.

Additionally, we've added the capability to link libraries at runtime, in both Windows and Mac OS X builds. A simple example for how to use this with the glewfw library is to create a header which you include instead of the glewfw.h header as follows:

#pragma once

#include <GL/glewfw.h>
#include "RuntimeLinkLibrary.h"

#ifdef _WIN32
RUNTIME_COMPILER_LINKLIBRARY( "glewfw.lib");
RUNTIME_COMPILER_LINKLIBRARY( "opengl32.lib");
#endif
#ifdef __APPLE__
    RUNTIME_COMPILER_LINKLIBRARY( "-framework glewfw");
    RUNTIME_COMPILER_LINKLIBRARY( "-framework OpenGL");
#endif

When included by runtime compiled code, the library will be linked during compilation as and when required. Note that for OS X, we need to specify frameworks with "-framework [name]" and "-l[name]" where [name] is the framework or library (see ld documentation on Apple's developer site). You can link to both static and dynamic libraries this way, but the use of dynamic libraries should be preferred. With dynamic libraries only one instance will be loaded by the executable, and resources (such as graphics data) can be preserved without serialization between runtime compiles.

Being able to use libraries is a significant improvement in functionality. For example, core rendering functionality can now be runtime compiled.

Furthermore, we've fixed up the Visual Studio property sheets so the solution will now build with Visual Studio 2012 (including the express version), though you'll need to upgrade the solution so that it uses the default VS 2012 toolset (called v110) if you want or need to. Visual Studio should prompt you to upgrade the solution when opened.

There's quite a lot more, For full list check out the commits..