Binary File Versioning on Windows

Open discussion on the "DLL hell" problem with OSG and beyond. Oh, and a tentative solution, too.

The Problem

All Windows programmers know how difficult it is to make different versions of the same shared library live together happily. The well-known expression "DLL hell" describes that better than a thousand words. In a nutshell, there is no easy way to ensure that an application will pick up the DLLs? it needs if these are located in a shared location, because other applications may overwrite them with their own. Different versions of a library often result in binaries that are mutually incompatible. Even if the source code of the library is the same, building it with a different compiler may be sufficient to produce a binary file that is not compatible with those generated by other compilers.

Currently, the only safe way to deliver an OSG-based application on Windows platforms is to include all required OSG DLLs? in the application's package, putting them in a private directory not included in the search path (usually it is the same directory as that of the application's main executable). This works but is very, very space-consuming. After all, why did they call them "shared libraries" if you can't actually share them between applications?

The Goal

That said, our goal is to establish a versioning system for OSG so that different versions and variants of OSG DLLs? can co-exist in a shared location without interfering one with each other. It doesn't take a rocket scientist to imagine how this can be done: simply apply a consistent naming scheme to DLL files so that different versions of the same DLL have different names (such as osgUtil-win32-vc7-x.y.z.dll), and applications that were linked against some version x.y.z will actually load that DLL.

As it usually happens in the Windows world, this is easier said than done. A relatively simple task like that described above is not so easy to be turned into practice because MS development tools were just not designed to do that. This is a limitation that often doesn't exist in the Linux world, where the development effort is concentrated into smaller, finer-grained tools that can be combined together beyond the original developer's expectations.

The Theory

If you're a Windows programmer, you already know how DLL libraries work. When you build a DLL you actually get two files: the DLL itself and an "import library", a sort of fake static library that acts like an "index" for accessing the code units (functions, variables, etc.) in the DLL. If you have a program that needs the functions exported by the DLL you need to link it against the import library.

We want import libraries to have an unversioned name, as they currently have, so that the library user doesn't have to change the project settings of his/her application every time OSG is updated. So we need to keep the original name of the import library but rename the DLL to some sort of versioned name (see above). The problem is, if you rename the DLL, the import library will still reference the original name, so any application linked against such import library will not load the "versioned" DLL.

The Solution

If it can't be done with MS tools it doesn't mean it's impossible. We make software after all. I've written a small utility that I'm currently testing on some of my libraries and it seems to be working fine so far. This utility performs the following steps:

determines a versioned name (it can be specified on the command line or it can be read from the DLL itself if it has version information)
copies mylib.dll to mylib-versioned-name.dll
recreates the import library mylib.lib from scratch so that it references mylib-versioned-name.dll instead of just mylib.dll

From the user's point of view this process is completely transparent: applications still need to be linked against mylib.lib, just like before. The difference is that when the application is run, it will try to load mylib-versioned-name.dll, which is exactly the one it needs. This utility can be easily integrated into the VC7?/8 build system as a Post-Build event. Please note that it's better to keep unversioned targets (mylib.dll) and intermediate files in place to avoid breaking the dependency checks.

Embedding Version Information

Step 1 listed above requires a versioned name to be specified. Passing it on the command line requires the command line to be modified every time the library version numbers change. This can be tedious to do manually, especially when you have many libraries as in OpenSceneGraph. Windows executables and DLLs? can have version information embedded in them, so that version numbers can be easily recognized programmatically in a consistent way. I've written another small utility that:

reads a C/C++ header file such as include/osg/Version
extracts version numbers from the header file
opens an existing resource script (.rc) file that contains a version information resource
replaces the version numbers in the .rc file with those extacted from the header file
writes the resource script with new version numbers

This utility can be integrated into the build system as a Pre-Build event. By combining these two utilities one can have consistent and automatic versioning based just on the content of header files like osg/Version.

Putting It All Together

Of course, pre-build and post-build events like those described above must be entered in the project files. A script (yet to be written) can take the original set of project files and create a new, "versioned" set. During this step one should be able to specify any additional tags to be appended to the versioned name (to take customizations into account). For example I might want to build DLLs? with names like osgUtil-win64-vc8-1.2.0-marco.dll where marco indicates that the library was customized by me.

The future build system will probably generate VisualStudio? project files automatically from a set of input files. If the versioning method described in this page actually works, it could be worth considering to implement it in the new build system.

The Utilities

The utilities I've described are updatever and deploybin. Both source code and precompiled binaries are available for download to those who wish to try them out.

Source code: vertools-src.zip
Binaries: vertools-bin.zip

updatever

updatever is the program you need to update a resource script (.rc) file with version numbers extracted from a C/C++ header file. To see a list of options, type updatever -? on the command line.

In order to use this utility you need a C/C++ header that defines version number as follows:

 #define <identifier-for-first-version-number>     <first-version-number>
 #define <identifier-for-second-version-number>    <second-version-number>
 #define <identifier-for-third-version-number>     <third-version-number>
 #define <identifier-for-fourth-version-number>    <fourth-version-number>

The file include/osg/Version follows such scheme. Then you need a resource script file that is going to be compiled and embedded into the target DLL or EXE file. Such .rc file must already contain a version information resource. You can then invoke updatever as follows (note that version.rc doesn't currently exist in the OSG distribution):

 updatever -h path/to/osg/include/osg/Version -r path/to/osg/VisualStudio?/osg/version.rc 
   --id1 OSG_VERSION_MAJOR --id2 OSG_VERSION_MINOR --id3 OSG_VERSION_RELEASE --id4 OSG_VERSION_REVISION

The arguments of options --id1, --id2 etc. must match the identifiers that the header file uses to define version numbers. The result of executing the above command line is a possibly updated version.rc file whose version resource carries the same version numbers as those specified in include/osg/Version.

deploybin

deploybin does the interesting work. It takes a DLL or EXE file, extracts version information from it (if available), makes a copy of it with a "versioned" name and rebuilds the import library so that it "points" to the versioned DLL.

The version tag (the text that is appended to the file name to obtain the versioned name) can be specified on the command line with option -v. Within the user-supplied version tag, any occurence of (n) (where 1 <= n <= 4) is replaced with the corresponding version number extracted from the binary file itself. For example, the following command:

 deploybin -t path/to/osg/bin/win32/osg.dll -v "win32-vc7-1.2"

will create an import library named osg.lib that points to a DLL file named osg-win32-vc7-1.2.dll, while this command:

 deploybin -t path/to/osg/bin/win32/osg.dll -v "win32-vc7-(1).(2).(3)-rev(4)"

will read version information from osg.dll (provided that a version resource was embedded into the DLL, see updatever) and will create an import library that points to osg-win32-vc7-1.2.0-rev0.dll (numbers are those defined in my copy of include/osg/Version). If you want deploybin to make a "versioned" copy of the DLL, just add the -c option followed by the directory where you want the DLL to be stored (or an empty string to use the original DLL's directory).

NOTE: deploybin invokes the DUMPBIN and LIB programs, so make sure they are reachable by the current PATH! This condition should be always satisfied if deploybin is being executed as a Post-Build step from within Microsoft VisualStudio?.

Let's Discuss!

The idea and tentative implementation I've described in this page is something that popped into my mind one morning and that I turned into code on the same day, so it might even be totally wrong. That's why I've put this page on the wiki: I'd like other users to give their impressions and suggestions and report past experience with the versioning problem on Windows. I encourage anyone who is interested in the topic to discuss it on the mailing list and report the most significant parts of the discussion here.

Marco Jez, November 2006