C++ Include test: full matrix

Finally I am able to present some include experiment results! Previously I wrote about Code Generator and now I can actually run this tool and get some numbers out of it. I compared VC11.0 (Visual Studio 2012 For Desktop) and GCC 4.7.2 (using MinGW32 4.7.2 and DevCpp).

Basics of experiment

• VC11.0 - Visual Studio 2012 For Desktop, Release Mode, 32bit, no optimizations.
  • I turned build timings on to have detailed build performance.
• GCC 4.7.2 32bit - MinGW 4.7.2 version and run from DevCpp 5.4.1
  • g++.exe -c testHeaders.cpp -o testHeaders.o -ftime-report
• The machine: Core i5, 4 cores, 4GB RAM, Windows 8 64bit
  • compilation will probably take place only on one core only (there will be one translation unit only)

I run each test 3 times and then compute the average. At the end of post there is a link to detailed spreadsheet.

Code structure

The overall code structure is not a real case scenario. It was my first attempt to do such experiment and it was actually easy to create such hierarchy.

• testHeader.cpp includes N header files
• m-th header file includes N-1 other header files (so we have "cross" include)
• each header file has its proper include guard

Note:  For 199 GCC will return an error "#include nested too deeply". You can read more about it in gamesfromwithin.com. In general GCC thinks that 199 include level is unreal and will block it.

Test N headers

A header can look like this in this tests:

#ifndef _INCLUDED_HEADER_HEADER_5_H
    #define _INCLUDED_HEADER_HEADER_5_H

    #include "header_0.h"
    #include "header_1.h"
    #include "header_2.h"
    #include "..."

generator.exe N 100 includeTestOutput/

N = 100

GCC	3,95s
GCC Compilation	2,99s
VC11.0	2,90s
VC11.0 Compilation	2,68s

N = 132

GCC	5,37s
GCC Compilation	3,98s
VC11.0	4,31s
VC11.0 Compilation	4,11s

N = 164

GCC	6,49s
GCC Compilation	4,92s
VC11.0	6,10s
VC11.0 Compilation	5,91s

N = 192

GCC	7,40s
GCC Compilation	5,77s
VC11.0	7,98s
VC11.0 Compilation	7,77s

Test N headers - additional ifDef

A header can look like this in this tests:

#ifndef _INCLUDED_HEADER_HEADER_5_H
    #define _INCLUDED_HEADER_HEADER_5_H

    #ifndef _INCLUDED_HEADER_HEADER_0_H
        #include "header_0.h"
    #endif
    #ifndef _INCLUDED_HEADER_HEADER_1_H
        #include "header_1.h"
    #endif
    #ifndef _INCLUDED_HEADER_HEADER_2_H
        #include "header_2.h"
    #endif
    #include "..."

generator.exe N 100 includeTestOutput/ ifDef

N = 100

GCC	3,91s
GCC Compilation	2,96s
VC11.0	1,44s
VC11.0 Compilation	1,22s

N = 132

GCC	5,35s
GCC Compilation	3,91s
VC11.0	1,71s
VC11.0 Compilation	1,51s

N = 164

GCC	6,41s
GCC Compilation	4,86s
VC11.0	1,98s
VC11.0 Compilation	1,77s

N = 192

GCC	7,31s
GCC Compilation	5,69s
VC11.0	2,16s
VC11.0 Compilation	1,96s

Test N headers - #pragma once

A header can look like this in this tests:

#pragma once
    #ifndef _INCLUDED_HEADER_HEADER_5_H
    #define _INCLUDED_HEADER_HEADER_5_H

    #include "header_0.h"
    #include "header_1.h"
    #include "header_2.h"
    #include "..."

generator.exe N 100 includeTestOutput/ pragmaOnce

N = 100

GCC	4,02s
GCC Compilation	3,08s
VC11.0	1,48s
VC11.0 Compilation	1,28s

N = 132

GCC	5,42s
GCC Compilation	4,06s
VC11.0	1,84s
VC11.0 Compilation	1,65s

N = 164

GCC	6,64s
GCC Compilation	5,08s
VC11.0	2,06s
VC11.0 Compilation	1,86s

N = 192

GCC	7,60s
GCC Compilation	5,98s
VC11.0	2,39s
VC11.0 Compilation	2,20s

Conclusion

• The code structure is rather theoretical and does not represent 'common' structures that may appear in projects.
• GCC and VC build code a bit different. GCC linker phase is much longer than in VC.
• GCC uses lots of optimization for header files. There is almost no need to do any 'tricks' with includes. Header guards are enough.
• VC likes header files 'tricks'!
  • There is above 2X to 3X speedup using additional include guards or pragma once
  • Pragma once for such code structure seems to be a bit slower than additional include guards: from 5% to even 10% slower.
• All in all VC11.0 is faster than GCC.

What I have learnt

• VC Release mode is not the same as GCC basic compilation
• Make sure you set up proper experiment base
• Automate as much as possible
• Tool for spreadsheet is a must have-software :)
• It is valuable to figure out how to break the compiler

Links

Link to repo
Link to spreadsheet
Article is posted also on the CodeProject

Reposted with permission from Bartłomiej Filipek's blog