============================================================================= Notes about performance benchmarks ============================================================================= For each benchmark, here is a brief description and notes about its strengths and weaknesses. ----------------------------------------------------------------------------- Artificial stress tests ----------------------------------------------------------------------------- bigcode1, bigcode2: - Description: Executes a lot of (nonsensical) code. - Strengths: Demonstrates the cost of translation which is a large part of runtime, particularly on larger programs. - Weaknesses: Highly artificial. heap: - Description: Does a lot of heap allocation and deallocation, and has a lot of heap blocks live while doing so. - Strengths: Stress test for an important sub-system; bug #105039 showed that inefficiencies in heap allocation can make a big difference to programs that allocate a lot. - Weaknesses: Highly artificial -- allocation pattern is not real, and only a few different size allocations are used. sarp: - Description: Does a lot of stack allocation and deallocation. - Strengths: Tests for a specific performance bug that existed in 3.1.0 and all earlier versions. - Weaknesses: Highly artificial. ----------------------------------------------------------------------------- Real programs ----------------------------------------------------------------------------- bz2: - Description: Burrows-Wheeler compression and decompression. - Strengths: A real, widely used program, very similar to the 256.bzip2 SPEC2000 benchmark. Not dominated by any code, the hottest 55 blocks account for only 90% of execution. Has lots of short blocks and stresses the memory system hard. - Weaknesses: None, really, it's a good benchmark. fbench: - Description: Does some ray-tracing. - Strengths: Moderately realistic program. - Weaknesses: Dominated by sin and cos, which are not widely used, and are hardware-supported on x86 but not on other platforms such as PPC. ffbench: - Description: Does a Fast Fourier Transform (FFT). - Strengths: Tests common FP ops (mostly adding and multiplying array elements), FFT is a very important operation. - Weaknesses: Dominated by the inner loop, which is quite long and flatters Valgrind due to the small dispatcher overhead. tinycc: - Description: A very small and fast C compiler. A munged version of Fabrice Bellard's TinyCC compiling itself multiple times. - Strengths: A real program, lots of code (top 100 blocks only account for 47% of execution), involves large irregular data structures (presumably, since it's a compiler). Does lots of malloc/free calls and so changes that make a big improvement to perf/heap typically cause a small improvement. - Weaknesses None, really, it's a good benchmark.