think 10 The ‘telco’ benchmark
A simplified billing application

This benchmark was devised in order to investigate the balance between Input and Output (I/O) time and calculation time in a simple program which captures the essence of a telephone company billing application. It does not attempt to cover a wide spectrum of decimal operations and so is not suitable for benchmarking decimal implementations; modern applications often need much greater precision and also alignment, division, conversion, and rounding operations which do not appear in this benchmark.

In summary, the benchmark reads a large input file containing a suitably distributed list of telephone call durations (each in seconds). For each call, a charging rate is chosen and the price calculated and rounded to hundreths. One or two taxes are applied (depending on the type of call) and the total cost is converted to a character string and written to an output file. Running totals of the total cost and taxes are kept; these are displayed at the end of the benchmark for verification.

Full details and a discussion of the design parameters are included in The ‘telco’ benchmark specification. Links to the sample data files may be found below.

The following table gives some initial results, derived using the sample benchmark programs linked from the table. Comments and corrections are invited.

Initial results

In this table, each implementation is shown on a separate row. The columns are:

  1. The language used in the implementation.
  2. A link to the sample source code for the benchmark.
  3. The percentange of the benchmark time which is spent in the decimal arithmetic calculations (including roundings and conversions). For measurements in this column the input and output files are on a local disk.
  4. As the third column, except that the input and output files are on a different machine (connected via fast Ethernet). This scenario gives a ‘worst case’ I/O time (and hence the ‘best case’ calculation percentage). A database implementation would be between the two cases.
  5. Implementation-specific notes.
Language Source Local File I/O Ethernet I/O Notes
C cTelco.c 72.2% 50.3% Compiler: VisualAge(TM) for C++ for Windows(R), Version 3.5
Input file: expon180.1e6
Uses the decNumber.c package for calculations. DECDPUN=4.
C cTelco754r.c 73.4% 50.8% Compiler: GCC-3.2-core-20020817-1 (MinGW 2.0.0)
Input file: expon180.1e6r (decimal64 format numbers)
Uses the decNumber.c package for calculations. DECDPUN=4.
C# csTelco.cs 76.4% 56.9% Compiler: Visual C# .NET Compiler version 7.00.9466
Input file: expon180.1e6b
Uses the native decimal type (implemented by the .Net Decimal class) for calculations.
Java 93.2% 88.2% Compiler: javac (in Java 1.3)
Input file: expon180.1e6b
Uses the java.math.BigDecimal class for calculations.

General Notes:

  1. All timings used the best of three measurements.
  2. The local machine for all tests except cTelco754r was an IBM Intellistation Model 6889 (450MHz Pentium II, SCSI disks, 256MB RAM). For the cTelco754r test, a Shuttle X (3GHz, Pentium 4, ATA disk, 1GB) was used.
  3. For the Ethernet I/O tests, the data files were held on a Thinkpad T21 (800MHz Pentium III, IDE disk, 256MB RAM), connected by a 100MHz Ethernet link via one hub.
  4. The I/O time (local I/O) averaged over all three implementations on the Intellistation was 2.75 microseconds per number.
  5. The tax calculations were between 35% and 50% of the total calculation time, depending on the implementation.

Other results

Timings summary   (local file I/O unless stated)
Author Language Results
William M. Klein Standard COBOL with calculations: 23.08s
‘skipping’ calculations: 8.31s
calculations time: 64%
Using COMP-5 binary temporaries with calculations: 13.59s
‘skipping’ calculations: 7.29s
calculations time: 46%
Glenn Someone Turbo Pascal 7.0, using scaled long ints with calculations: 29.55s
‘skipping’ calculations: 21.80s
calculations time: 26%

Data files

Two data collections are available for use with the benchmark source files; one is a short test file comprising ten numbers; the other contains one million numbers (7.63MBytes). Each file is available in three formats: 8-byte packed decimal, 8-byte binary, and the decimal64 format from the IEEE 754r draft (one number in every eight bytes in all three cases).

All files were generated with an exponential distribution with mean set to 180; for the larger files the actual mean is 180.2 and the maximum value is 2641.

Here are the links to the files. Be sure to use ‘Save As’ or ‘Save Target As’ for the test files (as well as for the zipped files) to avoid line-ending problems due to browser/platform behaviour.
  Packed decimal Binary decimal64
10-number test file telco.test telco.testb telco.testr
Million-number file
expon180.1e6 expon180.1e6b expon180.1e6r

Here is a worked example (using the numbers from the telco.test files) to aid verification of new implementations of the benchmark:

    Time Rate | Price   Btax   Dtax  | Output
     308   L  |  0.40   0.02         |  0.42
      76   L  |  0.10   0.00         |  0.10
      68   L  |  0.09   0.00         |  0.09
     323   D  |  2.89   0.19   0.09  |  3.17
      82   L  |  0.11   0.00         |  0.11
     329   D  |  2.94   0.19   0.10  |  3.23
      14   L  |  0.02   0.00         |  0.02
     101   D  |  0.90   0.06   0.03  |  0.99
     522   L  |  0.68   0.04         |  0.72
      50   L  |  0.06   0.00         |  0.06
     Totals:  |  8.19   0.50   0.22  |  8.91
To verify correct rounding rules, etc., the final totals for the million-number files must also match the values shown in the specification.

Note: Not all rounding cases are covered in the small test file. A round-down-to-even case is covered in the last record (50 × 0.0013 → 0.065 which is truncated to 0.06 because 6 is even), but a round-up-to-even case is not covered, for example 550 × 0.0013 → 0.715 which should be rounded up to 0.72. The latter example can be found as the 5873rd entry in the main test file (at offset hex 16F0).

Please send any comments or corrections to Mike Cowlishaw,
Copyright © Mike Cowlishaw 2002, 2012. All rights reserved.
Parts Copyright © IBM Corporation 2002, 2006. All rights reserved.
Java is a trademark of Sun Microsystems Inc.