First tests

The benchmark suite was introduced to test claims that were being made by vendors that their BASIC was much faster than others running on the same machines. The authors were unable to find a set of standardized benchmarks and decided to write their own.^[1] The tests deliberately ignored string and floating point performance, as many BASICs of the era, especially those descended from Tiny BASIC, lacked these features. Strings were further hindered by major differences in syntax between different versions.^{[2][lower-alpha 1]}

The two ran the benchmarks on every machine they could find, typically at friends' houses. All of the major 8-bit CPUs were tested, including the Intel 8080, Zilog Z80, Motorola 6800 and MOS 6502. For comparison, they also ran it on a CDC Cyber 174 supercomputer, which is so fast that they had to add code to time the run using the system's clock as they could not operate a stopwatch quickly enough. They used this machine for two reasons; one was simply to see how fast such a machine ran, and the other was to ensure no one micro would end up at the top of the list and then claim they had been proven to be the fastest.^[3]

The article makes a special note of the BASIC that shipped with IMSAI computers, as it basically didn't work. The machine was very new at the time and the BASIC was described by the company as "preliminary". During this period, different IMSAI machines were delivered with different versions of BASIC. Each version had different features, but none of them were able to run even the majority of the programs.^[4]

The Apple II, using the original Integer BASIC, finished well in front of all the other machines. As the tests did not make use of any floating point features, this result was not surprising given the much simpler internal representation of numbers.^{[lower-alpha 2]} The next fastest, albeit significantly slower, was Zapple BASIC on a Zilog Z80 add-in card in an Altair 8800. The rest of the list contained a number of very closely spaced entries, dominated by what would later be known as Microsoft BASIC. Across the entire suite of 8080 and Z80 machines and versions of BASICs, the spread was only 20%. In contrast, the 6800-based machines were isolated entirely at the bottom of the list, 30 to 40% slower than the 8080 and Z80 entries.^[4]

As the article went to press, the editors of Kilobaud visited several local computer stores to test the programs on newer machines. This process added the production version of IMSAI BASIC, North Star BASIC and the 11 kB BASIC on the Poly-88. IMSAI's results were at the bottom of the pack, while North Star and Poly were in the middle.^[5]

Follow-up

Rugg and Feldman revisited the suite in the October 1977 issue. The new article opened by noting they had been deluged with mail about the original article. Among them was a letter from Bill Gates, who they introduce as the author of "Altair BASIC (8080 and 6800 versions), OSI BASIC and PET BASIC". The last entry refers to a member of the "1977 trinity" of machines, the Commodore PET, which was made available in prototype form for inclusion in the article.^[6]

Gates complained that the original test series "let an integer BASIC be compared against... more powerful BASIC... using floating point." He suggested that the test include a DEFINT A-Z at the start, which would make newer versions of Altair BASIC use integer math as well. Instead, Rugg and Feldman took another approach and eliminated all integer-only BASICs from the new test results.^[6] This did not eliminate the Apple II, which by this time had introduced the MS-derived Applesoft BASIC on cassette.^[7]

Gates also noted that the results for the 6800 machine were not indicative of this processor. The machine they used, the Altair 680, runs the CPU at half its rated speed. He suggested a more typical 6800-based machine would be slightly faster than the 8080. However, the 6800 once again put in a poor showing even on newer machines running at higher speeds.^[6]

Gates also suggested that the Cyber 173's time was likely due to it being a compiler rather than an interpreter. The authors point out this was not the case for the TRW BASIC they used in the original test, and then use this as a segue to compare the differences between compilers and interpreters.^[6]

Gates concluded his letter by noting that a number of the results were identical in spite of being on different machines. He suggests this is because the BASICs in question contained "signatures" from Altair BASIC and are thus "illegitimate software". He did not specify which ones he claimed were stolen, and the authors responded by saying that if "Bill can stop people from selling them through legal means, we'll stop listing them."^[8]

Rugg and Feldman conclude the article by mentioning some of the other concerns that were raised after the original article. One common issue was the lack of more advanced mathematical functions, which they acknowledge, but suggest this is something best left to the reader. The other was the lack of string manipulation, but they note that the syntax of string handling differed considerably between platforms and thus could not be made in a single version.^{[8][lower-alpha 3]}

In this series of tests, the list was topped by the OSI Challenger, a 6502-based machine that had been "souped up" to 2 MHz, double that of typical 1 MHz 6502 machines of the era like the Apple II and PET. When running at its normal 1 MHz speed, the Challenger was just beaten by Zapple BASIC on Z80 machines running at 4 MHz. PET BASIC was next, only slightly behind the Challenger.^[9] They conclude that the 6502 is the highest performing of the CPUs, agreeing with comments Gates had made in his letter. The 6800 once again ends up in last place.^[10]

PCW version

As part of a longer article discussing new entries into the computer market, including the TRS-80, John Coll used the Rugg/Feldman tests to benchmark a variety of machines available to him in the UK in October 1977. He added an eighth test to exercise the math routines, and provided the resulting run times both on their own as well as the additional time compared to Test 7, in keeping with the earlier concept of each test modifying the last.^[11] The results were published in the first issue of Personal Computer World in February 1978,^[12] with a short follow-up in their November 1978 issue.^[13]

Use

As one of the earliest sets of BASIC benchmarks, the F/R tests were seen primarily in the late 1970s and early 1980s. It was a standard among reviews in Kilobaud, used to compare the many new varieties of BASIC that continued to appear for early microcomputers.^[14] Compute! used it for their 1979 review of the Challenger 1P,^[15] and 68 Journal used it to demonstrate the extremely high performance of BASIC09.^[16] InfoWorld used it for their 1981 review of a new BASIC for the TRS-80,^[17] and the TRS-80 Color Computer as a whole.^[18]

After that point the Byte Sieve began to become popular and the number of articles referring to the F-R benchmark become less common, but it could be found even such rarefied sources as the HP Journal.^[19]

The PCW versions remained very popular in the UK, and can be found in many reviews of UK-centric machines like the Grundy NewBrain, various Sinclair Research machines,^[20] and the BBC Micro.^[21]

Benchmark 1

300 PRINT"S"
400 FOR K=1 TO 1000
500 NEXT K
700 PRINT"E"
800 END

FOR-NEXT loops are one of the most fundamental constructs in the BASIC language, and if the performance of these loops is slow it is highly likely that any program running in that BASIC will be slow as well. A famous example of this is Atari BASIC, which had several problems that greatly slowed the performance of FOR-NEXT loops compared to contemporary examples, and BASIC programs on the Atari were generally very slow as a result.^[23]

Benchmark 2

300 PRINT"S"
400 K=0
500 K=K+1
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

This test is ultimately performing the same operations as test 1, but in this case it uses an explicit test and jump rather than using the built-in FOR-NEXT construct. Generally, this program runs much slower than 1 because most BASICs push the address of the FOR on a stack so the NEXT is relatively fast, whereas using a GOTO requires the interpreter to scan through the program for the corresponding line number, in this case 500. Although this test does not show it due to its small size, this search takes increasingly long times as the program length grows. Some versions of BASIC optimized GOTO using explicit labels,^[24] or pushing the GOTO targets on a stack to make them perform like NEXT.^[25] Turbo-Basic XL did both, and ran much faster than any similar BASIC as a result.^[24]

Benchmark 3

300 PRINT"S"
400 K=0
500 K=K+1
510 LET A=K/K*K+K-K
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

Test 3 is an expansion of test 2, this time adding some basic mathematics and variable access. By comparing the time to run tests 2 and 3, one can get an idea of the performance of the language's math library.^[1]

Benchmark 4

300 PRINT"S"
400 K=0
500 K=K+1
510 LET A=K/2*3+4-5
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

The same as test 3 except that the variable K is replaced by numeric constants. This requires the interpreter to convert the values from their textual representation into their internal storage format, which takes time (unless this is performed once and for all, before execution, as in some more advanced BASICs). Some idea of the performance of this conversion functionality can be determined by comparing this time to benchmark 3.^[1]

Benchmark 5

300 PRINT"S"
400 K=0
500 K=K+1
510 LET A=K/2*3+4-5
520 GOSUB 820
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

Test 5 introduces a subroutine call. Long programs in early versions of BASIC would make extensive use of subroutines and thus the efficiency of the calling mechanism was important. Depending on the way the system worked, the return might cache the location of the calling line in a fashion similar to NEXT, and thus run very quickly. Others might store the line number of the calling line, and thus require the code to scan the program listings to RETURN.^[1]

Benchmark 6

300 PRINT"S"
400 K=0
430 DIM M(5)
500 K=K+1
510 LET A=K/2*3+4-5
520 GOSUB 820
530 FOR L=1 TO 5
540 NEXT L
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

Test 6 defines a small array at the start and adds another FOR-NEXT loop inside the main loop. This has little effect on the code, but is used to set a baseline for test 7.^[1]

Benchmark 7

300 PRINT"S"
400 K=0
430 DIM M(5)
500 K=K+1
510 LET A=K/2*3+4-5
520 GOSUB 820
530 FOR L=1 TO 5
535 M(L)=A
540 NEXT L
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

This assigns a value to each of the array elements every time through the loop. Comparing the time needed to run 7 to 6 indicates the efficiency of array access.^[1]

Benchmark 8

300 PRINT"S"
400 K=0
500 K=K+1
530 A=K^2
540 B=LOG(K)
550 C=SIN(K)
600 IF K<100 THEN 500
700 PRINT"E"
800 END

Test 8 was added by PCW,^[11] performing several transcendental functions in order to test their performance. This code does not include the code from test 7, which breaks the original pattern of each test adding to the last. Instead, the associated article listed both the time to run 8 and 7 and 8 combined.^[12]

Citations

Bibliography

• Rugg, Tom; Feldman, Phil (June 1977). "BASIC Timing Comparisons... information for speed freaks". Kilobaud. pp. 66–70.
• Rugg, Tom; Feldman, Phil (October 1977). "BASIC timing comparisons... revised and updated". Kilobaud. pp. 20–25.
• Coll, John (February 1978). "Direct Addressing: Where to get your personal computer". Personal Computer World. pp. 55–58.
• Coll, John (November 1978). "Benchmarks Again". Personal Computer World. p. 52.