REX -- A Command Programming Language 18 Feb 1981 """"""""""""""""""""""""""""""""""""" SHARE 56 Abstract """""""" Languages which allow you to combine useful sequences of commands to create new commands are increasing in impor- tance in modern Operating Systems. In CMS, for example, most commands are issued with the aid of EXECs, and editor macros increase the power and flexibility of our Editors. REX is an experimental command programming language which has the syntax and structure of higher level lan- guages, yet which maintains the advantages and power of EXEC 2. EXECs are much more readable when written in REX, and are more easily maintained. The powerful expression evalu- ation and string parsing facilities can greatly reduce the length of EXECs, and the enhanced (and interactive) tracing facilities make debugging easier. What is REX? """""""""""" REX is a command programming language. It can be used as a direct replacement for or alternative to the CMS EXEC and EXEC 2 languages, and as a "Macro processor" for editors etc. REX is a language with useful control structures, string parsing, and general expression evaluation. It also has good tracing facilities (including a powerful interactive debug mode), and is easy to learn and use. It seems that both "end users" and programmers find REX a simple and ef- fective language. Like EXEC 2, REX has several advantages over the original CMS EXEC language: it has considerably enhanced function, it does not tokenise data, and if the Exec involves loops of any kind, then it is significantly faster. The language itself is PL/I-like, and essentially system in- dependent. In its CMS implementation it is usually in- stalled as an alternative to the standard processors. In this mode one can write Execs or Editor Macros in one of three languages: EXEC (standard CMS), EXEC 2 (its replace- ment), or REX. The REX interface will examine the file and pass it on to the appropriate interpreter. (If the file begins with a REX comment it will be interpreted by REX, etc.). This means that REX coexists with both EXEC and EXEC 2 and users may gradually convert to REX without having to change any of their existing Execs or Macros. REX can also be invoked from a program with the data to be interpreted held in stor- age, so avoiding File System overheads. Why REX was designed """""""""""""""""""" The CMS EXEC language (which has since been extended and im- proved upon by EXEC 2) is based on the common macro language principle that variables and controls should be distin- guished (by "&") and literals should exist in "plain text". When Execs consisted mainly of strings of commands, with very little logic in between, this was a fair and sensible choice: however a quick scan through the Execs of almost any modern user quickly shows that the majority of words in use are symbolic (that is, they begin with "&"). This observa- tion must cast some doubt on the validity of using this syn- tax. A further argument is the increasing use of "complicated" strings in Execs: for example embedded blanks are heavily used in Editor Macros; full screen displays; and so on. EXEC 2 handles these quite well, but EXEC cannot manipulate them at all: the user is reduced to unreadable manipulations of the Underscore character or other machinations to achieve the desired result. In REX, blanks can be handled as easily as any other character. Thirdly, the necessity of using upper case characters throughout the EXEC languages makes them awkward to type and difficult to read: it is clear that programs typed in mixed case are, like this paper, easier to follow. Finally, the underlying syntax of the Exec languages makes the efficient interpretation (and, perhaps, compilation) of modern control structures extremely difficult, if not impos- sible. However, such facilities are necessary in order to easily enhance and maintain Execs and macros once they have been written. Therefore there is, perhaps, some justification in investi- gating an alternative macro command language which uses the "more conventional" notation used by the higher level pro- gramming languages such as PL/I, Pascal, ADA, and so on. Experience suggests that a language with this type of syntax will be easier to learn and use than one which is derived from a programmers' "Macro language". Although REX is espe- cially attractive to those who are used to programming, many people in IBM who before would not learn a command or pro- gramming language now use REX. The use of the more conventional syntax will naturally cause users to draw comparisons with their usual programming lan- guages. This inevitably will lead them to expect a corre- sponding improvement in the facilities available in the command language: this in turn would seem to imply that the interpreter might be larger and probably slower than either EXEC or EXEC 2. Size (within reason) is not often a problem on modern virtual machines, however a severe performance penalty would be unacceptable in most environments. Some effort has therefore been made to ensure good performance. During implementation it has been found that REX is of simi- lar size to the existing interpreters (currently it is about 18000 bytes, 10% of which are the error messages). The REX interpreter is somewhat slower than EXEC 2 for triv- ial operations, but for many tasks it is faster. It is usu- ally very much faster than EXEC. What then are the major desirable features for a command macro language? My choices included: 1. Structured flow control statements, some equivalent of If-then-else, Do (Iteration/Until/While)-end, Select- when-end being the most important. 2. Free format, yet not requiring a terminator for every statement. 3. Literal shorthand: unknown "tokens" assumed to be en- closed in quotes, with a natural concatenation mech- anism. 4. Some automatic case translation as required by the host, and no requirement that keywords and variable names be typed in upper case, etc. 5. "Complex" expressions (i.e. parentheses, multiple opera- tors) 6. Built-in parsing facilities for character strings. 7. In line "function" calls to other Execs or Modules. 8. No requirement for self-modifying Execs. 9. "Peer" communication between Execs and programs. These and other features are outlined in the foils that fol- low this paper. <1993: not available now in softcopy--sorry> Mike Cowlishaw, IBM U.K. Laboratories, Hursley Park, Winchester, England. Currently on Sabbatical at IBM T.J.Watson Research Center.