Some files may exist independently of the execution of a program, so that a program can create a file that can later be accessed by another program. Thus, files can provide continuity from one program execution to another.

Every file has a name, which is represented in a MAINSAIL program as a STRING. The correspondence between files and STRINGs may not be one-to-one.

MAINSAIL distinguishes between text files and data files. The two kinds of files differ in how non-character, non-STRING data are read and written from the file: a text file translates to or from human-readable STRING representations of the data, whereas data files read or write the data using their native machine representations.

MAINSAIL distinguishes two methods of access to a file: sequential and random. The current position in a sequential file is updated to point to the next datum as each datum is read or written, in order, starting from the beginning. The current position of a random file may be explicitly changed to be anywhere within the file.

Before a file can be used by a program, it must be opened by a call to the system PROCEDURE open. Arguments to the open PROCEDURE specify the file name and indicate how the file is to be accessed (sequentially or randomly, for input and/or output, etc.).

A file is closed by a call to the system PROCEDURE close to indicate that the program no longer intends to use it (unless the program reopens the file later).

A file is referenced in a MAINSAIL program by means of a POINTER returned by the open PROCEDURE. The POINTER belongs to one of the predeclared CLASSes textFile and dataFile.

The system PROCEDUREs shown in Table 22–1 may be used to manipulate files.

22.1. File Names

On every operating system, the name TTY (case is not distinguished) refers to the operating-system-dependent primary input or output file, usually the user's terminal. TTY is a text file that may be opened either for sequential input or sequential output.

Valid MAINSAIL identifiers (other than TTY) of six or fewer characters are guaranteed to be valid operating-system-dependent file names on any system. It is not guaranteed whether the operating system distinguishes case; e.g., ABC and abc may or may not refer to distinct operating system files. The $attributes bit $fileNamesAreCaseSensitive (see Section 30.26) is set if the operating system treats files names of different case as names of different files.

In some cases (e.g., with open MODULE libraries to see if a library is already open), MAINSAIL must compare two file names to see if they are the same. In such cases, MAINSAIL always uses a case-sensitive comparison, regardless of the setting of the bit $fileNamesAreCaseSensitive in $attributes.

22.2. The CLASSes file, textFile, and dataFile

The only fields of the CLASS file available to a user program are the fields name, $deviceAttributes, and $strArea. $strArea is described in Chapter 25. name is set to be the name specified to the open PROCEDURE (or the substituted name if a logical name or searchpath substitution occurs) when the file is opened. The name may be further changed by some device MODULEs to reflect the “real” name of the file opened; the documentation for any device MODULE that modifies the name field explains the modifications made.

As an example, if outFile is a POINTER declared as:

POINTER(textFile) outFile;

and a file named RESULT is opened as:

open(outFile,"RESULT",...)

then:

outFile.name = "RESULT"

assuming there is no logical name or searchpath that matched RESULT.

The $deviceAttributes field is a LONG BITS field that describes the characteristics of the device on which the file resides. Currently, the only bit that may be set in this field is $hasFileVersions; if set, the device supports multiple numbered versions of files, as on the VAX/VMS operating system.

The effect of altering the name and $deviceAttributes fields or of accessing undocumented fields of the CLASSes file, textFile, and dataFile is not defined.

22.3. Text Files

When (LONG) INTEGER, (LONG) REAL, or (LONG) BITS data are written to a text file (with the system PROCEDURE write), an automatic conversion is made to the appropriate STRING representation of those data, and then the STRING is written to the file. For example, if r is a REAL variable with the value 123.8, and f is a text file, then write(f,r) converts the number 123.8 to the STRING "123.8" and writes the STRING to the file. Only the five characters of the STRING representation are written; i.e., MAINSAIL does not automatically write any spaces or end-of-line characters to separate numbers.

A STRING is written to a text file by writing its characters into the file. For example, write(f,"a4$") writes the three characters of the STRING "a4$" into the text file f. An eol is not automatically appended; eol may be explicitly written to a text file like any other STRING.

When the non-STRING data types are read from a text file (by means of the system PROCEDURE read), a scan for an appropriate STRING representation takes place, and when found, a conversion is made to the appropriate internal representation. For example, if i is an INTEGER variable, then read(f,i) causes a scan of the file referenced by f for a proper STRING representation of an INTEGER (that is, one or more digits, possibly preceded by -). If the digits found are 123, then the number 123 is assigned to the INTEGER variable i. The scan skips over characters that do not form part of the numeric representation. See the description of read in Section 46.7 for more details.

A STRING may be read from a text file using read, fldRead, or scan.

On input, certain characters (typically NUL and, on some systems, carriage return) are discarded by default. The setting of the keepNul bit, passed to open, controls this behavior; see Section 43.5 for details.

22.4. Data Files

Since no conversion to STRING is necessary, input and output of BOOLEAN, (LONG) INTEGER, (LONG) REAL, and (LONG) BITS values to data files are usually more efficient than to text files.

Characters written to data files as parts of STRINGs or individually with the PROCEDURE cWrite are stored one per character unit, as in a text file. See Sections 32.66 and 32.97.

22.5. Input and Output

A file opened for sequential access may be opened for either input or output, but not both at once. A file opened for random access may be opened for input or output or both.

22.6. Sequential and Random Access

When a file already exists and is opened for sequential access, it can be opened only for input (unless the entire file is replaced). Each read from the file obtains data starting from the current position (starting with the beginning position of the file) and then sets the file pointer to the position following the data read, as determined by the data type being read.

A random file allows access to any position within the file, as specified by the positioning PROCEDUREs setPos, relPos, and getPos. The positioning may be interspersed with normal sequential reads and/or writes. A random file is like a sequential file except that it may allow both reads and writes, an existing file can be altered without replacing the entire file, and the position within the file can be controlled by explicit positioning PROCEDUREs.

A file opened for random output is automatically extended if the current position is set to be beyond the end of the file.

Not all file formats allow random access. An error message is generated when a random open is attempted on a file that does not permit random access.

The contents of unwritten locations in a random access file are Zero.

Some operating system file systems have separate file organizations for sequential and random files. It may not be possible to open a sequentially organized file for random access. MAINSAIL guarantees to use the random file organization only if a file is opened for random access when it is originally created.

22.7. End-of-File Positions

Some operating systems have a separate file organization for sequential files (“record-oriented files”). On such files, the end-of-file position can be found, but not without reading the entire contents of the file. For such files, the system PROCEDURE $getEofPos (see Section 36.10) is undefined; however, the system PROCEDURE $gotValue (see Section 36.21) works correctly for such files.

22.8. Opening a File

open(inFile,"notes",input)

opens the file named notes for sequential text input. input is a predefined BITS constant used by the open PROCEDURE. The open PROCEDURE produces as its first argument a POINTER belonging to one of the predeclared CLASSes textFile or dataFile. That POINTER is used for subsequent references to the file. For example, if s is a STRING variable, then:

read(inFile,s)

reads the next line from the file notes into the STRING s.

A file name can be specified during execution. If an error occurs while a file is being opened, then by default, MAINSAIL prompts for and reads a new file name from cmdFile. The prompt bit may also be specified so that MAINSAIL always prompts, as described in Section 43.5.

The same file may be opened more than once (i.e., several calls to open may be made for the same file, resulting in several different file POINTERs referring to the same file), provided that none of the file POINTERs is opened for write access; otherwise, the effects are undefined.

Some operating systems do not permit the creation of a zero-length file. Thus, if a file is opened for create access but no data are written to it before it is closed, the file is not guaranteed to exist after it is closed.

File names may be logical names. A real file name is substituted for a logical file name when the file is opened. Logical file names allow redirection of specified files and the use of system-independent file name forms. Logical names are established by a call to enterLogicalName and examined by a call to lookUpLogicalName.

Any STRING may be used as a logical file name. XIDAK's own logical file names are written enclosed in parentheses. To avoid conflict with XIDAK logical names, it is suggested that each organization adopt its own convention; e.g., XYZ Corporation might choose names like:

XYZ:main library

Logical file names may contain spaces, although some XIDAK utilities use space as a separator character. Real file names must be specified to such utilities if the corresponding logical file name contains a space.

In addition to logical file names, file searchpaths affect the name actually used to open a file. A searchpath is set up using the MAINEX subcommand SEARCHPATH; see Chapter 21 of the MAINSAIL Utilities User's Guide.

22.9. Closing a File

MAINSAIL automatically closes all files that remain open at the end of a MAINSAIL execution.

22.10. Terminal I/O and Primary Input and Output

Terminal input and output may have operating-system-dependent characteristics; consult the appropriate operating-system-dependent MAINSAIL user's guide.

Most operating systems provide editing commands (e.g., a backspace key to correct mistakes) that may be applied to a line before it is entered from a keyboard, but MAINSAIL does not guarantee that this is the case.

If an end-of-file is detected by ttyRead, and no text was read by the invocation of ttyRead before the end-of-file, then the exception $ttyEofExcpt is raised. If it is not handled and $raiseReturn is not called, then MAINSAIL exits with the message:

Eof on TTY: exiting

This helps avoid infinite loops in batch jobs that mistakenly loop while reading beyond the end of the batch script. To deal explicitly with a terminal end-of-file in a program, do something like the following:

$HANDLE s := ttyRead
$WITHB
    IF $exceptionName NEQ $ttyEofExcpt THEN $raise;
    s := "" END; # and fall out of handler

The device MODULE name is specified as a part of the file name. It is separated from the rest of the file name with the device MODULE break character, $devModBrk. For example, on most systems, where $devModBrk is the character >, the file name foo>s specifies that the device MODULE FOO be used with the file s.

$devModBrk may be different on different operating systems, so the character > (the most common value for $devModBrk) should not be hardwired into file names in programs.

Some device MODULEs accept special file name syntax; refer to the documentation on the device MODULE (e.g., Chapter 22 of the MAINSAIL Utilities User's Guide and Chapter 25 of the MAINSAIL Utilities User's Guide).

Some files use a prefix that looks syntactically like a device MODULE prefix, but is in fact mapped to a MODULE of another name. This is transparent to the user of the device MODULE. The terms “device prefix” and “device MODULE” are used interchangeably.

22.12. cmdFile and logFile and MAINSAIL Standard Input and Output

cmdFile and logFile are the “standard” input and output, i.e., the main medium of interaction (or simulated interaction) with the user as established within MAINSAIL. They are distinct from TTY, which is “primary” input and output, i.e., the main medium of interaction with the user as established by the operating system when MAINSAIL is invoked from the operating system level. To redirect TTY, you must write a $ttyCls MODULE and set the system variable $ttyMod to point to its data section (see Chapter 23); by contrast, you can use MAINEX subcommands to redirect cmdFile and logFile to any file.

cmdFile and logFile are used to redirect the MAINSAIL standard input or output stream by opening some other text input file as cmdFile, or by opening some other text output file as logFile. For example, a program may use the calls:

open(cmdFile,"Command file: ",prompt!input);
open(logFile,"Logging file: ",prompt!create!output)

to allow the user to specify the files to use for MAINSAIL standard input and output.

MAINEX provides subcommands to redirect cmdFile and logFile; see Sections 21.2.7, 21.2.23, 21.2.18, and 21.2.17 of the MAINSAIL Utilities User's Guide.

The system PROCEDUREs ttyRead, ttyWrite, and ttycWrite communicate directly with TTY, and cannot be redirected by MAINSAIL to communicate with some other file.

Closing cmdFile or logFile has the effect of reopening it to TTY. When end-of-file is reached on cmdFile, the following occurs unless the configuration bit $noAutoCmdFileSwitching is set (see Chapter 6 of the MAINSAIL Utilities User's Guide).

1. The predefined exception $cmdFileEofExcpt is raised.

2. If the exception is handled, execution proceeds with no change to the status of cmdFile.

3. If the exception is not handled:
   • If cmdFile is TTY, MAINSAIL exits.

   • Otherwise, cmdFile is closed and reopened to TTY.

22.13. errorOK and File I/O

• Invalid open bits, e.g., the create bit set without the output bit.

• A file closed twice, or I/O attempted on a closed file.

• An invalid format prefix, e.g., a VAR prefix for a file opened for random output.

• A file that cannot be closed.

• Buffered I/O attempted on an unbuffered file.

• Unable to perform a read or write on an open file not at end-of-file.

Windows NT supports two standards for the end-of-line sequence within a text file: carriage return, then linefeed (CRLF); or just linefeed (LF). MAINSAIL supports either standard for most text file operations, and the LF standard for all operations. For data files, only the LF standard is supported.

The LF standard is the default for text files on both on Windows NT and UNIX. The MAINSAIL STRING constant eol consists of a single character, linefeed, on both systems.

22.14.1. CRLF Device Prefix

"CRLF" & $devModBrkStr

($devModBrkStr is currently > everywhere, but could be different on some future operating system). Subsequently, eol characters written to the file are translated to the carriage-return-linefeed sequence. When a carriage return is read from the file, it is discarded (see Section 39.1.1); since a carriage return usually precedes a linefeed (eol) character, this leaves just the linefeed.

The use of the CRLF device prefix usually presents no problems for sequential files. For random-access files, however, problems arise if a program performs arithmetic on file positions based on the number of characters it has read from or written to a file, and any of the characters read or written is a carriage return or linefeed. The arithmetic may produce incorrect results in such cases because of the translation of a one-character sequence to a two-character sequence (on output) or the discarding of characters (on input). CRLF should not be used for files where such calculations are performed.

For random output files opened using CRLF, it is also possible to position so as to overwrite half of a two-character carriage-return-linefeed sequence, a potential source of additional confusion.

Because files opened with the CRLF prefix have so many restrictions, CRLF is not the default device prefix on NTPNT, even though most Windows NT utilities create such files. However, most Windows NT programs seem to process LF-standard files correctly as well.

It is possible to change MAINSAIL's default text file prefix to CRLF, as described in Section 22.14.3. You may wish to do so if you know that your MAINSAIL program will not perform random I/O on CRLF files in such a way as to encounter the problems mentioned above.

The CRLF prefix is incompatible with the keepNul bit, which specifies that carriage returns are to be retained when read from a file; an error message is issued when both keepNul and the CRLF prefix are specified when a file is opened.

22.14.2. LF Device Prefix

GLOBALSYMBOL defaultTextFileDevMod CRLF

in your v1620.prm parameter file):

• defaultTextFileDevMod: The value of this STRING is the default prefix to use for text files (without the $devModBrk character). For example, to set the default prefix to CRLF, set defaultTextFileDevMod to CRLF.

• defaultInputTextFileDevMod: If a text file is opened only for input, use this prefix for it by default. As with defaultTextFileDevMod, do not include $devModBrk in the STRING. Overrides defaultTextFileDevMod if both are set.

• defaultOutputTextFileDevMod: If a text file is opened only for output, use this prefix for it by default. As with defaultTextFileDevMod, do not include $devModBrk in the STRING. Overrides defaultTextFileDevMod if both are set.

The default prefix can always be overridden by an explicit prefix as part of the file name passed to open.

22.14.4. Converting CRLF Files to LF Files and Vice Versa

*copier<eol>
Text File Copier
Input file (just <eol> to stop): LF>noCR<eol>
Output file: CRLF>hasCR<eol>

22.15. cmdFile and logFile Echoing

The configuration bit $echoIfRedirected (value 'H80) causes input to cmdFile to be echoed to TTY if cmdFile or logFile is not the file TTY, and output to logFile to be echoed to TTY if logFile is not the file TTY. The configuration bit $echoCmdFile (value 'H100) always echoes cmdFile to logFile.

The file TTY is the MAINSAIL file TTY. If operating-system-dependent standard input is redirected for a MAINSAIL process, MAINSAIL still considers that the operating-system-dependent standard input is the MAINSAIL file TTY. The $echoIfRedirected bit does not take effect unless logFile and cmdFile are redirected from within MAINSAIL.

There are some instances when input routines look one character ahead to determine that an input operation has terminated. This character is then input the next time a read occurs from the file. In such cases, the character is echoed twice, once when it was used in the one-character lookahead, and once when it was actually read. This can occur at present (if f is cmdFile) for read(f,x), where x is a (LONG) INTEGER, (LONG) REAL, or (LONG) BITS, and for scan(f,...,proceed...). This bug may be fixed in a future release.

22.16. Caching of Files

22.16.1. Background: Buffered I/O

MAINSAIL file I/O is usually buffered, meaning that when something is written to a file, it is not immediately sent to a file, but temporarily stored in memory in a file buffer. File buffers are written when they become full, but since file buffers are usually much larger than the data written by a single write operation, data are sent to a file less frequently than would be the case if every system PROCEDURE that wrote data to a file forced it to the file immediately. Since sending data to a file is usually an expensive operation, file buffering provides substantial performance improvement in most cases.

In some cases, file buffering is not desired; in this case, the bit $unbuffered can be set in the call to open.

The exact size of a buffer depends on the file and device type.

If a file is opened for random access, its buffers may be cached; i.e., more than one buffer may be maintained in memory for random access files. This increases the chance that when a setPos is done, the portion of the file to which the file position is changed may be found in a buffer in memory, thereby avoiding unnecessary low-level file I/O.

22.16.2. Introduction to the File Cache

• The ability to specify that a file not be cached

• The ability to cache non-bytestream files opened for random access

• The ability to cache buffers privately for a particular file

• The ability to remove all of a file's buffers from the cache

• The ability to control whether or not dirty buffers (i.e., buffers that have been modified but not yet written to the file) are written to the file when removed from the cache

• More efficient buffer lookup

The file cache improves I/O performance for buffered random access files; sequential files are not cached.

A file cache is a collection of file buffers that are maintained in memory rather than on a device. All such buffers are referred to as cached buffers. Since information which resides in memory can be accessed faster than information which resides on a device, I/O performance is improved.

A buffered random access file can be globally cached, privately cached, or not cached, except that if its buffer size is different from the size of the buffers in the global cache, it cannot be globally cached. The global cache reuses buffers among all globally cached files while a private cache reuses only a particular file's cached buffers. A buffered random access file is automatically globally cached if its buffer size is the same as the size of the buffers in the global cache; otherwise, it is automatically privately cached.

The major benefit of the global cache is that it does not force an application to predict file usage. Which buffers reside in the global cache is dynamically adjusted to file usage; i.e., more buffers are cached for more heavily-accessed files, and fewer buffers are cached for less heavily-accessed files. A private cache is meant for use only for key files that are very heavily used and for files with a buffer size other than that of the buffers in the global cache. An application program should therefore normally let bytestream files be globally cached.

All buffers (including the current buffer) for a cached file reside in a hash table associated with that file. The hash function is based on the file position. All buffers except the current buffer also reside in a list ordered from most-recently-used to least-recently-used, subsequently referred to as an LRU list. The cache parameters requestedMinCacheSize and requestedMaxCacheSize refer to the size of the LRU list (a file can be cached even though requestedMaxCacheSize is Zero, since requestedMaxCacheSize refers to the LRU list and the current buffer is cached but does not reside in the LRU list).

A cached file is implicitly updated when a cached buffer is reused (i.e., written out and the storage occupied by the buffer reused for data at another file position) and when the file is closed. It can be explicitly updated by calling a PROCEDURE that clears the cache.

Maintenance of a cache is governed by three parameters:

1. The requested minimum number of buffers,
2. The requested maximum number of buffers, and
3. The requested cache hit percent.

The default parameters for the global cache can be changed programmatically. Default values for private cache parameters are used if private cache parameter values are not provided by the application program. The number of buffers in the LRU list is kept between the requested minimum and maximum size whenever possible; the LRU list size may be smaller than the requested minimum size but is not allowed to grow larger than the requested maximum size.

When a request for a new buffer is made, the cache is searched (this is a hash table lookup). If the requested buffer is found, it becomes the file's current buffer. If it is not found, either a new cache entry is created, the least-recently-used cache entry is reused, or the file's current buffer is reused. Which action is taken is governed by the file cache parameters.

22.16.3. File Cache PROCEDUREs

$queryFileCacheParms returns information about the cache, optionally associated with a particular file.

$setFileCacheParms sets the parameters for the cache, optionally associated with a particular file.

$clearFileCache removes some or all of a file's buffers from the LRU list, optionally writes dirty buffers, and optionally uncaches the file.

22.17. Disk I/O and the BIGMSLBUFFERS Environment Variable

XIDAK recommends that you set the environment variable BIGMSLBUFFERS to 1 whenever you run MAINSAIL, unless experiment shows that performance is better without this variable set.

It is possible that larger file buffers may become the default in a future version of MAINSAIL, in which case it would no longer be necessary to set the BIGMSLBUFFERS environment variable.

22.18. Partial Data Reads

read(f,bb)

where bb is a LONG BITS variable, causes a partial data read.

When a partial data read occurs, an error message is issued. The user can handle the error and obtain information about what happened. In particular, the errMsg call that occurs if a data read gets only part of a data type is:

errMsg($partialDataRead,
    "of " & cvs(i) & " chars from file " & f.name,msgMyCaller);

where $partialDataRead is a predefined identifier for the STRING constant partial data read.

To handle partial data read errors, put a $HANDLE statement around any suspect read. In the handler, check for:

$exceptionName = $systemExcpt AND
    $exceptionStringArg1 = $partialDataRead

If this condition is TRUE, then cvi($exceptionStringArg2) is the number of characters actually read, and the last “word” in $exceptionStringArg2 is the file name (if the file name contained no spaces). In order to obtain the partial datum, the handler must call $raiseReturn, which will cause the error message not to be printed, and then the read will return the partial datum in the PRODUCES variable (left- or right-justified depending upon the host machine's characteristics). $gotValue(f) will be FALSE and the file will be positioned immediately after the characters that were read.