# Standard text processing reference

This appendix is a reference of the standard text processing module.

Pattern

The `Pattern` class is a pattern matching class based either on regular expression or balanced string. In the regex mode, the pattern is defined with a regex and a matching is said to occur when a regex match is achieved. In the balanced string mode, the pattern is defined with a start pattern and end pattern strings. The balanced mode can be a single or recursive. Additionally, an escape character can be associated with the class. A name and a tag is also bound to the pattern object as a mean to ease the integration within a scanner.

pattern-p
Object
Pattern
none

The `Pattern` constructor creates an empty pattern.

Pattern
String|Regex

The `Pattern` constructor creates a pattern object associated with a regular expression. The argument can be either a string or a regular expression object. If the argument is a string, it is converted into a regular expression object.

Pattern
String String

The `Pattern` constructor creates a balanced pattern. The first argument is the start pattern string. The second argument is the end balanced string.

Pattern
String String Character

The `Pattern` constructor creates a balanced pattern with an escape character. The first argument is the start pattern string. The second argument is the end balanced string. The third character is the escape character.

Pattern
String String Boolean

The `Pattern` constructor creates a recursive balanced pattern. The first argument is the start pattern string. The second argument is the end balanced string.

REGEX

The `REGEX` constant indicates that the pattern is a regular expression.

BALANCED

The `BALANCED` constant indicates that the pattern is a balanced pattern.

RECURSIVE

The `RECURSIVE` constant indicates that the pattern is a recursive balanced pattern.

check
Boolean
String

The `check` method checks the pattern against the input string. If the verification is successful, the method returns true, false otherwise.

match
String
String|InputStream

The `match` method attempts to match an input string or an input stream. If the matching occurs, the matching string is returned. If the input is a string, the end of string is used as an end condition. If the input stream is used, the end of stream is used as an end condition.

set-tag
none
Integer

The `set-tag` method sets the pattern tag. The tag can be further used inside a scanner.

get-tag
Integer
none

The `get-tag` method returns the pattern tag.

set-name
none
String

The `set-name` method sets the pattern name. The name is symbol identifier for that pattern.

get-name
String
none

The `get-name` method returns the pattern name.

set-regex
none
String|Regex

The `set-regex` method sets the pattern regex either with a string or with a regex object. If the method is successfully completed, the pattern type is switched to the REGEX type.

set-escape
none
Character

The `set-escape` method sets the pattern escape character. The escape character is used only in balanced mode.

get-escape
Character
none

The `get-escape` method returns the escape character.

set-balanced
none
String| String String

The `set-balanced` method sets the pattern balanced string. With one argument, the same balanced string is used for starting and ending. With two arguments, the first argument is the starting string and the second is the ending string.

Lexeme

The `Lexeme` class is a literal object that is designed to hold a matching pattern. A lexeme consists in string (i.e. the lexeme value), a tag and eventually a source name (i.e. file name) and a source index (line number).

lexeme-p
Literal
Lexeme
none

The `Lexeme` constructor creates an empty lexeme.

Lexeme
String

The `Lexeme` constructor creates a lexeme by value. The string argument is the lexeme value.

set-tag
none
Integer

The `set-tag` method sets the lexeme tag. The tag can be further used inside a scanner.

get-tag
Integer
none

The `get-tag` method returns the lexeme tag.

set-value
none
String

The `set-value` method sets the lexeme value. The lexeme value is generally the result of a matching operation.

get-value
String
none

The `get-value` method returns the lexeme value.

set-index
none
Integer

The `set-index` method sets the lexeme source index. The lexeme source index can be for instance the source line number.

get-index
Integer
none

The `get-index` method returns the lexeme source index.

set-source
none
String

The `set-source` method sets the lexeme source name. The lexeme source name can be for instance the source file name.

get-source
String
none

The `get-source` method returns the lexeme source name.

Scanner

The `Scanner` class is a text scanner or lexical analyzer that operates on an input stream and permits to match one or several patterns. The scanner is built by adding patterns to the scanner object. With an input stream, the scanner object attempts to build a buffer that match at least one pattern. When such matching occurs, a lexeme is built. When building a lexeme, the pattern tag is used to mark the lexeme.

scanner-p
Object
Scanner
none

The `Scanner` constructor creates an empty scanner.

add
none
Pattern*

The `add` method adds 0 or more pattern objects to the scanner. The priority of the pattern is determined by the order in which the patterns are added.

length
Integer
none

The `length` method returns the number of pattern objects in this scanner.

get
Pattern
Integer

The `get` method returns a pattern object by index.

check
Lexeme
String

The `check` method checks that a string is matched by the scanner and returns the associated lexeme.

scan
Lexeme
InputStream

The `scan` method scans an input stream until a pattern is matched. When a matching occurs, the associated lexeme is returned.

Literate

The `Literate` class is transliteration mapping class. Transliteration is the process of changing characters my mapping one to another one. The transliteration process operates with a character source and produces a target character with the help of a mapping table. This transliteration object can also operate with an escape table. In the presence of an escape character, an escape mapping table is used instead of the regular one.

literate-p
Object
Literate
none

The `Literate` constructor creates a default transliteration object.

Literate
Character

The `Literate` constructor creates a default transliteration object with an escape character. The argument is the escape character.

read
Character
InputStream

The `read` method reads a character from the input stream and translate it with the help of the mapping table. A second character might be consumed from the stream if the first character is an escape character.

getu
Character
InputStream

The `getu` method reads a Unicode character from the input stream and translate it with the help of the mapping table. A second character might be consumed from the stream if the first character is an escape character.

reset
none
none

The `reset` method resets all the mapping table and install a default identity one.

set-map
none
Character Character

The `set-map` method set the mapping table by using a source and target character. The first character is the source character. The second character is the target character.

get-map
Character
Character

The `get-map` method returns the mapping character by character. The source character is the argument.

translate
String
String

The `translate` method translate a string by transliteration and returns a new string.

set-escape
none
Character

The `set-escape` method set the escape character.

get-escape
Character
none

The `get-escape` method returns the escape character.

set-escape-map
none
Character Character

The `set-escape-map` method set the escape mapping table by using a source and target character. The first character is the source character. The second character is the target character.

get-escape-map
Character
Character

The `get-escape-map` method returns the escape mapping character by character. The source character is the argument.

sort-ascent
none
Vector

The `sort-ascent` function sorts in ascending order the vector argument. The vector is sorted in place.

sort-descent
none
Vector

The `sort-descent` function sorts in descending order the vector argument. The vector is sorted in place.

sort-lexical
none
Vector

The `sort-lexical` function sorts in lexicographic order the vector argument. The vector is sorted in place.