Module Token

module Token: sig  end

Lexers for Camlp4 grammars.

This module defines the Camlp4 lexer type to be used in extensible grammars (see module Grammar). It also provides some useful functions to create lexers (this module should be renamed Glexer one day).

type pattern = string * string 

Token patterns come from the EXTEND statement.

• The first string is the constructor name (must start with an uppercase character). When it is empty, the second string is supposed to be a keyword.
• The second string is the constructor parameter. Empty if it has no parameter.
• The way tokens patterns are interpreted to parse tokens is done by the lexer, function tok_match below.

exception Error of string

An lexing error exception to be used by lexers.

type location = int * int 

type location_function = int -> location 

The type for a function associating a number of a token in a stream (starting from 0) to its source location.

type 'a lexer_func = char Stream.t -> 'a Stream.t * location_function 

The type for a lexer function. The character stream is the input stream to be lexed. The result is a pair of a token stream and a location function for this tokens stream.

The type for a lexer used by Camlp4 grammars.

• The field tok_func is the main lexer function. See lexer_func type above. This function may be created from a char stream parser or for an ocamllex function using the functions below.
• The field tok_using is a function telling the lexer that the grammar uses this token (pattern). The lexer can check that its constructor is correct, and interpret some kind of tokens as keywords (to record them in its tables). Called by EXTEND statements.
• The field tok_removing is a function telling the lexer that the grammar does not uses the given token (pattern) any more. If the lexer has a notion of "keywords", it can release it from its tables. Called by DELETE_RULE statements.
• The field tok_match is a function taking a pattern and returning a function matching a token against the pattern. Warning: for efficency, write it as a function returning functions according to the values of the pattern, not a function with two parameters.
• The field tok_text returns the name of some token pattern, used in error messages.
• The field tok_comm if not None asks the lexer to record the locations of the comments.

val lexer_text : pattern -> string

A simple tok_text function for lexers

val default_match : pattern -> string * string -> string

A simple tok_match function for lexers, appling to token type (string * string)

          { Token.tok_func = f;
            Token.tok_using = (fun _ -> ());
            Token.tok_removing = (fun _ -> ());
            Token.tok_match = Token.default_match;
            Token.tok_text = Token.lexer_text }
   

val lexer_func_of_parser : (char Stream.t -> 'a * location) -> 'a lexer_func

A lexer function from a lexer written as a char stream parser returning the next token and its location.

val lexer_func_of_ocamllex : (Lexing.lexbuf -> 'a) -> 'a lexer_func

A lexer function from a lexer created by ocamllex

val make_stream_and_location : (unit -> 'a * location) -> 'a Stream.t * location_function

General function

val eval_char : string -> char

val eval_string : string -> string

Convert a char or a string token, where the backslashes had not been interpreted into a real char or string; raise Failure if bad backslash sequence found; Token.eval_char (Char.escaped c) returns c and Token.eval_string (String.escaped s) returns s