Lexical Analysis and Scanning


Click here to start

Table of Contents

Lexical Analysis and Scanning

The Input

The Output

Free form vs Fixed form





More on Identifier Tables

String Literals

Character Literals

Numeric Literals

Handling Comments

Case Equivalence

Issues to Address

General Approach

Interface to Lexical Analyzer

Implementation of Scanner

Relevant Formalisms

Regular Grammars

Regular Grammars

Regular Expressions

Specifying REs in Unix Tools

Finite State Machines

Definitions of FSM

Building FSM from Grammar

Building FSMs from REs

Non-Deterministic FSM

Deterministic FSM

Dealing with ND FSM

Converting to Deterministic

Implementing the Scanner

Ad Hoc Code (forget FSMs)

Using FSM Formalisms

Using FSM formalisms, cont

Implementing FSM (Method 1)

Implementing FSM (Method 2)

Implementing FSM (Method 3)

Automatic FSM Generation

Flex General Format

An Example of a Flex scanner

Flex Example (continued)

Assembling the flex program

Running flex

Choice Between Methods?

The GNAT Scanner

More on the GNAT Scanner

More on GNAT Scanner

DTL (Dewar Trivial Language)

DTL (Continued)

DTL Example



Nasty Glitches

Author: Robert Berriedale Keith Dewar

Email: dewar@gnat.com