## Jet scripts: document and sentence processing

When Jet is run in batch mode, the basic unit of processing is the document. In addition, the Jet console can be used to enter and process individual sentences.

Document and sentence processing is controlled by a set of scripts. Each script specifies a series of actions to be performed on a portion of the document. The actions correspond for the most part to the tools or annotators provided with Jet.

Scripts are defined in the Jet configuration file. A script definition has the form

> scriptName = action1, action2, ...

The following actions are allowed:

tokenize | divide text into tokens
tag(XMLtag) | assign annotation XMLtag to text enclosed in `<XMLtag> ... </XMLtag>`
sentenceSplit | divide text into sentences
lexLookup | look up tokens in lexicon
pruneTags | use HMM POS tagger to select most likely POS from those given in lexicon
tagPOS | use HMM POS tagger to assign Penn POS tags
tagJet | use HMM POS tagger to assign Jet POS tags
tagNames | use HMM name tagger to assign MUC name tags
chunk | use Maxent tagger to identify noun groups
parse | parse sentence using context-free parser
statParse | parse sentence using TreeBank-trained probabilistic parser
pat(patternSetName) | apply patterns in patternSetName
resolve | resolve references
tag : scriptName | apply script scriptName to every instance of text annotated with tag tag

Note the last action: it allows one script to invoke another script over a portion of the text.

In batch mode (or when the processDocuments menu item is selected from the console), the `processDocument` script is applied to each input document. When a sentence is typed in directly at the console, the `processSentence` script is applied. The following scripts are defined by default:

> `processDocument = tag(TEXT), TEXT:processTextZone`
> `processTextZone = tokenize, sentenceSplit, sentence:processSentence`

This means that when Jet processes a document, it looks for tags of the form `<TEXT> ... </TEXT>` in the document, and adds an annotation of type text to the enclosed text. Then it runs the script `processTextZone` on each such text. The script `processTextZone` runs the tokenizer and then the sentence splitter on that text. The sentence splitter adds annotations of type sentence to the text. Then, for the text subsumed by each sentence annotation, we run the script `processSentence`.

There is no default for the `processSentence` script. A simple example, to look words up in the dictionary and the run the parser, would be

> `processSentence = lexLookup, parse`

To look up words, prune the result using the POS tagger, and then apply two sets of patterns, for `dates` and `names`, one would write

> `processSentence = lexLookup, pruneTags, pat(dates), pat(names)`