dk_tokenizer_t Struct Reference

Tokenizer. More...

#include <dktypes.h>


Data Fields

char * token_buffer
 Buffer for token.
size_t buffer_size
 Buffer size for token.
size_t buffer_used
 Number of bytes used in token buffer.
char ** quotes
 Pairs of opening and closing quotes.
char * sct
 Single character tokens.
char * whsp
 Whitespaces.
char * nl
 Newlines.
char comment
 Comment introducer.
char * endquote
 This ends the current quote.
int error_code
 Last error occured.
dk_fct_tokenizerfct
 Function to process a token.
void * user_data
 Additional data for function.
int state
 Current state of the tokenizer.
unsigned long lineno
 Line number in input file.


Detailed Description

Tokenizer.

This structure can be used to split input into a sequence of tokens.


Field Documentation

Buffer size for token.

Number of bytes used in token buffer.

Comment introducer.

This ends the current quote.

Last error occured.

Function to process a token.

unsigned long dk_tokenizer_t::lineno

Line number in input file.

Newlines.

Pairs of opening and closing quotes.

Single character tokens.

Current state of the tokenizer.

Buffer for token.

Additional data for function.

Whitespaces.


The documentation for this struct was generated from the following file:

Generated on Thu Feb 25 13:48:25 2010 for dklibs by  doxygen 1.5.8