#include <dktypes.h>
Data Fields | |
char * | token_buffer |
Buffer for token. | |
size_t | buffer_size |
Buffer size for token. | |
size_t | buffer_used |
Number of bytes used in token buffer. | |
char ** | quotes |
Pairs of opening and closing quotes. | |
char * | sct |
Single character tokens. | |
char * | whsp |
Whitespaces. | |
char * | nl |
Newlines. | |
char | comment |
Comment introducer. | |
char * | endquote |
This ends the current quote. | |
int | error_code |
Last error occured. | |
dk_fct_tokenizer * | fct |
Function to process a token. | |
void * | user_data |
Additional data for function. | |
int | state |
Current state of the tokenizer. | |
unsigned long | lineno |
Line number in input file. |
This structure can be used to split input into a sequence of tokens.
Buffer for token.
size_t dk_tokenizer_t::buffer_size |
Buffer size for token.
size_t dk_tokenizer_t::buffer_used |
Number of bytes used in token buffer.
char** dk_tokenizer_t::quotes |
Pairs of opening and closing quotes.
char* dk_tokenizer_t::sct |
Single character tokens.
char* dk_tokenizer_t::whsp |
Whitespaces.
char* dk_tokenizer_t::nl |
Newlines.
Comment introducer.
char* dk_tokenizer_t::endquote |
This ends the current quote.
Last error occured.
Function to process a token.
Additional data for function.
Current state of the tokenizer.
unsigned long dk_tokenizer_t::lineno |
Line number in input file.