PA_Tokenize

version 2003

PA_Tokenize (toTokenize; len; tokens) → long

Parameter	Type		Description
toTokenize	char *	→	Line of code to be tokenized
len	long	→	Length of toTokenize
tokens	char *	←	Tokens of toTokenize, or 0 to read the tokens size
Function result	long	←	Length of tokens (0 if an error occured)

Description

The routine PA_Tokenize allows the 4D plug-in to tokenize any 4th Dimension statement or expression.


   
    
     PA_Tokenize

allows the 4D plug-in to tokenize only one line of code.


   toTokenize

cannot contain several lines, such as a "Case of" or a "For" block. If you need to tokenize several lines of 4th Dimension code, write the code as a 4th Dimension global method or function, then tokenize a call to this method or function.

After the call, tokens contains the tokenized statement or expression. If the code to be tokenized is incorrect (contains a syntax error), the functions returns 0, an empty string in tokens and an error code in PA_GetLastError .

Pre-tokenization of a 4D formula optimizes in terms of execution time. Let's look at the following examples:

- A user inserts a formula into a 4D Write document such as Substring ([aTable]aField;1;5). This formula stays the same, as long as the user does not change it. During this time, 4D Write may need to evaluate the expression several times (document is reflowed, updated, current record has changed, etc.). For each of these calls, the tokenization phase prior to evaluation can be saved because the syntax of the expression has not changed. This is exactly what PA_Tokenize allows the 4D plug-in to do -- it pre-tokenizes the statement or the expression; then the 4D plug-in calls the PA_ExecuteTokens or PA_ExecuteTokensAsFunction each time it is needed for execution or evaluation.

Consequently, if the developer knows that he will need to execute or evaluate a 4th Dimension statement or expression whose syntax does not change several times, he can use PA_Tokenize and then PA_ExecuteTokens or PA_ExecuteTokensAsFunction instead of PA_ExecuteMethod or PA_ExecuteFunction . For example, if the 4D plug-in calls a 4th Dimension formula 100 times, this allows the 4D plug-in to save the time expended for ninety-nine identical tokenizations.

Example

Tokenize a calculation using project methods.

 
   char   toTokenize[] = "(3.14 * MySinus(1, 2, 3)) / (MyRealRandom(0.1, 0.5))\0"; // as C String
   long   len;
   char*   tokens;
   
   len = PA_Tokenize( toTokenize, strlen( toTokenize ), 0 );
   tokens = malloc( len );
   PA_Tokenize( toTokenize, strlen( toTokenize ), tokens );

We can compute this expression faster elsewhere in the code:

   /* . . . some code . . . */
   PA_ExecuteTokens( tokens, len );

Error Handling

Use PA_GetLastError to see if an error occurred