Page 1 of 1
Flex analyses the way of Token
Posted: Wed Jan 08, 2003 7:05 pm
by Xinyan Liu
Could you tell me when Flex analyses the word(Token) ,it is the disposable reading into its inside buffer area of a lot of characters , Analyse the buffer content of district ,resolve and publish piece of Tokens; If reach the end of buffering area,read into again
Re:Flex analyses the way of Token
Posted: Wed Jan 08, 2003 7:12 pm
by jrfritz
This would be better in the Programming forum....
Re:Flex analyses the way of Token
Posted: Thu Jan 09, 2003 3:52 am
by Pype.Clicker
the core of Flex is a finite state automaton (check google for some theory about it). every new character is checked against a character table and will lead to a new state where new characters are available.
For instance, imagine you want to translate ":-)" into "joy.gif", ":-(" into "sad.gif", ">:[" in "angry.gif" and "=]" in "l337.gif"
Code: Select all
state0 : {':' --> state1 , '>' --> state2, '=' --> state3}
state1 : {'-' --> state4}
state2 : {':' --> state5}
state3 : {']' --> accept("l337.gif")}
state4 : {')' --> accept("joy.gif"), '(' --> accept("sad.gif")}
state5 : {'-' --> state6}
state6 : {'[' --> accept("angry.gif")}
that's it ... you start in state0 and try to follow the existing chars. If you can't, just output the first char of the current string and go on with the second one (so that ::-) is translated into :joy.gif)
Re:Flex analyses the way of Token
Posted: Thu Jan 09, 2003 11:26 pm
by Xinyan Liu
Thanks! This my dear friend speaks very much above! Flex is really a method to use finite state, It will get the state transition diagram(Transition disgram) according to Regular Expression, Produce the state and change form(some array forms)s according to this transition diagram afterwards, And cooperate with a corresponding word and discern that the procedure discerns Token.
Re:Flex analyses the way of Token
Posted: Fri Jan 10, 2003 2:52 am
by jack
All there is Look Ahead this concept in morphological analysis and syntactic analysis. Whether Look Ahead in morphological analysis just needs Look Ahead in order to match the longest Pattern , But it need and recognise next Token either,why recognise The Next Token of the yytext among Flex? ? ? ? ?
Re:Flex analyses the way of Token
Posted: Tue Jan 14, 2003 2:17 am
by jack
Thanks a lot! Please give me a forgiveness for my bad english
There is the concept named Look Ahead both parser and Lexer. In order to match the longest Token in all pattern
matching rules in lexer，lexer need looking ahead，but not need get the next Token! why do we get the next Token in yytext in flex?