Flex analyses the way of Token

[Xinyan Liu]

Could you tell me when Flex analyses the word(Token) ,it is the disposable reading into its inside buffer area of a lot of characters , Analyse the buffer content of district ,resolve and publish piece of Tokens; If reach the end of buffering area,read into again

[jrfritz]

This would be better in the Programming forum....

[Pype.Clicker]

the core of Flex is a finite state automaton (check google for some theory about it). every new character is checked against a character table and will lead to a new state where new characters are available.

For instance, imagine you want to translate ":-)" into "joy.gif", ":-(" into "sad.gif", ">:[" in "angry.gif" and "=]" in "l337.gif"

Code: Select all

  state0 : {':' --> state1 , '>' --> state2, '=' --> state3}
  state1 : {'-' --> state4}
  state2 : {':' --> state5}
  state3 : {']' --> accept("l337.gif")}
  state4 : {')' --> accept("joy.gif"), '(' --> accept("sad.gif")}
  state5 : {'-' --> state6}
  state6 : {'[' --> accept("angry.gif")}

that's it ... you start in state0 and try to follow the existing chars. If you can't, just output the first char of the current string and go on with the second one (so that ::-) is translated into :joy.gif)

[Xinyan Liu]

Thanks! This my dear friend speaks very much above! Flex is really a method to use finite state, It will get the state transition diagram(Transition disgram) according to Regular Expression, Produce the state and change form(some array forms)s according to this transition diagram afterwards, And cooperate with a corresponding word and discern that the procedure discerns Token.

[jack]

All there is Look Ahead this concept in morphological analysis and syntactic analysis. Whether Look Ahead in morphological analysis just needs Look Ahead in order to match the longest Pattern , But it need and recognise next Token either,why recognise The Next Token of the yytext among Flex? ? ? ? ?

[jack]

Thanks a lot! Please give me a forgiveness for my bad english
There is the concept named Look Ahead both parser and Lexer. In order to match the longest Token in all pattern

matching rules in lexer，lexer need looking ahead，but not need get the next Token! why do we get the next Token in yytext in flex?