javascript - Grammar rules for comments -

i working reflect.js (a nice javascript parser) zach carter on github; trying modify behavior of parser handle comments normal tokens should parsed else. default behavior of reflect.js keep track of comments (the lexer grabs them tokens) , append list of them end of ast (abstract syntax tree) creates.

however, these comments included in-place in ast. believe change involve adding grammar rules grammar.y file here . there no rules comments -- if understanding correct, why ignored main parsing code.

how write rules include comments in ast?

the naive version modifies each rule of original grammer:

      lhs = rhs1 rhs2 ... rhsn ; 

to be:

      lhs =  rhs1 comments rhs2 comments ... comments rhsn ; 

while works in abstract, screw parser generator if ll or lalr based, because can't see far enough ahead next token decide do. you'd have switch more powerful parser generator such glr.

a smarter version replaces (only and) every terminal t nonterminal:

      t  =  comments t ; 

and modifies orginal lexer trivally emit t instead of t. still have lookahead troubles.

but gives basis real solution.

a more sophisticated version of cause lexer collect comments seen before token , attach them next token emits; in essence, implementing terminal rule modification of grammar, in lexer.

now parser (you don't have switch technologies) sees tokens saw; tokens carry comments annotations. you'll find useful divide comments attach previous token, , attach next, won't able make better heuristic, because there no practical way decide token comments belong.

you'll find fun figure out how capture positioning information on tokens , comments, enable regeneration of original text ("comments in proper locations"). you'll find more fun regenerate text appropriate radix values, character string escapes, etc., in way doesn't break language syntax rules.

we our general language processing tools , works reasonably well. amazing how work straight, can focus on transformation task. people underestimate lot.