Parsing Converter 3: Lexical part of handwritten PHP to Python compiler

这篇文章解析转换器3：手写PHP转Python编译器的词法部分

一时技痒，自然而然地想搞个大家伙，把整个PHP程序转成Python。不比模板，可以用正则匹配偷懒，这次非写一个Php编译器不可。

上网搜了一下，发现大部分Python to xxx的transpile都是直接基于AST，省略了最重要的Tokenizer，Parser。直接写个Visitor了事。要不然就是基于Antlr之类的生成器，搞一大堆代码，看得令人心烦。

既然大家都不想做这个苦力，我就来试试，手工写一个Php编译器。分Tokenizer，Parser，Visitor三个部分来实现。

翻出《龙书》《虎书》做参考，仔细学了一回PHP，不学不知道，原来PHP有那么多特性，做个编译器真心累人。

词法部分很简单，就是一个自动机。设计了一个结构存放自动机，然后简单粗暴地在自动机上编程，也顾不上什么性能了，就是个一锤子买卖。

写得还算快，调试不是很顺，不过我是不会说的，哈

自动机不复杂，发上来大家看看，敬请指正。

self.statemachine = {
            'current': {
                'state': 'default', 'content': '', 'line': 0},
            'default': [
                {'name': 'open', 'next': 'php', 'extra': 0, 'start': 0, 'end': 0, 'cache': '',
                 &#39;token&#39;: r&#39;<\?&#39;},
                {&#39;name&#39;: &#39;open&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;<\?php&#39;}],
            &#39;php&#39;: [
                {&#39;name&#39;: &#39;close&#39;, &#39;next&#39;: &#39;default&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\?>&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;lnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[0-9]+&#39;},
                {&#39;name&#39;: &#39;dnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;([0-9]*\.[0-9]+)|([0-9]+\.[0-9]*)&#39;},
                {&#39;name&#39;: &#39;exponent&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;(([0-9]+|([0-9]*\.[0-9]+)|([0-9]+\.[0-9]*))[eE][+-]?[0-9]+)&#39;},
                {&#39;name&#39;: &#39;hnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;0x[0-9a-fA-F]+&#39;},
                {&#39;name&#39;: &#39;bnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;0b[01]+&#39;},
                {&#39;name&#39;: &#39;label&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;commentline&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;//&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;commentline&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;#&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;comment&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;/\*&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;"&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;symbol&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[\\\{\};:,\.\[\]\(\)\|\^&\+-/\*=%!~$<>\?@]&#39;}],
            &#39;string1&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;escape1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\\&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;escape1&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;.&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;string2&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;escape2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\\&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;escape2&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;.&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;commentline&#39;: [
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;(\r|\n|\r\n)&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;comment&#39;: [
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\*/&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}]}

The above is the detailed content of Parsing Converter 3: Lexical part of handwritten PHP to Python compiler. For more information, please follow other related articles on the PHP Chinese website!