Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]
-
Upload
python-meetup -
Category
Software
-
view
257 -
download
3
Transcript of Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]
![Page 1: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/1.jpg)
PYTHON AST
MODULE
Между исходным текстом и байткодом
Николай Карелин
![Page 2: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/2.jpg)
Что это?
• Abstract Syntax Tree:
• Представление абстрактной структуры кода языка
программирования в виде дерева объектов
• В отличие от дерева разбора (parse tree) остаются только
элементы синтаксиса языка
![Page 3: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/3.jpg)
Компиляция кода
• Python 2.5 и позже
• By Armin Ronacher
.py
Parse
Tree
CFG*
bytecode
AST
*CFG = Control Flow Graph
![Page 4: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/4.jpg)
Пример 1
![Page 5: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/5.jpg)
Module
FunctionDef
Name
Expr Pass
body
arguments
body
Module(
body=[
FunctionDef(name='func', args=arguments(
args=[Name(id='x', ctx=Param())],
vararg=None, kwarg=None,
defaults=[]),
body=[Expr(value=Str(s='My function')),
Pass()],
decorator_list=[])])
Str
valueParam
ctx
![Page 6: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/6.jpg)
Пример 2
https://github.com/titusjan/astviewer
Attributes
Fields
![Page 7: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/7.jpg)
Структура модуля• from _ast import *
• Часть С-кода
• Определение всех объектов
• Helper functions
• Получение
• Обработка
• Манипуляции
• Visitor/Transformer class prototypes
![Page 8: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/8.jpg)
Иерархия объектовAST
comprehension
boolop cmpop
arguments excepthandler
alias expr_context
exprmod
operator
keyword
unaryopstmt slice
103 classes in total
![Page 9: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/9.jpg)
Иерархия объектов
• Attribute, BinOp, BoolOp,
• Call, Compare, Dict,
• DictComp, GeneratorExp,
• IfExp, Lambda, List,
• ListComp, Name, Num,
• Repr, Set, SetComp,
• Str, Subscript, Tuple,
• UnaryOp, Yield
• Assert, Assign,
• AugAssign, Break,
• ClassDef, Continue, Delete,
• Exec, Expr, For,
• FunctionDef, Global,
• If, Import, ImportFrom,
• Pass, Print, Raise,
• Return, TryExcept,
• TryFinally, While, With
expr stmt
![Page 10: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/10.jpg)
Синтаксис Python
![Page 11: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/11.jpg)
Синтаксис Python
ASDL (Zephyr Abstract Syntax Definition Language) format
Использует Python скрипт для генерации C-кода ;)
Опционально0 или более
![Page 12: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/12.jpg)
Примеры
• blaze, cffi, enaml, Ipython
• numpy (SafeEval), pandas, scipy.weave, sympy
• pep8, pip, pycparser pylint pyflakes rope
![Page 13: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/13.jpg)
Применение: компиляция
• Numba, http://numba.pydata.org/
• HOPE, http://hope.phys.ethz.ch
• PyJS (Pyjamas), http://pyjs.org,
https://github.com/pyjs/pyjs
.py
Parse
Tree
CFG
bytecode
AST
LLVM
![Page 14: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/14.jpg)
Visitor
GreenTreeSnakes
![Page 15: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/15.jpg)
Свой язык в Python
• Hy ;)
.py
Parse
Tree
CFG
bytecode
AST
![Page 16: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/16.jpg)
Создание AST
![Page 17: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/17.jpg)
Hy: HyASTCompiler.compile()
![Page 18: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/18.jpg)
Сode instrumenting
• Pyflakes, …
• Macros (http://www.pocoo.org/projects/karnickel/)
.py
Parse
Tree
CFG
bytecode
AST
?
![Page 19: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/19.jpg)
numpy.utils.safe_eval()
…
См. также ast.literal_eval()
![Page 20: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/20.jpg)
Осторожно!
There Might be Dragons!
![Page 21: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/21.jpg)
AST - Предосторожности
• Странные ошибки (segfaults)
• Нет гарантии стабильности API
• Разная структура для
• 2.x <-> 3.x
• Разные версии ветки 3.x
• Cpython, PyPy, Jython, …
![Page 22: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/22.jpg)
Что дальше
• Hacker’s Guide to Python
• PEP 339 / Python Dev. Guide
• Green Tree Snakes documentation,
https://greentreesnakes.readthedocs.org/en/latest/
• ast module documentation
• http://pyvideo.org/video/419/pycon-2011--what-would-you-
do-with-an-ast
• http://www.dalkescientific.com/writings/diary/archive/2010/
02/22/instrumenting_the_ast.html
![Page 23: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/23.jpg)
Hacker’s Guide to Python
Table of contents
1. Starting your project
2. Modules and libraries
3. Documentation
4. Distribution
5. Virtual environments
6. Unit testing
7. Methods and decorators
8. Functional programming
9. The AST
10. Performances and optimizations
11. Scaling and architecture
12. RDBMS and ORM
13. Python 3 support strategies
14. Write less, code more
https://julien.danjou.info/books/the-hacker-guide-to-python
![Page 24: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/24.jpg)
THE END
![Page 25: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/25.jpg)
На будущее
• Sphinx и ReST для документирования проектов
• Python для численных расчетов (Numba, GPU, …)
• ctypesgen – обертки над внешними библиотеками
???
![Page 26: Python AST / Николай Карелин / VPI Development Center [Python Meetup 27.03.15]](https://reader033.fdocument.pub/reader033/viewer/2022042701/55a660491a28ab4c538b4607/html5/thumbnails/26.jpg)
Ищется помощь по белорусификации
http://pymorphy2.readthedocs.org/en/latest/
https://github.com/kmike/pymorphy2