Python parser #4

HEL · 2026-05-26T08:27:48Z

HEL commented

2026-05-26 08:27:48 +00:00

This PR adds a simple parser which takes in a Python Module (an AST node) and converts it to our own internal AST structure.
This is done for two reasons:

Parse custom type annotation into a more semantically explicit structure (Frame[name: Type] is a Frame with a column named name of type Type, not a subscript with a slice)
Simplify Python constructs to expose typed operations (e.g. assignments, annotations, function definitions and calls, etc.)

The latter might not be fully required, as it involves a lot of redundancy with Python's own AST nodes, but it provides a more consistent interface with custom types

This PR also adds a utility gen/gen.py script to help generate AST node classes from simple templates.
Finally, it also adds a new utility command midas utils highlight -o OUTPUT SOURCE_FILE which runs the Midas or Python parser against the provided source file and outputs the highlighted code as a standalone HTML file.

This PR adds a simple parser which takes in a Python Module (an AST node) and converts it to our own internal AST structure. This is done for two reasons: - Parse custom type annotation into a more semantically explicit structure (`Frame[name: Type]` is a `Frame` with a column named `name` of type `Type`, not a subscript with a slice) - Simplify Python constructs to expose typed operations (e.g. assignments, annotations, function definitions and calls, etc.) The latter might not be fully required, as it involves a lot of redundancy with Python's own AST nodes, but it provides a more consistent interface with custom types This PR also adds a utility `gen/gen.py` script to help generate AST node classes from simple templates. Finally, it also adds a new utility command `midas utils highlight -o OUTPUT SOURCE_FILE` which runs the Midas or Python parser against the provided source file and outputs the highlighted code as a standalone HTML file.

HEL self-assigned this 2026-05-26 08:27:48 +00:00

HEL added 27 commits 2026-05-26 08:27:48 +00:00

refactor: move source files in subdirectory 6e0190a378

feat: add pyproject.toml 8a9bb6ef4e

feat(cli): add basic CLI entrypoint eb79cf6dc3

chore: complete pyproject.toml 5a112332f2

feat(cli): add compile command to read python AST e2f3cabe15

feat(parser): add AST nodes for python 7ce2840f03

feat(parser): add simple Python parser f4d2be3b1b

feat(parser): add pretty-printer for python AST 4f799caaf5

feat(cli): add option to run python parser 3d599b3462

fix: use generic Difference type in example 832c350b61

feat(parser): parse type constraints in python 8d7c115432

feat(parser): parse functions in python 5aedddfabb

feat(parser): store locations in parsed nodes d0c54db33a

feat(cli): add highlight command 9b59058881

feat(parser): add location to midas AST nodes e94db2181f

feat(cli): add midas highlighter 0e0a1b26f2

fix(parser): update ast gen script a735113466

refactor(parser): improve AST class generator 939e5af4ce

make the generation script more flexible

feat(parser): generate python AST classes 0bbdf04621

use the generation script to create Python AST node classes, also distinguish between Midas type annotation nodes and statements

feat(parser): add Python AST nodes ecab1b74a4

refactor(parser): remove inheritance from NodeVisitor c64d626d1c

remove the parent NodeVisitor class from PythonParser and implement all custom recursive methods instead

feat(parser): adapt AST printer with new nodes 4d23e8840e

feat(cli): update highlighter with new nodes bbd0e3ae8d

feat(parser): parse assignments 8a9b4f3989

feat(parser): parse python expressions 0b3f33d7fe

fix(parser): add call keywords attribute in gen definition 170101aa37

feat(cli): highlight midas keywords 6524b3591a

HEL merged commit ec079f32ca into main

2026-05-26 08:28:42 +00:00

HEL deleted branch feat/python-parser

2026-05-26 08:28:42 +00:00

HEL referenced this issue from a commit

2026-05-26 08:28:43 +00:00

Merge pull request 'Python parser' (#4) from feat/python-parser into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: HEL/midas#4