HEL/midas
1
0
Fork 0
Code Issues Pull Requests Releases Activity

Compare commits

base: HEL:main
Branches Tags
HEL:main
HEL:v0.0.1-proto
..
compare: HEL:11422d43649a3a61c94fbdbba3ec802228c2406b
Branches Tags
HEL:main
HEL:v0.0.1-proto

5 Commits
main ... 11422d4364

Author SHA1 Message Date
LordBaryhobal
11422d4364 feat(cli): add stubs command 2026-06-17 12:19:54 +02:00
LordBaryhobal
e8f8a5ca2f feat(gen): add base for stubs generator 2026-06-17 12:19:32 +02:00
LordBaryhobal
df8d71c0a9 fix(checker): handle calls to AliasType 2026-06-17 12:14:20 +02:00
LordBaryhobal
e4fb142f99 fix(checker): allow substitutyping type vars in GenericType and TopType 2026-06-17 12:12:07 +02:00
LordBaryhobal
2f8f9d633b fix(checker): store member kind in registry 2026-06-17 12:11:16 +02:00
87 changed files with 2631 additions and 17302 deletions
Expand all files Collapse all files

99
README.md
View File

@@ -1,4 +1,4 @@
<h1>Midas</h1>
# Midas
*Midas* is a type system to _Maintain Integrity of Data with Annotated Structures_. In Greek mythology, [Midas](https://en.wikipedia.org/wiki/Midas) was a Phrygian king who was blessed with the gift of turning everything he touched into gold.
@@ -6,25 +6,6 @@
This framework is being developed as part of a Bachelor's Thesis by Louis Heredero at HEI Sion.
<details>
<summary><strong>Table of Contents</strong></summary>
- [Requirements](#requirements)
- [Installation](#installation)
- [Commands](#commands)
- [Type Checking](#type-checking)
- [Compiling](#compiling)
- [Formatting](#formatting)
- [Highlighting](#highlighting)
- [Dumping the AST](#dumping-the-ast)
- [Dumping the Registry](#dumping-the-registry)
- [Generating Stubs](#generating-stubs)
- [Showing Type Judgements](#showing-type-judgements)
- [Validating Definitions](#validating-definitions)
- [Tests](#tests)
</details>
## Requirements
- Python 3.11+
@@ -51,49 +32,25 @@ This framework is being developed as part of a Bachelor's Thesis by Louis Herede
## Commands
<!--
check
compile
format
highlight
parse
dump_registry
types
validate
-->
### Type Checking
```shell
midas check -t types.midas source.py
```
This command parses the given files and run the type checkers against the Midas definitions and Python program. Diagnostics are then printed showing warnings and errors.
### Compiling
> [!NOTE]
> In the current state of the project, the `compile` command doesn't generate any runnable code, it only runs the parsers and type checker on the provided files
```shell
midas compile -t types.midas source.py
```
With the `compile` command, you can process a source Python file, with any number of custom type definition files (`-t FILE` option), and the type checker will verify the coherence of your program and generate the runnable code with valid syntax and runtime assertions.
### Formatting
```shell
midas format types.midas
midas format types.midas -o formatted.midas
```
This command parses the given Midas file and outputs a pretty printed file from the AST.
The optional `-l FILE` option lets you produce a highlighted version of the source code showing diagnostics from the type checker (see [Highlighting](#highlighting))
### Highlighting
```shell
midas highlight source.py
midas highlight source.py -o highlighted.html
midas highlight types.midas
midas highlight types.midas -o highlighted.html
midas utils highlight source.py
# or
midas utils highlight types.midas
```
The `highlight` command takes in a source file (Python or Midas), runs the appropriate parser and outputs an HTML file containing the source code with added highlighting. This highlighting takes the form of hoverable annotations showing some of the parsed structures (e.g. a function definition, an assignment, a generic type, etc.)
@@ -103,43 +60,14 @@ The optional `-o FILE` option can be used to specify an output path. By default,
### Dumping the AST
```shell
midas parse source.py
midas parse types.midas
midas utils dump-ast source.py
# or
midas utils dump-ast types.midas
```
For debugging purposes, you can output the AST parsed from a Python or Midas file. For Python files, the `--raw` flags lets you toggle the custom AST parsing. With `--raw`, the raw AST is returned, as produced by the builtin `ast` module. This flag has no effect on Midas files.
For debugging purposes, you can output the AST parsed from a Python or Midas file. For Python files, the `-p` flags lets you toggle the custom AST parsing. Without `-p`, the raw AST is returned, as produced by the builtin `ast` module. This flag has no effect on Midas files.
### Dumping the Registry
```shell
midas dump-registry -t types.midas
```
This command processes the given Midas definitions and dumps the contents of the types registry.
### Generating Stubs
```shell
midas stubs types.midas -o stubs.pyi
```
This command generate Python stubs from a Midas definition file
### Showing Type Judgements
```shell
midas types -t types.midas source.py
```
This command type checks the given Python source file and logs all typing judgements made by the type checker.
### Validating Definitions
```shell
midas validate types.midas
```
This command lets you validate a Midas definition file by running the parser and type checker, verifying syntax and references.
The optional `-o FILE` option can be used to specify an output path. By default, the file is printed in stdout (equivalent to `-o -`).
## Tests
@@ -149,7 +77,6 @@ Several snapshot tests are available to assert the good behaviour of the parsers
uv run -m tests.midas run -a
uv run -m tests.python run -a
uv run -m tests.checker run -a
uv run -m tests.generator run -a
```
**Available subcommands:**

117
assets/icon.svg
View File

@@ -1,117 +0,0 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
width="128"
height="128"
viewBox="0 0 128 128"
version="1.1"
id="svg1"
inkscape:export-filename="logo.png"
inkscape:export-xdpi="96"
inkscape:export-ydpi="96"
inkscape:version="1.4.4 (1:1.4.4+202605061436+dcaf3e7d9e)"
sodipodi:docname="logo.svg"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg">
<sodipodi:namedview
id="namedview1"
pagecolor="#ffffff"
bordercolor="#000000"
borderopacity="0.25"
inkscape:showpageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#d1d1d1"
inkscape:document-units="mm"
showgrid="true"
inkscape:zoom="1.9332778"
inkscape:cx="-8.2760999"
inkscape:cy="112.2446"
inkscape:window-width="2584"
inkscape:window-height="1028"
inkscape:window-x="0"
inkscape:window-y="24"
inkscape:window-maximized="1"
inkscape:current-layer="layer1">
<inkscape:grid
id="grid1"
units="px"
originx="0"
originy="0"
spacingx="4"
spacingy="4"
empcolor="#0099e5"
empopacity="0.30196078"
color="#0099e5"
opacity="0.14901961"
empspacing="4"
enabled="true"
visible="true" />
</sodipodi:namedview>
<defs
id="defs1">
<linearGradient
inkscape:collect="always"
xlink:href="#linearGradient4689"
id="linearGradient1478"
gradientUnits="userSpaceOnUse"
gradientTransform="matrix(0.562541,0,0,0.567972,-9.399749,-5.305317)"
x1="26.648937"
y1="20.603781"
x2="135.66525"
y2="114.39767" />
<linearGradient
id="linearGradient4689">
<stop
style="stop-color:#e1be1e;stop-opacity:1;"
offset="0"
id="stop4691" />
<stop
style="stop-color:#ffeb82;stop-opacity:1;"
offset="1"
id="stop4693" />
</linearGradient>
<linearGradient
inkscape:collect="always"
xlink:href="#linearGradient4671"
id="linearGradient1475"
gradientUnits="userSpaceOnUse"
gradientTransform="matrix(0.562541,0,0,0.567972,-9.399749,-5.305317)"
x1="150.96111"
y1="192.35176"
x2="112.03144"
y2="137.27299" />
<linearGradient
id="linearGradient4671">
<stop
style="stop-color:#ffdc21;stop-opacity:1;"
offset="0"
id="stop4673" />
<stop
style="stop-color:#ffeb82;stop-opacity:1;"
offset="1"
id="stop4675" />
</linearGradient>
</defs>
<g
inkscape:label="Calque 1"
inkscape:groupmode="layer"
id="layer1">
<g
id="g1"
transform="translate(2.911719,3.414527)">
<path
style="fill:url(#linearGradient1478);fill-opacity:1"
d="m 60.510156,6.3979729 c -4.583653,0.021298 -8.960939,0.4122177 -12.8125,1.09375 C 36.35144,9.4962267 34.291407,13.691825 34.291406,21.429223 v 10.21875 h 26.8125 v 3.40625 h -26.8125 -10.0625 c -7.792459,0 -14.6157592,4.683717 -16.7500002,13.59375 -2.46182,10.212966 -2.5710151,16.586023 0,27.25 1.9059283,7.937852 6.4575432,13.593748 14.2500002,13.59375 h 9.21875 v -12.25 c 0,-8.849902 7.657144,-16.656248 16.75,-16.65625 h 26.78125 c 7.454951,0 13.406253,-6.138164 13.40625,-13.625 v -25.53125 c 0,-7.266339 -6.12998,-12.7247775 -13.40625,-13.9375001 -4.605987,-0.7667253 -9.385097,-1.1150483 -13.96875,-1.09375 z m -14.5,8.2187501 c 2.769547,0 5.03125,2.298646 5.03125,5.125 -2e-6,2.816336 -2.261703,5.09375 -5.03125,5.09375 -2.779476,-1e-6 -5.03125,-2.277415 -5.03125,-5.09375 -1e-6,-2.826353 2.251774,-5.125 5.03125,-5.125 z"
id="path1948" />
<path
style="fill:url(#linearGradient1475);fill-opacity:1"
d="m 91.228906,35.054223 v 11.90625 c 0,9.230755 -7.825895,16.999999 -16.75,17 h -26.78125 c -7.335833,0 -13.406249,6.278483 -13.40625,13.625 v 25.531247 c 0,7.26634 6.318588,11.54032 13.40625,13.625 8.487331,2.49561 16.626237,2.94663 26.78125,0 6.750155,-1.95439 13.406253,-5.88761 13.40625,-13.625 V 92.897973 h -26.78125 v -3.40625 h 26.78125 13.406254 c 7.79246,0 10.69625,-5.435408 13.40624,-13.59375 2.79933,-8.398886 2.68022,-16.475776 0,-27.25 -1.92578,-7.757441 -5.60387,-13.59375 -13.40624,-13.59375 z m -15.0625,64.65625 c 2.779478,3e-6 5.03125,2.277417 5.03125,5.093747 -2e-6,2.82635 -2.251775,5.125 -5.03125,5.125 -2.76955,0 -5.03125,-2.29865 -5.03125,-5.125 2e-6,-2.81633 2.261697,-5.093747 5.03125,-5.093747 z"
id="path1950" />
</g>
</g>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.7 KiB

809
docs/manual.typ
View File

@@ -1,809 +0,0 @@
//#import "@preview/codly:1.3.0": codly, codly-init
// Fix unaligned highlights in v0.15.0 ()
// See https://github.com/Dherse/codly/pull/132
#import "@local/codly:1.3.1": codly, codly-init
#import "@preview/codly-languages:0.1.10": codly-languages
#import "template.typ": TODO, project
#import "@preview/gentle-clues:1.3.1" as gc
#let midas-version = toml("../pyproject.toml").project.version
#let head-ref = read("../.git/HEAD").split(":").at(1).trim()
#let commit-hash = read("../.git/" + head-ref).slice(0, 8)
#show: project.with(
title: [Midas User Manual],
author: "Louis Heredero",
version: midas-version,
hash: commit-hash,
icon-path: path("../assets/icon.svg"),
)
#show: codly-init
#codly(
languages: codly-languages
+ (
midas: (
name: "Midas",
color: rgb("#eedd47"),
icon: box(
image(
"../assets/icon.svg",
height: 130%,
fit: "contain",
),
),
),
),
)
= Introduction
Python is a very popular programming language, especially in data sciences.
However, it has been designed for simplicity, distancing itself from typed languages such as Java or C to embrace dynamic typing.
What this means is that in Python, type checks are deferred to runtime when operations are concretely executed.
For developers, it might seem like a great way of simplifying the language and making it very flexible, but it does come with a cost.
Indeed, type errors are very easy to make in Python. While passing an integer where a string is expected might not be an issue in some cases, these are the sort of thing that can cause crashes or incorrect results without a clear diagnostic to help the user fix it.
Fortunately, developers using IDEs or properly configured text editors can benefit from external type checkers such as MyPy which will perform static type analysis of their Python code. Some can also be configured to be very strict, forcing the user to make the whole code typeable statically, thus avoiding any runtime type errors.
This is not the end of the problem though. Some parts of a program, especially in data related fields, may not be available at "compile-time". For example, a dataset can be loaded from an external file, or data can be fetched from an API, with no guarantees of having the expected format when analyzing the code statically.
In turn, that can cause a range of loud and silent errors at runtime. A malformed number will probably crash the program when trying to convert it, but a NaN in a series of value might just produce wrong results without any exception. Combine this with often long-running data-processing pipelines and this is how developers can waste hours of precious computation time.
Midas is a type system which can be used on top of Python to provide better type checking capabilities and gradual typing.
It aims at providing optional but strict type annotations and casting operations which can produce runtime assertions. It also allows the user to define dependent types with value constraints that are translated into runtime checks.
= Installation
Midas comes as a very light Python package that you can install on your system in a few simple steps.
== Requirements
Here below are the requirements for installing Midas. All Python dependencies will be installed by `uv` in the installation process described in @install-steps.
- Python 3.11+
- `uv`
== Steps <install-steps>
1. Clone the repository
```bash
git clone https://git.kb28.ch/HEL/midas.git
```
2. Navigate inside the directory
```bash
cd midas
```
3. Install Midas as a tool in your local user space
```bash
uv tool install .
```
And that's it ! You can now use Midas commands anywhere, like this:
```bash
midas --help
```
= Quick Start
This chapter will give you the keys to quickly start using Midas in your project.
== Defining custom types
To begin with, you might want to define some custom types for your project, to avoid handling anonymous float values everywhere. To do so, create a `*.midas` file in your project, and write some definitions for your types. See @midas-ref for more information on syntax and features.
@qs-midas shows a simple example of what it might look like.
#codly(header: [types.midas])
#figure(
```midas
type Meter = float
extend Meter {
def __add__: fn(Meter, /) -> Meter
def __sub__: fn(Meter, /) -> Meter
}
type Coordinate = object
extend Coordinate {
prop x: Meter
prop y: Meter
}
```,
caption: [Example Midas type definitions],
) <qs-midas>
You can check for any syntax error using the following command:
```bash
midas validate types.midas
```
When you are happy with your definitions, you can generate Python stubs to use in your source code. This allows other type checkers like MyPy to recognize your custom types and avoid reporting them as undefined. It can also help catch some type errors in your IDE.
```bash
midas stubs types.midas -o stubs.pyi
```
This command will generate a file as shown in @qs-stubs, providing stub classes to represent the type lattice including methods and properties.
#codly(header: [stubs.pyi])
#figure(
```pyi
from __future__ import annotations
class Meter(float):
def __add__(self, _0: Meter, /) -> Meter: ...
def __sub__(self, _0: Meter, /) -> Meter: ...
class Coordinate(object):
x: Meter
y: Meter
```,
caption: [Generated stubs from example definitions of @qs-midas],
) <qs-stubs>
== Using Midas in Python
You can now write your Python program as you would normally. You can import your custom types from the generated stubs file and use them in type annotations.
You can also import the `cast` and `unsafe_cast` functions from `midas.typing` to explicitly cast a value to a specific type (see @cast for more information).
An example Python script is shown in @qs-python, demonstrating how you can use custom types in type annotations. Notice the comments describing errors that will be caught by the type checker in @qs-type-checking.
#codly(header: [script.py])
#figure(
```python
from lib import load_coordinate
from midas.typing import cast
from stubs import Coordinate, Meter
p1 = cast(Coordinate, load_coordinate(0))
p2 = cast(Coordinate, load_coordinate(1))
diff_x = p2.x - p1.x
diff_y = p2.y - p1.y
dist = diff_x + diff_y
p2.x += cast(Meter, 1)
p2.y = True # invalid, wrong type
p2.z = 3 # invalid, no property 'z' on Coordinate
p2.x.a = 3 # invalid, no properties on Meter
```,
caption: [Example Python script],
) <qs-python>
== Type checking <qs-type-checking>
Now that you have defined some types and written a script, you can run the type checker with the following command. You can also skip this step and directly run the compilation command in @qs-compilation.
```bash
midas check -t types.midas script.py
```
== Compiling <qs-compilation>
The final step is to compile your code. This step will produce a runnable Python script, including runtime assertions generated by `cast` expressions.
```bash
midas compile -t types.midas script.py
python3 build/midas/script.py
```
= Midas Language Reference <midas-ref>
In this chapter, you will find a complete reference for the Midas definition language.
A `*.midas` file contains a number of statements, which can be:
- *`alias`* statements (see @alias-stmt): to define a new type alias
- *`type`* statements (see @type-stmt): to define a new type
- *`extend`* statements (see @extend-stmt): to define member of a type
- *`predicate`* statements (see @predicate-stmt): to define named predicates that can be used in constraint types
== Alias Statement <alias-stmt>
An *`alias`* statement lets you define a new type alias. It requires a unique name and base type.
While a `type` statement (see @type-stmt) allows generic definitions, aliases are purely a for givin an alternative name to a type.
#figure(
```midas
alias MyType = float
```,
caption: [Simple `alias` statement declaring a new type "`MyType`" equivalent to `float`],
) <midas-simple-alias>
This statement defines a new type called `MyType` which is equivalent to `float`. `MyType` and `float` can be used interchangeably.
== Type Statement <type-stmt>
A *`type`* statement lets you define a new type. It requires a unique name and base type.
The simplest form of a *`type`* statement is:
#figure(
```midas
type MyType = float
```,
caption: [Simple `type` statement declaring a new type "`MyType`" as a subtype of `float`],
) <midas-simple-type>
This statement defines a new type called `MyType` which is a subtype of `float`. `MyType` is a `float` but a `float` is not necessarily `MyType`.
=== Builtin / base types
A number of base types are provided out of the box, which can be used to derive other types.
They correspond to Python's builtin types:
```py object```,
```py str```,
```py float```,
```py int```,
```py bool```,
```py list```,
```py dict```,
```py None```.
Some differences are to be noted however.
1. ```py bool``` is not a subtype of ```py int```
2. ```py list``` are homogeneous, i.e. all items must be of the same type
3. ```py dict``` keys and values are homogeneous, i.e. all keys must be of the same type and all values must be of the same type (can be different from keys).
=== Function types
A function type is written in a similar notation to Python function definitions:
#figure(
```midas
type Repeater = fn(text: str, count: int) -> str
```,
caption: [Simple function type definition],
)
Midas supports positional-only, keyword-only and mixed arguments (using the `/` and `*` separators). You may omit the name of positional-only arguments. The return type is required.
Optional parameters can be indicated by adding a question mark (`?`) after their type:
#figure(
```midas
type Repeater = fn(text: str, count: int, *, sep: str?) -> str
```,
caption: [Function type definition with an optional keyword-only parameter],
)
#gc.warning[
Sink arguments (`*args`, `**kwargs`) are not currently supported.
]
=== Constraint types
A useful feature provided by Midas is the possibility to combine types with custom value constraints. For example, you might want to define a type for positive amounts of money:
#figure(
```midas
type Money = float
type Income = Money where _ >= 0
```,
caption: [Simple constraint type definition],
)
Constraints can be combined with any type using the `where` keyword, followed by a constraint expression (see @constraint-expr).
=== Generic types
For more complex types, you might want to use type parameters. For example, to define a container, we might write:
#figure(
```midas
type Container[T] = object
```,
caption: [Simple generic container type definition],
)
To better refine a generic type, you can also bound type parameters using the following syntax:
#figure(
```midas
type Container[T <: float] = object
```,
caption: [Generic container type definition with a bound],
)
This can be read as "`Container` is a generic type which takes one type parameter `T` that must be a subtype of `float`".\
You can use a generic type, i.e. instantiate it, by using a similar syntax with concrete type as arguments:
#figure(
```midas
type MyContainer = Container[MyType]
```,
caption: [Application of a generic type],
)
Generic types can also take multiple parameters, which are then separated by commas:
#figure(
```midas
type ZipCodeRegistry = dict[int, str]
```,
caption: [Application of a multi-parameter generic type],
)
The _body_ of a generic type, i.e. the right-hand side of the definition, can contain or even be equal to any number of its parameters.#footnote[The latter is not something that is expressible in standard Python, yet it brings a semantic distinction on top of structurally equivalent values.] For example, the following is a valid type statement:
#figure(
```midas
type Price[T <: Currency] = T where _ > 0
```,
caption: [Type parameters in a generic type's body],
)
=== `Column` / `Frame` types
To provide useful type-checking for data engineers, Midas offers two special types: `Column` and `Frame`.
Their goal is to help type check Pandas' `Series` and `DataFrame` respectively.
==== `Column`
The `Column` type is a generic type used to represent a `pandas.Series` object.
You can use it like any other generic type and it will provide type checking for some common methods and attributes offered by Pandas.
#figure(
```midas
type Temperature = float
alias Temperatures = Column[Temperature]
```,
caption: [Simple column type definition],
)
==== `Frame` <frame-type>
The `Frame` type is a super-powered generic type used to represent a `pandas.DataFrame` object.
In place of type arguments, `Frame` accepts a schema, i.e. a series of column definitions.
@simple-frame show how you can define a simple frame type with 3 columns:
- `name`: a column of `Name` values
- `age`: a column of `int` values
- `height`: a column of `float where _ >= 0` values
Notice that you don't need to specify `Column` types.
#figure(
```midas
type Name = str where len(_) != 0
alias Data = Frame[
name: Name,
age: int,
height: float where _ >= 0
]
```,
) <simple-frame>
#pagebreak()
== Extend Statement <extend-stmt>
Type statements allow you to define new types, kind of like type aliases. However, a type might have properties or methods of its own. These might override those of the parent type or be brand new members.
This is where the `extend` statement comes into play. It allows defining members on a given type. Members can either be properties (`prop`) or methods (`def`). The only difference between the two is that methods must be functions and can be overloaded.
Here is a simple example showing how to define a property and a method on a custom type:
#figure(
```midas
type MyType = float
extend MyType {
prop norm: float
def double: fn() -> MyType
}
```,
caption: [Simple `extend` statement defining a property and a method],
)
An `extend` statement can appear anywhere after the type it extends has been defined.
You may want to override Python's dunder methods to implement type checking for some basic operators, like `__add__` for the `+` operator.
#figure(
```midas
type Money = float
extend Money {
def __add__(Money, /) -> Money
def __mul__(float, /) -> Money
}
```,
caption: [Simple `extend` statement overriding some dunder methods],
)
When extending generic type, you must specify the whole type, including its parameter(s):
#figure(
```midas
type Container[T <: float] = object
extend Container[T <: float] {
prop content: T
def set_content: fn(content: T) -> None
}
```,
caption: [Generic `extend` statement using type parameters in the declared members],
)
#pagebreak()
== Predicate Statement <predicate-stmt>
A *`predicate`* statement lets you define a named constraint expression, like a function, which can then be used in other constraint expressions (either in other predicate statements or in constraint types). See @constraint-expr for more information about the syntax of constraint expressions.
The left-hand side of a predicate statement is written as a function signature, without a return type. The right-hand side is a constraint expression. For example:
#figure(
```midas
predicate is_positive(v: float) = v >= 0
```,
caption: [Simple `predicate` statement defining an `is_positive` predicate],
)
The left-hand side can also be curried to allow partial application. For example:
#figure(
```midas
predicate in_range(mn: float, mx: float)(v: float) = mn <= v & v <= mx
predicate is_ratio = in_range(0.0, 1.0)
```,
caption: [Curried `predicate` statement and partial application],
) <midas-predicate-partial>
Notice that the second predicate statement doesn't take any parameters. This is simply a partial application of another predicate, kind of like an alias. You can use it in other expressions to finalize the call:
#figure(
```midas
type Efficiency = float where is_ratio(_)
```,
caption: [Constraint type definition using the partially applied predicate from @midas-predicate-partial],
)
Of course you can also directly call `in_range`:
#figure(
```midas
type Efficiency = float where in_range(0.0, 1.0)(_)
```,
caption: [Full call of curried predicate from @midas-predicate-partial],
)
When compiled, named predicates are translated to Python functions which are used in runtime assertions. Only predicates that are referenced are compiled.
#pagebreak()
== Constraint Expressions <constraint-expr>
*Constraint expressions* are Python-like expressions which can appear in *`predicate`* statements or in constraint types.
They can contain comparisons, simple computations, logical operations and must evaluate to a boolean value.
Context is quite restricted inside these expressions. You can only reference some builtin functions, such as type constructors (`float(...)`, `str(...)`, etc.), parameters of predicate statements, and named predicates. In constraint type, the special variable `_` can be used to reference the value targeted by the type. For example:
#figure(
```midas
predicate not_nan(v: float) = v != float("nan")
type RealFloat = float where not_nan(_)
```,
caption: [Example constraint expressions],
) <ex-constraint-expr>
In the predicate statement (@ex-constraint-expr:1), we reference the parameter `v` and the builtin `float` function.
In the constraint type definition (@ex-constraint-expr:2), we then reference the named predicate `not_nan`, passing the value targeted by the type itself ( `_` )
= Supported Python Syntax <python-ref>
Midas integrates naturally in Python via type annotations. Through generated stubs, even other type checker can detect your custom types (see @cmd-stubs).
It has been designed to leave the user free of typing any amount of their code but be strict about the parts that are annotated. By default, any untyped Python expression is assigned `UnknownType`.
Any operation is permitted on `UnknownType` and will result in `UnknownType` values.
The moment an expression can be typed, that be thanks to an annotation or a literal value, the type checker kicks in and will validate your statements.
Because Python is very flexible language with many features, some expressions and statements might be more complex to properly type check, thus only a subset of the Python language is fully supported. This chapter lists all supported features of Python and how they affect type checking.
Some examples are presented in the following sections in the form of code blocks. Highlights in the code blocks indicate the type assigned to each expression by the type checker. Some types may be omitted for readability. For example:
#codly(
highlights: (
(
line: 1,
start: 5,
fill: green,
tag: [_int_],
),
(
line: 2,
start: 7,
end: 7,
fill: green,
tag: [_int_],
),
),
)
```python
v = 3
print(v)
```
== Literals
Literal Python values are type checked using builtin types. Lists and dictionaries of literals are also typed liked literals. This does not include comprehension lists/dicts (```py [. for . in .]```), nor formatted strings (```py f"..."```). @supported-literals shows the list of supported literal values and their type.
#let supported-literals = table(
columns: 2,
table.header[*Example value*][*Judged Type*],
```py 42```, ```py int```,
```py 3.14```, ```py float```,
```py True```, ```py bool```,
```py "Midas"```, ```py str```,
```py None```, ```py None```,
```py [1, 2, 3]```, ```py list[int]```,
```py {1: "One", 2: "Two"}```, ```py dict[int, str]```,
```py ("1", 1, True)```, ```py tuple[str, int, bool]```,
)
#figure(
supported-literals,
caption: [Supported literal values and their judged types],
) <supported-literals>
== Assignments
Variable assignments allow assigning a new value to a variable. For the type checker, this implies two things:
1. If the variable was not already declared in the current scope, it is declared at that point with the type of the right-hand side expression
2. If the variable was already declared, the type of the right-hand side expression is checked against the declared type of the variable. Only a subtype of the variable's type can be assigned to it
Once a variable has been given a type, it cannot be changed in the same scope.
The walrus operator (```py :=```) is not currently supported.
A simple annotation declaration, without assigning a value, is enough to declare a variable. For example:
#figure(
```python
var: float
```,
caption: [Bare Python variable annotation without assignment],
)
Because unpacking is not supported, assigning to multiple values is also not handled by the type checker.
For more information about type annotations, see @type-annotations
== Arithmetic
- All basic binary operators are supported, through dunder methods.
- All comparison operators except ```py in``` are supported.
- All unary operators are supported (`+`, `-`, `~`).
- All logical operators are supported (```py and```, ```py or```, ```py not```).
== Ternary operator
The ternary operator ```py . if . else .``` is supported. As for `if` statements (see @if-else), the test expression must be a boolean. Additionally, both branches must be of the same type.
For example:
#codly(
highlights: (
(
line: 1,
start: 10,
end: 44,
tag: [_str_],
fill: blue,
),
(
line: 1,
start: 11,
end: 16,
tag: [_str_],
fill: green,
),
(
line: 1,
start: 39,
end: 43,
tag: [_str_],
fill: green,
),
(
line: 1,
start: 21,
end: 32,
tag: [_bool_],
fill: green,
),
),
)
#figure(
```python
parity = ("even" if num % 2 == 0 else "odd")
```,
caption: [Typing of ternary operator],
)
== Control flow
Some control flow features are supported. For the limited code of this project, not all constructs are supported. The following are those currently handled and typ checked by Midas.
=== `if` / `elif` / `else` <if-else>
Conditional statements are checked relatively strictly by Midas. The test expression, i.e. what comes after the ```py if``` keyword, must be a boolean. While Python allows introducing and leaking new variables from inside an ```py if``` statement, Midas will strictly forbid leaks by restraining bindings to the scope they are defined in. For example, the following Python code will not compile with Midas:
#figure(
```python
age = 22
if age >= 18:
msg = "You're an adult"
else:
msg = "You're still a child"
print(msg) # -> unknown variable 'msg'
```,
caption: [`if`/`else` statement cannot leak variables],
)
=== `for` loops
Simple forms of `for` loops can be used, that is using a single variable and iterating over an object implementing the `__getitem__` method. Like above in @if-else, leaking variables from inside the loop is ignored.
`for`-`else` statements are not supported. `while` loops are also not supported.
== Functions
You can define functions as usual and the type checker will do its best to type it. Apart from argument sinks (`*args`, `**kwargs`), all forms of parameter specifications are supported (positional-only, keyword-only, mixed, optional).
As for the rest of your code, type annotations are optional, but recommended. If you omit the return type hint, the type checker will try to infer it from the function body and its return statements. If you did specify a return type, all return paths must return values that are subtypes of the type hint.
#codly(
highlights: (
(
line: 2,
start: 12,
end: 16,
tag: [_float_],
fill: green,
),
(
line: 2,
start: 12,
tag: [_float_],
fill: blue,
),
(
line: 3,
start: 10,
end: 15,
tag: [_(value: float) -> float_],
fill: green,
),
(
line: 3,
start: 17,
end: 19,
tag: [_float_],
fill: green,
),
(
line: 3,
start: 10,
tag: [_float_],
fill: blue,
),
),
)
#figure(
```python
def double(value: float) -> float:
return value * 2
result = double(4.0)
```,
caption: [Typing of function's body and call],
)
Anonymous functions (```py lambda```) are not yet supported
== Casts <cast>
#gc.info[
The functions discussed in this section are provided by the `midas.typing` submodule. You can import them in your script like so:
#figure(
```python
from midas.typing import cast, unsafe_cast
```,
caption: [Importing cast functions],
)
]
Sometimes, you may want to use a value whose type is not known to the type checker in a place where it expects a particular type. In that case, if you do know that the runtime type will correspond to what is expected, you can use a `cast` expression.
Similar to the `cast` function from the `typing` package of Python's Standard Library, it allows telling the type checker that a value has a given type. While `typing`'s function doesn't have any runtime side-effect, Midas' will generate runtime assertions, ensuring that your statement is true when running the code. What cannot be checked statically is checked at runtime.
In the following example, a runtime check would be generated to ensure that the value is indeed a `float` and that it satisfies the type's constraint (i.e. `>= 0`):
#codly(
highlights: (
(
line: 1,
start: 35,
end: 47,
tag: [_UnknownType_],
fill: red,
),
(
line: 2,
start: 7,
end: 17,
tag: [_PositiveFloat_],
fill: green,
),
),
)
#figure(
```python
typed_value = cast(PositiveFloat, unknown_value)
print(typed_value)
```,
caption: [Typing of `cast` expression],
)
#gc.warning[
Assertions are statements inserted just before a statement using a `cast` expression. This means that the expression is evaluated _before_ its actual intended usage location, which might cause issues if you rely on logical operator short-circuiting. See @eager-eval for more information.
]
There may be some cases where the cost of checking a value at runtime is simply not worth the safety, for example when dealing with a big dataset. If do wish so, you can use `unsafe_cast` which will only tell the type checker the type of the value, without generating a runtime assertion. This maps to the default behavior of `typing`'s own `cast` function.
If the value passed to `cast` or `unsafe_cast` is a literal (e.g. an integer, a string, a list of literals, etc.), the assertion is evaluated _at compile-time_ and no runtime assertion is generated.
== Annotations / Type Hints <type-annotations>
Vanilla Python already lets you use type hints to specify the type of variables and function parameters.
Midas use them to type check your code. Additionally, it allows you to use a special syntax to define a `Frame` types directly in these annotations.
Because these annotations are not interpretable by Python, your integrated type checker might complain loudly about them being invalid.
A workaround is to silence it by adding a type comment at the end of the line, as shown in @silence-errors.
#figure(
```python
var: Frame[name: str, age: float] # type: ignore # noqa: F821
```,
caption: [MyPy's and Pylance's complaints about custom type annotation can be silenced with type comments],
) <silence-errors>
=== Frame type annotation
The syntax is similar to how you can define frame types in the Midas language (see @frame-type). The only difference is that types can only be name references; you cannot inline constraint types.
The example of @python-frame-type shows how you can annotate a dataframe with some columns directly in Python.
#figure(
```python
df: Frame[name: Name, age: float, height: Length[Meter]] = ...
```,
caption: [Frame type annotation in Python],
) <python-frame-type>
= Commands <commands>
#TODO
== Type Checking (`check`) <cmd-check>
== Compiling (`compile`) <cmd-compile>
== Formatting (`format`) <cmd-format>
== Highlighting (`highlight`) <cmd-highlight>
== Dumping the AST (`parse`) <cmd-parse>
== Dumping the Registry (`dump-registry`) <cmd-registry>
== Generating Stubs (`stubs`) <cmd-stubs>
== Showing Type Judgements (`types`) <cmd-types>
== Validating Definitions (`validate`) <cmd-validate>
= Known limitations <limitations>
== Eager evaluation in runtime assertions <eager-eval>
The process of generating assertions to ensure safety at runtime, mainly for `cast` expressions, leads to the creation of aliases for the expressions being casted. These alias definitions eagerly evaluate before the assertion, and most importantly before the real usage location. This means that you should avoid using `cast` expressions inside logical expressions like `and` or `or`, because the normal "short-circuit" behavior will be irrelevant to the evaluations of the operands.
For example:
#figure(
```py
def foo():
print("Foo")
return True
def bar():
print("Bar")
return True
result = foo() or bar()
# Foo
# Bar
```,
caption: [Runtime assertions may eagerly evaluate expressions and bypass logical operator's short-circuit],
)

211
docs/midas.sublime-syntax
View File

@@ -1,211 +0,0 @@
%YAML 1.2
---
name: Midas
file_extensions:
- midas
scope: source.midas
variables:
identifier: "[a-zA-Z_][a-zA-Z0-9_]*"
contexts:
prototype:
- include: comments
main:
- include: keywords
- include: types
comments:
- match: "//"
scope: punctuation.definition.comment.midas
push:
- meta_scope: comment.line.midas
- match: $
pop: true
- match: /\*
scope: punctuation.definition.comment.midas
push:
- meta_scope: comment.block.midas
- match: \*/
pop: true
string:
- meta_include_prototype: false
- meta_scope: string.quoted.double.c
- match: '"'
pop: true
keywords:
- match: \balias\b
scope: keyword.declaration.midas
push: alias-stmt
- match: \btype\b
scope: keyword.declaration.midas
push: type-stmt
- match: \bextend\b
scope: keyword.declaration.midas
push: extend-stmt
- match: \bpredicate\b
scope: keyword.declaration.midas
push: predicate-stmt
alias-stmt:
- match: "{{identifier}}"
scope: entity.name.type
- match: "="
scope: keyword.operator.equal.midas
push: type-expr
- match: $
pop: true
type-stmt:
- match: "{{identifier}}"
scope: entity.name.type
- match: \[
push: type-params
- match: "="
scope: keyword.operator.equal.midas
push: type-expr
- match: $
pop: true
type-expr:
- match: \b(fn)\s*(\()
captures:
1: keyword.other.midas
2: punctuation.section.group.begin
push: fn-params
- match: \b(where)\b
scope: keyword.other.midas
set: constraint
- match: "Frame"
scope: entity.name.type
push:
- match: \[
push: frame-schema
- match: $
pop: true
- match: "{{identifier}}"
scope: entity.name.type
- match: $
pop: 2
fn-params:
- match: "({{identifier}})(:)"
captures:
1: variable.parameter.midas
2: punctuation.separator.annotation.midas
push:
- include: type-expr
- match: \?
scope: keyword.operator.qmark.midas
- match: "(?=,)"
scope: punctuation.separator.midas
pop: true
- match: '(?=\))'
pop: true
- include: type-expr
- match: '\)'
set:
- match: "->"
scope: keyword.operator.arrow.midas
set: type-expr
constraint:
- match: $
pop: 2
- match: \d+(\.\d+)?
scope: constant.numeric.midas
- match: \b(true|false|none)\b
scope: constant.language.midas
- match: '"'
push: string
- match: (<=|>=|<|>|==|!=|&)
scope: keyword.operator
- match: _
scope: variable.language.midas
- match: '{{identifier}}(?=\s*\()'
scope: variable.function.midas
- match: "{{identifier}}"
scope: variable.other.readwrite.midas
type-params:
- match: "<:"
scope: keyword.operator.subtype.midas
- match: "[a-zA-Z][a-zA-Z_0-9]*"
scope: entity.name.type
- match: "]"
pop: true
extend-stmt:
- match: "{{identifier}}"
scope: entity.name.type
- match: \[
push: type-params
- match: \{
scope: punctuation.section.block.begin
set: extend-body
extend-body:
- include: member-stmt
- match: \}
scope: punctuation.section.block.end
pop: true
member-stmt:
- match: \b(prop|def)\b
scope: keyword.other.midas
push:
- match: "{{identifier}}"
scope: variable.other.member
- match: ":"
push: type-expr
- match: $
pop: true
predicate-stmt:
- match: "{{identifier}}"
scope: entity.name.function.midas
- match: '\('
push: predicate-params
- match: "="
scope: keyword.operator.equal.midas
set: constraint
- match: $
pop: true
predicate-params:
- match: "({{identifier}})(:)"
captures:
1: variable.parameter.midas
2: punctuation.separator.annotation.midas
push:
- include: type-expr
- match: "(?=,)"
scope: punctuation.separator.midas
pop: true
- match: '(?=\))'
pop: true
- match: '\)'
pop: true
frame-schema:
- include: frame-column
- match: \]
# scope: punctuation.section.block.end
pop: true
frame-column:
- match: "{{identifier}}"
scope: variable.other.member
- match: ":"
push: type-expr

143
docs/template.typ
View File

@@ -1,143 +0,0 @@
#import "@preview/modpattern:0.2.0": modpattern
#let TODO = block(
width: 6em,
height: 3em,
stroke: red,
fill: modpattern(
size: (10pt, 10pt),
line(
start: (0%, 0%),
end: (100%, 100%),
stroke: gray.transparentize(60%) + 2pt,
),
),
align(
center + horizon,
text(fill: red, size: 1.5em)[*TODO*],
),
)
#let _render-header(version, hash) = {
let last-heading = query(heading.where(level: 1).before(here())).last(default: none)
let next-heading = query(heading.where(level: 1).after(here())).first(default: none)
let current-heading = if next-heading != none and next-heading.location().page() == here().page() {
next-heading
} else if last-heading != none {
last-heading
} else { none }
let chapter = if current-heading != none {
let body = current-heading.body
if current-heading.numbering != none {
let num = counter(heading).display(current-heading.numbering, at: current-heading.location())
body = [#num #body]
}
body
} else []
grid(
columns: (1fr, auto, 1fr),
align: (left, center, right),
document.title, [v#version - #hash], chapter,
)
}
#let _unshift-prefix(prefix, content) = context {
pad(left: -measure(prefix).width, prefix + content)
}
#let project(
title: none,
author: none,
version: "0.0.1",
hash: "abcdefgh",
icon-path: none,
doc,
) = {
assert(title != none, message: "Please provide a title")
set document(
title: title,
author: author,
)
set text(
font: "Source Sans 3",
)
set raw(syntaxes: path("midas.sublime-syntax"))
let front-page() = {
align(center)[
#{
set text(size: 1.5em)
std.title()
}
v#version - #hash
#if icon-path != none {
v(1cm)
image(icon-path)
}
]
pagebreak()
}
let outlines() = {
outline()
pagebreak()
outline(
title: [List of Listings],
target: figure.where(kind: raw),
)
outline(
title: [List of Tables],
target: figure.where(kind: table),
)
}
let main() = {
// Adapted from https://github.com/hei-templates/hei-synd-thesis/blob/7d2b941197babae0bf3afd4e5914754e09a64001/lib/template-thesis.typ#L242-L261
show heading.where(level: 1): it => {
pagebreak()
set text(size: 1.5em)
set block(above: 1.2em, below: 1.2em)
if it.numbering != none {
let num = numbering(it.numbering, ..counter(heading).at(it.location()))
let prefix = num + h(1em)
_unshift-prefix(prefix, it.body)
} else {
it
}
}
show heading.where(level: 2): it => {
if it.numbering != none {
let num = numbering(it.numbering, ..counter(heading).at(it.location()))
_unshift-prefix(num + h(0.8em), it.body)
} else {
it
}
}
set page(
header: context _render-header(version, hash),
footer: context if page.numbering != none {
align(center, counter(page).display(page.numbering, both: true))
},
numbering: "1 / 1",
)
show heading: set heading(numbering: "I.1.")
counter(page).update(1)
doc
}
front-page()
outlines()
main()
}

15
examples/02_demonstration/demo.midas
View File

@@ -1,15 +0,0 @@
predicate in_range(min: float, max: float)(v: float) = min <= v & v <= max
predicate is_ratio = in_range(0, 1)
type Currency = float
type Price[T <: Currency] = T where _ >= 0
extend Price[T <: Currency] {
def __add__: fn(Price[T], /) -> Price[T]
}
type EUR = Currency
type USD = Currency
type CHF = Currency
type Discount = float where is_ratio(_)

35
examples/02_demonstration/demo.py
View File

@@ -1,35 +0,0 @@
from typing import TypeVar
from demo_stubs import CHF, EUR, USD, Currency, Discount, Price
from midas.typing import cast, unsafe_cast
T = TypeVar("T", bound=Currency)
def apply_discount(amount: Price[T], discount: Discount) -> Price[T]:
return cast(Price[T], (1.0 - discount) * amount)
a1 = cast(Price[EUR], 3.2)
a2 = cast(Price[USD], 10.4)
r1 = cast(Discount, 0.2)
print(apply_discount(a1, r1))
print(apply_discount(a2, r1))
a3 = a1 + a1
a4 = a1 + a2 # cannot add euros and dollars
a3 = a2 # cannot change variable type
dyn_price = float(input("Price (CHF): "))
dyn_discount = float(input("Discount (0.0-1.0): "))
discounted = apply_discount(
cast(Price[CHF], dyn_price),
cast(Discount, dyn_discount),
)
print(f"Discounted: CHF {discounted}")
large_data = [i * 10 for i in range(100)]
prices = unsafe_cast(list[Price[EUR]], large_data)

14
examples/02_demonstration/demo_stubs.pyi
View File

@@ -1,14 +0,0 @@
from __future__ import annotations
from typing import Generic, TypeVar
class Currency(float): ...
_T0 = TypeVar("_T0", bound=Currency, covariant=True)
class Price(Currency, Generic[_T0]):
def __add__(self, _0: Price[_T0], /) -> Price[_T0]: ...
class EUR(Currency): ...
class USD(Currency): ...
class CHF(Currency): ...
class Discount(float): ...

6
gen/gen.py
View File

@@ -1,9 +1,3 @@
"""
Helper script to generate AST nodes for Midas and Python.
Takes in simple templates and generates full dataclasses and a visitor interface
"""
import re
from pathlib import Path

40
gen/midas.py
View File

@@ -26,14 +26,6 @@ class MemberKind(Enum):
METHOD = auto()
@dataclass(frozen=True, kw_only=True)
class ParamSpec:
l_paren: Token
pos: list[FunctionType.Parameter]
mixed: list[FunctionType.Parameter]
kw: list[FunctionType.Parameter]
###<
@@ -44,11 +36,6 @@ class TypeStmt:
type: Type
class AliasStmt:
name: Token
type: Type
class MemberStmt:
name: Token
type: Type
@@ -63,8 +50,9 @@ class ExtendStmt:
class PredicateStmt:
name: Token
params: list[ParamSpec]
body: Expr
subject: Token
type: Type
condition: Expr
###<
@@ -90,12 +78,6 @@ class UnaryExpr:
right: Expr
class CallExpr:
callee: Expr
arguments: list[Expr]
keywords: dict[str, Expr]
class GetExpr:
expr: Expr
name: Token
@@ -146,25 +128,17 @@ class ExtensionType:
class FunctionType:
params: ParamSpec
pos_args: list[Argument]
args: list[Argument]
kw_args: list[Argument]
returns: Type
@dataclass(frozen=True, kw_only=True)
class Parameter:
class Argument:
location: Optional[Location] = None
name: Optional[Token]
type: Type
required: bool
class FrameType:
columns: list[Column]
@dataclass(frozen=True, kw_only=True)
class Column:
location: Optional[Location] = None
name: Token
type: Type
###<

34
gen/python.py
View File

@@ -12,25 +12,10 @@ from midas.ast.location import Location
###<
###> Preamble
@dataclass(frozen=True, kw_only=True)
class ParamSpec:
pos: list[Function.Parameter]
mixed: list[Function.Parameter]
kw: list[Function.Parameter]
@property
def all(self) -> list[Function.Parameter]:
return self.pos + self.mixed + self.kw
###<
###> MidasType | Type annotations | node
class BaseType:
base: str
args: tuple[MidasType, ...]
param: Optional[MidasType]
class ConstraintType:
@@ -57,17 +42,25 @@ class ExpressionStmt:
class Function:
name: str
params: ParamSpec
posonlyargs: list[Argument]
args: list[Argument]
sink: Optional[Argument]
kwonlyargs: list[Argument]
kw_sink: Optional[Argument]
returns: Optional[MidasType]
body: list[Stmt]
@dataclass(frozen=True, kw_only=True)
class Parameter:
class Argument:
location: Optional[Location] = None
name: str
type: Optional[MidasType]
default: Optional[Expr]
@property
def all_args(self) -> list[Argument]:
return self.posonlyargs + self.args + self.kwonlyargs
class TypeAssign:
name: str
@@ -152,7 +145,6 @@ class LogicalExpr:
class CastExpr:
type: MidasType
expr: Expr
unsafe: bool
class TernaryExpr:
@@ -181,10 +173,6 @@ class SliceExpr:
step: Optional[Expr]
class TupleExpr:
items: tuple[Expr, ...]
class RawExpr:
expr: ast.expr

12
midas/ast/location.py
View File

@@ -13,8 +13,6 @@ class HasLocation(Protocol):
@dataclass(frozen=True, kw_only=True)
class Location:
"""Information about the location of an AST node"""
lineno: int
col_offset: int
end_lineno: Optional[int]
@@ -31,16 +29,6 @@ class Location:
@staticmethod
def span(start: Location, end: Location) -> Location:
"""Create a new location spanning from one location to another
Args:
start (Location): the starting location
end (Location): the end location
Returns:
Location: a new location spanning from the start of `start`
to the end of `end`
"""
return Location(
lineno=start.lineno,
col_offset=start.col_offset,

61
midas/ast/midas.py
View File

@@ -27,14 +27,6 @@ class MemberKind(Enum):
METHOD = auto()
@dataclass(frozen=True, kw_only=True)
class ParamSpec:
l_paren: Token
pos: list[FunctionType.Parameter]
mixed: list[FunctionType.Parameter]
kw: list[FunctionType.Parameter]
##############
# Statements #
##############
@@ -51,9 +43,6 @@ class Stmt(ABC):
@abstractmethod
def visit_type_stmt(self, stmt: TypeStmt) -> T: ...
@abstractmethod
def visit_alias_stmt(self, stmt: AliasStmt) -> T: ...
@abstractmethod
def visit_member_stmt(self, stmt: MemberStmt) -> T: ...
@@ -74,15 +63,6 @@ class TypeStmt(Stmt):
return visitor.visit_type_stmt(self)
@dataclass(frozen=True)
class AliasStmt(Stmt):
name: Token
type: Type
def accept(self, visitor: Stmt.Visitor[T]) -> T:
return visitor.visit_alias_stmt(self)
@dataclass(frozen=True)
class MemberStmt(Stmt):
name: Token
@@ -106,8 +86,9 @@ class ExtendStmt(Stmt):
@dataclass(frozen=True)
class PredicateStmt(Stmt):
name: Token
params: list[ParamSpec]
body: Expr
subject: Token
type: Type
condition: Expr
def accept(self, visitor: Stmt.Visitor[T]) -> T:
return visitor.visit_predicate_stmt(self)
@@ -135,9 +116,6 @@ class Expr(ABC):
@abstractmethod
def visit_unary_expr(self, expr: UnaryExpr) -> T: ...
@abstractmethod
def visit_call_expr(self, expr: CallExpr) -> T: ...
@abstractmethod
def visit_get_expr(self, expr: GetExpr) -> T: ...
@@ -183,16 +161,6 @@ class UnaryExpr(Expr):
return visitor.visit_unary_expr(self)
@dataclass(frozen=True)
class CallExpr(Expr):
callee: Expr
arguments: list[Expr]
keywords: dict[str, Expr]
def accept(self, visitor: Expr.Visitor[T]) -> T:
return visitor.visit_call_expr(self)
@dataclass(frozen=True)
class GetExpr(Expr):
expr: Expr
@@ -265,9 +233,6 @@ class Type(ABC):
@abstractmethod
def visit_function_type(self, type: FunctionType) -> T: ...
@abstractmethod
def visit_frame_type(self, type: FrameType) -> T: ...
@dataclass(frozen=True)
class NamedType(Type):
@@ -314,11 +279,13 @@ class ExtensionType(Type):
@dataclass(frozen=True)
class FunctionType(Type):
params: ParamSpec
pos_args: list[Argument]
args: list[Argument]
kw_args: list[Argument]
returns: Type
@dataclass(frozen=True, kw_only=True)
class Parameter:
class Argument:
location: Optional[Location] = None
name: Optional[Token]
type: Type
@@ -326,17 +293,3 @@ class FunctionType(Type):
def accept(self, visitor: Type.Visitor[T]) -> T:
return visitor.visit_function_type(self)
@dataclass(frozen=True)
class FrameType(Type):
columns: list[Column]
@dataclass(frozen=True, kw_only=True)
class Column:
location: Optional[Location] = None
name: Token
type: Type
def accept(self, visitor: Type.Visitor[T]) -> T:
return visitor.visit_frame_type(self)

789
midas/ast/printer.py Normal file
View File

@@ -0,0 +1,789 @@
from __future__ import annotations
import ast
import io
from contextlib import contextmanager
from enum import Enum, auto
from typing import Generator, Generic, Optional, Protocol, TypeVar
import midas.ast.midas as m
import midas.ast.python as p
class _Level(Enum):
EMPTY = auto()
ACTIVE = auto()
LAST = auto()
class Expr(Protocol):
def accept(self, printer: AstPrinter) -> None: ...
T = TypeVar("T", bound=Expr)
class AstPrinter(Generic[T]):
LAST_CHILD = "└── "
CHILD = "├── "
VERTICAL = "│ "
EMPTY = " "
def __init__(self):
self._levels: list[_Level] = []
self._idx: Optional[int] = None
self._buf: io.StringIO = io.StringIO()
def print(self, expr: T):
self._buf = io.StringIO()
expr.accept(self)
return self._buf.getvalue()
@contextmanager
def _child_level(self, single: bool = False) -> Generator[None, None, None]:
self._levels.append(_Level.LAST if single else _Level.ACTIVE)
try:
yield
finally:
self._levels.pop()
def _mark_last(self):
if self._levels:
self._levels[-1] = _Level.LAST
def _write_line(self, text: str, *, last: bool = False):
if last:
self._mark_last()
indent: str = self._build_indent()
if self._idx is not None:
text = f"[{self._idx}] {text}"
self._idx = None
self._buf.write(indent + text + "\n")
def _build_indent(self) -> str:
parts: list[str] = []
for level in self._levels[:-1]:
parts.append(self.EMPTY if level == _Level.EMPTY else self.VERTICAL)
if self._levels:
if self._levels[-1] == _Level.LAST:
parts.append(self.LAST_CHILD)
self._levels[-1] = _Level.EMPTY
else:
parts.append(self.CHILD)
return "".join(parts)
def _write_optional_child(
self, label: str, child: Optional[T], *, last: bool = False
):
if last:
self._mark_last()
if child is None:
self._write_line(f"{label}: None")
else:
self._write_line(label)
with self._child_level(single=True):
child.accept(self)
class MidasAstPrinter(
AstPrinter, m.Expr.Visitor[None], m.Stmt.Visitor[None], m.Type.Visitor[None]
):
# Statements
def visit_type_stmt(self, stmt: m.TypeStmt) -> None:
self._write_line("TypeStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line("params")
with self._child_level():
for i, param in enumerate(stmt.params):
self._idx = i
if i == len(stmt.params) - 1:
self._mark_last()
self._print_type_param(param)
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def _print_type_param(self, param: m.TypeParam) -> None:
self._write_line("Param")
with self._child_level():
self._write_line(f'name: "{param.name.lexeme}"')
self._write_optional_child("bound", param.bound, last=True)
def visit_member_stmt(self, stmt: m.MemberStmt):
self._write_line("MemberStmt")
with self._child_level():
self._write_line(f"kind: {stmt.kind.name}")
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def visit_extend_stmt(self, stmt: m.ExtendStmt) -> None:
self._write_line("ExtendStmt")
with self._child_level():
self._write_line("params")
with self._child_level():
for i, param in enumerate(stmt.params):
self._idx = i
if i == len(stmt.params) - 1:
self._mark_last()
self._print_type_param(param)
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line("params")
with self._child_level():
for i, param in enumerate(stmt.params):
self._idx = i
if i == len(stmt.params) - 1:
self._mark_last()
self._print_type_param(param)
self._write_line("members", last=True)
with self._child_level():
for i, member in enumerate(stmt.members):
self._idx = i
if i == len(stmt.members) - 1:
self._mark_last()
member.accept(self)
def visit_predicate_stmt(self, stmt: m.PredicateStmt):
self._write_line("PredicateStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line(f'subject: "{stmt.subject.lexeme}"')
self._write_line("type")
with self._child_level(single=True):
stmt.type.accept(self)
self._write_line("condition", last=True)
with self._child_level(single=True):
stmt.condition.accept(self)
# Expressions
def visit_logical_expr(self, expr: m.LogicalExpr):
self._write_line("LogicalExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_binary_expr(self, expr: m.BinaryExpr):
self._write_line("BinaryExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_unary_expr(self, expr: m.UnaryExpr):
self._write_line("UnaryExpr")
with self._child_level():
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_get_expr(self, expr: m.GetExpr):
self._write_line("GetExpr")
with self._child_level():
self._write_line("expr")
with self._child_level(single=True):
expr.expr.accept(self)
self._write_line(f'name: "{expr.name.lexeme}"', last=True)
def visit_variable_expr(self, expr: m.VariableExpr):
self._write_line("VariableExpr")
with self._child_level():
self._write_line(f'name: "{expr.name.lexeme}"', last=True)
def visit_grouping_expr(self, expr: m.GroupingExpr):
self._write_line("GroupingExpr")
with self._child_level():
self._write_line("expr", last=True)
with self._child_level(single=True):
expr.expr.accept(self)
def visit_literal_expr(self, expr: m.LiteralExpr) -> None:
self._write_line("LiteralExpr")
with self._child_level():
self._write_line(f"value: {expr.value}", last=True)
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> None:
self._write_line("WildcardExpr")
def visit_named_type(self, type: m.NamedType) -> None:
self._write_line("NamedType")
with self._child_level():
self._write_line(f'name: "{type.name.lexeme}"', last=True)
def visit_generic_type(self, type: m.GenericType) -> None:
self._write_line("GenericType")
with self._child_level():
self._write_line("type")
with self._child_level():
type.type.accept(self)
self._write_line("args", last=True)
with self._child_level():
for i, param in enumerate(type.args):
self._idx = i
if i == len(type.args) - 1:
self._mark_last()
param.accept(self)
def visit_constraint_type(self, type: m.ConstraintType) -> None:
self._write_line("ConstraintType")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
type.type.accept(self)
self._write_line("constraint", last=True)
with self._child_level(single=True):
type.constraint.accept(self)
def visit_complex_type(self, type: m.ComplexType) -> None:
self._write_line("ComplexType")
with self._child_level():
self._write_line("members", last=True)
with self._child_level():
for i, member in enumerate(type.members):
self._idx = i
if i == len(type.members) - 1:
self._mark_last()
member.accept(self)
def visit_extension_type(self, type: m.ExtensionType) -> None:
self._write_line("ExtensionType")
with self._child_level():
self._write_line("base")
with self._child_level(single=True):
type.base.accept(self)
self._write_line("extension", last=True)
with self._child_level(single=True):
type.extension.accept(self)
def visit_function_type(self, type: m.FunctionType) -> None:
self._write_line("FunctionType")
with self._child_level():
self._write_line("pos_args")
with self._child_level():
for i, arg in enumerate(type.pos_args):
self._idx = i
if i == len(type.pos_args) - 1:
self._mark_last()
self._print_function_arg(arg)
self._write_line("args")
with self._child_level():
for i, arg in enumerate(type.args):
self._idx = i
if i == len(type.args) - 1:
self._mark_last()
self._print_function_arg(arg)
self._write_line("kw_args")
with self._child_level():
for i, arg in enumerate(type.kw_args):
self._idx = i
if i == len(type.kw_args) - 1:
self._mark_last()
self._print_function_arg(arg)
self._write_line("returns", last=True)
with self._child_level(single=True):
type.returns.accept(self)
def _print_function_arg(self, arg: m.FunctionType.Argument) -> None:
self._write_line("Argument")
with self._child_level():
name: str = "None"
if arg.name is not None:
name = f'"{arg.name.lexeme}"'
self._write_line(f"name: {name}")
self._write_line("type")
with self._child_level(single=True):
arg.type.accept(self)
self._write_line(f"required: {arg.required}", last=True)
class MidasPrinter(m.Expr.Visitor[str], m.Stmt.Visitor[str], m.Type.Visitor[str]):
def __init__(self, indent: int = 4):
self.indent: int = indent
self.level: int = 0
def indented(self, text: str) -> str:
return " " * (self.level * self.indent) + text
def print(self, expr: m.Expr | m.Stmt | m.Type) -> str:
self.level = 0
return expr.accept(self)
def visit_type_stmt(self, stmt: m.TypeStmt) -> str:
template: str = ""
if len(stmt.params) != 0:
params: list[str] = [self._print_type_param(param) for param in stmt.params]
template = f"[{', '.join(params)}]"
res: str = f"type {stmt.name.lexeme}{template} = {stmt.type.accept(self)}"
return self.indented(res)
def _print_type_param(self, param: m.TypeParam) -> str:
res: str = param.name.lexeme
if param.bound is not None:
res += "<:" + param.bound.accept(self)
return res
def visit_member_stmt(self, stmt: m.MemberStmt):
keyword: str = {
m.MemberKind.PROPERTY: "prop",
m.MemberKind.METHOD: "def",
}.get(stmt.kind, "")
res: str = f"{keyword} {stmt.name.lexeme}: {stmt.type.accept(self)}"
return self.indented(res)
def visit_extend_stmt(self, stmt: m.ExtendStmt):
template: str = ""
if len(stmt.params) != 0:
params: list[str] = [self._print_type_param(param) for param in stmt.params]
template = f"[{', '.join(params)}]"
res: str = self.indented(f"extend {stmt.name.lexeme}{template}")
res += " {\n"
self.level += 1
for member in stmt.members:
res += member.accept(self) + "\n"
self.level -= 1
res += self.indented("}")
return res
def visit_predicate_stmt(self, stmt: m.PredicateStmt):
name: str = stmt.name.lexeme
subject: str = stmt.subject.lexeme
type: str = stmt.type.accept(self)
condition: str = stmt.condition.accept(self)
return self.indented(f"predicate {name}({subject}: {type}) = {condition}")
def visit_logical_expr(self, expr: m.LogicalExpr):
left: str = expr.left.accept(self)
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{left} {operator} {right}"
def visit_binary_expr(self, expr: m.BinaryExpr):
left: str = expr.left.accept(self)
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{left} {operator} {right}"
def visit_unary_expr(self, expr: m.UnaryExpr):
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{operator}{right}"
def visit_get_expr(self, expr: m.GetExpr):
expr_: str = expr.expr.accept(self)
name: str = expr.name.lexeme
return f"{expr_}.{name}"
def visit_variable_expr(self, expr: m.VariableExpr):
return expr.name.lexeme
def visit_grouping_expr(self, expr: m.GroupingExpr):
expr_: str = expr.expr.accept(self)
return f"({expr_})"
def visit_literal_expr(self, expr: m.LiteralExpr):
return str(expr.value)
def visit_wildcard_expr(self, expr: m.WildcardExpr):
return "_"
def visit_named_type(self, type: m.NamedType) -> str:
return type.name.lexeme
def visit_generic_type(self, type: m.GenericType) -> str:
res: str = type.type.accept(self)
if len(type.args) != 0:
args: list[str] = [param.accept(self) for param in type.args]
res += f"[{', '.join(args)}]"
return res
def visit_constraint_type(self, type: m.ConstraintType) -> str:
res: str = type.type.accept(self)
res += " where " + type.constraint.accept(self)
return res
def visit_complex_type(self, type: m.ComplexType) -> str:
res: str = "{\n"
self.level += 1
for member in type.members:
res += member.accept(self)
res += "\n"
self.level -= 1
res += self.indented("}")
return res
def visit_extension_type(self, type: m.ExtensionType) -> str:
return f"{type.base.accept(self)} & {type.extension.accept(self)}"
def visit_function_type(self, type: m.FunctionType) -> str:
pos_args: list[str] = [self._print_arg(arg) for arg in type.pos_args]
mixed_args: list[str] = [self._print_arg(arg) for arg in type.args]
kw_args: list[str] = [self._print_arg(arg) for arg in type.kw_args]
args: list[str] = pos_args
if len(pos_args) != 0:
args.append("/")
args += mixed_args
if len(kw_args) != 0:
args.append("*")
args += kw_args
return f"fn ({', '.join(args)}) -> {type.returns.accept(self)}"
def _print_arg(self, arg: m.FunctionType.Argument) -> str:
res: str = ""
if arg.name is not None:
res += arg.name.lexeme
res += ": "
res += arg.type.accept(self)
if not arg.required:
res += "?"
return res
class PythonAstPrinter(
AstPrinter,
p.MidasType.Visitor[None],
p.Stmt.Visitor[None],
p.Expr.Visitor[None],
):
def visit_base_type(self, node: p.BaseType) -> None:
self._write_line("BaseType")
with self._child_level():
self._write_line(f"base: {node.base}")
self._write_optional_child("param", node.param, last=True)
def visit_constraint_type(self, node: p.ConstraintType) -> None:
self._write_line("ConstraintType")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
node.type.accept(self)
self._write_line(f"constraint: {ast.unparse(node.constraint)}", last=True)
def visit_frame_column(self, node: p.FrameColumn) -> None:
self._write_line("FrameColumn")
with self._child_level():
self._write_line(f"name: {node.name}")
self._write_optional_child("type", node.type, last=True)
def visit_frame_type(self, node: p.FrameType) -> None:
self._write_line("FrameType")
with self._child_level():
self._write_line("columns", last=True)
with self._child_level():
for i, col in enumerate(node.columns):
self._idx = i
if i == len(node.columns) - 1:
self._mark_last()
col.accept(self)
def visit_expression_stmt(self, stmt: p.ExpressionStmt) -> None:
stmt.expr.accept(self)
def visit_function(self, stmt: p.Function) -> None:
self._write_line("Function")
with self._child_level():
self._write_line(f"name: {stmt.name}")
self._write_line("posonlyargs")
with self._child_level():
for i, arg in enumerate(stmt.posonlyargs):
self._idx = i
if i == len(stmt.posonlyargs) - 1:
self._mark_last()
self._print_argument(arg)
self._write_line("args")
with self._child_level():
for i, arg in enumerate(stmt.args):
self._idx = i
if i == len(stmt.args) - 1:
self._mark_last()
self._print_argument(arg)
self._write_line("kwonlyargs")
with self._child_level():
for i, arg in enumerate(stmt.kwonlyargs):
self._idx = i
if i == len(stmt.kwonlyargs) - 1:
self._mark_last()
self._print_argument(arg)
self._write_optional_child("returns", stmt.returns)
self._write_line("body", last=True)
with self._child_level():
for i, body_stmt in enumerate(stmt.body):
self._idx = i
if i == len(stmt.body) - 1:
self._mark_last()
body_stmt.accept(self)
def _print_argument(self, arg: p.Function.Argument) -> None:
self._write_line("FunctionArgument")
with self._child_level():
self._write_line(f"name: {arg.name}")
self._write_optional_child("type", arg.type, last=True)
def visit_type_assign(self, stmt: p.TypeAssign) -> None:
self._write_line("TypeAssign")
with self._child_level():
self._write_line(f"name: {stmt.name}")
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def visit_assign_stmt(self, stmt: p.AssignStmt) -> None:
self._write_line("AssignStmt")
with self._child_level():
self._write_line("targets")
with self._child_level():
for i, target in enumerate(stmt.targets):
self._idx = i
if i == len(stmt.targets) - 1:
self._mark_last()
target.accept(self)
self._write_line("value", last=True)
with self._child_level(single=True):
stmt.value.accept(self)
def visit_return_stmt(self, stmt: p.ReturnStmt) -> None:
self._write_line("ReturnStmt")
with self._child_level():
self._write_optional_child("value", stmt.value, last=True)
def visit_if_stmt(self, stmt: p.IfStmt) -> None:
self._write_line("IfStmt")
with self._child_level():
self._write_line("test")
with self._child_level(single=True):
stmt.test.accept(self)
self._write_line("body")
with self._child_level():
for i, body_stmt in enumerate(stmt.body):
self._idx = i
if i == len(stmt.body) - 1:
self._mark_last()
body_stmt.accept(self)
self._write_line("orelse", last=True)
with self._child_level():
for i, else_stmt in enumerate(stmt.orelse):
self._idx = i
if i == len(stmt.orelse) - 1:
self._mark_last()
else_stmt.accept(self)
def visit_pass(self, stmt: p.Pass) -> None:
self._write_line("Pass")
def visit_for_stmt(self, stmt: p.ForStmt) -> None:
self._write_line("ForStmt")
with self._child_level():
self._write_line("target")
with self._child_level(single=True):
stmt.target.accept(self)
self._write_line("iterator")
with self._child_level(single=True):
stmt.iterator.accept(self)
self._write_line("body", last=True)
with self._child_level():
for i, body_stmt in enumerate(stmt.body):
self._idx = i
if i == len(stmt.body) - 1:
self._mark_last()
body_stmt.accept(self)
def visit_raw_stmt(self, stmt: p.RawStmt) -> None:
self._write_line("RawStmt")
with self._child_level(single=True):
self._write_line(f"stmt: {ast.unparse(stmt.stmt)}")
def visit_binary_expr(self, expr: p.BinaryExpr) -> None:
self._write_line("BinaryExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_compare_expr(self, expr: p.CompareExpr) -> None:
self._write_line("CompareExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_unary_expr(self, expr: p.UnaryExpr) -> None:
self._write_line("UnaryExpr")
with self._child_level():
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_call_expr(self, expr: p.CallExpr) -> None:
self._write_line("CallExpr")
with self._child_level():
self._write_line("callee")
with self._child_level(single=True):
expr.callee.accept(self)
self._write_line("arguments")
with self._child_level():
for i, arg in enumerate(expr.arguments):
self._idx = i
if i == len(expr.arguments) - 1:
self._mark_last()
arg.accept(self)
self._write_line("keywords", last=True)
with self._child_level():
for i, (name, arg) in enumerate(expr.keywords.items()):
self._idx = i
if i == len(expr.keywords) - 1:
self._mark_last()
self._write_line(name)
with self._child_level(single=True):
arg.accept(self)
def visit_get_expr(self, expr: p.GetExpr) -> None:
self._write_line("GetExpr")
with self._child_level():
self._write_line("object")
with self._child_level(single=True):
expr.object.accept(self)
self._write_line(f"name: {expr.name}", last=True)
def visit_literal_expr(self, expr: p.LiteralExpr) -> None:
self._write_line("LiteralExpr")
with self._child_level(single=True):
self._write_line(f"value: {expr.value!r}")
def visit_variable_expr(self, expr: p.VariableExpr) -> None:
self._write_line("VariableExpr")
with self._child_level(single=True):
self._write_line(f"name: {expr.name}")
def visit_logical_expr(self, expr: p.LogicalExpr) -> None:
self._write_line("LogicalExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_cast_expr(self, expr: p.CastExpr) -> None:
self._write_line("CastExpr")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
expr.type.accept(self)
self._write_line("expr", last=True)
with self._child_level(single=True):
expr.expr.accept(self)
def visit_ternary_expr(self, expr: p.TernaryExpr) -> None:
self._write_line("TernaryExpr")
with self._child_level():
self._write_line("test")
with self._child_level(single=True):
expr.test.accept(self)
self._write_line("if_true")
with self._child_level(single=True):
expr.if_true.accept(self)
self._write_line("if_false", last=True)
with self._child_level(single=True):
expr.if_false.accept(self)
def visit_list_expr(self, expr: p.ListExpr) -> None:
self._write_line("ListExpr")
with self._child_level():
self._write_line("items", last=True)
with self._child_level():
for i, item in enumerate(expr.items):
self._idx = i
if i == len(expr.items) - 1:
self._mark_last()
item.accept(self)
def visit_dict_expr(self, expr: p.DictExpr) -> None:
self._write_line("DictExpr")
with self._child_level():
self._write_line("keys")
with self._child_level():
for i, key in enumerate(expr.keys):
self._idx = i
if i == len(expr.keys) - 1:
self._mark_last()
if key is None:
self._write_line("None")
else:
key.accept(self)
self._write_line("values", last=True)
with self._child_level():
for i, value in enumerate(expr.values):
self._idx = i
if i == len(expr.values) - 1:
self._mark_last()
value.accept(self)
def visit_subscript_expr(self, expr: p.SubscriptExpr) -> None:
self._write_line("SubscriptExpr")
with self._child_level():
self._write_line("object")
with self._child_level(single=True):
expr.object.accept(self)
self._write_line("index", last=True)
with self._child_level(single=True):
expr.index.accept(self)
def visit_slice_expr(self, expr: p.SliceExpr) -> None:
self._write_line("SliceExpr")
with self._child_level():
self._write_optional_child("lower", expr.lower)
self._write_optional_child("upper", expr.upper)
self._write_optional_child("step", expr.step, last=True)
def visit_raw_expr(self, expr: p.RawExpr) -> None:
self._write_line("RawExpr")
with self._child_level(single=True):
self._write_line(f"expr: {ast.unparse(expr.expr)}")

3
midas/ast/printer/__init__.py
View File

@@ -1,3 +0,0 @@
from .midas import MidasPrinter as MidasPrinter
from .midas_ast import MidasAstPrinter as MidasAstPrinter
from .python_ast import PythonAstPrinter as PythonAstPrinter

103
midas/ast/printer/base.py
View File

@@ -1,103 +0,0 @@
from __future__ import annotations
import io
from contextlib import contextmanager
from enum import Enum, auto
from typing import Callable, Generator, Generic, Optional, Protocol, Sequence, TypeVar
class _Level(Enum):
EMPTY = auto()
ACTIVE = auto()
LAST = auto()
class Expr(Protocol):
def accept(self, printer: AstPrinter) -> None: ...
T = TypeVar("T", bound=Expr)
class AstPrinter(Generic[T]):
LAST_CHILD = "└── "
CHILD = "├── "
VERTICAL = "│ "
EMPTY = " "
def __init__(self):
self._levels: list[_Level] = []
self._idx: Optional[int] = None
self._buf: io.StringIO = io.StringIO()
def print(self, expr: T):
self._buf = io.StringIO()
expr.accept(self)
return self._buf.getvalue()
@contextmanager
def _child_level(self, single: bool = False) -> Generator[None, None, None]:
self._levels.append(_Level.LAST if single else _Level.ACTIVE)
try:
yield
finally:
self._levels.pop()
def _mark_last(self):
if self._levels:
self._levels[-1] = _Level.LAST
def _write_line(self, text: str, *, last: bool = False):
if last:
self._mark_last()
indent: str = self._build_indent()
if self._idx is not None:
text = f"[{self._idx}] {text}"
self._idx = None
self._buf.write(indent + text + "\n")
def _build_indent(self) -> str:
parts: list[str] = []
for level in self._levels[:-1]:
parts.append(self.EMPTY if level == _Level.EMPTY else self.VERTICAL)
if self._levels:
if self._levels[-1] == _Level.LAST:
parts.append(self.LAST_CHILD)
self._levels[-1] = _Level.EMPTY
else:
parts.append(self.CHILD)
return "".join(parts)
def _write_optional_child(
self, label: str, child: Optional[T], *, last: bool = False
):
if last:
self._mark_last()
if child is None:
self._write_line(f"{label}: None")
else:
self._write_line(label)
with self._child_level(single=True):
child.accept(self)
def _write_sequence(
self,
label: str,
list_: Sequence[T],
*,
last: bool = False,
print_func: Optional[Callable[[T], None]] = None,
):
if last:
self._mark_last()
self._write_line(label)
with self._child_level():
for i, item in enumerate(list_):
self._idx = i
if i == len(list_) - 1:
self._mark_last()
if print_func is not None:
print_func(item)
else:
item.accept(self)

183
midas/ast/printer/midas.py
View File

@@ -1,183 +0,0 @@
import midas.ast.midas as m
class MidasPrinter(
m.Expr.Visitor[str],
m.Stmt.Visitor[str],
m.Type.Visitor[str],
):
def __init__(self, indent: int = 4):
self.indent: int = indent
self.level: int = 0
def indented(self, text: str) -> str:
return " " * (self.level * self.indent) + text
def print(self, expr: m.Expr | m.Stmt | m.Type) -> str:
self.level = 0
return expr.accept(self)
# Statements
def visit_type_stmt(self, stmt: m.TypeStmt) -> str:
template: str = ""
if len(stmt.params) != 0:
params: list[str] = [self._print_type_param(param) for param in stmt.params]
template = f"[{', '.join(params)}]"
res: str = f"type {stmt.name.lexeme}{template} = {stmt.type.accept(self)}"
return self.indented(res)
def visit_alias_stmt(self, stmt: m.AliasStmt) -> str:
return self.indented(f"alias {stmt.name.lexeme} = {stmt.type.accept(self)}")
def _print_type_param(self, param: m.TypeParam) -> str:
res: str = param.name.lexeme
if param.bound is not None:
res += "<:" + param.bound.accept(self)
return res
def visit_member_stmt(self, stmt: m.MemberStmt):
keyword: str = {
m.MemberKind.PROPERTY: "prop",
m.MemberKind.METHOD: "def",
}.get(stmt.kind, "")
res: str = f"{keyword} {stmt.name.lexeme}: {stmt.type.accept(self)}"
return self.indented(res)
def visit_extend_stmt(self, stmt: m.ExtendStmt):
template: str = ""
if len(stmt.params) != 0:
params: list[str] = [self._print_type_param(param) for param in stmt.params]
template = f"[{', '.join(params)}]"
res: str = self.indented(f"extend {stmt.name.lexeme}{template}")
res += " {\n"
self.level += 1
for member in stmt.members:
res += member.accept(self) + "\n"
self.level -= 1
res += self.indented("}")
return res
def visit_predicate_stmt(self, stmt: m.PredicateStmt):
name: str = stmt.name.lexeme
sig: str = "".join(self._visit_param_spec(spec) for spec in stmt.params)
body: str = stmt.body.accept(self)
return self.indented(f"predicate {name}{sig} = {body}")
# Expressions
def visit_logical_expr(self, expr: m.LogicalExpr):
left: str = expr.left.accept(self)
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{left} {operator} {right}"
def visit_binary_expr(self, expr: m.BinaryExpr):
left: str = expr.left.accept(self)
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{left} {operator} {right}"
def visit_unary_expr(self, expr: m.UnaryExpr):
operator: str = expr.operator.lexeme
right: str = expr.right.accept(self)
return f"{operator}{right}"
def visit_call_expr(self, expr: m.CallExpr) -> str:
args: list[str] = [arg.accept(self) for arg in expr.arguments] + [
f"{name}={arg.accept(self)}" for name, arg in expr.keywords.items()
]
return f"{expr.callee.accept(self)}({', '.join(args)})"
def visit_get_expr(self, expr: m.GetExpr):
expr_: str = expr.expr.accept(self)
name: str = expr.name.lexeme
return f"{expr_}.{name}"
def visit_variable_expr(self, expr: m.VariableExpr):
return expr.name.lexeme
def visit_grouping_expr(self, expr: m.GroupingExpr):
expr_: str = expr.expr.accept(self)
return f"({expr_})"
def visit_literal_expr(self, expr: m.LiteralExpr):
return str(expr.value)
def visit_wildcard_expr(self, expr: m.WildcardExpr):
return "_"
# Types
def visit_named_type(self, type: m.NamedType) -> str:
return type.name.lexeme
def visit_generic_type(self, type: m.GenericType) -> str:
res: str = type.type.accept(self)
if len(type.args) != 0:
args: list[str] = [param.accept(self) for param in type.args]
res += f"[{', '.join(args)}]"
return res
def visit_constraint_type(self, type: m.ConstraintType) -> str:
res: str = type.type.accept(self)
res += " where " + type.constraint.accept(self)
return res
def visit_complex_type(self, type: m.ComplexType) -> str:
res: str = "{\n"
self.level += 1
for member in type.members:
res += member.accept(self)
res += "\n"
self.level -= 1
res += self.indented("}")
return res
def visit_extension_type(self, type: m.ExtensionType) -> str:
return f"{type.base.accept(self)} & {type.extension.accept(self)}"
def visit_function_type(self, type: m.FunctionType) -> str:
spec: str = self._visit_param_spec(type.params)
return f"fn {spec} -> {type.returns.accept(self)}"
def _visit_param_spec(self, spec: m.ParamSpec) -> str:
pos: list[str] = [self._print_param(param) for param in spec.pos]
mixed: list[str] = [self._print_param(param) for param in spec.mixed]
kw: list[str] = [self._print_param(param) for param in spec.kw]
params: list[str] = pos
if len(pos) != 0:
params.append("/")
params += mixed
if len(kw) != 0:
params.append("*")
params += kw
return f"({', '.join(params)})"
def _print_param(self, param: m.FunctionType.Parameter) -> str:
res: str = ""
if param.name is not None:
res += param.name.lexeme
res += ": "
res += param.type.accept(self)
if not param.required:
res += "?"
return res
def visit_frame_type(self, type: m.FrameType) -> str:
res: str = self.indented("Frame[")
if len(type.columns) != 0:
res += "\n"
self.level += 1
columns: list[str] = []
for column in type.columns:
columns.append(self.indented(self._print_frame_column(column)))
res += ",\n".join(columns)
self.level -= 1
res += "\n"
res += "]"
return res
def _print_frame_column(self, column: m.FrameType.Column) -> str:
return f"{column.name.lexeme}: {column.type.accept(self)}"

253
midas/ast/printer/midas_ast.py
View File

@@ -1,253 +0,0 @@
import midas.ast.midas as m
from midas.ast.printer.base import AstPrinter
class MidasAstPrinter(
AstPrinter,
m.Expr.Visitor[None],
m.Stmt.Visitor[None],
m.Type.Visitor[None],
):
# Statements
def visit_type_stmt(self, stmt: m.TypeStmt) -> None:
self._write_line("TypeStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_sequence(
"params",
stmt.params,
print_func=self._print_type_param,
)
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def visit_alias_stmt(self, stmt: m.AliasStmt) -> None:
self._write_line("AliasStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def _print_type_param(self, param: m.TypeParam) -> None:
self._write_line("Param")
with self._child_level():
self._write_line(f'name: "{param.name.lexeme}"')
self._write_optional_child("bound", param.bound, last=True)
def visit_member_stmt(self, stmt: m.MemberStmt):
self._write_line("MemberStmt")
with self._child_level():
self._write_line(f"kind: {stmt.kind.name}")
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def visit_extend_stmt(self, stmt: m.ExtendStmt) -> None:
self._write_line("ExtendStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_sequence(
"params",
stmt.params,
print_func=self._print_type_param,
)
self._write_sequence("members", stmt.members, last=True)
def visit_predicate_stmt(self, stmt: m.PredicateStmt):
self._write_line("PredicateStmt")
with self._child_level():
self._write_line(f'name: "{stmt.name.lexeme}"')
self._write_sequence(
"params",
stmt.params,
print_func=self._visit_param_spec,
)
self._write_line("body", last=True)
with self._child_level(single=True):
stmt.body.accept(self)
# Expressions
def visit_logical_expr(self, expr: m.LogicalExpr):
self._write_line("LogicalExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_binary_expr(self, expr: m.BinaryExpr):
self._write_line("BinaryExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_unary_expr(self, expr: m.UnaryExpr):
self._write_line("UnaryExpr")
with self._child_level():
self._write_line(f"operator: {expr.operator.lexeme}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_call_expr(self, expr: m.CallExpr) -> None:
self._write_line("CallExpr")
with self._child_level():
self._write_line("callee")
with self._child_level(single=True):
expr.callee.accept(self)
self._write_sequence("arguments", expr.arguments)
self._write_line("keywords", last=True)
with self._child_level():
for i, (name, arg) in enumerate(expr.keywords.items()):
self._idx = i
if i == len(expr.keywords) - 1:
self._mark_last()
self._write_line(name)
with self._child_level(single=True):
arg.accept(self)
def visit_get_expr(self, expr: m.GetExpr):
self._write_line("GetExpr")
with self._child_level():
self._write_line("expr")
with self._child_level(single=True):
expr.expr.accept(self)
self._write_line(f'name: "{expr.name.lexeme}"', last=True)
def visit_variable_expr(self, expr: m.VariableExpr):
self._write_line("VariableExpr")
with self._child_level():
self._write_line(f'name: "{expr.name.lexeme}"', last=True)
def visit_grouping_expr(self, expr: m.GroupingExpr):
self._write_line("GroupingExpr")
with self._child_level():
self._write_line("expr", last=True)
with self._child_level(single=True):
expr.expr.accept(self)
def visit_literal_expr(self, expr: m.LiteralExpr) -> None:
self._write_line("LiteralExpr")
with self._child_level():
self._write_line(f"value: {expr.value}", last=True)
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> None:
self._write_line("WildcardExpr")
# Types
def visit_named_type(self, type: m.NamedType) -> None:
self._write_line("NamedType")
with self._child_level():
self._write_line(f'name: "{type.name.lexeme}"', last=True)
def visit_generic_type(self, type: m.GenericType) -> None:
self._write_line("GenericType")
with self._child_level():
self._write_line("type")
with self._child_level():
type.type.accept(self)
self._write_sequence("args", type.args, last=True)
def visit_constraint_type(self, type: m.ConstraintType) -> None:
self._write_line("ConstraintType")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
type.type.accept(self)
self._write_line("constraint", last=True)
with self._child_level(single=True):
type.constraint.accept(self)
def visit_complex_type(self, type: m.ComplexType) -> None:
self._write_line("ComplexType")
with self._child_level():
self._write_sequence("members", type.members, last=True)
def visit_extension_type(self, type: m.ExtensionType) -> None:
self._write_line("ExtensionType")
with self._child_level():
self._write_line("base")
with self._child_level(single=True):
type.base.accept(self)
self._write_line("extension", last=True)
with self._child_level(single=True):
type.extension.accept(self)
def visit_function_type(self, type: m.FunctionType) -> None:
self._write_line("FunctionType")
with self._child_level():
self._write_line("params")
with self._child_level(single=True):
self._visit_param_spec(type.params)
self._write_line("returns", last=True)
with self._child_level(single=True):
type.returns.accept(self)
def _visit_param_spec(self, spec: m.ParamSpec) -> None:
self._write_line("ParamSpec")
with self._child_level():
self._write_sequence(
"pos",
spec.pos,
print_func=self._print_param,
)
self._write_sequence(
"mixed",
spec.mixed,
print_func=self._print_param,
)
self._write_sequence(
"kw",
spec.kw,
print_func=self._print_param,
last=True,
)
def _print_param(self, param: m.FunctionType.Parameter) -> None:
self._write_line("Parameter")
with self._child_level():
name: str = "None"
if param.name is not None:
name = f'"{param.name.lexeme}"'
self._write_line(f"name: {name}")
self._write_line("type")
with self._child_level(single=True):
param.type.accept(self)
self._write_line(f"required: {param.required}", last=True)
def visit_frame_type(self, type: m.FrameType) -> None:
self._write_line("FrameType")
with self._child_level(single=True):
self._write_sequence(
"columns",
type.columns,
print_func=self._print_frame_column,
)
def _print_frame_column(self, column: m.FrameType.Column) -> None:
self._write_line("Column")
with self._child_level():
self._write_line(f'name: "{column.name.lexeme}"')
self._write_line("type")
with self._child_level(single=True):
column.type.accept(self)

285
midas/ast/printer/python_ast.py
View File

@@ -1,285 +0,0 @@
import ast
import midas.ast.python as p
from midas.ast.printer.base import AstPrinter
class PythonAstPrinter(
AstPrinter,
p.MidasType.Visitor[None],
p.Stmt.Visitor[None],
p.Expr.Visitor[None],
):
# Types
def visit_base_type(self, node: p.BaseType) -> None:
self._write_line("BaseType")
with self._child_level():
self._write_line(f"base: {node.base}")
self._write_sequence("args", node.args, last=True)
def visit_constraint_type(self, node: p.ConstraintType) -> None:
self._write_line("ConstraintType")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
node.type.accept(self)
self._write_line(f"constraint: {ast.unparse(node.constraint)}", last=True)
def visit_frame_column(self, node: p.FrameColumn) -> None:
self._write_line("FrameColumn")
with self._child_level():
self._write_line(f"name: {node.name}")
self._write_optional_child("type", node.type, last=True)
def visit_frame_type(self, node: p.FrameType) -> None:
self._write_line("FrameType")
with self._child_level(single=True):
self._write_sequence("columns", node.columns)
# Statements
def visit_expression_stmt(self, stmt: p.ExpressionStmt) -> None:
stmt.expr.accept(self)
def visit_function(self, stmt: p.Function) -> None:
self._write_line("Function")
with self._child_level():
self._write_line(f"name: {stmt.name}")
self._write_line("params")
with self._child_level():
self._print_param_spec(stmt.params)
self._write_optional_child("returns", stmt.returns)
self._write_sequence("body", stmt.body, last=True)
def _print_param_spec(self, spec: p.ParamSpec) -> None:
self._write_line("ParamSpec")
with self._child_level():
self._write_sequence(
"pos",
spec.pos,
print_func=self._print_param,
)
self._write_sequence(
"mixed",
spec.mixed,
print_func=self._print_param,
)
self._write_sequence(
"kw",
spec.kw,
print_func=self._print_param,
last=True,
)
def _print_param(self, param: p.Function.Parameter) -> None:
self._write_line("Parameter")
with self._child_level():
self._write_line(f"name: {param.name}")
self._write_optional_child("type", param.type, last=True)
def visit_type_assign(self, stmt: p.TypeAssign) -> None:
self._write_line("TypeAssign")
with self._child_level():
self._write_line(f"name: {stmt.name}")
self._write_line("type", last=True)
with self._child_level(single=True):
stmt.type.accept(self)
def visit_assign_stmt(self, stmt: p.AssignStmt) -> None:
self._write_line("AssignStmt")
with self._child_level():
self._write_sequence("targets", stmt.targets)
self._write_line("value", last=True)
with self._child_level(single=True):
stmt.value.accept(self)
def visit_return_stmt(self, stmt: p.ReturnStmt) -> None:
self._write_line("ReturnStmt")
with self._child_level():
self._write_optional_child("value", stmt.value, last=True)
def visit_if_stmt(self, stmt: p.IfStmt) -> None:
self._write_line("IfStmt")
with self._child_level():
self._write_line("test")
with self._child_level(single=True):
stmt.test.accept(self)
self._write_sequence("body", stmt.body)
self._write_sequence("orelse", stmt.orelse, last=True)
def visit_pass(self, stmt: p.Pass) -> None:
self._write_line("Pass")
def visit_for_stmt(self, stmt: p.ForStmt) -> None:
self._write_line("ForStmt")
with self._child_level():
self._write_line("target")
with self._child_level(single=True):
stmt.target.accept(self)
self._write_line("iterator")
with self._child_level(single=True):
stmt.iterator.accept(self)
self._write_sequence("body", stmt.body, last=True)
def visit_raw_stmt(self, stmt: p.RawStmt) -> None:
self._write_line("RawStmt")
with self._child_level(single=True):
self._write_line(f"stmt: {ast.unparse(stmt.stmt)}")
# Expressions
def visit_binary_expr(self, expr: p.BinaryExpr) -> None:
self._write_line("BinaryExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_compare_expr(self, expr: p.CompareExpr) -> None:
self._write_line("CompareExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_unary_expr(self, expr: p.UnaryExpr) -> None:
self._write_line("UnaryExpr")
with self._child_level():
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_call_expr(self, expr: p.CallExpr) -> None:
self._write_line("CallExpr")
with self._child_level():
self._write_line("callee")
with self._child_level(single=True):
expr.callee.accept(self)
self._write_sequence("arguments", expr.arguments)
self._write_line("keywords", last=True)
with self._child_level():
for i, (name, arg) in enumerate(expr.keywords.items()):
self._idx = i
if i == len(expr.keywords) - 1:
self._mark_last()
self._write_line(name)
with self._child_level(single=True):
arg.accept(self)
def visit_get_expr(self, expr: p.GetExpr) -> None:
self._write_line("GetExpr")
with self._child_level():
self._write_line("object")
with self._child_level(single=True):
expr.object.accept(self)
self._write_line(f"name: {expr.name}", last=True)
def visit_literal_expr(self, expr: p.LiteralExpr) -> None:
self._write_line("LiteralExpr")
with self._child_level(single=True):
self._write_line(f"value: {expr.value!r}")
def visit_variable_expr(self, expr: p.VariableExpr) -> None:
self._write_line("VariableExpr")
with self._child_level(single=True):
self._write_line(f"name: {expr.name}")
def visit_logical_expr(self, expr: p.LogicalExpr) -> None:
self._write_line("LogicalExpr")
with self._child_level():
self._write_line("left")
with self._child_level(single=True):
expr.left.accept(self)
self._write_line(f"operator: {expr.operator.__class__.__name__}")
self._write_line("right", last=True)
with self._child_level(single=True):
expr.right.accept(self)
def visit_cast_expr(self, expr: p.CastExpr) -> None:
self._write_line("CastExpr")
with self._child_level():
self._write_line("type")
with self._child_level(single=True):
expr.type.accept(self)
self._write_line("expr")
with self._child_level(single=True):
expr.expr.accept(self)
self._write_line(f"unsafe: {expr.unsafe}", last=True)
def visit_ternary_expr(self, expr: p.TernaryExpr) -> None:
self._write_line("TernaryExpr")
with self._child_level():
self._write_line("test")
with self._child_level(single=True):
expr.test.accept(self)
self._write_line("if_true")
with self._child_level(single=True):
expr.if_true.accept(self)
self._write_line("if_false", last=True)
with self._child_level(single=True):
expr.if_false.accept(self)
def visit_list_expr(self, expr: p.ListExpr) -> None:
self._write_line("ListExpr")
with self._child_level():
self._write_sequence("items", expr.items, last=True)
def visit_dict_expr(self, expr: p.DictExpr) -> None:
self._write_line("DictExpr")
with self._child_level():
self._write_sequence(
"keys",
expr.keys,
print_func=lambda k: (
self._write_line("None") if k is None else k.accept(self)
),
)
self._write_sequence("values", expr.values, last=True)
def visit_subscript_expr(self, expr: p.SubscriptExpr) -> None:
self._write_line("SubscriptExpr")
with self._child_level():
self._write_line("object")
with self._child_level(single=True):
expr.object.accept(self)
self._write_line("index", last=True)
with self._child_level(single=True):
expr.index.accept(self)
def visit_slice_expr(self, expr: p.SliceExpr) -> None:
self._write_line("SliceExpr")
with self._child_level():
self._write_optional_child("lower", expr.lower)
self._write_optional_child("upper", expr.upper)
self._write_optional_child("step", expr.step, last=True)
def visit_tuple_expr(self, expr: p.TupleExpr) -> None:
self._write_line("TupleExpr")
with self._child_level():
self._write_sequence("items", expr.items, last=True)
def visit_raw_expr(self, expr: p.RawExpr) -> None:
self._write_line("RawExpr")
with self._child_level(single=True):
self._write_line(f"expr: {ast.unparse(expr.expr)}")

36
midas/ast/python.py
View File

@@ -14,16 +14,6 @@ from midas.ast.location import Location
T = TypeVar("T")
@dataclass(frozen=True, kw_only=True)
class ParamSpec:
pos: list[Function.Parameter]
mixed: list[Function.Parameter]
kw: list[Function.Parameter]
@property
def all(self) -> list[Function.Parameter]:
return self.pos + self.mixed + self.kw
####################
# Type annotations #
@@ -54,7 +44,7 @@ class MidasType(ABC):
@dataclass(frozen=True)
class BaseType(MidasType):
base: str
args: tuple[MidasType, ...]
param: Optional[MidasType]
def accept(self, visitor: MidasType.Visitor[T]) -> T:
return visitor.visit_base_type(self)
@@ -138,17 +128,25 @@ class ExpressionStmt(Stmt):
@dataclass(frozen=True)
class Function(Stmt):
name: str
params: ParamSpec
posonlyargs: list[Argument]
args: list[Argument]
sink: Optional[Argument]
kwonlyargs: list[Argument]
kw_sink: Optional[Argument]
returns: Optional[MidasType]
body: list[Stmt]
@dataclass(frozen=True, kw_only=True)
class Parameter:
class Argument:
location: Optional[Location] = None
name: str
type: Optional[MidasType]
default: Optional[Expr]
@property
def all_args(self) -> list[Argument]:
return self.posonlyargs + self.args + self.kwonlyargs
def accept(self, visitor: Stmt.Visitor[T]) -> T:
return visitor.visit_function(self)
@@ -270,9 +268,6 @@ class Expr(ABC):
@abstractmethod
def visit_slice_expr(self, expr: SliceExpr) -> T: ...
@abstractmethod
def visit_tuple_expr(self, expr: TupleExpr) -> T: ...
@abstractmethod
def visit_raw_expr(self, expr: RawExpr) -> T: ...
@@ -355,7 +350,6 @@ class LogicalExpr(Expr):
class CastExpr(Expr):
type: MidasType
expr: Expr
unsafe: bool
def accept(self, visitor: Expr.Visitor[T]) -> T:
return visitor.visit_cast_expr(self)
@@ -407,14 +401,6 @@ class SliceExpr(Expr):
return visitor.visit_slice_expr(self)
@dataclass(frozen=True)
class TupleExpr(Expr):
items: tuple[Expr, ...]
def accept(self, visitor: Expr.Visitor[T]) -> T:
return visitor.visit_tuple_expr(self)
@dataclass(frozen=True)
class RawExpr(Expr):
expr: ast.expr

98
midas/checker/builtins.midas
View File

@@ -178,100 +178,4 @@ extend dict[K, V] {
// def __ior__: fn(value: SupportsKeysAndGetItem[K, V], /) -> dict[K, V]
// def __ior__: fn(value: Iterable[tuple[K, V]], /) -> dict[K, V]
}
extend str {
def capitalize: fn() -> str
def casefold: fn() -> str
def center: fn(width: int, fillchar: str?, /) -> str
def count: fn(sub: str, start: None?, end: None?, /) -> int
def count: fn(sub: str, start: int, end: None?, /) -> int
def count: fn(sub: str, start: None, end: int, /) -> int
def count: fn(sub: str, start: int, end: int, /) -> int
def encode: fn(encoding: str?, errors: str?) -> bytes
def endswith: fn(suffix: str, start: None?, end: None?, /) -> bool
def endswith: fn(suffix: str, start: int, end: None?, /) -> bool
def endswith: fn(suffix: str, start: None, end: int, /) -> bool
def endswith: fn(suffix: str, start: int, end: int, /) -> bool
def expandtabs: fn(tabsize: int?) -> str
def find: fn(sub: str, start: None?, end: None?, /) -> int
def find: fn(sub: str, start: int, end: None?, /) -> int
def find: fn(sub: str, start: None, end: int, /) -> int
def find: fn(sub: str, start: int, end: int, /) -> int
// def format: fn(*args: object, **kwargs: object) -> str
// def format_map: fn(mapping: _FormatMapMapping, /) -> str
def index: fn(sub: str, start: None?, end: None?, /) -> int
def index: fn(sub: str, start: int, end: None?, /) -> int
def index: fn(sub: str, start: None, end: int, /) -> int
def index: fn(sub: str, start: int, end: int, /) -> int
def isalnum: fn() -> bool
def isalpha: fn() -> bool
def isascii: fn() -> bool
def isdecimal: fn() -> bool
def isdigit: fn() -> bool
def isidentifier: fn() -> bool
def islower: fn() -> bool
def isnumeric: fn() -> bool
def isprintable: fn() -> bool
def isspace: fn() -> bool
def istitle: fn() -> bool
def isupper: fn() -> bool
def join: fn(iterable: list[str], /) -> str // TODO: use Iterable
def ljust: fn(width: int, fillchar: str?, /) -> str
def lower: fn() -> str
def lstrip: fn(chars: None?, /) -> str
def lstrip: fn(chars: str, /) -> str
def partition: fn(sep: str, /) -> tuple[str, str, str]
def replace: fn(old: str, new: str, count: int?, /) -> str
def removeprefix: fn(prefix: str, /) -> str
def removesuffix: fn(suffix: str, /) -> str
def rfind: fn(sub: str, start: None?, end: None?, /) -> int
def rfind: fn(sub: str, start: int, end: None?, /) -> int
def rfind: fn(sub: str, start: None, end: int, /) -> int
def rfind: fn(sub: str, start: int, end: int, /) -> int
def rindex: fn(sub: str, start: None?, end: None?, /) -> int
def rindex: fn(sub: str, start: int, end: None?, /) -> int
def rindex: fn(sub: str, start: None, end: int, /) -> int
def rindex: fn(sub: str, start: int, end: int, /) -> int
def rjust: fn(width: int, fillchar: str?, /) -> str
def rpartition: fn(sep: str, /) -> tuple[str, str, str]
def rsplit: fn(sep: None?, maxsplit: int?) -> list[str]
def rsplit: fn(sep: str, maxsplit: int?) -> list[str]
def rstrip: fn(chars: None?, /) -> str
def rstrip: fn(chars: str, /) -> str
def split: fn(sep: None?, maxsplit: int?) -> list[str]
def split: fn(sep: str, maxsplit: int?) -> list[str]
def splitlines: fn(keepends: bool?) -> list[str]
def startswith: fn(prefix: str, start: None?, end: None?, /) -> bool
def startswith: fn(prefix: str, start: int, end: None?, /) -> bool
def startswith: fn(prefix: str, start: None, end: int, /) -> bool
def startswith: fn(prefix: str, start: int, end: int, /) -> bool
def strip: fn(chars: None?, /) -> str
def strip: fn(chars: str, /) -> str
def swapcase: fn() -> str
def title: fn() -> str
// def translate: fn(table: _TranslateTable, /) -> str
def upper: fn() -> str
def zfill: fn(width: int, /) -> str
def __add__: fn(value: str, /) -> str
// Incompatible with Sequence.__contains__
def __contains__: fn(key: str, /) -> bool
def __eq__: fn(value: object, /) -> bool
def __ge__: fn(value: str, /) -> bool
def __getitem__: fn(key: slice, /) -> str
def __getitem__: fn(key: int, /) -> str
def __gt__: fn(value: str, /) -> bool
def __hash__: fn() -> int
// def __iter__: fn() -> Iterator[str]
def __le__: fn(value: str, /) -> bool
def __len__: fn() -> int
def __lt__: fn(value: str, /) -> bool
def __mod__: fn(value: Any, /) -> str
def __mul__: fn(value: int, /) -> str
def __ne__: fn(value: object, /) -> bool
def __rmul__: fn(value: int, /) -> str
def __getnewargs__: fn() -> tuple[str]
def __format__: fn(format_spec: str, /) -> str
}
}

10
midas/checker/builtins.py
View File

@@ -15,14 +15,9 @@ if TYPE_CHECKING:
BUILTIN_SUBTYPES: dict[str, set[str]] = {
"object": {"float", "list", "dict", "str", "bytes", "tuple"},
"float": {"int"},
"int": {"bool"},
}
"""
Hard-coded subtype relationships between builtin types
Circular dependencies and diamond inheritance MUST be avoided
"""
def define_builtins(reg: TypesRegistry):
@@ -30,15 +25,12 @@ def define_builtins(reg: TypesRegistry):
any = reg.define_type("Any", TopType())
unit = reg.define_type("None", UnitType())
object = reg.define_type("object", BaseType(name="object"))
bytes = reg.define_type("bytes", BaseType(name="bytes"))
bool = reg.define_type("bool", BaseType(name="bool"))
int = reg.define_type("int", BaseType(name="int"))
float = reg.define_type("float", BaseType(name="float"))
str = reg.define_type("str", BaseType(name="str"))
slice = reg.define_type("slice", BaseType(name="slice"))
tuple = reg.define_type("tuple", BaseType(name="tuple"))
list = reg.define_type(
"list",
GenericType(

5
midas/checker/checker.py
View File

@@ -10,11 +10,6 @@ from midas.utils import TypedAST
class TypeChecker:
"""Type checking dispatcher
Contains a typer for Midas and one for Python, as well as the types registry
"""
def __init__(self):
self.types: TypesRegistry = TypesRegistry()
self.reporter: Reporter = Reporter()

19
midas/checker/diagnostic.py
View File

@@ -9,17 +9,10 @@ class DiagnosticType(StrEnum):
ERROR = "Error"
WARNING = "Warning"
INFO = "Info"
DEBUG = "Debug"
@dataclass(frozen=True)
class Diagnostic:
"""Information about a diagnostic (warning, errors, etc.)
Holds a location, a diagnostic type and a message.
Optionally bound to a file path
"""
file_path: Optional[str]
location: Location
type: DiagnosticType
@@ -27,18 +20,6 @@ class Diagnostic:
@property
def location_str(self) -> str:
"""Get diagnostic type and location as a human readable string
The location is formatted as "<Type> in <file> from L<start_line>:<start_col> to <end_line>:<end_col>",
for example: "Error in /home/user/Desktop/script.py from L12:5 to L12:8"
If the file is `None`, the "in ..." section is excluded from the result.<br>
If the location's end is not specified, the formulation "at L<start_line>:<start_col>" is used.
Returns:
str: the formatted type and location string
"""
start_loc: str = f"L{self.location.lineno}:{self.location.col_offset+1}"
end_loc: Optional[str] = ""
if (

526
midas/checker/dispatcher.py
View File

@@ -1,526 +0,0 @@
import logging
from dataclasses import dataclass
from enum import StrEnum
from typing import Generic, Optional, Protocol, TypeVar, Union
from midas.ast.location import Location
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter
from midas.checker.types import (
AppliedType,
DerivedType,
Function,
GenericType,
OverloadedFunction,
Type,
UnknownType,
)
from midas.checker.unifier import Unifier
class HasLocation(Protocol):
@property
def location(self) -> Location: ...
E = TypeVar("E", bound=HasLocation)
TypedExpr = tuple[E, Type]
"""An expression and its type"""
@dataclass(frozen=True, kw_only=True)
class MappedArgument(Generic[E]):
"""An argument passed in a call and the corresponding parameter"""
arg_expr: E
arg_type: Type
parameter: Function.Parameter
@dataclass(frozen=True, kw_only=True)
class OverloadCandidate:
"""An overloaded function call candidate with its mapped arguments"""
function: Function
mapped: list[MappedArgument]
class CallError(StrEnum):
"""Reason of a call error"""
INVALID_ARGS = "Invalid arguments"
NO_MATCHING_OVERLOAD = "No matching overload"
IMPOSSIBLE_UNIFICATION = "Parameters unification failed"
NOT_CALLABLE = "Not callable"
@dataclass(frozen=True, kw_only=True)
class CallResult:
"""The result of a function call
Holds a return type, an optional error reason and message
"""
error: Optional[CallError] = None
"""The reason of the error, if there is one"""
result: Type = UnknownType()
"""The result type. `UnknownType()` if the call is invalid"""
message: Optional[str] = None
"""An optional error message"""
@property
def is_valid(self) -> bool:
"""Whether the call is valid (i.e. no error)"""
return self.error is None
@property
def error_message(self) -> str:
"""A descriptive message for the error, if there is one"""
if self.message is not None:
return self.message
if self.error is not None:
return str(self.error)
return ""
class CallDispatcher(Generic[E]):
"""Helper class to handle dispatching calls and mapping arguments
This class is responsible for mapping call-site arguments to function
parameters, verifying the validity of calls and computing their
return types
:class:`CallDispatcher` is generic to handle AST nodes from both Midas and Python
"""
def __init__(self, types: TypesRegistry, reporter: FileReporter) -> None:
self.types: TypesRegistry = types
self.reporter: FileReporter = reporter
self.logger: logging.Logger = logging.getLogger("CallDispatcher")
def set_reporter(self, reporter: FileReporter):
self.reporter = reporter
def get_result(
self,
location: Location,
callee: Type,
positional: list[TypedExpr[E]],
keywords: dict[str, TypedExpr[E]],
report_errors: bool = True,
) -> CallResult:
"""Get the result type of a function call
If the callee has overloads, this function will try to resolve the
appropriate signature.
Argument types are matched with the defined parameters.
This function doesn't take the raw expression as a parameter to
accommodate for desugared calls such as for operators.
Args:
location (Location): the call location
callee (Type): the called function
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
report_errors (bool, optional): whether type errors should be reported as diagnostics. Defaults to True.
Returns:
CallResult: the call result, either a type or an error
"""
match callee:
case Function() as function:
valid: bool
mapped: list[MappedArgument[E]]
valid, mapped = self.map_call_arguments(
function, location, positional, keywords
)
valid = valid and self._are_arguments_valid(mapped, report_errors)
if not valid:
return CallResult(error=CallError.INVALID_ARGS)
return CallResult(result=function.returns)
case OverloadedFunction(overloads=overloads):
res = self._match_overload(
overloads, location, positional, keywords, report_errors
)
if res[0] is None:
return CallResult(
error=CallError.NO_MATCHING_OVERLOAD,
message=res[1],
)
return CallResult(result=res[0].returns)
case AppliedType(body=body):
return self.get_result(
location, body, positional, keywords, report_errors
)
case UnknownType():
return CallResult(result=UnknownType())
case DerivedType(type=base):
return self.get_result(
location, base, positional, keywords, report_errors
)
case GenericType():
unifier: Unifier = Unifier(self.types)
pos: list[Type] = [a[1] for a in positional]
kw: dict[str, Type] = {k: v[1] for k, v in keywords.items()}
unified: Optional[Type] = unifier.unify_call(callee, pos, kw)
if unified is None:
pos_str: str = ", ".join(str(t) for t in pos)
kw_str: str = ", ".join(f"{k}: {v}" for k, v in kw.items())
message: str = (
f"Could not unify {callee}={callee.body} with pos=[{pos_str}] and kw={{{kw_str}}}"
)
if report_errors:
self.reporter.error(location, message)
return CallResult(
error=CallError.IMPOSSIBLE_UNIFICATION,
message=message,
)
return self.get_result(
location,
unified,
positional,
keywords,
report_errors,
)
case _:
message: str = f"{callee} ({callee.__class__.__name__}) is not callable"
if report_errors:
self.reporter.error(location, message)
return CallResult(
error=CallError.NOT_CALLABLE,
message=message,
)
def _unwrap_function(
self,
callee: Type,
positional: list[TypedExpr[E]],
keywords: dict[str, TypedExpr[E]],
) -> Union[tuple[Function, None], tuple[None, CallError]]:
"""Unwrap a type to get a callable `Function`
Args:
callee (Type): the called type
positional (list[TypedExpr[E]]): the list of positional arguments
keywords (dict[str, TypedExpr[E]]): the map of keyword arguments
Returns:
Union[tuple[Function, None], tuple[None, CallError]]: a tuple
containing the callable `Function` type, or `None` if it could
not be unwrapped, and an error, or `None` if there was none.
"""
match callee:
case DerivedType(type=base):
return self._unwrap_function(base, positional, keywords)
case GenericType():
unifier: Unifier = Unifier(self.types)
unified: Optional[Type] = unifier.unify_call(
callee,
[a[1] for a in positional],
{k: v[1] for k, v in keywords.items()},
)
if unified is None:
return None, CallError.IMPOSSIBLE_UNIFICATION
return self._unwrap_function(unified, positional, keywords)
case Function():
return callee, None
case AppliedType(body=body):
return self._unwrap_function(body, positional, keywords)
case _:
return None, CallError.NOT_CALLABLE
def _are_arguments_valid(
self,
arguments: list[MappedArgument[E]],
report_errors: bool = True,
) -> bool:
"""Check whether the passed argument types correspond to their matched parameter definitions
Args:
arguments (list[MappedArgument]): the list of argument/parameter pairs
report_errors (bool, optional): whether type errors should be reported as diagnostics. Defaults to True.
Returns:
bool: True if all arguments fit the matching parameter definitions, False otherwise
"""
valid: bool = True
for arg in arguments:
if not self.types.is_subtype(arg.arg_type, arg.parameter.type):
if report_errors:
self.reporter.error(
arg.arg_expr.location,
f"Wrong type for argument '{arg.parameter.name}', expected {arg.parameter.type}, got {arg.arg_type}",
)
valid = False
return valid
def _match_overload(
self,
overloads: list[Type],
location: Location,
positional: list[TypedExpr[E]],
keywords: dict[str, TypedExpr[E]],
report_errors: bool = True,
) -> Union[tuple[Function, None], tuple[None, str]]:
"""Try and resolve the appropriate overload for the given arguments
Args:
overloads (list[Type]): the list of possible overloads
location (Location): the call location
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keywords arguments
report_errors (bool, optional): whether type errors should be reported as diagnostics. Defaults to True.
Returns:
Union[tuple[Function, None], tuple[None, str]]: a tuple containing
the resolved function signature if it can be determined
unambiguously, or `None`, and an error message, or `None`
"""
candidates: list[OverloadCandidate] = []
errors: list[CallError] = []
for overload in overloads:
function, unwrap_error = self._unwrap_function(
overload, positional, keywords
)
if function is None:
errors.append(unwrap_error) # type: ignore
continue
valid, mapped = self.map_call_arguments(
function=function,
location=location,
positional=positional,
keywords=keywords,
report_errors=False,
)
if valid and self._are_arguments_valid(mapped, report_errors=False):
candidates.append(
OverloadCandidate(
function=function,
mapped=mapped,
)
)
pos_types: str = ", ".join(str(type) for _, type in positional)
kw_types: str = ", ".join(
f"{name}: {type}" for name, (_, type) in keywords.items()
)
for_args: str = f"for arguments pos=[{pos_types}] and kw={{{kw_types}}}"
n_candidates: int = len(candidates)
# Exactly 1 match -> return it
if n_candidates == 1:
return candidates[0].function, None
# No match -> invalid call
if n_candidates == 0:
overloads_str: str = ", ".join(map(str, overloads))
errors_str: str = ", ".join(errors)
message: str = (
f"No matching overload in [{overloads_str}] {for_args} (errors: {errors_str})"
)
if report_errors:
self.reporter.error(location, message)
return None, message
# Multiple matches -> see if one <: all others (more specific)
for i1, c1 in enumerate(candidates):
mapped1: list[MappedArgument[E]] = c1.mapped
best_match: bool = True
for i2, c2 in enumerate(candidates):
if i1 == i2:
continue
mapped2: list[MappedArgument[E]] = c2.mapped
if not self._are_mapped_subtypes(mapped1, mapped2):
best_match = False
break
self.logger.debug(f"{c1.function} is a full overload of {c2.function}")
if best_match:
return c1.function, None
candidates_str: str = ", ".join(
str(candidate.function) for candidate in candidates
)
message: str = f"Multiple matching overloads {for_args}: {candidates_str}"
if report_errors:
self.reporter.error(location, message)
return None, message
def map_call_arguments(
self,
function: Function,
location: Location,
positional: list[TypedExpr[E]],
keywords: dict[str, TypedExpr[E]],
report_errors: bool = True,
) -> tuple[bool, list[MappedArgument]]:
"""Map call arguments to a function's parameters as defined in its signature
This method maps positional-only, keyword-only and mixed parameter definitions
with the arguments passed at the call site
Any mismatched, missing or unexpected argument is reported as a diagnostic,
unless `report_errors` is set to `False`
Args:
function (Function): the function definition
location (Location): the call location
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
report_errors (bool, optional): whether type errors should be reported as diagnostics. Defaults to True.
Returns:
tuple[bool, list[MappedArgument]]: a boolean reporting whether
the call is valid and the list of mapped arguments
"""
set_params: set[str] = set()
required_positional: list[str] = [
param.name
for param in function.params.pos + function.params.mixed
if param.required
]
required_keyword: list[str] = [
param.name for param in function.params.kw if param.required
]
mapped: list[MappedArgument[E]] = []
pos_params: list[Function.Parameter] = list(function.params.pos)
mixed_params: list[Function.Parameter] = list(function.params.mixed)
kw_params: dict[str, Function.Parameter] = {
param.name: param for param in function.params.kw
}
valid_call: bool = True
# TODO: handle *args and **kwargs sinks
for arg in positional:
param: Function.Parameter
if len(pos_params) != 0:
param = pos_params.pop(0)
elif len(mixed_params) != 0:
param = mixed_params.pop(0)
else:
if report_errors:
self.reporter.error(
arg[0].location, "Too many positional arguments"
)
valid_call = False
break
name: str = param.name
if name in required_positional:
required_positional.remove(name)
if name in required_keyword:
required_keyword.remove(name)
set_params.add(name)
mapped.append(
MappedArgument(
arg_expr=arg[0],
arg_type=arg[1],
parameter=param,
)
)
kw_params.update({param.name: param for param in mixed_params})
for name, arg in keywords.items():
param: Function.Parameter
if name not in kw_params:
if report_errors:
if name in set_params:
self.reporter.error(
arg[0].location, f"Multiple values for parameter '{name}'"
)
else:
self.reporter.error(
arg[0].location, f"Unknown keyword parameter '{name}'"
)
valid_call = False
continue
param = kw_params.pop(name)
if name in required_positional:
required_positional.remove(name)
if name in required_keyword:
required_keyword.remove(name)
set_params.add(name)
mapped.append(
MappedArgument(
arg_expr=arg[0],
arg_type=arg[1],
parameter=param,
)
)
def join_params(params: list[str]) -> str:
params = list(map(lambda p: f"'{p}'", params))
if len(params) == 0:
return ""
if len(params) == 1:
return params[0]
return ", ".join(params[:-1]) + " and " + params[-1]
if len(required_positional) != 0:
plural: str = "" if len(required_positional) == 1 else "s"
params: str = join_params(required_positional)
if report_errors:
self.reporter.error(
location,
f"Missing required positional argument{plural}: {params}",
)
valid_call = False
if len(required_keyword) != 0:
plural: str = "" if len(required_keyword) == 1 else "s"
params: str = join_params(required_keyword)
if report_errors:
self.reporter.error(
location,
f"Missing required keyword argument{plural}: {params}",
)
valid_call = False
return valid_call, mapped
def _are_mapped_subtypes(
self, mapped1: list[MappedArgument[E]], mapped2: list[MappedArgument[E]]
) -> bool:
"""Check whether the given argument mappings are subtype/supertype of one another
This function checks whether the argument mappings `mapped1` are subtypes
of `mapped2`. If any of the parameter type in `mapped1` is not a subtype
of the corresponding parameter in `mapped2`, `False` is returned.
This is used to check whether a given overload is a more specific
function / a subtype of another.
Args:
mapped1 (list[MappedArgument]): the first argument mappings (subtype)
mapped2 (list[MappedArgument]): the second argument mappings (supertype)
Returns:
bool: `True` if `mapped1` is a subtype of `mapped2`, `False` otherwise
"""
by_expr: dict[E, Type] = {}
for arg in mapped1:
by_expr[arg.arg_expr] = arg.parameter.type
for arg in mapped2:
type2: Type = arg.parameter.type
type1: Type = by_expr[arg.arg_expr]
if not self.types.is_subtype(type1, type2):
return False
return True

240
midas/checker/evaluator.py
View File

@@ -1,240 +0,0 @@
from dataclasses import dataclass
from typing import Any, Callable, Optional
import midas.ast.midas as m
from midas.checker.preamble import Preamble
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter
from midas.checker.types import Function, Predicate
from midas.lexer.token import TokenType
@dataclass(frozen=True, kw_only=True)
class PartialPredicate(Predicate):
"""A partially applied predicate"""
scope: dict[str, Any]
"""A dictionary of already applied parameters"""
class Evaluator(m.Expr.Visitor[Any]):
"""Helper class to evaluate an expression
This class is used to evaluate constraint types on literals at compile-time.
"""
def __init__(self, types: TypesRegistry, reporter: Optional[FileReporter] = None):
self.types: TypesRegistry = types
self.reporter: Optional[FileReporter] = reporter
self.preamble: Preamble = Preamble(self.types)
self.scopes: list[dict[str, Any]] = [{}]
def evaluate(self, expr: m.Expr) -> Any:
"""Evaluate the given expression
Args:
expr (m.Expr): the expression to evaluate
Returns:
Any: the value of the expression
"""
value: Any = expr.accept(self)
if self.reporter is not None:
self.reporter.debug(expr.location, f"Value: {value}")
return value
def get_value(self, name: str) -> Any:
"""Get the value of a variable in the current scope
Args:
name (str): the name of the variable
Raises:
KeyError: if the variable is not defined
Returns:
Any: the value of the variable
"""
scope: dict[str, Any] = self.scopes[-1]
return scope[name]
def set_value(self, name: str, value: Any, force_declare: bool = False):
"""Set the value of a variable
If `force_declare` is `False`, this function first tries to find the
closest scope in which the variable is defined and assign the value in
that scope, if it can find one.
If `force_declare` is `True` or if the variable is not defined in any
scope, it is declare and assigned in the current scope
Args:
name (str): the name of the variable
value (Any): the value of the variable
force_declare (bool, optional): if `False` and the variable is
defined in a scope, the value is assigned in that scope (the
closest if there are multiple declarations). Defaults to False.
"""
if not force_declare:
for scope in reversed(self.scopes):
if name in scope:
scope[name] = value
return
self.scopes[-1][name] = value
def visit_logical_expr(self, expr: m.LogicalExpr) -> Any:
def left():
return self.evaluate(expr.left)
def right():
return self.evaluate(expr.right)
match expr.operator.type:
case TokenType.AND:
return left() and right()
case _:
raise NotImplementedError
def visit_binary_expr(self, expr: m.BinaryExpr) -> Any:
left: Any = self.evaluate(expr.left)
right: Any = self.evaluate(expr.right)
match expr.operator.type:
case TokenType.MINUS:
return left - right
case TokenType.STAR:
return left * right
case TokenType.SLASH:
return left / right
case TokenType.GREATER:
return left > right
case TokenType.GREATER_EQUAL:
return left >= right
case TokenType.LESS:
return left < right
case TokenType.LESS_EQUAL:
return left <= right
case TokenType.EQUAL_EQUAL:
return left == right
case TokenType.BANG_EQUAL:
return left != right
case _:
raise NotImplementedError
def visit_unary_expr(self, expr: m.UnaryExpr) -> Any:
right: Any = self.evaluate(expr.right)
match expr.operator.type:
case TokenType.MINUS:
return -right
case _:
raise NotImplementedError
def visit_call_expr(self, expr: m.CallExpr) -> Any:
callee: Any = self.evaluate(expr.callee)
args: list[Any] = [self.evaluate(arg) for arg in expr.arguments]
kwargs: dict[str, Any] = {
name: self.evaluate(arg) for name, arg in expr.keywords.items()
}
match callee:
case Predicate():
return self._evaluate_predicate(callee, args, kwargs)
case _ if callable(callee):
return callee(*args, **kwargs)
case _:
return NotImplementedError
def visit_get_expr(self, expr: m.GetExpr) -> Any:
obj: Any = self.evaluate(expr.expr)
return getattr(obj, expr.name.lexeme)
def visit_variable_expr(self, expr: m.VariableExpr) -> Any:
name: str = expr.name.lexeme
for scope in reversed(self.scopes):
if name in scope:
return scope[name]
predicate: Optional[Predicate] = self.types.lookup_predicate(name)
if predicate is not None:
if predicate.alias:
return self.evaluate(predicate.body)
return predicate
glob: Optional[Callable] = self.preamble.get_py_func(name)
if glob is not None:
return glob
raise NameError(f"Unknown variable '{name}'")
def visit_grouping_expr(self, expr: m.GroupingExpr) -> Any:
return self.evaluate(expr.expr)
def visit_literal_expr(self, expr: m.LiteralExpr) -> Any:
return expr.value
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> Any:
return self.get_value("_")
def _evaluate_predicate(
self,
predicate: Predicate,
args: list[Any],
kwargs: dict[str, Any],
) -> Any:
"""Evaluate a predicate function call
Args:
predicate (Predicate): the predicate to evaluate
args (list[Any]): a list of positional arguments
kwargs (dict[str, Any]): a map of keyword arguments
Returns:
Any: the value returned by the predicate call
"""
res: Any = None
if isinstance(predicate, PartialPredicate):
self.scopes.append(predicate.scope)
else:
self.scopes.append({})
match predicate.type:
case Function(returns=Function() as inner):
self._map_args(predicate.type, args, kwargs)
res = PartialPredicate(
type=inner,
body=predicate.body,
alias=False,
scope=self.scopes[-1],
)
case Function():
self._map_args(predicate.type, args, kwargs)
res = self.evaluate(predicate.body)
case _:
raise NotImplementedError
self.scopes.pop()
return res
def _map_args(self, function: Function, args: list[Any], kwargs: dict[str, Any]):
"""Map call arguments to a function's parameters and set their values in context
Each argument is mapped to a parameter of the function, then its value
is set in the context using :func:`set_value` with the parameter's name
Args:
function (Function): the called function
args (list[Any]): a list of positional arguments
kwargs (dict[str, Any]): a map of keyword arguments
"""
positional: list[Function.Parameter] = (
function.params.pos + function.params.mixed
)
keywords: dict[str, Function.Parameter] = {
param.name: param for param in function.params.mixed + function.params.kw
}
for i, arg in enumerate(args):
param: Function.Parameter = positional[i]
self.set_value(param.name, arg)
for name, arg in kwargs.items():
param: Function.Parameter = keywords[name]
self.set_value(param.name, arg)

229
midas/checker/frames/column_groupby_methods.py
View File

@@ -1,229 +0,0 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.dispatcher import CallResult
from midas.checker.frames.utils import MethodRegistry, method
from midas.checker.types import (
ColumnGroupBy,
ColumnType,
Function,
ParamSpec,
TopType,
Type,
)
if TYPE_CHECKING:
from midas.checker.python import TypedExpr
@dataclass(frozen=True, kw_only=True)
class Call:
"""A column group-by method call, implements :class:`utils.MethodCall`"""
location: Location
call_expr: p.Expr
groupby: ColumnGroupBy
groupby_expr: p.Expr
positional: list[TypedExpr]
keywords: dict[str, TypedExpr]
@property
def subject(self) -> TypedExpr:
return (self.groupby_expr, self.groupby)
class ColumnGroupByMethodRegistry(MethodRegistry[Call]):
"""The method registry for column group-by types"""
NAMED_ARGS: dict[str, str] = {
"numeric_only": "bool",
"skipna": "bool",
"engine": "str",
"engine_kwargs": "dict",
}
def _aggregate(
self,
call: Call,
params: list[str | tuple[str, str, bool]] = [],
*,
preserve_inner_type: bool = False,
) -> Type:
"""Compute the result type of an aggregate method call
Args:
call (Call): the call object
params (list[str | tuple[str, str, bool], optional): a list of extra
mixed parameters. The list can contain strings to include
parameters predefined in `NAMED_ARGS`, or tuples containing the
parameter's name, type and required flag. Defaults to [].
preserve_inner_type (bool, optional): If `True`, the result type
will preserve the column's inner type (e.g. for `min`/`max`),
otherwise the inner type is widened to `TopType`. Defaults to False.
Returns:
Type: the result type
"""
real_params: list[Function.Parameter] = []
for i, param in enumerate(params):
match param:
case str() as name:
param = Function.Parameter(
pos=i,
name=name,
type=self.types.get_type(self.NAMED_ARGS[name]),
required=False,
)
case (name, type, required):
param = Function.Parameter(
pos=i,
name=name,
type=self.types.get_type(type),
required=required,
)
real_params.append(param)
signature = Function(
params=ParamSpec(mixed=real_params),
returns=(
call.groupby.column
if preserve_inner_type
else ColumnType(type=TopType())
),
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method()
def kurt(self, call: Call) -> Type:
return self._aggregate(
call,
["skipna", "numeric_only"],
)
@method()
def max(self, call: Call) -> Type:
return self._aggregate(
call,
[
"numeric_only",
(
"min_count",
"int",
False,
),
"skipna",
"engine",
"engine_kwargs",
],
preserve_inner_type=True,
)
@method()
def mean(self, call: Call) -> Type:
return self._aggregate(
call,
["numeric_only", "skipna", "engine", "engine_kwargs"],
)
@method()
def median(self, call: Call) -> Type:
return self._aggregate(
call,
["numeric_only", "skipna"],
preserve_inner_type=True,
)
@method()
def min(self, call: Call) -> Type:
return self._aggregate(
call,
[
"numeric_only",
(
"min_count",
"int",
False,
),
"skipna",
"engine",
"engine_kwargs",
],
preserve_inner_type=True,
)
@method()
def prod(self, call: Call) -> Type:
return self._aggregate(
call,
[
"numeric_only",
(
"min_count",
"int",
False,
),
"skipna",
],
)
@method()
def std(self, call: Call) -> Type:
return self._aggregate(
call,
[
(
"ddof",
"int",
False,
),
"engine",
"engine_kwargs",
"numeric_only",
"skipna",
],
)
@method()
def sum(self, call: Call) -> Type:
return self._aggregate(
call,
[
"numeric_only",
(
"min_count",
"int",
False,
),
"skipna",
"engine",
"engine_kwargs",
],
)
@method()
def var(self, call: Call) -> Type:
return self._aggregate(
call,
[
(
"var",
"int",
False,
),
"engine",
"engine_kwargs",
"numeric_only",
"skipna",
],
)

182
midas/checker/frames/column_manager.py
View File

@@ -1,182 +0,0 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Optional
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.dispatcher import CallResult
from midas.checker.frames.column_groupby_methods import Call as GroupByCall
from midas.checker.frames.column_groupby_methods import ColumnGroupByMethodRegistry
from midas.checker.frames.column_methods import Call, ColumnMethodRegistry
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter
from midas.checker.types import (
ColumnGroupBy,
ColumnType,
Function,
OverloadedFunction,
ParamSpec,
Type,
)
if TYPE_CHECKING:
from midas.checker.python import PythonTyper, TypedExpr
class ColumnManager:
"""Helper class to handle methods and subscripts on column types"""
def __init__(self, typer: PythonTyper) -> None:
self.typer: PythonTyper = typer
self.method_resolver: ColumnMethodRegistry = ColumnMethodRegistry(self.typer)
self.groupby_method_resolver: ColumnGroupByMethodRegistry = (
ColumnGroupByMethodRegistry(self.typer)
)
def get(
self,
reporter: FileReporter,
location: Location,
column: ColumnType,
index: TypedExpr,
) -> Type:
"""Compute the type of a subscript access
Args:
reporter (FileReporter): the file reporter to use for diagnostics
location (Location): the subscript's location
column (DataFrameType): the column type
index (TypedExpr): the index
Returns:
Type: the resulting type
"""
single = Function(
params=ParamSpec(
pos=[
Function.Parameter(
pos=0,
name="index",
type=self.typer.types.get_type("int"),
required=True,
)
]
),
returns=column.type,
)
slice = Function(
params=ParamSpec(
pos=[
Function.Parameter(
pos=0,
name="slice",
type=self.typer.types.get_type("slice"),
required=True,
)
]
),
returns=column,
)
overload = OverloadedFunction(overloads=[single, slice])
result: CallResult = self.typer.dispatcher.get_result(
location=location,
callee=overload,
positional=[index],
keywords={},
)
return result.result
def call(
self,
method: str,
location: Location,
call_expr: p.Expr,
column: ColumnType,
column_expr: p.Expr,
positional: list[TypedExpr],
keywords: dict[str, TypedExpr],
) -> Type:
"""Compute the result type of a column's method call
Args:
method (str): the method name
location (Location): the call's location
call_expr (p.Expr): the call expression
column (ColumnType): the column type
column_expr (p.Expr): the column expression
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
Returns:
Type: the result type
"""
call: Call = Call(
location=location,
call_expr=call_expr,
column=column,
column_expr=column_expr,
positional=positional,
keywords=keywords,
)
return self.method_resolver.call(method, call)
def groupby_call(
self,
method: str,
location: Location,
call_expr: p.Expr,
groupby: ColumnGroupBy,
groupby_expr: p.Expr,
positional: list[TypedExpr],
keywords: dict[str, TypedExpr],
) -> Type:
"""Compute the result type of a column group-by's method call
Args:
method (str): the method name
location (Location): the call's location
call_expr (p.Expr): the call expression
groupby (ColumnGroupBy): the column group-by object
groupby_expr (p.Expr): the column group-by expression
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
Returns:
Type: the result type
"""
call: GroupByCall = GroupByCall(
location=location,
call_expr=call_expr,
groupby=groupby,
groupby_expr=groupby_expr,
positional=positional,
keywords=keywords,
)
return self.groupby_method_resolver.call(method, call)
def get_attribute(self, column: ColumnType, name: str) -> Optional[Type]:
"""Get the type of a column's attribute
Args:
column (ColumnType): the column type
name (str): the attribute's name
Returns:
Optional[Type]: the attribute's type, or `None` if it doesn't exist
"""
types: TypesRegistry = self.typer.types
match name:
case "ndim" | "size":
return types.get_type("int")
case "shape":
return types.tuple_of("int")
case "T":
return column
case _:
return None

437
midas/checker/frames/column_methods.py
View File

@@ -1,437 +0,0 @@
from __future__ import annotations
import ast
from dataclasses import dataclass
from typing import TYPE_CHECKING, Optional
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.dispatcher import CallResult
from midas.checker.frames.utils import MethodRegistry, method
from midas.checker.types import (
ColumnGroupBy,
ColumnType,
Function,
GenericType,
ParamSpec,
TopType,
Type,
TypeVar,
UnknownType,
unfold_type,
)
if TYPE_CHECKING:
from midas.checker.python import TypedExpr
@dataclass(frozen=True, kw_only=True)
class Call:
"""A column method call, implements :class:`utils.MethodCall`"""
location: Location
call_expr: p.Expr
column: ColumnType
column_expr: p.Expr
positional: list[TypedExpr]
keywords: dict[str, TypedExpr]
@property
def subject(self) -> TypedExpr:
return (self.column_expr, self.column)
class ColumnMethodRegistry(MethodRegistry[Call]):
"""The method registry for column types"""
def _element_binary_op(self, call: Call, method: str) -> ColumnType:
"""Compute the result of an element-wise binary operation
This function delegates to the inner types for computing the resulting
type.
Args:
call (Call): the call that triggered this resolution
method (str): the method name
Returns:
ColumnType: the resulting column type
"""
column2: Optional[ColumnType] = None
col_type1: Type = call.column.type
new_column: Type = ColumnType(type=UnknownType())
if len(call.positional) != 0:
other: Type = call.positional[0][1]
unfolded_other: Type = unfold_type(other)
if isinstance(unfolded_other, ColumnType):
column2 = unfolded_other
col_type2: Type = column2.type
new_inner_type = self.typer.result_of_binary_op(
location=call.location,
expr=call.call_expr,
left=(call.column_expr, col_type1),
right=(call.positional[0][0], col_type2),
method=method,
)
new_column = ColumnType(type=new_inner_type)
return new_column
def _element_wise(self, call: Call, method: str) -> Type:
"""Compute the result of an element-wise method call
If the call is valid, this method also generates an assertion to check
that both operands have the same length at runtime
Args:
call (Call): the call object
method (str): the method's name
Returns:
Type: the result type
"""
# TODO: support add with scalar
# Build signature with new column type and generic operand
param_type: TypeVar = TypeVar(name="T", bound=None)
signature = GenericType(
name="add",
params=[param_type],
body=Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="other",
type=ColumnType(type=param_type),
required=True,
),
],
),
returns=self._element_binary_op(call, method),
),
)
# Map arguments and compute result type
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
if result.is_valid:
self._assert_same_length(
call.call_expr, call.column_expr, call.positional[0][0]
)
return result.result
@method("add", "__add__")
def add(self, call: Call) -> Type:
return self._element_wise(call, "__add__")
@method("sub", "__sub__")
def sub(self, call: Call) -> Type:
return self._element_wise(call, "__sub__")
@method("mul", "__mul__")
def mul(self, call: Call) -> Type:
return self._element_wise(call, "__mul__")
@method("div", "truediv", "__truediv__")
def truediv(self, call: Call) -> Type:
return self._element_wise(call, "__truediv__")
@method("floordiv", "__floordiv__")
def floordiv(self, call: Call) -> Type:
return self._element_wise(call, "__floordiv__")
@method("mod", "__mod__")
def mod(self, call: Call) -> Type:
return self._element_wise(call, "__mod__")
@method("pow", "__pow__")
def pow(self, call: Call) -> Type:
return self._element_wise(call, "__pow__")
@method("lt", "__lt__")
def lt(self, call: Call) -> Type:
return self._element_wise(call, "__lt__")
@method("gt", "__gt__")
def gt(self, call: Call) -> Type:
return self._element_wise(call, "__gt__")
@method("le", "__le__")
def le(self, call: Call) -> Type:
return self._element_wise(call, "__le__")
@method("ge", "__ge__")
def ge(self, call: Call) -> Type:
return self._element_wise(call, "__ge__")
@method("ne", "__ne__")
def ne(self, call: Call) -> Type:
return self._element_wise(call, "__ne__")
@method("eq", "__eq__")
def eq(self, call: Call) -> Type:
return self._element_wise(call, "__eq__")
def _aggregate(
self,
call: Call,
kwargs: list[Function.Parameter] = [],
*,
preserve_inner_type: bool = False,
) -> Type:
"""Compute the result type of an aggregate method call
Args:
call (Call): the call object
kwargs (list[Function.Parameter], optional): a list of extra
keyword-only parameters. Defaults to [].
preserve_inner_type (bool, optional): If `True`, the result type
will preserve the column's inner type (e.g. for `min`/`max`),
otherwise the inner type is widened to `TopType`. Defaults to False.
Returns:
Type: the result type
"""
signature = Function(
params=ParamSpec(
kw=[
Function.Parameter(
pos=0,
name="axis",
type=TopType(),
required=False,
),
*kwargs,
],
),
returns=call.column if preserve_inner_type else ColumnType(type=TopType()),
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method("kurtosis", "kurt")
def kurtosis(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def max(self, call: Call) -> Type:
return self._aggregate(call, preserve_inner_type=True)
@method()
def mean(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def median(self, call: Call) -> Type:
return self._aggregate(call, preserve_inner_type=True)
@method()
def min(self, call: Call) -> Type:
return self._aggregate(call, preserve_inner_type=True)
@method()
def mode(self, call: Call) -> Type:
return self._aggregate(call, preserve_inner_type=True)
@method("product", "prod")
def product(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def std(self, call: Call) -> Type:
return self._aggregate(
call,
[
Function.Parameter(
pos=1,
name="ddof",
type=self.types.get_type("int"),
required=False,
)
],
)
@method()
def sum(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def var(self, call: Call) -> Type:
return self._aggregate(
call,
[
Function.Parameter(
pos=1,
name="var",
type=self.types.get_type("int"),
required=False,
)
],
)
@method()
def head(self, call: Call) -> Type:
signature = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="n",
type=self.types.get_type("int"),
required=False,
),
],
),
returns=call.column,
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method()
def tail(self, call: Call) -> Type:
signature = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="n",
type=self.types.get_type("int"),
required=False,
),
],
),
returns=call.column,
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method()
def groupby(self, call: Call) -> Type:
bool_: Type = self.types.get_type("bool")
function: Function = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="by",
type=TopType(),
required=False,
),
Function.Parameter(
pos=1,
name="level",
type=TopType(),
required=False,
),
],
kw=[
Function.Parameter(
pos=i + 2,
name=name,
type=bool_,
required=False,
)
for i, name in enumerate(
["as_index", "sort", "group_keys", "observed", "dropna"]
)
],
),
returns=ColumnGroupBy(column=call.column),
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=function,
positional=call.positional,
keywords=call.keywords,
)
return result.result
def _assert_same_length(self, call_expr: p.Expr, column1: p.Expr, column2: p.Expr):
"""Generate an assertion to check that two columns have the same length
Args:
call_expr (p.Expr): the call expression, to insert the assertion
at the right place
column1 (p.Expr): the first column expression
column2 (p.Expr): the second column expression
"""
func_name: str = "__midas_column_same_length__"
# Efficiently compute length
# https://stackoverflow.com/a/15943975/11109181
def len_of_col(col: ast.expr) -> ast.expr:
return ast.Call(
func=ast.Name(id="len"),
args=[
ast.Attribute(
value=col,
attr="index",
)
],
keywords=[],
)
self.assertions.define(
func_name,
ast.FunctionDef(
name=func_name,
args=ast.arguments(
posonlyargs=[],
args=[
ast.arg(arg="column1"),
ast.arg(arg="column2"),
],
kwonlyargs=[],
defaults=[],
kw_defaults=[],
),
body=[
ast.Return(
value=ast.Compare(
left=len_of_col(ast.Name(id="column1")),
ops=[ast.Eq()],
comparators=[
len_of_col(ast.Name(id="column2")),
],
)
)
],
decorator_list=[],
),
)
self.assertions.add(
bound_expr=call_expr,
inputs=[column1, column2],
builder=lambda c1, c2: ast.Call(
func=ast.Name(id=func_name),
args=[c1, c2],
keywords=[],
),
message="Columns must have the same length",
)

109
midas/checker/frames/frame_groupby_methods.py
View File

@@ -1,109 +0,0 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.frames.utils import MethodRegistry, method
from midas.checker.types import (
ColumnGroupBy,
ColumnType,
DataFrameType,
FrameGroupBy,
Type,
UnknownType,
)
if TYPE_CHECKING:
from midas.checker.python import TypedExpr
@dataclass(frozen=True, kw_only=True)
class Call:
"""A frame group-by method call, implements :class:`utils.MethodCall`"""
location: Location
call_expr: p.Expr
groupby: FrameGroupBy
groupby_expr: p.Expr
positional: list[TypedExpr]
keywords: dict[str, TypedExpr]
@property
def subject(self) -> TypedExpr:
return (self.groupby_expr, self.groupby)
class FrameGroupByMethodRegistry(MethodRegistry[Call]):
"""The method registry for frame group-by types"""
def _aggregate(self, call: Call, method: str) -> Type:
"""Compute the result type of an aggregate method call
Args:
call (Call): the call object
method (str): the method's name
Returns:
Type: the result type
"""
new_columns: list[DataFrameType.Column] = []
for column in call.groupby.frame.columns:
column_groupby: ColumnGroupBy = ColumnGroupBy(column=column.type)
result_type: Type = self.typer.call_method(
location=call.location,
call_expr=call.call_expr,
obj=(call.groupby_expr, column_groupby),
method_name=method,
positional=call.positional,
keywords=call.keywords,
)
if not isinstance(result_type, ColumnType):
result_type = ColumnType(type=UnknownType())
new_columns.append(
DataFrameType.Column(
index=column.index,
name=column.name,
type=result_type,
)
)
return DataFrameType(columns=new_columns)
@method()
def kurt(self, call: Call) -> Type:
return self._aggregate(call, "kurt")
@method()
def max(self, call: Call) -> Type:
return self._aggregate(call, "max")
@method()
def mean(self, call: Call) -> Type:
return self._aggregate(call, "mean")
@method()
def median(self, call: Call) -> Type:
return self._aggregate(call, "median")
@method()
def min(self, call: Call) -> Type:
return self._aggregate(call, "min")
@method()
def prod(self, call: Call) -> Type:
return self._aggregate(call, "prod")
@method()
def std(self, call: Call) -> Type:
return self._aggregate(call, "std")
@method()
def sum(self, call: Call) -> Type:
return self._aggregate(call, "sum")
@method()
def var(self, call: Call) -> Type:
return self._aggregate(call, "var")

386
midas/checker/frames/frame_manager.py
View File

@@ -1,386 +0,0 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Optional, TypeGuard, cast
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.frames.frame_groupby_methods import Call as GroupByCall
from midas.checker.frames.frame_groupby_methods import FrameGroupByMethodRegistry
from midas.checker.frames.frame_methods import Call, FrameMethodRegistry
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter
from midas.checker.types import (
ColumnGroupBy,
ColumnType,
DataFrameType,
FrameGroupBy,
TupleType,
Type,
UnknownType,
)
if TYPE_CHECKING:
from midas.checker.python import PythonTyper, TypedExpr
def is_list_of_literals(exprs: list[p.Expr]) -> TypeGuard[list[p.LiteralExpr]]:
"""Check whether the given list only contains literal expressions
Args:
exprs (list[p.Expr]): the list to check
Returns:
TypeGuard[list[p.LiteralExpr]]: whether `exprs` only contains literal expressions
"""
return all(isinstance(expr, p.LiteralExpr) for expr in exprs)
class FrameManager:
"""Helper class to handle methods and subscripts on frame types"""
def __init__(self, typer: PythonTyper) -> None:
self.typer: PythonTyper = typer
self.method_resolver: FrameMethodRegistry = FrameMethodRegistry(self.typer)
self.groupby_method_resolver: FrameGroupByMethodRegistry = (
FrameGroupByMethodRegistry(self.typer)
)
def assign(
self,
reporter: FileReporter,
location: Location,
frame: DataFrameType,
index: p.Expr,
value_type: Type,
) -> Type:
"""Compute the new frame type after assigning a value to an index
Args:
reporter (FileReporter): the file reporter to use for diagnostics
location (Location): the assignment's location
frame (DataFrameType): the frame type
index (p.Expr): the index expression
value_type (Type): the assigned value
Returns:
Type: the resulting frame type
"""
match index:
case p.LiteralExpr(value=str() as name):
return self.assign_column(reporter, location, frame, name, value_type)
case p.ListExpr(items=indices) if is_list_of_literals(indices) and all(
isinstance(index.value, str) for index in indices
):
names: list[str] = [cast(str, index.value) for index in indices]
if not isinstance(value_type, TupleType):
reporter.error(
location,
f"Cannot assign {type} to dataframe columns. Must be a tuple of columns",
)
return UnknownType()
if len(names) != len(value_type.items):
reporter.error(
location,
f"Wrong number of columns. Cannot assign {len(value_type.items)} to {len(names)} targets",
)
return UnknownType()
new_frame: Type = frame
for name, value in zip(names, value_type.items):
new_frame = self.assign_column(
reporter,
location,
new_frame,
name,
value,
)
if not isinstance(new_frame, DataFrameType):
return new_frame
return new_frame
case _:
reporter.error(
location, f"Invalid index type {index} on {frame} (assignment)"
)
return UnknownType()
def assign_column(
self,
reporter: FileReporter,
location: Location,
frame: DataFrameType,
name: str,
type: Type,
) -> Type:
"""Compute the new frame type after assigning a single value to a column
Args:
reporter (FileReporter): the file reporter to use for diagnostics
location (Location): the assignment's location
frame (DataFrameType): the frame type
name (str): the column name
type (Type): the assigned value type
Returns:
Type: the resulting frame type
"""
if not isinstance(type, ColumnType):
reporter.error(
location,
f"Cannot assign {type} to dataframe column. Must be a ColumnType",
)
return frame
return self._set_column(frame, name, type)
def get(
self,
reporter: FileReporter,
location: Location,
frame: DataFrameType,
index: p.Expr,
) -> Type:
"""Compute the type of a subscript access
Args:
reporter (FileReporter): the file reporter to use for diagnostics
location (Location): the subscript's location
frame (DataFrameType): the frame type
index (p.Expr): the index expression
Returns:
Type: the resulting type
"""
match index:
case p.LiteralExpr(value=str() as name):
column: Optional[ColumnType] = FrameManager._get_column(frame, name)
if column is None:
reporter.error(location, f"Unknown column '{name}' on {frame}")
return UnknownType()
return column
case p.ListExpr(items=indices) if is_list_of_literals(indices) and all(
isinstance(index.value, str) for index in indices
):
names: list[str] = [cast(str, index.value) for index in indices]
columns: list[ColumnType] = []
for name in names:
column: Optional[ColumnType] = FrameManager._get_column(frame, name)
if column is None:
reporter.error(location, f"Unknown column '{name}' on {frame}")
return UnknownType()
columns.append(column)
return TupleType(items=tuple(columns))
case _:
reporter.error(
location, f"Invalid index type {index} on {frame} (access)"
)
return UnknownType()
def groupby_get(
self,
reporter: FileReporter,
location: Location,
groupby: FrameGroupBy,
index: p.Expr,
) -> Type:
"""Compute the type of a subscript access on a frame group-by object
Args:
reporter (FileReporter): the file reporter to use for diagnostics
location (Location): the subscript's location
groupby (FrameGroupBy): the group-by object
index (p.Expr): the index expression
Returns:
Type: the resulting type
"""
result: Type = self.get(reporter, location, groupby.frame, index)
match result:
case ColumnType():
result = ColumnGroupBy(column=result)
case TupleType(items=columns):
result = TupleType(
items=tuple(
ColumnGroupBy(column=cast(ColumnType, column))
for column in columns
)
)
return result
@classmethod
def _set_column(
cls, frame: DataFrameType, name: str, column: ColumnType
) -> DataFrameType:
"""Set a frame's column to the given type
Args:
frame (DataFrameType): the frame type
name (str): the column's name
column (ColumnType): the new column's type
Returns:
DataFrameType: the new frame type
"""
new_columns: list[DataFrameType.Column] = []
index: int = len(frame.columns)
replace: bool = False
for i, col in enumerate(frame.columns):
if col.name == name:
index = i
replace = True
# TODO: check column type here to prevent changing it
new_columns.append(col)
new_col: DataFrameType.Column = DataFrameType.Column(
index=index,
name=name,
type=column,
)
if replace:
new_columns[index] = new_col
else:
new_columns.append(new_col)
return DataFrameType(columns=new_columns)
@classmethod
def _set_columns(
cls, frame: DataFrameType, names: list[str], columns: list[ColumnType]
) -> DataFrameType:
"""Set multiple columns of a frame to the given types
Args:
frame (DataFrameType): the frame type
names (list[str]): the column names
columns (list[ColumnType]): the new column types
Returns:
DataFrameType: the new frame type
"""
for name, col in zip(names, columns):
frame = cls._set_column(frame, name, col)
return frame
@classmethod
def _get_column(cls, frame: DataFrameType, name: str) -> Optional[ColumnType]:
"""Get a column's type by name
Args:
frame (DataFrameType): the frame type
name (str): the column's name
Returns:
Optional[ColumnType]: the column's type, or `None` if it doesn't exist
"""
for col in frame.columns:
if col.name == name:
return col.type
return None
@classmethod
def _get_columns(
cls, frame: DataFrameType, names: list[str]
) -> list[Optional[ColumnType]]:
"""Get multiple column types by name
Args:
frame (DataFrameType): the frame type
names (list[str]): the column names
Returns:
list[Optional[ColumnType]]: the column types (see :func:`_get_column`)
"""
return [cls._get_column(frame, name) for name in names]
def call(
self,
method: str,
location: Location,
call_expr: p.Expr,
frame: DataFrameType,
frame_expr: p.Expr,
positional: list[TypedExpr],
keywords: dict[str, TypedExpr],
) -> Type:
"""Compute the result type of a frame's method call
Args:
method (str): the method name
location (Location): the call's location
call_expr (p.Expr): the call expression
frame (DataFrameType): the frame type
frame_expr (p.Expr): the frame expression
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
Returns:
Type: the result type
"""
call: Call = Call(
location=location,
call_expr=call_expr,
frame=frame,
frame_expr=frame_expr,
positional=positional,
keywords=keywords,
)
return self.method_resolver.call(method, call)
def groupby_call(
self,
method: str,
location: Location,
call_expr: p.Expr,
groupby: FrameGroupBy,
groupby_expr: p.Expr,
positional: list[TypedExpr],
keywords: dict[str, TypedExpr],
) -> Type:
"""Compute the result type of a frame group-by's method call
Args:
method (str): the method name
location (Location): the call's location
call_expr (p.Expr): the call expression
groupby (FrameGroupBy): the frame group-by object
groupby_expr (p.Expr): the frame group-by expression
positional (list[TypedExpr]): the list of positional arguments
keywords (dict[str, TypedExpr]): the map of keyword arguments
Returns:
Type: the result type
"""
call: GroupByCall = GroupByCall(
location=location,
call_expr=call_expr,
groupby=groupby,
groupby_expr=groupby_expr,
positional=positional,
keywords=keywords,
)
return self.groupby_method_resolver.call(method, call)
def get_attribute(self, frame: DataFrameType, name: str) -> Optional[Type]:
"""Get the type of a frame's attribute
Args:
frame (DataFrameType): the frame type
name (str): the attribute's name
Returns:
Optional[Type]: the attribute's type, or `None` if it doesn't exist
"""
types: TypesRegistry = self.typer.types
match name:
case "ndim" | "size":
return types.get_type("int")
case "shape":
return types.tuple_of("int", "int")
case _:
return None

513
midas/checker/frames/frame_methods.py
View File

@@ -1,513 +0,0 @@
from __future__ import annotations
import ast
from dataclasses import dataclass
from typing import TYPE_CHECKING, Optional
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.dispatcher import CallResult
from midas.checker.frames.utils import MethodRegistry, method
from midas.checker.types import (
ColumnType,
DataFrameType,
FrameGroupBy,
Function,
OverloadedFunction,
ParamSpec,
TopType,
Type,
UnknownType,
unfold_type,
)
if TYPE_CHECKING:
from midas.checker.python import TypedExpr
@dataclass(frozen=True, kw_only=True)
class Call:
"""A frame method call, implements :class:`utils.MethodCall`"""
location: Location
call_expr: p.Expr
frame: DataFrameType
frame_expr: p.Expr
positional: list[TypedExpr]
keywords: dict[str, TypedExpr]
@property
def subject(self) -> TypedExpr:
return (self.frame_expr, self.frame)
class FrameMethodRegistry(MethodRegistry[Call]):
"""The method registry for frame types"""
def _get_method_result(
self,
call: Call,
column1: ColumnType,
column2: ColumnType,
method: str,
) -> ColumnType:
"""Get the result of calling a method on a column, passing a second
This function delegates to the main typer the resolution of the method
member, as well as computing the result type. Because we don't have any
AST expression for the individual columns, the frame expressions are
used instead.
Args:
call (Call): the call that triggered this resolution
column1 (ColumnType): the first column, i.e. left operand
column2 (ColumnType): the second column, i.e. right operand
method (str): the method name
Returns:
ColumnType: the resulting column.
If the operation is invalid / doesn't exist,
`ColumnType(type=UnknownType())` is returned
"""
result: Type = self.typer.result_of_binary_op(
location=call.location,
expr=call.call_expr,
left=(call.frame_expr, column1),
right=(call.positional[0][0], column2),
method=method,
)
if not isinstance(result, ColumnType):
return ColumnType(type=UnknownType())
return result
def _element_binary_op(self, call: Call, method: str) -> DataFrameType:
"""Compute the result of an element-wise binary operation
This function delegates to the matching columns for computing resulting
types. Any column only present in one of the frames is forwarded as a
generic `ColumnType(type=UnknownType())`. Columns only in the second
frame are append at the end of the schema.
Args:
call (Call): the call that triggered this resolution
method (str): the method name
Returns:
DataFrameType: the resulting frame type
"""
new_columns: list[DataFrameType.Column] = []
by_name: dict[str, DataFrameType.Column] = {}
frame2: Optional[DataFrameType] = None
# Get map of operand's columns by name, if there is at least 1 operand, which is a dataframe
if len(call.positional) != 0:
operand: TypedExpr = call.positional[0]
unfolded_other: Type = unfold_type(operand[1])
if isinstance(unfolded_other, DataFrameType):
frame2 = unfolded_other
by_name = {
col.name: col for col in frame2.columns if col.name is not None
}
# Compute new schema:
# Step 1: for all columns in frame1:
# - if present in frame2 -> delegate operation to columns
# - if not -> add to schema as unknown
in_frame1: set[str] = set()
for column in call.frame.columns:
if column.name is not None:
in_frame1.add(column.name)
col_type1: ColumnType = column.type
col_type: ColumnType = ColumnType(type=UnknownType())
if column.name in by_name:
column2 = by_name[column.name]
col_type2: ColumnType = column2.type
col_type = self._get_method_result(call, col_type1, col_type2, method)
new_column = DataFrameType.Column(
index=column.index,
name=column.name,
type=col_type,
)
new_columns.append(new_column)
# Step 2: for all columns in frame2
# - if not in frame1 -> add to schema as unknown
if frame2 is not None:
for column in frame2.columns:
if column.name in in_frame1:
continue
new_columns.append(
DataFrameType.Column(
index=len(new_columns),
name=column.name,
type=ColumnType(type=UnknownType()),
)
)
return DataFrameType(columns=new_columns)
def _element_wise(self, call: Call, method: str) -> Type:
"""Compute the result of an element-wise method call
If the call is valid, this method also generates an assertion to check
that both operands have the same length at runtime
Args:
call (Call): the call object
method (str): the method's name
Returns:
Type: the result type
"""
# TODO: support scalar, sequence, Series, dict operand
# Build signature with new schema and generic operand
signature = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="other",
type=DataFrameType(columns=[]),
required=True,
),
],
),
returns=self._element_binary_op(call, method),
)
# Map arguments and compute result type
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
if result.is_valid:
self._assert_same_length(
call.call_expr, call.frame_expr, call.positional[0][0]
)
return result.result
@method("add", "__add__")
def add(self, call: Call) -> Type:
return self._element_wise(call, "__add__")
@method("sub", "__sub__")
def sub(self, call: Call) -> Type:
return self._element_wise(call, "__sub__")
@method("mul", "__mul__")
def mul(self, call: Call) -> Type:
return self._element_wise(call, "__mul__")
@method("div", "truediv", "__truediv__")
def truediv(self, call: Call) -> Type:
return self._element_wise(call, "__truediv__")
@method("floordiv", "__floordiv__")
def floordiv(self, call: Call) -> Type:
return self._element_wise(call, "__floordiv__")
@method("mod", "__mod__")
def mod(self, call: Call) -> Type:
return self._element_wise(call, "__mod__")
@method("pow", "__pow__")
def pow(self, call: Call) -> Type:
return self._element_wise(call, "__pow__")
@method("lt", "__lt__")
def lt(self, call: Call) -> Type:
return self._element_wise(call, "__lt__")
@method("gt", "__gt__")
def gt(self, call: Call) -> Type:
return self._element_wise(call, "__gt__")
@method("le", "__le__")
def le(self, call: Call) -> Type:
return self._element_wise(call, "__le__")
@method("ge", "__ge__")
def ge(self, call: Call) -> Type:
return self._element_wise(call, "__ge__")
@method("ne", "__ne__")
def ne(self, call: Call) -> Type:
return self._element_wise(call, "__ne__")
@method("eq", "__eq__")
def eq(self, call: Call) -> Type:
return self._element_wise(call, "__eq__")
def _aggregate(self, call: Call, kwargs: list[Function.Parameter] = []) -> Type:
"""Compute the result type of an aggregate method call
Args:
call (Call): the call object
kwargs (list[Function.Parameter], optional): a list of extra
keyword-only parameters. Defaults to [].
Returns:
Type: the result type
"""
with_axis = Function(
params=ParamSpec(
kw=[
Function.Parameter(
pos=0,
name="axis",
type=self.types.get_type("int"),
required=False,
),
*kwargs,
],
),
returns=ColumnType(type=TopType()),
)
without_axis = Function(
params=ParamSpec(
kw=[
Function.Parameter(
pos=0,
name="axis",
type=self.types.get_type("None"),
required=True,
),
*kwargs,
],
),
returns=TopType(),
)
overload = OverloadedFunction(
overloads=[
with_axis,
without_axis,
]
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=overload,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method("kurtosis", "kurt")
def kurtosis(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def max(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def mean(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def median(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def min(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def mode(self, call: Call) -> Type:
return self._aggregate(call)
@method("product", "prod")
def product(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def std(self, call: Call) -> Type:
return self._aggregate(
call,
[
Function.Parameter(
pos=1,
name="ddof",
type=self.types.get_type("int"),
required=False,
)
],
)
@method()
def sum(self, call: Call) -> Type:
return self._aggregate(call)
@method()
def var(self, call: Call) -> Type:
return self._aggregate(
call,
[
Function.Parameter(
pos=1,
name="var",
type=self.types.get_type("int"),
required=False,
)
],
)
@method()
def head(self, call: Call) -> Type:
signature = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="n",
type=self.types.get_type("int"),
required=False,
),
],
),
returns=call.frame,
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method()
def tail(self, call: Call) -> Type:
signature = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="n",
type=self.types.get_type("int"),
required=False,
),
],
),
returns=call.frame,
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=signature,
positional=call.positional,
keywords=call.keywords,
)
return result.result
@method()
def groupby(self, call: Call) -> Type:
bool_: Type = self.types.get_type("bool")
function: Function = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="by",
type=TopType(),
required=False,
),
Function.Parameter(
pos=1,
name="level",
type=TopType(),
required=False,
),
],
kw=[
Function.Parameter(
pos=i + 2,
name=name,
type=bool_,
required=False,
)
for i, name in enumerate(
["as_index", "sort", "group_keys", "observed", "dropna"]
)
],
),
returns=FrameGroupBy(frame=call.frame),
)
result: CallResult = self.dispatcher.get_result(
location=call.location,
callee=function,
positional=call.positional,
keywords=call.keywords,
)
return result.result
def _assert_same_length(self, call_expr: p.Expr, frame1: p.Expr, frame2: p.Expr):
"""Generate an assertion to check that two frames have the same length
Args:
call_expr (p.Expr): the call expression, to insert the assertion
at the right place
frame1 (p.Expr): the first frame expression
frame2 (p.Expr): the second frame expression
"""
func_name: str = "__midas_frame_same_length__"
# Efficiently compute length
# https://stackoverflow.com/a/15943975/11109181
def len_of_df(df: ast.expr) -> ast.expr:
return ast.Call(
func=ast.Name(id="len"),
args=[
ast.Attribute(
value=df,
attr="index",
)
],
keywords=[],
)
self.assertions.define(
func_name,
ast.FunctionDef(
name=func_name,
args=ast.arguments(
posonlyargs=[],
args=[
ast.arg(arg="frame1"),
ast.arg(arg="frame2"),
],
kwonlyargs=[],
defaults=[],
kw_defaults=[],
),
body=[
ast.Return(
value=ast.Compare(
left=len_of_df(ast.Name(id="frame1")),
ops=[ast.Eq()],
comparators=[len_of_df(ast.Name(id="frame2"))],
)
)
],
decorator_list=[],
),
)
self.assertions.add(
bound_expr=call_expr,
inputs=[frame1, frame2],
builder=lambda f1, f2: ast.Call(
func=ast.Name(id=func_name),
args=[f1, f2],
keywords=[],
),
message="DataFrames must have the same length",
)

129
midas/checker/frames/utils.py
View File

@@ -1,129 +0,0 @@
from __future__ import annotations
from typing import (
TYPE_CHECKING,
Any,
Callable,
Generic,
Optional,
Protocol,
Self,
TypeVar,
)
import midas.ast.python as p
from midas.ast.location import Location
from midas.checker.dispatcher import CallDispatcher
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter
from midas.checker.types import Type, UnknownType
from midas.generator.collector import AssertionCollector
if TYPE_CHECKING:
from midas.checker.python import PythonTyper, TypedExpr
class _MethodRegistryMeta(type):
"""Meta-class for :class:`MethodRegistry`
Collects methods marked with the :func:`method` decorator into a dictionary
named `_methods` on the class itself
"""
_methods: dict[str, Callable[..., Type]] = {}
def __new__(
cls,
name: str,
bases: tuple[type, ...],
namespace: dict[str, Any],
):
new_class = super().__new__(cls, name, bases, namespace)
new_class._methods = {}
for attr in namespace.values():
if callable(attr) and hasattr(attr, "__method_names__"):
for name in attr.__method_names__: # type: ignore
new_class._methods[name] = attr # type: ignore
return new_class
class MethodCall(Protocol):
"""A method call object
Must have at least `location`, `call_expr` and `subject` properties
"""
@property
def location(self) -> Location: ...
@property
def call_expr(self) -> p.Expr: ...
@property
def subject(self) -> TypedExpr: ...
T = TypeVar("T", bound=MethodCall)
class MethodRegistry(Generic[T], metaclass=_MethodRegistryMeta):
"""A registry of methods"""
def __init__(self, typer: PythonTyper) -> None:
self.typer: PythonTyper = typer
@property
def reporter(self) -> FileReporter:
return self.typer.reporter
@property
def types(self) -> TypesRegistry:
return self.typer.types
@property
def dispatcher(self) -> CallDispatcher[p.Expr]:
return self.typer.dispatcher
@property
def assertions(self) -> AssertionCollector:
return self.typer.assertions
def call(self, method: str, call: T) -> Type:
"""Compute the result type of a call to the given method
Args:
method (str): the method's name
call (T): the call
Returns:
Type: the result type
"""
func: Optional[Callable[[Self, T], Type]] = self._methods.get(method)
if func is None:
self.reporter.warning(
call.location, f"Unknown method {method} on {call.subject[1]}"
)
return UnknownType()
return func(self, call)
_Self = TypeVar("_Self", bound=MethodRegistry[Any])
Method = Callable[[_Self, T], Type]
def method(*names: str) -> Callable[[Method[_Self, T]], Method[_Self, T]]:
"""Simple decorator to mark a method as part of the registry
Args:
names (str): names by which the method can be called. If left empty, the
Python method's name will be used
"""
def wrapper(func: Method[_Self, T]) -> Method[_Self, T]:
names_: tuple[str, ...] = names
if len(names_) == 0:
names_ = (func.__name__,)
setattr(func, "__method_names__", names_)
return func
return wrapper

338
midas/checker/midas.py
View File

@@ -3,79 +3,42 @@ from pathlib import Path
from typing import Optional
import midas.ast.midas as m
from midas.ast.location import Location
from midas.checker.builtins import define_builtins
from midas.checker.dispatcher import CallDispatcher, CallResult
from midas.checker.environment import Environment
from midas.checker.operators import MIDAS_BINARY_METHODS, MIDAS_UNARY_METHODS
from midas.checker.preamble import Preamble
from midas.checker.registry import TypesRegistry
from midas.checker.reporter import FileReporter, Reporter
from midas.checker.types import (
ColumnType,
AliasType,
ComplexType,
ConstraintType,
DataFrameType,
DerivedType,
ExtensionType,
Function,
GenericType,
ParamSpec,
Predicate,
Type,
TypeVar,
UnknownType,
)
from midas.checker.variance import VarianceInferrer
from midas.lexer.midas import MidasLexer
from midas.lexer.token import Token
from midas.parser.midas import MidasParser
class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type]):
class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[None], m.Type.Visitor[Type]):
"""A resolver which evaluates Midas type definitions and build a registry"""
def __init__(self, types: TypesRegistry, reporter: Reporter) -> None:
self.logger: logging.Logger = logging.getLogger("MidasTyper")
self.reporter: FileReporter = reporter.for_file(None)
self.types: TypesRegistry = types
self.dispatcher: CallDispatcher[m.Expr] = CallDispatcher[m.Expr](
self.types, self.reporter
)
self._local_variables: dict[str, TypeVar] = {}
self._predicate_params: dict[str, Type] = {}
self._current_name: Optional[str] = None
define_builtins(self.types)
builtins_path: Path = (Path(__file__).parent / "builtins.midas").resolve()
self.process(builtins_path.read_text(), str(builtins_path))
self._bool: Type = self.get_type("bool")
self._preamble: Environment = Preamble(self.types)
def set_reporter(self, reporter: FileReporter):
"""Set the file reporter to use for diagnostics
Args:
reporter (FileReporter): the file reporter
"""
self.reporter = reporter
self.dispatcher.set_reporter(reporter)
def process(self, source: str, path: Optional[str]):
"""Process some Midas source code
Args:
source (str): the Midas source code
path (Optional[str]): the path of the source file, if known
"""
reporter: FileReporter = self.reporter.for_file(path)
self.set_reporter(reporter)
self.reporter = self.reporter.for_file(path)
lexer: MidasLexer = MidasLexer(source)
tokens: list[Token] = lexer.process()
parser: MidasParser = MidasParser(tokens)
@@ -84,18 +47,6 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
self.reporter.error(error.token.get_location(), error.message)
self.resolve(stmts)
def type_of(self, expr: m.Expr) -> Type:
"""Compute the type of the given expression
Args:
expr (m.Expr): the expression to type
Returns:
Type: the type of the expression
"""
type: Type = expr.accept(self)
return type
def get_type(self, name: str) -> Type:
"""Get a type from its name
@@ -112,34 +63,6 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
return self._local_variables[name]
return self.types.get_type(name)
def get_variable(self, name: str) -> Type:
"""Get the type of a variable
This function will first look into the current predicate's parameters if
we are in a predicate definition.
The the variable is looked up in the preamble (i.e. global environment)
Args:
name (str): the name of the variable
Raises:
NameError: if the variable cannot be found
Returns:
Type: the type of the variable
"""
if name in self._predicate_params:
return self._predicate_params[name]
predicate: Optional[Predicate] = self.types.lookup_predicate(name)
if predicate is not None:
return predicate.type
global_: Optional[Type] = self._preamble.get(name)
if global_ is not None:
return global_
raise NameError(f"Unknown variable '{name}'")
def resolve(self, stmts: list[m.Stmt]):
"""Process a sequence of statements
@@ -149,21 +72,6 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
for stmt in stmts:
stmt.accept(self)
for name, type in self.types._types.items():
if isinstance(type, GenericType):
inferrer = VarianceInferrer(self.types)
self.types._types[name] = inferrer.infer(type)
def assert_bool(self, expr: m.Expr):
"""Check that the given expression is a subtype of `bool` or report an error
Args:
expr (m.Expr): the expression to check
"""
type: Type = self.type_of(expr)
if not self.types.is_subtype(type, self._bool):
self.reporter.error(expr.location, f"Must be a boolean but is {type}")
def visit_type_stmt(self, stmt: m.TypeStmt) -> None:
name: str = stmt.name.lexeme
self._current_name = name
@@ -173,18 +81,11 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
if len(params) != 0:
type = GenericType(name=name, params=params, body=type)
else:
type = DerivedType(name=name, type=type)
type = AliasType(name=name, type=type)
self.types.define_type(name, type)
self._local_variables.clear()
self._current_name = None
def visit_alias_stmt(self, stmt: m.AliasStmt) -> None:
name: str = stmt.name.lexeme
self._current_name = name
type: Type = stmt.type.accept(self)
self.types.define_type(name, type)
self._current_name = None
def visit_member_stmt(self, stmt: m.MemberStmt) -> None: ...
def visit_extend_stmt(self, stmt: m.ExtendStmt) -> None:
@@ -205,171 +106,31 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
)
def visit_predicate_stmt(self, stmt: m.PredicateStmt) -> None:
for spec in stmt.params:
for param in spec.mixed:
assert param.name is not None
self._predicate_params[param.name.lexeme] = param.type.accept(self)
self.reporter.warning(stmt.location, "PredicateStmt not yet supported")
type: Type = self.type_of(stmt.body)
params: list[ParamSpec] = [self._visit_param_spec(spec) for spec in stmt.params]
def visit_logical_expr(self, expr: m.LogicalExpr) -> None:
self.reporter.warning(expr.location, "LogicalExpr not yet supported")
if not self._is_valid_predicate(type):
self.reporter.error(
stmt.body.location,
f"Predicate function body must evaluate to a boolean, got {type}",
)
if len(params) != 0:
type = self._bool
for spec in reversed(params):
type = Function(
params=spec,
returns=type,
)
self._predicate_params = {}
self.types.define_predicate(
stmt.name.lexeme,
Predicate(
type=type,
body=stmt.body,
alias=len(params) == 0,
),
)
def visit_binary_expr(self, expr: m.BinaryExpr) -> None:
self.reporter.warning(expr.location, "BinaryExpr not yet supported")
def _is_valid_predicate(self, body: Type) -> bool:
"""Check whether the given type is valid as a predicate's body
def visit_unary_expr(self, expr: m.UnaryExpr) -> None:
self.reporter.warning(expr.location, "UnaryExpr not yet supported")
Accepted types are either subtypes of `bool` or valid predicates
def visit_get_expr(self, expr: m.GetExpr) -> None:
self.reporter.warning(expr.location, "GetExpr not yet supported")
Args:
body (Type): the potential predicate body
def visit_variable_expr(self, expr: m.VariableExpr) -> None:
self.reporter.warning(expr.location, "VariableExpr not yet supported")
Returns:
bool: `True` if `body` can be a predicate body, `False` otherwise
"""
match body:
case Function(returns=returns):
return self._is_valid_predicate(returns)
case _ if self.types.is_subtype(body, self._bool):
return True
case _:
return False
def visit_logical_expr(self, expr: m.LogicalExpr) -> Type:
self.assert_bool(expr.left)
self.assert_bool(expr.right)
return self._bool
def visit_binary_expr(self, expr: m.BinaryExpr) -> Type:
method: Optional[str] = MIDAS_BINARY_METHODS.get(expr.operator.type)
if method is None:
self.logger.warning(f"Unsupported operator {expr.operator.lexeme}")
self.reporter.warning(
expr.location, f"Unsupported operator {expr.operator.lexeme}"
)
return UnknownType()
return self._visit_binary_expr(expr.location, expr.left, expr.right, method)
def _visit_binary_expr(
self,
location: Location,
left_expr: m.Expr,
right_expr: m.Expr,
method: str,
) -> Type:
left: Type = self.type_of(left_expr)
right: Type = self.type_of(right_expr)
operation: Optional[Type] = self.types.lookup_member(left, method)
if operation is None:
self.reporter.error(
location,
f"Undefined operation {method} between {left} and {right}",
)
return UnknownType()
result: CallResult = self.dispatcher.get_result(
location=location,
callee=operation,
positional=[(right_expr, right)],
keywords={},
)
return result.result
def visit_unary_expr(self, expr: m.UnaryExpr) -> Type:
method: Optional[str] = MIDAS_UNARY_METHODS.get(expr.operator.type)
if method is None:
self.logger.warning(f"Unsupported operator {expr.operator.lexeme}")
self.reporter.warning(
expr.location, f"Unsupported operator {expr.operator.lexeme}"
)
return UnknownType()
operand: Type = self.type_of(expr.right)
operation: Optional[Type] = self.types.lookup_member(operand, method)
if operation is None:
self.reporter.error(
expr.location,
f"Undefined operation {method} for {operand}",
)
return UnknownType()
result: CallResult = self.dispatcher.get_result(
location=expr.location,
callee=operation,
positional=[],
keywords={},
)
return result.result
def visit_call_expr(self, expr: m.CallExpr) -> Type:
callee: Type = expr.callee.accept(self)
positional: list[tuple[m.Expr, Type]] = [
(arg, self.type_of(arg)) for arg in expr.arguments
]
keywords: dict[str, tuple[m.Expr, Type]] = {
name: (arg, self.type_of(arg)) for name, arg in expr.keywords.items()
}
result: CallResult = self.dispatcher.get_result(
location=expr.location,
callee=callee,
positional=positional,
keywords=keywords,
)
return result.result
def visit_get_expr(self, expr: m.GetExpr) -> Type:
object: Type = expr.expr.accept(self)
member: Optional[Type] = self.types.lookup_member(object, expr.name.lexeme)
if member is None:
self.reporter.error(
expr.location, f"Unknown member '{expr.name.lexeme}' of {object}"
)
return UnknownType()
return member
def visit_variable_expr(self, expr: m.VariableExpr) -> Type:
return self.get_variable(expr.name.lexeme)
def visit_grouping_expr(self, expr: m.GroupingExpr) -> Type:
def visit_grouping_expr(self, expr: m.GroupingExpr) -> None:
return expr.expr.accept(self)
def visit_literal_expr(self, expr: m.LiteralExpr) -> Type:
match expr.value:
case bool(): # Must be before int
return self.types.get_type("bool")
case int():
return self.types.get_type("int")
case float():
return self.types.get_type("float")
case str():
return self.types.get_type("str")
case _:
self.reporter.warning(expr.location, f"Unknown literal {expr}")
return UnknownType()
def visit_literal_expr(self, expr: m.LiteralExpr) -> None:
self.reporter.warning(expr.location, "LiteralExpr not yet supported")
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> Type:
return self.get_variable("_")
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> None:
self.reporter.warning(expr.location, "WildcardExpr not yet supported")
def visit_named_type(self, type: m.NamedType) -> Type:
name: str = type.name.lexeme
@@ -392,10 +153,10 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
return UnknownType()
def visit_constraint_type(self, type: m.ConstraintType) -> Type:
return ConstraintType(
type=type.type.accept(self),
constraint=type.constraint,
)
type_: Type = type.type.accept(self)
type.constraint.accept(self)
# TODO
return UnknownType()
def visit_complex_type(self, type: m.ComplexType) -> ComplexType:
return ComplexType(
@@ -411,46 +172,25 @@ class MidasTyper(m.Stmt.Visitor[None], m.Expr.Visitor[Type], m.Type.Visitor[Type
)
def visit_function_type(self, type: m.FunctionType) -> Type:
return Function(
params=self._visit_param_spec(type.params),
returns=type.returns.accept(self),
)
n_pos_args: int = len(type.pos_args)
n_args: int = len(type.args)
def _visit_param_spec(self, spec: m.ParamSpec) -> ParamSpec:
n_pos: int = len(spec.pos)
n_mixed: int = len(spec.mixed)
def process_param(
param: m.FunctionType.Parameter, i: int
) -> Function.Parameter:
return Function.Parameter(
def process_arg(arg: m.FunctionType.Argument, i: int) -> Function.Argument:
return Function.Argument(
pos=i,
name=param.name.lexeme if param.name is not None else str(i),
type=param.type.accept(self),
required=param.required,
name=arg.name.lexeme if arg.name is not None else str(i),
type=arg.type.accept(self),
required=arg.required,
)
return ParamSpec(
pos=[process_param(param, i) for i, param in enumerate(spec.pos)],
mixed=[
process_param(param, i + n_pos) for i, param in enumerate(spec.mixed)
return Function(
pos_args=[process_arg(arg, i) for i, arg in enumerate(type.pos_args)],
args=[process_arg(arg, i + n_pos_args) for i, arg in enumerate(type.args)],
kw_args=[
process_arg(arg, i + n_pos_args + n_args)
for i, arg in enumerate(type.kw_args)
],
kw=[
process_param(param, i + n_pos + n_mixed)
for i, param in enumerate(spec.kw)
],
)
def visit_frame_type(self, type: m.FrameType) -> Type:
def process_column(i: int, col: m.FrameType.Column) -> DataFrameType.Column:
return DataFrameType.Column(
index=i,
name=col.name.lexeme,
type=ColumnType(type=col.type.accept(self)),
)
return DataFrameType(
columns=[process_column(i, col) for i, col in enumerate(type.columns)]
returns=type.returns.accept(self),
)
def _resolve_type_params(self, params: list[m.TypeParam]):

41
midas/checker/operators.py
View File

@@ -1,9 +1,7 @@
import ast
from typing import Type
from midas.lexer.token import TokenType
PY_OPERATOR_METHODS: dict[Type[ast.operator], str] = {
OPERATOR_METHODS: dict[Type[ast.operator], str] = {
ast.Add: "__add__",
ast.Sub: "__sub__",
ast.Mult: "__mul__",
@@ -19,9 +17,9 @@ PY_OPERATOR_METHODS: dict[Type[ast.operator], str] = {
ast.FloorDiv: "__floordiv__",
}
PY_COMPARATOR_METHODS: dict[Type[ast.cmpop], str] = {
COMPARATOR_METHODS: dict[Type[ast.cmpop], str] = {
ast.Eq: "__eq__",
ast.NotEq: "__eq__",
# ast.NotEq: "__noteq__",
ast.Lt: "__lt__",
ast.LtE: "__le__",
ast.Gt: "__gt__",
@@ -32,40 +30,9 @@ PY_COMPARATOR_METHODS: dict[Type[ast.cmpop], str] = {
# ast.NotIn: "__notin__",
}
PY_UNARY_METHODS: dict[Type[ast.unaryop], str] = {
UNARY_METHODS: dict[Type[ast.unaryop], str] = {
ast.Invert: "__invert__",
# ast.Not: "",
ast.UAdd: "__pos__",
ast.USub: "__neg__",
}
MIDAS_BINARY_METHODS: dict[TokenType, str] = {
TokenType.PLUS: "__add__",
TokenType.MINUS: "__sub__",
TokenType.STAR: "__mul__",
TokenType.SLASH: "__truediv__",
# TokenType.MODULO: "__mod__",
# TokenType.POW: "__pow__",
# ast.BitOr: "__or__",
# ast.BitXor: "__xor__",
# ast.BitAnd: "__and__",
# ast.FloorDiv: "__floordiv__",
TokenType.EQUAL_EQUAL: "__eq__",
TokenType.BANG_EQUAL: "__eq__",
TokenType.LESS: "__lt__",
TokenType.LESS_EQUAL: "__le__",
TokenType.GREATER: "__gt__",
TokenType.GREATER_EQUAL: "__ge__",
# ast.Is: "__is__",
# ast.IsNot: "__isnot__",
# ast.In: "__in__",
# ast.NotIn: "__notin__",
}
MIDAS_UNARY_METHODS: dict[TokenType, str] = {
# ast.Invert: "__invert__",
# ast.Not: "",
# TokenType.PLUS: "__pos__",
TokenType.MINUS: "__neg__",
}

124
midas/checker/preamble.py
View File

@@ -1,18 +1,8 @@
from dataclasses import dataclass
from typing import Any, Callable, Optional
from midas.checker.environment import Environment
from midas.checker.registry import TypesRegistry
from midas.checker.types import (
Function,
GenericType,
OverloadedFunction,
ParamSpec,
TopType,
Type,
TypeVar,
UnitType,
)
from midas.checker.types import Function, GenericType, TopType, Type, TypeVar, UnitType
@dataclass(frozen=True)
@@ -23,31 +13,26 @@ class Param:
class Preamble(Environment):
"""The initial environment containing some of Python's builtin functions"""
def __init__(self, types: TypesRegistry) -> None:
super().__init__()
self._types: TypesRegistry = types
self._python_funcs: dict[str, Callable[..., Any]] = {}
self._def_type_constructor("object", object)
self._def_type_constructor("float", float)
self._def_type_constructor("int", int)
self._def_type_constructor("bool", bool)
self._def_type_constructor("str", str)
self._def_type_constructor("object")
self._def_type_constructor("float")
self._def_type_constructor("int")
self._def_type_constructor("bool")
self._def_type_constructor("str")
self._def_function(
name="list",
pos=[Param("object", TopType())],
returns=self._list_of(TopType()),
py_function=list,
)
# TODO: use sink
self._def_function(
name="print",
pos=[Param("object", TopType(), required=False)],
pos=[Param("object", TopType())],
returns=UnitType(),
py_function=print,
)
map_in = TypeVar(name="T", bound=None)
@@ -67,62 +52,17 @@ class Preamble(Environment):
),
],
returns=self._list_of(map_out), # TODO: replace with Iterable[U]
type_vars=[map_in, map_out],
py_function=map,
)
self._def_function(
name="input",
pos=[Param("prompt", TopType(), required=False)],
returns=self._types.get_type("str"),
)
self._def_function(
name="len",
pos=[Param("object", TopType())],
returns=self._types.get_type("int"),
)
T = TypeVar(name="T", bound=None)
self._def_overloads(
name="max",
py_function=max,
signatures=[
(
[Param("arg1", T), Param("arg2", T)],
[],
[],
T,
[T],
),
([Param("iterable", self._list_of(T))], [], [], T, [T]),
],
)
self._def_overloads(
name="min",
py_function=min,
signatures=[
(
[Param("arg1", T), Param("arg2", T)],
[],
[],
T,
[T],
),
([Param("iterable", self._list_of(T))], [], [], T, [T]),
],
)
def _list_of(self, item_type: Type) -> Type:
return self._types.apply_generic(self._types.get_type("list"), [item_type])
def _list_of(self, item_type: str | Type) -> Type:
return self._types.list_of(item_type)
def _def_type_constructor(
self, name: str, py_function: Optional[Callable[..., Any]] = None
):
def _def_type_constructor(self, name: str):
# TODO: more specific arg types
self._def_function(
name=name,
pos=[Param("object", TopType(), required=False)],
returns=self._types.get_type(name),
py_function=py_function,
)
def _make_function(
@@ -135,9 +75,9 @@ class Preamble(Environment):
returns: Type = UnitType(),
type_vars: list[TypeVar] = [],
) -> Type:
def map_params(params: list[Param], offset: int) -> list[Function.Parameter]:
def map_args(params: list[Param], offset: int) -> list[Function.Argument]:
return [
Function.Parameter(
Function.Argument(
pos=i + offset,
name=param.name,
type=param.type,
@@ -147,11 +87,9 @@ class Preamble(Environment):
]
function = Function(
params=ParamSpec(
pos=map_params(pos, 0),
mixed=map_params(mixed, len(pos)),
kw=map_params(kw, len(pos) + len(mixed)),
),
pos_args=map_args(pos, 0),
args=map_args(mixed, len(pos)),
kw_args=map_args(kw, len(pos) + len(mixed)),
returns=returns,
)
if len(type_vars) != 0:
@@ -171,7 +109,6 @@ class Preamble(Environment):
kw: list[Param] = [],
returns: Type = UnitType(),
type_vars: list[TypeVar] = [],
py_function: Optional[Callable[..., Any]] = None,
):
function: Type = self._make_function(
name=name,
@@ -182,34 +119,3 @@ class Preamble(Environment):
type_vars=type_vars,
)
self.define(name, function)
if py_function is not None:
self._python_funcs[name] = py_function
def _def_overloads(
self,
*,
name: str,
signatures: list[
tuple[list[Param], list[Param], list[Param], Type, list[TypeVar]]
],
py_function: Optional[Callable[..., Any]] = None,
):
overloads: list[Type] = []
for pos, mixed, kw, returns, type_vars in signatures:
overloads.append(
self._make_function(
name=name,
pos=pos,
mixed=mixed,
kw=kw,
returns=returns,
type_vars=type_vars,
)
)
function: Type = OverloadedFunction(overloads=overloads)
self.define(name, function)
if py_function is not None:
self._python_funcs[name] = py_function
def get_py_func(self, name: str) -> Optional[Callable[..., Any]]:
return self._python_funcs.get(name)

1240
midas/checker/python.py
View File

File diff suppressed because it is too large Load Diff

357
midas/checker/registry.py
View File

@@ -5,44 +5,33 @@ from typing import Optional
from midas.ast.midas import MemberKind
from midas.checker.builtins import BUILTIN_SUBTYPES
from midas.checker.types import (
AliasType,
AppliedType,
BaseType,
ColumnType,
ComplexType,
ConstraintType,
DataFrameType,
DerivedType,
ExtensionType,
Function,
GenericType,
OverloadedFunction,
Predicate,
TopType,
TupleType,
Type,
TypeVar,
UnknownType,
Variance,
substitute_typevars,
)
@dataclass
class Member:
"""A member of a type (property or method)"""
kind: MemberKind
type: Type
class TypesRegistry:
"""A registry of types, type members and predicates"""
def __init__(self) -> None:
self.logger: logging.Logger = logging.getLogger("TypesRegistry")
self._types: dict[str, Type] = {}
self._members: dict[str, dict[str, Member]] = {}
self._predicates: dict[str, Predicate] = {}
def get_type(self, name: str) -> Type:
"""Get a type from its name
@@ -85,25 +74,6 @@ class TypesRegistry:
member_type: Type,
kind: MemberKind,
):
"""Define a member on a type
If the member is a method and a member with the same name is already
defined on the given type, the two are combined into an :class:`OverloadedFunction`.
If the member is a property and a member with the same name is already
defined on the given type, the new definition is dropped and an error
is reported.
In any case, if a member with the same name but a different kind is
already defined on the given type, the new definition is dropped and
an error is reported.
Args:
type_name (str): the name of the type on which the member is defined
member_name (str): the name of the new member
member_type (Type): the type of the new member
kind (MemberKind): the kind of member to define (property or method)
"""
members: dict[str, Member] = self._members.setdefault(type_name, {})
if member_name in members:
current: Member = members[member_name]
@@ -131,38 +101,6 @@ class TypesRegistry:
else:
members[member_name] = Member(kind=kind, type=member_type)
def define_predicate(self, name: str, predicate: Predicate):
"""Define a predicate
Args:
name (str): the name of the new predicate
predicate (Predicate): the predicate to define
Raises:
ValueError: if a predicate with the same name is already defined
"""
if name in self._predicates:
raise ValueError(f"Predicate {name} already defined")
self._predicates[name] = predicate
def is_builtin_subtype(self, name1: str, name2: str) -> bool:
"""Check whether a type is a subtype of another base on builtin subtype rules
Args:
name1 (str): the name of the potential subtype
name2 (str): the name of the potential supertype
Returns:
bool: _description_
"""
subtypes: set[str] = BUILTIN_SUBTYPES.get(name2, set())
if name1 in subtypes:
return True
for subtype in subtypes:
if self.is_builtin_subtype(name1, subtype):
return True
return False
def is_subtype(self, type1: Type, type2: Type) -> bool:
"""Check whether `type1` is a subtype of `type2`
@@ -183,24 +121,11 @@ class TypesRegistry:
case (_, TopType()):
return True
case (_, UnknownType()):
return True
case (TypeVar(bound=bound), _):
if bound is None:
return False
return self.is_subtype(bound, type2)
case (_, TypeVar(bound=bound)):
if bound is None:
return True
return self.is_subtype(type1, bound)
case (DerivedType(type=base1), _):
case (AliasType(type=base1), _):
return self.is_subtype(base1, type2)
case (BaseType(name=name1), BaseType(name=name2)):
return self.is_builtin_subtype(name1, name2)
return name1 in BUILTIN_SUBTYPES.get(name2, set())
case (ComplexType(properties=props1), ComplexType(properties=props2)):
for k, t in props2.items():
@@ -210,66 +135,16 @@ class TypesRegistry:
return False
return True
case (DataFrameType(columns=columns1), DataFrameType(columns=columns2)):
# TODO: check order?
by_name1: dict[str, DataFrameType.Column] = {
col.name: col for col in columns1 if col.name is not None
}
for col2 in columns2:
if col2.name not in by_name1:
return False
if not self.is_subtype(by_name1[col2.name].type, col2.type):
return False
return True
case (ColumnType(type=inner1), ColumnType(type=inner2)):
# TODO: invariant, replace ColumnType with simple GenericType
if not self.are_equivalent(inner1, inner2):
return False
return True
case (Function(), Function()):
return self.is_func_subtype(type1, type2)
case (ConstraintType(type=base1), _):
return self.is_subtype(base1, type2)
case (
AppliedType(name=name1, args=args1),
AppliedType(name=name2, args=args2),
) if (
name1 == name2
):
generic: Type = self.get_type(name1)
assert isinstance(generic, GenericType)
for param, arg1, arg2 in zip(generic.params, args1, args2):
variance: Variance = param.variance
if variance in {Variance.INVARIANT, Variance.COVARIANT}:
if not self.is_subtype(arg1, arg2):
return False
if variance in {Variance.INVARIANT, Variance.CONTRAVARIANT}:
if not self.is_subtype(arg2, arg1):
return False
return True
# TODO: verify legitimacy
case (AppliedType(body=body), _):
return self.is_subtype(body, type2)
case (TypeVar(bound=bound), _):
if bound is None:
return False
return self.is_subtype(bound, type2)
return False
def are_equivalent(self, type1: Type, type2: Type) -> bool:
"""Check whether two types are equivalent (T <: S and S <: T)
Args:
type1 (Type): the first type
type2 (Type): the second type
Returns:
bool: whether `type1` is a subtype and a supertype of `type2`
"""
return self.is_subtype(type1, type2) and self.is_subtype(type2, type1)
# TODO: verify the logic in here
def is_func_subtype(self, func1: Function, func2: Function) -> bool:
"""Check whether a function is a subtype of another
@@ -284,121 +159,101 @@ class TypesRegistry:
if not self.is_subtype(func1.returns, func2.returns):
return False
pos1: list[Function.Parameter] = func1.params.pos
mixed1: list[Function.Parameter] = func1.params.mixed
kw1: dict[str, Function.Parameter] = {
param.name: param for param in func1.params.kw
}
pos2: list[Function.Parameter] = func2.params.pos
mixed2: list[Function.Parameter] = func2.params.mixed
kw2: dict[str, Function.Parameter] = {
param.name: param for param in func2.params.kw
}
pos1: list[Function.Argument] = func1.pos_args
mixed1: list[Function.Argument] = func1.args
kw1: dict[str, Function.Argument] = {a.name: a for a in func1.kw_args}
pos2: list[Function.Argument] = func2.pos_args
mixed2: list[Function.Argument] = func2.args
kw2: dict[str, Function.Argument] = {a.name: a for a in func2.kw_args}
mixed_by_pos: dict[int, Function.Parameter] = {
param.pos: param for param in mixed2
}
mixed_by_name: dict[str, Function.Parameter] = {
param.name: param for param in mixed2
}
mixed_by_pos: dict[int, Function.Argument] = {arg.pos: arg for arg in mixed2}
mixed_by_name: dict[str, Function.Argument] = {arg.name: arg for arg in mixed2}
def is_arg_subtype(sub: Function.Parameter, sup: Function.Parameter) -> bool:
def is_arg_subtype(sub: Function.Argument, sup: Function.Argument) -> bool:
if not self.is_subtype(sub.type, sup.type):
return False
if not sup.required and sub.required:
return False
return True
for param1 in pos1:
param2: Function.Parameter
if param1.pos < len(pos2):
param2 = pos2[param1.pos]
elif param1.pos in mixed_by_pos:
param2 = mixed_by_pos[param1.pos]
elif not param1.required:
for arg1 in pos1:
arg2: Function.Argument
if arg1.pos < len(pos2):
arg2 = pos2[arg1.pos]
elif arg1.pos in mixed_by_pos:
arg2 = mixed_by_pos[arg1.pos]
elif not arg1.required:
continue
else:
return False
if not is_arg_subtype(param2, param1):
if not is_arg_subtype(arg2, arg1):
return False
for name, param1 in kw1.items():
param2: Function.Parameter
for name, arg1 in kw1.items():
arg2: Function.Argument
if name in kw2:
param2 = kw2[name]
arg2 = kw2[name]
elif name in mixed_by_name:
param2 = mixed_by_name[name]
elif not param1.required:
arg2 = mixed_by_name[name]
elif not arg1.required:
continue
else:
return False
if not is_arg_subtype(param2, param1):
if not is_arg_subtype(arg2, arg1):
return False
for param1 in mixed1:
pos_param2: Optional[Function.Parameter] = None
kw_param2: Optional[Function.Parameter] = None
if param1.name in kw2:
kw_param2 = kw2[param1.name]
elif param1.name in mixed_by_name:
kw_param2 = mixed_by_name[param1.name]
if param1.pos < len(pos2):
pos_param2 = pos2[param1.pos]
elif param1.pos in mixed_by_pos:
pos_param2 = mixed_by_pos[param1.pos]
for arg1 in mixed1:
pos_arg2: Optional[Function.Argument] = None
kw_arg2: Optional[Function.Argument] = None
if arg1.name in kw2:
kw_arg2 = kw2[arg1.name]
elif arg1.name in mixed_by_name:
kw_arg2 = mixed_by_name[arg1.name]
if arg1.pos < len(pos2):
pos_arg2 = pos2[arg1.pos]
elif arg1.pos in mixed_by_pos:
pos_arg2 = mixed_by_pos[arg1.pos]
# No match in func2 and arg is required
if pos_param2 is None and kw_param2 is None and param1.required:
if pos_arg2 is None and kw_arg2 is None and arg1.required:
return False
# Matching keyword argument
if kw_param2 is not None and not is_arg_subtype(kw_param2, param1):
if kw_arg2 is not None and not is_arg_subtype(kw_arg2, arg1):
return False
# Matching positional argument
if pos_param2 is not None and not is_arg_subtype(pos_param2, param1):
if pos_arg2 is not None and not is_arg_subtype(pos_arg2, arg1):
return False
mixed_positions: set[int] = {param.pos for param in mixed1}
mixed_names: set[str] = {param.name for param in mixed1}
for param2 in pos2:
if not param2.required:
mixed_positions: set[int] = {a.pos for a in mixed1}
mixed_names: set[str] = {a.name for a in mixed1}
for arg2 in pos2:
if not arg2.required:
continue
if param2.pos >= len(pos1) and param2.pos not in mixed_positions:
if arg2.pos >= len(pos1) and arg2.pos not in mixed_positions:
return False
for name, param2 in kw2.items():
if not param2.required:
for name, arg2 in kw2.items():
if not arg2.required:
continue
if name not in kw1 and name not in mixed_names:
return False
for param2 in mixed2:
if param2.required:
for arg2 in mixed2:
if arg2.required:
continue
pos_match: bool = param2.pos < len(pos1) or param2.pos in mixed_positions
kw_match: bool = param2.name in kw1 or param2.name in mixed_names
pos_match: bool = arg2.pos < len(pos1) or arg2.pos in mixed_positions
kw_match: bool = arg2.name in kw1 or arg2.name in mixed_names
if not pos_match or not kw_match:
return False
return True
def apply_generic(self, type: Type, args: list[Type]) -> Type:
"""Instantiate a generic type with the given type arguments
Args:
type (Type): the generic
args (list[Type]): the type arguments
Raises:
ValueError: if the arguments are invalid (wrong number, bound violation)
Returns:
Type: the applied generic type
"""
match type:
case DerivedType(name=name, type=base):
return DerivedType(name=name, type=self.apply_generic(base, args))
case AliasType(name=name, type=base):
return AliasType(name=name, type=self.apply_generic(base, args))
case GenericType(name=name, params=type_vars, body=body):
n_args: int = len(args)
@@ -426,9 +281,6 @@ class TypesRegistry:
body=substitute_typevars(body, substitutions),
)
case BaseType(name="tuple"):
return TupleType(items=tuple(args))
case _:
raise ValueError(f"{type} is not a generic type")
@@ -461,19 +313,6 @@ class TypesRegistry:
return [types[i] for i in keep]
def lookup_member(self, type: Type, member_name: str) -> Optional[Type]:
"""Lookup a member by name on a given type
This function first looks up directly on the specified type, then
recurse through supertypes until it finds the member or reaches
the root type
Args:
type (Type): the type on which to lookup the member
member_name (str): the member's name
Returns:
Optional[Type]: the member's type, or `None` if it is not defined
"""
match type:
case BaseType(name=name):
if name in self._members:
@@ -481,7 +320,7 @@ class TypesRegistry:
return self._members[name][member_name].type
return None
case DerivedType(name=name, type=base):
case AliasType(name=name, type=base):
if name in self._members:
if member_name in self._members[name]:
return self._members[name][member_name].type
@@ -520,89 +359,9 @@ class TypesRegistry:
)
return self.lookup_member(base, member_name)
case ConstraintType(type=base):
return self.lookup_member(base, member_name)
case TypeVar(bound=bound) if bound is not None:
return self.lookup_member(bound, member_name)
case UnknownType():
return UnknownType()
case _:
self.logger.debug(f"Can't get member on {type}")
return None
def lookup_predicate(self, name: str) -> Optional[Predicate]:
"""Lookup a predicate by name
Args:
name (str): the name of the predicate
Returns:
Optional[Predicate]: the predicate, or `None` if is not defined
"""
return self._predicates.get(name)
def _by_name_or_type(self, name_or_type: str | Type) -> Type:
"""Get a type by name or return it as is
If `name_or_type` is a string, the associated type is looked up and returned.
Otherwise, the type is returned as is.
Args:
name_or_type (str | Type): the type or type's name
Returns:
Type: the type
"""
if isinstance(name_or_type, str):
return self.get_type(name_or_type)
return name_or_type
def list_of(self, item_type: str | Type) -> Type:
"""Helper method to type a list of a given item type
Args:
item_type (str | Type): the item type
Returns:
Type: the list type
"""
list_ = self.get_type("list")
return self.apply_generic(list_, [self._by_name_or_type(item_type)])
def tuple_of(self, *item_types: str | Type) -> Type:
"""Helper method to type a tuple of given item types
Args:
item_type (str | Type): the item types
Returns:
Type: the tuple type
"""
tuple_ = self.get_type("tuple")
return self.apply_generic(
tuple_,
[self._by_name_or_type(item_type) for item_type in item_types],
)
def dict_of(self, key_type: str | Type, value_type: str | Type) -> Type:
"""Helper method to type a dict of given key and value types
Args:
key_type (str | Type): the key type
value_type (str | Type): the value type
Returns:
Type: the dict type
"""
dict_ = self.get_type("dict")
return self.apply_generic(
dict_,
[
self._by_name_or_type(key_type),
self._by_name_or_type(value_type),
],
)

66
midas/checker/reporter.py
View File

@@ -7,8 +7,6 @@ from midas.checker.diagnostic import Diagnostic, DiagnosticType
class Reporter:
"""Helper class to store diagnostics"""
def __init__(self):
self.diagnostics: list[Diagnostic] = []
@@ -19,14 +17,6 @@ class Reporter:
location: Location,
message: str,
):
"""Create and record a diagnostic
Args:
path (Optional[str]): the path linked to this diagnostic
type (DiagnosticType): the type of diagnostic
location (Location): the location if the diagnostic in the file
message (str): the diagnostic's message
"""
self.diagnostics.append(
Diagnostic(
file_path=path,
@@ -37,52 +27,21 @@ class Reporter:
)
def for_file(self, path: Optional[str]) -> FileReporter:
"""Create a new file reporter for the given path using this reporter
Args:
path (Optional[str]): the path for the new file reporter
Returns:
FileReporter: the new file reporter, linked to this reporter
"""
return FileReporter(self, path)
class FileReporter:
"""Helper class to manage diagnostics for a file"""
def __init__(self, base_reporter: Reporter, path: Optional[str]) -> None:
self.base_reporter: Reporter = base_reporter
self.path: Optional[str] = path
def for_file(self, path: Optional[str]) -> FileReporter:
"""Create a new file reporter for the given path with the same base reporter
Args:
path (Optional[str]): the path for the new file reporter
Returns:
FileReporter: the file reporter
"""
return FileReporter(self.base_reporter, path)
def report(self, type: DiagnosticType, location: Location, message: str):
"""Report a diagnostic to the base reporter
Args:
type (DiagnosticType): the type of diagnostic
location (Location): the location of the diagnostic in the file
message (str): the diagnostic's message
"""
self.base_reporter.report(self.path, type, location, message)
def error(self, location: Location, message: str):
"""Report an error diagnostic
Args:
location (Location): the location of the diagnostic in the file
message (str): the diagnostic's message
"""
self.report(
type=DiagnosticType.ERROR,
location=location,
@@ -90,12 +49,6 @@ class FileReporter:
)
def warning(self, location: Location, message: str):
"""Report a warning diagnostic
Args:
location (Location): the location of the diagnostic in the file
message (str): the diagnostic's message
"""
self.report(
type=DiagnosticType.WARNING,
location=location,
@@ -103,27 +56,8 @@ class FileReporter:
)
def info(self, location: Location, message: str):
"""Report an info diagnostic
Args:
location (Location): the location of the diagnostic in the file
message (str): the diagnostic's message
"""
self.report(
type=DiagnosticType.INFO,
location=location,
message=message,
)
def debug(self, location: Location, message: str):
"""Report a debug diagnostic
Args:
location (Location): the location of the diagnostic in the file
message (str): the diagnostic's message
"""
self.report(
type=DiagnosticType.DEBUG,
location=location,
message=message,
)

18
midas/checker/resolver.py
View File

@@ -78,14 +78,6 @@ class Resolver(p.Stmt.Visitor[None], p.Expr.Visitor[None]):
return
def is_defined(self, name: str) -> bool:
"""Check whether the given variable is defined in any scope
Args:
name (str): the name of the variable
Returns:
bool: `True` if the variable is defined in a scope, `False` otherwise
"""
for scope in self.scopes:
if name in scope:
return True
@@ -101,7 +93,7 @@ class Resolver(p.Stmt.Visitor[None], p.Expr.Visitor[None]):
function (p.Function): the function to resolve
"""
self.begin_scope()
for param in function.params.all:
for param in function.all_args:
self.declare(param.name)
self.define(param.name)
self.resolve(*function.body)
@@ -136,10 +128,6 @@ class Resolver(p.Stmt.Visitor[None], p.Expr.Visitor[None]):
case p.GetExpr():
target.accept(self)
case p.SubscriptExpr():
target.accept(self)
case _:
raise Exception(f"Unsupported assignment to {target}")
@@ -244,9 +232,5 @@ class Resolver(p.Stmt.Visitor[None], p.Expr.Visitor[None]):
if expr.step is not None:
self.resolve(expr.step)
def visit_tuple_expr(self, expr: p.TupleExpr) -> None:
for item in expr.items:
self.resolve(item)
def visit_raw_expr(self, expr: p.RawExpr) -> None:
pass

356
midas/checker/types.py
View File

@@ -1,25 +1,17 @@
from __future__ import annotations
from dataclasses import dataclass, field
from enum import StrEnum
from typing import Optional, assert_never, cast
import midas.ast.midas as m
from midas.ast.printer import MidasPrinter
from typing import Optional
@dataclass(frozen=True, kw_only=True)
class TopType:
"""The top type (`Any`)"""
def __str__(self) -> str:
return "Any"
@dataclass(frozen=True, kw_only=True)
class BaseType:
"""A base / builtin type"""
name: str
def __str__(self) -> str:
@@ -27,9 +19,7 @@ class BaseType:
@dataclass(frozen=True, kw_only=True)
class DerivedType:
"""A derived type, i.e. a named subtype of another type"""
class AliasType:
name: str
type: Type
@@ -39,32 +29,40 @@ class DerivedType:
@dataclass(frozen=True, kw_only=True)
class UnknownType:
"""An unknown type"""
def __str__(self) -> str:
return "<Unknown>"
@dataclass(frozen=True, kw_only=True)
class UnitType:
"""The unit type (`None`)"""
def __str__(self) -> str:
return "None"
@dataclass(frozen=True, kw_only=True)
class Function:
"""A function type"""
params: ParamSpec
pos_args: list[Argument] = field(default_factory=list)
args: list[Argument] = field(default_factory=list)
kw_args: list[Argument] = field(default_factory=list)
returns: Type
def __str__(self) -> str:
return f"{self.params} -> {self.returns}"
args: list[str] = []
if len(self.pos_args) != 0:
args += list(map(str, self.pos_args))
args.append("/")
if len(self.args) != 0:
args += list(map(str, self.args))
if len(self.kw_args) != 0:
args.append("*")
args += list(map(str, self.kw_args))
return f"({', '.join(args)}) -> {self.returns}"
@dataclass(frozen=True, kw_only=True)
class Parameter:
class Argument:
pos: int
name: str
type: Type
@@ -75,34 +73,8 @@ class Function:
return f"{self.name}: {self.type}{opt}"
@dataclass(frozen=True, kw_only=True)
class ParamSpec:
"""A function's parameter spec"""
pos: list[Function.Parameter] = field(default_factory=list)
mixed: list[Function.Parameter] = field(default_factory=list)
kw: list[Function.Parameter] = field(default_factory=list)
def __str__(self) -> str:
params: list[str] = []
if len(self.pos) != 0:
params += list(map(str, self.pos))
params.append("/")
if len(self.mixed) != 0:
params += list(map(str, self.mixed))
if len(self.kw) != 0:
params.append("*")
params += list(map(str, self.kw))
return f"({', '.join(params)})"
@dataclass(frozen=True, kw_only=True)
class OverloadedFunction:
"""A list of method overloads"""
overloads: list[Type]
def __str__(self) -> str:
@@ -111,8 +83,6 @@ class OverloadedFunction:
@dataclass(frozen=True, kw_only=True)
class ComplexType:
"""A type with inline members"""
members: dict[str, Type]
def __str__(self) -> str:
@@ -122,8 +92,6 @@ class ComplexType:
@dataclass(frozen=True, kw_only=True)
class ExtensionType:
"""An extension of a type, adding members through a `ComplexType`"""
base: Type
extension: ComplexType
@@ -131,37 +99,19 @@ class ExtensionType:
return f"{self.base} & {self.extension}"
class Variance(StrEnum):
"""The variance of a :class:`TypeVar`"""
INVARIANT = "INVARIANT"
COVARIANT = "COVARIANT"
CONTRAVARIANT = "CONTRAVARIANT"
@dataclass(frozen=True, kw_only=True)
class TypeVar:
"""A type variable, often used as type parameters for a generic type"""
name: str
bound: Optional[Type]
variance: Variance = Variance.INVARIANT
def __str__(self) -> str:
variance: str = {
Variance.COVARIANT: "+",
Variance.CONTRAVARIANT: "-",
}.get(self.variance, "")
res: str = f"{variance}{self.name}"
if self.bound is not None:
res = f"{res} <: {self.bound}"
return res
return f"{self.name} <: {self.bound}"
return self.name
@dataclass(frozen=True, kw_only=True)
class GenericType:
"""A generic type, with type parameters and a generic body type"""
name: str
params: list[TypeVar]
body: Type
@@ -172,8 +122,6 @@ class GenericType:
@dataclass(frozen=True, kw_only=True)
class AppliedType:
"""An instance of a :class:`GenericType`, with concrete type arguments substituted in its body"""
name: str
args: list[Type]
body: Type
@@ -182,109 +130,13 @@ class AppliedType:
return f"{self.name}[{', '.join(map(str, self.args))}]"
@dataclass(frozen=True, kw_only=True)
class ConstraintType:
"""A type with a constraint expression"""
type: Type
constraint: m.Expr
def __str__(self) -> str:
printer = MidasPrinter()
return f"{self.type} where {printer.print(self.constraint)}"
@dataclass(frozen=True, kw_only=True)
class TupleType:
"""A tuple type, containing any number of ordered item types"""
items: tuple[Type, ...]
def __str__(self) -> str:
return f"({', '.join(map(str, self.items))})"
@dataclass(frozen=True, kw_only=True)
class ColumnType:
"""A column type containing items of a given unique type"""
type: Type
def __str__(self) -> str:
return f"Column[{self.type}]"
@dataclass(frozen=True, kw_only=True)
class DataFrameType:
"""A data-frame type, containing named columns of specific :class:`ColumnType`"""
columns: list[Column]
def __str__(self) -> str:
schema: list[str] = [f"{col.name}: {col.type}" for col in self.columns]
return f"Frame[{', '.join(schema)}]"
@dataclass(frozen=True, kw_only=True)
class Column:
index: int
name: Optional[str]
type: ColumnType
@dataclass(frozen=True, kw_only=True)
class FrameGroupBy:
"""A frame group-by object"""
frame: DataFrameType
def __str__(self) -> str:
return f"FrameGroupBy[{self.frame}]"
@dataclass(frozen=True, kw_only=True)
class ColumnGroupBy:
"""A column group-by object"""
column: ColumnType
def __str__(self) -> str:
return f"ColumnGroupBy[{self.column}]"
def substitute_typevars(type: Type, substitutions: dict[str, Type]) -> Type:
"""Substitute type variables in the given type
This function is called recursively on inner type structures
Args:
type (Type): the type in which to substitute type variables
substitutions (dict[str, Type]): a mapping of type variable names to
concrete types
Returns:
Type: the resulting type with substitutions applied
"""
def sub_parameter(param: Function.Parameter):
return Function.Parameter(
pos=param.pos,
name=param.name,
type=substitute_typevars(param.type, substitutions),
required=param.required,
)
def sub_param_spec(spec: ParamSpec):
return ParamSpec(
pos=list(map(sub_parameter, spec.pos)),
mixed=list(map(sub_parameter, spec.mixed)),
kw=list(map(sub_parameter, spec.kw)),
)
def sub_column(col: DataFrameType.Column):
return DataFrameType.Column(
index=col.index,
name=col.name,
type=cast(ColumnType, substitute_typevars(col.type, substitutions)),
def sub_argument(arg: Function.Argument):
return Function.Argument(
pos=arg.pos,
name=arg.name,
type=substitute_typevars(arg.type, substitutions),
required=arg.required,
)
match type:
@@ -297,17 +149,19 @@ def substitute_typevars(type: Type, substitutions: dict[str, Type]) -> Type:
case BaseType():
return type
case DerivedType(name=name, type=type2):
return DerivedType(
name=name, type=substitute_typevars(type2, substitutions)
)
case AliasType(name=name, type=type2):
return AliasType(name=name, type=substitute_typevars(type2, substitutions))
case Function(
params=params,
pos_args=pos_args,
args=args,
kw_args=kw_args,
returns=returns,
):
return Function(
params=sub_param_spec(params),
pos_args=list(map(sub_argument, pos_args)),
args=list(map(sub_argument, args)),
kw_args=list(map(sub_argument, kw_args)),
returns=substitute_typevars(returns, substitutions),
)
@@ -344,12 +198,6 @@ def substitute_typevars(type: Type, substitutions: dict[str, Type]) -> Type:
body=substitute_typevars(body, substitutions),
)
case ConstraintType():
return ConstraintType(
type=substitute_typevars(type.type, substitutions),
constraint=type.constraint,
)
case TypeVar(name=name):
if name in substitutions:
return substitutions[name]
@@ -370,147 +218,25 @@ def substitute_typevars(type: Type, substitutions: dict[str, Type]) -> Type:
body=substitute_typevars(body, substitutions),
)
case TupleType(items=items):
return TupleType(
items=tuple(substitute_typevars(item, substitutions) for item in items),
)
case ColumnType(type=items_type):
return ColumnType(
type=substitute_typevars(items_type, substitutions),
)
case DataFrameType(columns=columns):
return DataFrameType(
columns=list(map(sub_column, columns)),
)
case FrameGroupBy(frame=frame):
return FrameGroupBy(
frame=cast(DataFrameType, substitute_typevars(frame, substitutions))
)
case ColumnGroupBy(column=column):
return ColumnGroupBy(
column=cast(ColumnType, substitute_typevars(column, substitutions))
)
case UnknownType() | UnitType():
return type
case TopType() | GenericType():
raise NotImplementedError(f"Unsupported type {type}")
# Ensure exhaustiveness
case _:
assert_never(type)
raise NotImplementedError(f"Unsupported type {type}")
def unfold_type(type: Type) -> Type:
"""Unfold a chain of :class:`DerivedType` to get the root supertype
Args:
type (Type): the type to unfold
Returns:
Type: the root supertype
"""
match type:
case DerivedType(type=ref_type):
case AliasType(type=ref_type):
return unfold_type(ref_type)
case _:
return type
def to_annotation(type: Type) -> str:
"""Convert the given type to a Python annotation string
Args:
type (Type): the type to convert
Returns:
str: the annotation string
"""
def _params_annotation(spec: ParamSpec) -> str:
if len(spec.kw) != 0:
return "..."
params: str = ", ".join(
to_annotation(param.type) for param in spec.pos + spec.mixed
)
return f"[{params}]"
match type:
case TopType():
return "Any"
case BaseType(name=name):
return name
case DerivedType(name=name):
return name
case UnknownType():
return "Any"
case UnitType():
return "None"
case Function(params=params, returns=returns):
params_annot: str = _params_annotation(params)
return f"Callable[{params_annot}, {to_annotation(returns)}]"
case OverloadedFunction():
return "Callable"
case ComplexType() | ExtensionType():
raise NotImplementedError
case TypeVar(name=name):
return name
case GenericType(name=name, params=params):
return f"{name}[{', '.join(map(to_annotation, params))}]"
case AppliedType(name=name, args=args):
return f"{name}[{', '.join(map(to_annotation, args))}]"
case ConstraintType():
return str(type)
case TupleType(items=items):
return f"Tuple[{', '.join(map(to_annotation, items))}]"
case ColumnType():
return "pd.Series"
case DataFrameType():
return "pd.DataFrame"
case FrameGroupBy():
return "pd.api.typing.DataFrameGroupBy"
case ColumnGroupBy():
return "pd.api.typing.SeriesGroupBy"
case _:
assert_never(type)
@dataclass(frozen=True, kw_only=True)
class Predicate:
"""A predicate"""
type: Type
body: m.Expr
alias: bool
Type = (
TopType
| BaseType
| DerivedType
| AliasType
| UnknownType
| UnitType
| Function
@@ -520,10 +246,4 @@ Type = (
| TypeVar
| GenericType
| AppliedType
| ConstraintType
| TupleType
| ColumnType
| DataFrameType
| FrameGroupBy
| ColumnGroupBy
)

268
midas/checker/unifier.py
View File

@@ -1,268 +0,0 @@
import logging
from typing import Optional
from midas.checker.registry import TypesRegistry
from midas.checker.types import (
AppliedType,
ColumnType,
DataFrameType,
Function,
GenericType,
ParamSpec,
TopType,
Type,
TypeVar,
)
class UnificationError(Exception): ...
class Unifier:
"""
Helper class to unify generic types in concrete usages
This can be used for example when a generic function is called with concrete
arguments, at which point the type parameters of the function signature
should be resolvable
"""
def __init__(self, types: TypesRegistry) -> None:
self.types: TypesRegistry = types
self.logger: logging.Logger = logging.getLogger("Unifier")
def unify_call(
self,
type: GenericType,
positional: list[Type],
keywords: dict[str, Type],
) -> Optional[Type]:
"""Try and unify a generic function call given concrete arguments
Args:
type (GenericType): the generic function type
positional (list[Type]): the list of positional arguments
keywords (dict[str, Type]): the map of keyword arguments
Returns:
Optional[Type]: the concrete function type if unifiable, or `None`
"""
concrete_func: Function = Function(
params=ParamSpec(
pos=[
Function.Parameter(
pos=i,
name=str(i),
type=arg,
required=True,
)
for i, arg in enumerate(positional)
],
kw=[
Function.Parameter(
pos=len(positional) + i,
name=name,
type=arg,
required=True,
)
for i, (name, arg) in enumerate(keywords.items())
],
),
returns=TopType(), # TODO: use expected type
)
return self.unify_generic(type, concrete_func, match_return=False)
def unify_generic(
self,
template: GenericType,
concrete: Type,
match_return: bool = True,
) -> Optional[Type]:
"""Unify a generic type's parameters given a concrete usage
Args:
template (GenericType): the generic type
concrete (Type): a concrete usage
match_return (bool, optional): if `template` is a function type,
whether its return type must be matched (see :func:`match`).
Defaults to True.
Returns:
Optional[Type]: the concrete type if unifiable, or `None`
"""
substitutions: dict[str, Type]
try:
substitutions = self.match(template.body, concrete, match_return)
except UnificationError:
return None
args: list[Type] = []
for param in template.params:
if param.name not in substitutions:
return None
args.append(substitutions[param.name])
applied: Type = self.types.apply_generic(template, args)
return applied
def match(
self,
template: Type,
concrete: Type,
match_return: bool = True,
) -> dict[str, Type]:
"""Match a generic type with a concrete usage, recording parameter substitutions
Args:
template (Type): the generic type
concrete (Type): a concrete usage
match_return (bool, optional): if `template` and `concrete` are both
:class:`Function`, whether their return types are also matched.
Defaults to True.
Raises:
UnificationError: if there is a conflict in parameter substitutions
Returns:
dict[str, Type]: the parameter substitutions which,
applied to `template`, yield `concrete`
"""
# TODO: if concrete is Generic, record bound TypeVar. Then when merging
# substitutions, check that the constraint is respected
match (template, concrete):
case (TypeVar(name=name), _):
return {name: concrete}
case (
AppliedType(name=template_name, args=template_args),
AppliedType(name=concrete_name, args=concrete_args),
) if template_name == concrete_name and len(template_args) == len(
concrete_args
):
substitutions: dict[str, Type] = {}
for template_arg, concrete_arg in zip(template_args, concrete_args):
new_substistutions: dict[str, Type] = self.match(
template_arg, concrete_arg
)
substitutions = self.merge(substitutions, new_substistutions)
return substitutions
case (
DataFrameType(columns=template_columns),
DataFrameType(columns=concrete_columns),
) if len(template_columns) == len(concrete_columns):
substitutions: dict[str, Type] = {}
for template_column, concrete_column in zip(
template_columns, concrete_columns
):
if template_column.index != concrete_column or (
template_column.name != concrete_column.name
):
self.logger.debug(
f"Column mismatch: template={template_column}, concrete={concrete_column}"
)
raise UnificationError
new_substistutions: dict[str, Type] = self.match(
template_column.type, concrete_column.type
)
substitutions = self.merge(substitutions, new_substistutions)
return substitutions
case (ColumnType(type=template_column), ColumnType(type=concrete_column)):
return self.match(template_column, concrete_column)
case (Function(), Function()):
mapped: list[tuple[Function.Parameter, Function.Parameter]] = (
self.map_params(template, concrete)
)
substitutions: dict[str, Type] = {}
for template_arg, concrete_arg in mapped:
arg_subs: dict[str, Type] = self.match(
template_arg.type, concrete_arg.type
)
substitutions = self.merge(substitutions, arg_subs)
if match_return:
return_subs: dict[str, Type] = self.match(
template.returns, concrete.returns
)
substitutions = self.merge(substitutions, return_subs)
return substitutions
case _:
self.logger.debug(f"Can't match {concrete!r} with {template!r}")
return {}
def merge(self, subs1: dict[str, Type], subs2: dict[str, Type]) -> dict[str, Type]:
"""Merge two maps of substitutions and raise an error if incompatible
Args:
subs1 (dict[str, Type]): the first substitutions
subs2 (dict[str, Type]): the second substitutions
Raises:
UnificationError: if there is a conflict between the two maps
Returns:
dict[str, Type]: the merged map of substitutions
"""
merged: dict[str, Type] = subs1.copy()
for k, v in subs2.items():
if k in merged and merged[k] != v:
self.logger.debug(
f"Substitution already defined for {k} with type {merged[k]}, got {v}"
)
raise UnificationError
merged[k] = v
return merged
def map_params(
self, func1: Function, func2: Function
) -> list[tuple[Function.Parameter, Function.Parameter]]:
"""Map parameters of two functions
Args:
func1 (Function): the first function
func2 (Function): the second function
Returns:
list[tuple[Function.Parameter, Function.Parameter]]: the list of parameter pairs
"""
pos1: list[Function.Parameter] = func1.params.pos
mixed1: list[Function.Parameter] = func1.params.mixed
kw1: list[Function.Parameter] = func1.params.kw
pos2: list[Function.Parameter] = func2.params.pos
mixed2: list[Function.Parameter] = func2.params.mixed
kw2: list[Function.Parameter] = func2.params.kw
mapped: list[tuple[Function.Parameter, Function.Parameter]] = []
by_pos2: dict[int, Function.Parameter] = {
param.pos: param for param in pos2 + mixed2
}
by_name2: dict[str, Function.Parameter] = {
param.name: param for param in mixed2 + kw2
}
for arg1 in pos1:
if (arg2 := by_pos2.get(arg1.pos)) is not None:
mapped.append((arg1, arg2))
for arg1 in mixed1:
# Match both positionally and by name, conflicts are caught
# when merging substitutions
if (arg2 := by_pos2.get(arg1.pos)) is not None:
mapped.append((arg1, arg2))
if (arg2 := by_name2.get(arg1.name)) is not None:
mapped.append((arg1, arg2))
for arg1 in kw1:
if (arg2 := by_name2.get(arg1.name)) is not None:
mapped.append((arg1, arg2))
return mapped

181
midas/checker/variance.py
View File

@@ -1,181 +0,0 @@
from typing import Literal, Optional, cast
from midas.checker.registry import Member, TypesRegistry
from midas.checker.types import (
AppliedType,
ConstraintType,
Function,
GenericType,
OverloadedFunction,
Type,
TypeVar,
Variance,
)
Polarity = Literal[-1, 0, 1]
class Tracker:
"""Helper class to track the polarity of type parameter references and computer their variance"""
def __init__(self, vars: list[TypeVar]) -> None:
self.vars: list[TypeVar] = vars
self.refs: dict[str, set[Polarity]] = {var.name: set() for var in self.vars}
def record(self, var: TypeVar, polarity: Polarity):
"""Record a polarity of the given type parameter
Args:
var (TypeVar): the type parameter
polarity (Polarity): the polarity
"""
self.refs[var.name].add(polarity)
def get_updated_vars(self) -> list[TypeVar]:
"""Get a list of the tracked type variables with their recorded variance
Returns:
list[TypeVar]: the list of update type parameters
"""
return [
TypeVar(
name=var.name, bound=var.bound, variance=self.get_variance(var.name)
)
for var in self.vars
]
def get_variance(self, name: str) -> Variance:
"""Get the variance of a type parameter
If the type parameter is only referenced in positive positions, it is
covariant. If it is only referenced in negative positions, it is
contravariant. Otherwise, it is invariant
Args:
name (str): the name of the type parameter
Returns:
Variance: the variance of the type parameter
"""
refs: set[Polarity] = self.refs[name]
if refs == {-1}:
return Variance.CONTRAVARIANT
if refs == {1}:
return Variance.COVARIANT
return Variance.INVARIANT
def __contains__(self, item: TypeVar | str):
if isinstance(item, TypeVar):
return item.name in self
return item in self.refs
class VarianceInferrer:
"""Helper class to compute type parameter variance"""
def __init__(self, types: TypesRegistry) -> None:
self.types: TypesRegistry = types
self.tracker: Tracker = Tracker([])
def infer(self, type: GenericType) -> GenericType:
"""Infer the variance of a generic type's parameters
Args:
type (GenericType): the generic type
Returns:
GenericType: a new generic type with its parameters updated with
their inferred variance
"""
self.tracker = Tracker(type.params)
self.walk(type.body, 1, type.name)
members: dict[str, Member] = self.types._members.get(type.name, {})
for name, member in members.items():
self.walk(member.type, 1, type.name, [f"member:'{name}'"])
return GenericType(
name=type.name,
params=self.tracker.get_updated_vars(),
body=type.body,
)
def walk(
self,
type: Type,
polarity: Polarity,
base_name: str,
path: Optional[list[str]] = None,
):
"""Walk the type nodes and record variance
This function recurses into type substructures (e.g. function parameters,
overloads, constraint type bases, etc.)
When recursing, the polarity is flipped for consumer positions (e.g. function
parameters) or kept the same for producer positions (e.g. return type)
Args:
type (Type): the type to visit
polarity (Polarity): the current polarity
base_name (str): the root generic type name (used to detect and
handle cyclic references)
path (Optional[list[str]], optional): the path to reach the current
type from the root generic type (used for debugging). Defaults to None.
"""
if path is None:
path = []
match type:
# Arguments are negative positions -> flip polarity
# Return is positive position -> keep polarity
case Function(params=spec):
all_params: list[Function.Parameter] = spec.pos + spec.mixed + spec.kw
for param in all_params:
self.walk(
param.type,
-polarity,
base_name,
path + [f"param:'{param.name}'"],
)
self.walk(type.returns, polarity, base_name, path + ["return"])
# Walk all overloads
case OverloadedFunction(overloads=overloads):
for overload in overloads:
self.walk(overload, polarity, base_name, path)
# If same name as root generic -> skip
# Get inferred variance of parameters and multiply with current
# polarity to recurse through arguments
case AppliedType(name=name, args=args):
# TODO: handle mutually recursive types
if name == base_name:
return
generic: Type = self.types.get_type(name)
assert isinstance(generic, GenericType)
params: list[TypeVar] = generic.params
polarities: dict[Variance, Polarity] = {
Variance.INVARIANT: 0,
Variance.COVARIANT: 1,
Variance.CONTRAVARIANT: -1,
}
for arg, param in zip(args, params):
param_polarity: Polarity = polarities[param.variance]
self.walk(
arg,
cast(Polarity, polarity * param_polarity),
base_name,
path + [f"applied:'{name}'"],
)
# Walk base type
case ConstraintType(type=base):
self.walk(base, polarity, base_name, path + ["constraint"])
# Reached end
# If tracked, record polarity
case TypeVar():
if type in self.tracker:
self.tracker.record(type, polarity)

21
midas/cli/commands/compile.py
View File

@@ -5,7 +5,7 @@
import sys
from pathlib import Path
from typing import Optional, TextIO
from typing import TextIO
import click
@@ -19,33 +19,24 @@ from midas.utils import TypedAST
@click.command(help="Compile source")
@click.argument("file", type=click.File("r"))
@click.option("-t", "--types", type=click.File("r"), multiple=True)
@click.option("-s", "--stubs", type=str, multiple=True)
@click.option("--ignore-errors", is_flag=True)
def compile(
file: TextIO,
types: tuple[TextIO],
stubs: tuple[str],
ignore_errors: bool,
):
source: str = file.read()
source_path: Path = Path(file.name).resolve()
checker = TypeChecker()
type_files: list[tuple[Path, Optional[str]]] = []
for i, types_file in enumerate(types):
in_path: Path = Path(types_file.name).resolve()
checker.import_midas(in_path)
type_files.append((in_path, stubs[i] if i < len(stubs) else None))
for types_file in types:
checker.import_midas(Path(types_file.name).resolve())
typed_ast: TypedAST = checker.type_check_source(source, str(source_path))
diagnostics: list[Diagnostic] = checker.diagnostics.copy()
printer = DiagnosticPrinter()
printer.print_all(diagnostics)
if not ignore_errors and any(
map(lambda d: d.type == DiagnosticType.ERROR, diagnostics)
):
if any(map(lambda d: d.type == DiagnosticType.ERROR, diagnostics)):
sys.exit(1)
generator = Generator(workdir=source_path.parent, types=checker.types)
generator.generate(typed_ast, source_path, type_files=type_files)
generator = Generator(workdir=source_path.parent)
generator.generate(typed_ast, source_path)

34
midas/cli/commands/registry.py
View File

@@ -8,17 +8,15 @@ from typing import TextIO
import click
from midas.ast.printer import MidasPrinter
from midas.checker.checker import TypeChecker
from midas.checker.registry import Member
from midas.checker.types import AppliedType, BaseType, DerivedType, GenericType, Type
from midas.checker.types import AliasType, AppliedType, BaseType, GenericType, Type
def base_type(type: Type) -> Type:
match type:
case BaseType():
return type
case DerivedType(type=base):
case AliasType(type=base):
return base
case AppliedType(body=body):
return body
@@ -37,30 +35,10 @@ def dump_registry(
for types_file in types:
checker.import_midas(Path(types_file.name).resolve())
print("##### Types #####")
for name, type in checker.types._types.items():
members: dict[str, Member] = checker.types._members.get(name, {})
params: str = ""
if isinstance(type, GenericType):
params = ", ".join(map(str, type.params))
params = f"[{params}]"
print(f"{name}{params} = {base_type(type)}")
members: dict[str, Type] = checker.types._members.get(name, {})
print(f"{name} = {base_type(type)}")
if len(members) != 0:
print(" " * 4 + "Members:")
for member_name, member in members.items():
kind: str = member.kind.name
print(" " * 8 + f"({kind:8}) {member_name}: {member.type}")
print("##### Predicates #####")
printer = MidasPrinter()
for name, predicate in checker.types._predicates.items():
body: str = printer.print(predicate.body)
if predicate.alias:
print(f"{name}: {predicate.type} = {body}")
else:
print(f"{name}{predicate.type}:")
body = "\n".join(
" " + ("return " if i == 0 else "") + line
for i, line in enumerate(body.split("\n"))
)
print(body)
for member_name, member_type in members.items():
print(" " * 8 + f"{member_name}: {member_type}")

63
midas/cli/commands/stubs.py
View File

@@ -1,66 +1,27 @@
import ast
import time
from pathlib import Path
from typing import Optional, TextIO
from typing import TextIO
import black
import click
from watchdog.events import DirModifiedEvent, FileModifiedEvent, FileSystemEventHandler
from watchdog.observers import Observer
from midas.checker.checker import TypeChecker
from midas.generator.stubs import StubsGenerator
def generate_stubs(in_path: Path, out_path: Path):
@click.command(help="Generate stubs from Midas definitions")
@click.argument("file", type=click.File("r"))
@click.option("-o", "--output", type=click.File("w"), default="-")
def stubs(
file: TextIO,
output: TextIO,
):
source_path: Path = Path(file.name).resolve()
checker = TypeChecker()
checker.import_midas(in_path)
checker.import_midas(source_path)
generator = StubsGenerator(checker.types)
module: ast.Module = generator.generate_stubs()
module = ast.fix_missing_locations(module)
output: str = ast.unparse(module)
output = black.format_str(output, mode=black.Mode(is_pyi=True))
out_path.write_text(output)
class Handler(FileSystemEventHandler):
def __init__(self, in_path: Path, out_path: Path) -> None:
super().__init__()
self.in_path: Path = in_path
self.out_path: Path = out_path
def on_modified(self, event: DirModifiedEvent | FileModifiedEvent) -> None:
generate_stubs(self.in_path, self.out_path)
@click.command(help="Generate stubs from Midas definitions")
@click.argument("file", type=click.File("r"))
@click.option("-o", "--output", type=click.File("w"))
@click.option("-w", "--watch", is_flag=True)
def stubs(
file: TextIO,
output: Optional[TextIO],
watch: bool,
):
source_path: Path = Path(file.name).resolve()
out_path: Path = source_path.with_suffix(".pyi")
if output is not None:
out_path = Path(output.name).resolve()
generate_stubs(source_path, out_path)
if watch:
print(f"Watching {source_path}...")
print("Press CTRL+C to stop")
handler = Handler(source_path, out_path)
observer = Observer()
observer.schedule(handler, str(source_path))
observer.start()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
output.write(ast.unparse(module))

1
midas/cli/commands/types.py
View File

@@ -41,7 +41,6 @@ def types(
message=f"Type: {type}",
)
)
diagnostics.extend(checker.diagnostics)
printer = DiagnosticPrinter()
printer.print_all(diagnostics)

64
midas/cli/highlighter.py
View File

@@ -134,9 +134,9 @@ class PythonHighlighter(
def visit_base_type(self, node: p.BaseType) -> None:
self.wrap(node, "base-type")
for arg in node.args:
self.wrap(arg, "arg")
arg.accept(self)
if node.param is not None:
self.wrap(node.param, "param")
node.param.accept(self)
def visit_constraint_type(self, node: p.ConstraintType) -> None:
self.wrap(node, "constraint-type")
@@ -157,18 +157,15 @@ class PythonHighlighter(
def visit_function(self, stmt: p.Function) -> None:
self.wrap(stmt, "function")
self._highlight_param_spec(stmt.params)
for arg in stmt.posonlyargs + stmt.args + stmt.kwonlyargs:
self._highlight_function_argument(arg)
for body_stmt in stmt.body:
body_stmt.accept(self)
def _highlight_param_spec(self, spec: p.ParamSpec) -> None:
for param in spec.all:
self._highlight_function_param(param)
def _highlight_function_param(self, param: p.Function.Parameter) -> None:
self.wrap(param, "parameter")
if param.type is not None:
param.type.accept(self)
def _highlight_function_argument(self, arg: p.Function.Argument) -> None:
self.wrap(arg, "argument")
if arg.type is not None:
arg.type.accept(self)
def visit_type_assign(self, stmt: p.TypeAssign) -> None:
stmt.type.accept(self)
@@ -231,13 +228,6 @@ class PythonHighlighter(
for item in expr.items:
item.accept(self)
def visit_dict_expr(self, expr: p.DictExpr) -> None:
for key in expr.keys:
if key is not None:
key.accept(self)
for value in expr.values:
value.accept(self)
def visit_subscript_expr(self, expr: p.SubscriptExpr) -> None:
expr.object.accept(self)
expr.index.accept(self)
@@ -250,14 +240,6 @@ class PythonHighlighter(
if expr.step is not None:
expr.step.accept(self)
def visit_tuple_expr(self, expr: p.TupleExpr) -> None:
for item in expr.items:
item.accept(self)
def visit_raw_expr(self, expr: p.RawExpr) -> None: ...
def visit_raw_stmt(self, stmt: p.RawStmt) -> None: ...
class MidasHighlighter(
Highlighter, m.Stmt.Visitor[None], m.Expr.Visitor[None], m.Type.Visitor[None]
@@ -284,9 +266,8 @@ class MidasHighlighter(
def visit_predicate_stmt(self, stmt: m.PredicateStmt) -> None:
self.wrap(stmt, "predicate")
self.wrap(LocatableToken(stmt.name), "predicate-name")
for spec in stmt.params:
self._visit_param_spec(spec)
stmt.body.accept(self)
stmt.type.accept(self)
stmt.condition.accept(self)
def visit_logical_expr(self, expr: m.LogicalExpr) -> None:
self.wrap(expr, "logical-expr")
@@ -302,14 +283,6 @@ class MidasHighlighter(
self.wrap(expr, "unary-expr")
expr.right.accept(self)
def visit_call_expr(self, expr: m.CallExpr) -> None:
self.wrap(expr, "call-expr")
expr.callee.accept(self)
for arg in expr.arguments:
arg.accept(self)
for arg in expr.keywords.values():
arg.accept(self)
def visit_get_expr(self, expr: m.GetExpr) -> None:
self.wrap(expr, "get-expr")
expr.expr.accept(self)
@@ -345,7 +318,8 @@ class MidasHighlighter(
def visit_function_type(self, type: m.FunctionType) -> None:
self.wrap(type, "function")
self._visit_param_spec(type.params)
for arg in type.pos_args + type.args + type.kw_args:
arg.type.accept(self)
type.returns.accept(self)
def visit_extension_type(self, type: m.ExtensionType) -> None:
@@ -353,18 +327,6 @@ class MidasHighlighter(
type.base.accept(self)
type.extension.accept(self)
def _visit_param_spec(self, spec: m.ParamSpec) -> None:
for param in spec.pos + spec.mixed + spec.kw:
param.type.accept(self)
def visit_frame_type(self, type: m.FrameType) -> None:
self.wrap(type, "frame")
for column in type.columns:
self._visit_frame_column(column)
def _visit_frame_column(self, column: m.FrameType.Column) -> None:
self.wrap(column, "column")
class DiagnosticsHighlighter(Highlighter):
EXTRA_CSS_PATH: Optional[Path] = Path(__file__).parent / "hl_diagnostic.css"

4
midas/cli/hl_python.css
View File

@@ -3,7 +3,7 @@ span {
--col: 108, 233, 108;
}
&.arg {
&.param {
--col: 103, 192, 224;
}
@@ -23,7 +23,7 @@ span {
--col: 215, 103, 224;
}
&.parameter {
&.argument {
--col: 103, 192, 224;
}
}

55
midas/cli/utils.py
View File

@@ -1,4 +1,3 @@
from collections import defaultdict
from pathlib import Path
from typing import Optional
@@ -8,13 +7,6 @@ from midas.cli.ansi import Ansi
class DiagnosticPrinter:
COLORS: dict[DiagnosticType, int] = {
DiagnosticType.ERROR: Ansi.RED,
DiagnosticType.WARNING: Ansi.YELLOW,
DiagnosticType.INFO: Ansi.CYAN,
DiagnosticType.DEBUG: Ansi.MAGENTA,
}
def __init__(self) -> None:
self.files: dict[Optional[str], list[str]] = {}
@@ -30,25 +22,10 @@ class DiagnosticPrinter:
return self.files[filename]
def print_all(self, diagnostics: list[Diagnostic], indent: int = 4):
by_type: dict[DiagnosticType, int] = defaultdict(int)
for diagnostic in diagnostics:
filename: Optional[str] = diagnostic.file_path
lines = self.get_lines(filename)
self.print(lines, diagnostic, indent=indent)
by_type[diagnostic.type] += 1
if len(diagnostics) == 0:
return
counts: list[str] = []
for type in DiagnosticType:
if type not in by_type:
continue
count: int = by_type[type]
color: int = self.COLORS.get(type, Ansi.WHITE)
counts.append(f"{Ansi.FG(color)}{type.value}s{Ansi.RESET}: {count}")
print(" ".join(counts))
def print(self, lines: list[str], diagnostic: Diagnostic, indent: int = 4):
"""Pretty-print a diagnostic, showing some context if possible
@@ -68,7 +45,7 @@ class DiagnosticPrinter:
loc: Location = diagnostic.location
if loc.lineno != loc.end_lineno:
self.print_multiline(lines, diagnostic, indent)
print(diagnostic)
return
start_offset: int = loc.col_offset
@@ -78,7 +55,11 @@ class DiagnosticPrinter:
before: str = line[:start_offset]
after: str = line[end_offset:]
color: int = self.COLORS.get(diagnostic.type, Ansi.WHITE)
color: int = {
DiagnosticType.ERROR: Ansi.RED,
DiagnosticType.WARNING: Ansi.YELLOW,
DiagnosticType.INFO: Ansi.CYAN,
}.get(diagnostic.type, Ansi.WHITE)
subject: str = Ansi.FG(color) + line[start_offset:end_offset] + Ansi.RESET
cursor: str = (
@@ -95,27 +76,3 @@ class DiagnosticPrinter:
print(indent_str + before + subject + after)
print(indent_str + cursor)
print()
def print_multiline(
self, all_lines: list[str], diagnostic: Diagnostic, indent: int = 4
):
loc: Location = diagnostic.location
lines: list[str] = all_lines[loc.lineno - 1 : loc.end_lineno]
start_offset: int = loc.col_offset
end_offset: int = loc.end_col_offset or (start_offset + 1)
indent_str: str = " " * indent
color: int = self.COLORS.get(diagnostic.type, Ansi.WHITE)
res: str = indent_str + lines[0][:start_offset]
res += Ansi.FG(color) + lines[0][start_offset:]
for line in lines[1:-1]:
res += "\n" + indent_str + line
res += "\n" + indent_str + lines[-1][:end_offset]
res += Ansi.RESET + lines[-1][end_offset:]
print(diagnostic.location_str + ":")
print(res)
print()
print(Ansi.FG(color) + diagnostic.message + Ansi.RESET)
print()

59
midas/generator/collector.py
View File

@@ -1,59 +0,0 @@
import ast
from dataclasses import dataclass
from typing import Callable
import midas.ast.python as p
AssertionBuilder = Callable[..., ast.expr]
@dataclass
class Assertion:
bound_expr: p.Expr
inputs: list[p.Expr]
builder: AssertionBuilder
message: str
def is_bound_to(self, expr: p.Expr) -> bool:
return expr == self.bound_expr
class AssertionCollector:
def __init__(self):
self.assertions: list[Assertion] = []
self.definitions: dict[str, ast.stmt] = {}
def add(
self,
bound_expr: p.Expr,
inputs: list[p.Expr],
builder: AssertionBuilder,
message: str,
):
self.assertions.append(
Assertion(
bound_expr=bound_expr,
inputs=inputs,
builder=builder,
message=message,
)
)
def remove(self, assertion: Assertion):
try:
self.assertions.remove(assertion)
except ValueError:
pass
def define(self, name: str, stmt: ast.stmt):
if name not in self.definitions:
self.definitions[name] = stmt
def get_definitions(self) -> list[ast.stmt]:
return list(self.definitions.values())
def get_assertions(self) -> list[Assertion]:
return self.assertions
def get_assertions_for(self, expr: p.Expr) -> list[Assertion]:
return list(filter(lambda a: a.is_bound_to(expr), self.assertions))

225
midas/generator/constraints.py
View File

@@ -1,225 +0,0 @@
import ast
from typing import Optional
import midas.ast.midas as m
from midas.checker.registry import TypesRegistry
from midas.checker.types import (
Function,
ParamSpec,
Predicate,
Type,
to_annotation,
)
from midas.lexer.token import TokenType
LOGICAL_OPERATORS: dict[TokenType, type[ast.boolop]] = {
TokenType.AND: ast.And,
# TokenType.OR: ast.Or,
}
BINARY_OPERATORS: dict[TokenType, type[ast.operator]] = {
# TokenType.PLUS: ast.Add,
TokenType.MINUS: ast.Sub,
TokenType.STAR: ast.Mult,
TokenType.SLASH: ast.Div,
}
UNARY_OPERATORS: dict[TokenType, type[ast.unaryop]] = {
# TokenType.PLUS: ast.UAdd,
TokenType.MINUS: ast.USub,
}
COMPARISON_OPERATORS: dict[TokenType, type[ast.cmpop]] = {
TokenType.GREATER: ast.Gt,
TokenType.GREATER_EQUAL: ast.GtE,
TokenType.LESS: ast.Lt,
TokenType.LESS_EQUAL: ast.LtE,
TokenType.EQUAL_EQUAL: ast.Eq,
TokenType.BANG_EQUAL: ast.NotEq,
}
class ConstraintGenerator(m.Expr.Visitor[ast.expr]):
def __init__(self, types: TypesRegistry):
self.types: TypesRegistry = types
self._id: int = 0
self._definitions: list[ast.stmt] = []
self._aliases: dict[str, str] = {}
def get_definitions(self) -> list[ast.stmt]:
return self._definitions
def generate(self, expr: m.Expr) -> ast.expr:
match expr:
case m.VariableExpr():
return expr.accept(self)
case _:
func = Function(
params=ParamSpec(
mixed=[
Function.Parameter(
pos=0,
name="_",
type=self.types.get_type("Any"),
required=True,
)
],
),
returns=self.types.get_type("bool"),
)
alias: str = self.make_alias(None)
definition: ast.stmt = self.make_definition(
alias, Predicate(type=func, body=expr, alias=False)
)
self._definitions.append(definition)
return ast.Name(id=alias)
def make_alias(self, name: Optional[str]) -> str:
suffix: str
if name is None:
suffix = f"p{self._id}"
self._id += 1
else:
suffix = name
alias: str = f"__midas_{suffix}__"
return alias
def make_definition(self, name: str, predicate: Predicate) -> ast.stmt:
body: ast.expr = predicate.body.accept(self)
if predicate.alias:
return ast.Assign(
targets=[
ast.Name(id=name),
],
value=body,
)
return self.make_func(name, [ast.Return(value=body)], predicate.type)
def make_args(self, params: ParamSpec) -> ast.arguments:
return ast.arguments(
posonlyargs=[
ast.arg(
arg=param.name,
annotation=ast.Constant(value=to_annotation(param.type)),
)
for param in params.pos
],
args=[
ast.arg(
arg=param.name,
annotation=ast.Constant(value=to_annotation(param.type)),
)
for param in params.mixed
],
kwonlyargs=[
ast.arg(
arg=param.name,
annotation=ast.Constant(value=to_annotation(param.type)),
)
for param in params.kw
],
defaults=[],
kw_defaults=[],
)
def make_func(
self, name: str, inner_body: list[ast.stmt], type: Type, level: int = 0
) -> ast.stmt:
match type:
case Function(params=params, returns=Function()):
inner_name: str = f"inner{level}"
return ast.FunctionDef(
name=name,
args=self.make_args(params),
body=[
self.make_func(inner_name, inner_body, type.returns, level + 1),
ast.Return(value=ast.Name(id=inner_name)),
],
returns=ast.Constant(value=to_annotation(type.returns)),
decorator_list=[],
)
case Function(params=params):
return ast.FunctionDef(
name=name,
args=self.make_args(params),
body=inner_body,
returns=ast.Constant(value=to_annotation(type.returns)),
decorator_list=[],
)
case _:
raise ValueError(f"Expected function, got {type!r}")
def get_predicate(self, name: str) -> Optional[ast.expr]:
if name not in self._aliases:
predicate: Optional[Predicate] = self.types.lookup_predicate(name)
if predicate is None:
return None
alias: str = self.make_alias(name)
self._aliases[name] = alias
self._definitions.append(self.make_definition(alias, predicate))
return ast.Name(id=self._aliases[name])
def visit_logical_expr(self, expr: m.LogicalExpr) -> ast.expr:
return ast.BoolOp(
op=LOGICAL_OPERATORS[expr.operator.type](),
values=[
expr.left.accept(self),
expr.right.accept(self),
],
)
def visit_binary_expr(self, expr: m.BinaryExpr) -> ast.expr:
op: TokenType = expr.operator.type
if op in BINARY_OPERATORS:
return ast.BinOp(
left=expr.left.accept(self),
op=BINARY_OPERATORS[op](),
right=expr.right.accept(self),
)
if op in COMPARISON_OPERATORS:
return ast.Compare(
left=expr.left.accept(self),
ops=[COMPARISON_OPERATORS[op]()],
comparators=[expr.right.accept(self)],
)
raise ValueError(f"Unexpected binary operator {op}")
def visit_unary_expr(self, expr: m.UnaryExpr) -> ast.expr:
return ast.UnaryOp(
op=UNARY_OPERATORS[expr.operator.type](),
operand=expr.right.accept(self),
)
def visit_call_expr(self, expr: m.CallExpr) -> ast.expr:
return ast.Call(
func=expr.callee.accept(self),
args=[arg.accept(self) for arg in expr.arguments],
keywords=[
ast.keyword(arg=name, value=arg.accept(self))
for name, arg in expr.keywords.items()
],
)
def visit_get_expr(self, expr: m.GetExpr) -> ast.expr:
return ast.Attribute(
value=expr.expr.accept(self),
attr=expr.name.lexeme,
)
def visit_variable_expr(self, expr: m.VariableExpr) -> ast.expr:
name: str = expr.name.lexeme
if (p := self.get_predicate(name)) is not None:
return p
return ast.Name(id=name)
def visit_grouping_expr(self, expr: m.GroupingExpr) -> ast.expr:
return expr.accept(self)
def visit_literal_expr(self, expr: m.LiteralExpr) -> ast.expr:
return ast.Constant(value=expr.value)
def visit_wildcard_expr(self, expr: m.WildcardExpr) -> ast.expr:
return ast.Name(id="_")

549
midas/generator/generator.py
View File

@@ -1,108 +1,64 @@
import ast
import logging
import shutil
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional, assert_never
from typing import Optional
import midas.ast.midas as m
import midas.ast.python as p
from midas.ast.location import Location
from midas.ast.printer import MidasPrinter
from midas.checker.checker import TypeChecker
from midas.checker.registry import TypesRegistry
from midas.checker.types import (
AliasType,
AppliedType,
BaseType,
ColumnGroupBy,
ColumnType,
ComplexType,
ConstraintType,
DataFrameType,
DerivedType,
ExtensionType,
FrameGroupBy,
Function,
GenericType,
OverloadedFunction,
TopType,
TupleType,
Type,
TypeVar,
UnitType,
UnknownType,
)
from midas.generator.collector import Assertion, AssertionCollector
from midas.generator.constraints import ConstraintGenerator
from midas.generator.stubs import StubsGenerator
from midas.utils import TypedAST
@dataclass
class Scope:
pre_assertions: list[ast.stmt] = field(default_factory=list[ast.stmt])
aliases: list[str] = field(default_factory=list[str])
pre_assertions: list[ast.stmt] = field(default_factory=list)
aliases: list[str] = field(default_factory=list)
class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
IS_DATAFRAME_FUNC = "__midas_is_dataframe__"
IS_COLUMN_FUNC = "__midas_is_column__"
def __init__(self, workdir: Path, types: TypesRegistry) -> None:
def __init__(self, workdir: Path) -> None:
self.workdir: Path = workdir.resolve()
self.build_dir: Path = self.workdir / "build" / "midas"
if self.build_dir.exists():
shutil.rmtree(self.build_dir)
self.build_dir.mkdir(parents=True, exist_ok=True)
self.rel_src_path: Path = Path()
self.logger: logging.Logger = logging.getLogger("Generator")
self._typed_ast: TypedAST = TypedAST(
stmts=[],
judgements=[],
evaluated_casts=[],
assertions=AssertionCollector(),
)
self._alias_count: int = 0
self._predicate_count: int = 0
self._scopes: list[Scope] = []
self._aliases: list[tuple[p.Expr, ast.expr]] = []
self._constraint_generator: ConstraintGenerator = ConstraintGenerator(types)
self._constraints: list[tuple[m.Expr, ast.expr]] = []
self.define_is_dataframe: bool = False
self.define_is_column: bool = False
def set_src_path(self, path: Path):
self.rel_src_path = path.resolve().relative_to(self.workdir)
def generate_ast(self, typed_ast: TypedAST) -> ast.AST:
def generate_ast(self, typed_ast: TypedAST, src_path: Path) -> ast.AST:
self.rel_src_path = src_path.relative_to(self.workdir)
self._typed_ast = typed_ast
body: list[ast.stmt] = self._visit_body(typed_ast.stmts, can_be_empty=True)
predicates: list[ast.stmt] = self._constraint_generator.get_definitions()
body = predicates + body
if self.define_is_dataframe:
body = [self._is_dataframe_definition()] + body
if self.define_is_column:
body = [self._is_column_definition()] + body
body: list[ast.stmt] = self._visit_body(typed_ast.stmts)
module = ast.Module(body=body, type_ignores=[])
module = ast.fix_missing_locations(module)
return module
def generate(
self,
typed_ast: TypedAST,
src_path: Path,
out_path: Optional[Path] = None,
type_files: Optional[list[tuple[Path, Optional[str]]]] = None,
self, typed_ast: TypedAST, src_path: Path, out_path: Optional[Path] = None
) -> Path:
self.set_src_path(src_path)
module: ast.AST = self.generate_ast(typed_ast, src_path)
compiled: str = ast.unparse(module)
if out_path is None:
if self.build_dir.exists():
shutil.rmtree(self.build_dir)
self.build_dir.mkdir(parents=True, exist_ok=True)
out_path = (self.build_dir / self.rel_src_path).resolve()
try:
_ = out_path.relative_to(self.build_dir)
@@ -110,72 +66,43 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
raise ValueError(
f"Directory traversal, {self.rel_src_path} points outside of parent directory"
)
out_dir: Path = out_path.parent
out_dir.parent.mkdir(parents=True, exist_ok=True)
if type_files is not None:
for in_path, out_name in type_files:
if out_name is None:
out_name = in_path.stem
self.generate_stubs(in_path, out_dir / f"{out_name}.py")
module: ast.AST = self.generate_ast(typed_ast)
compiled: str = ast.unparse(module)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(compiled)
return out_path
def generate_stubs(self, in_path: Path, out_path: Path):
checker = TypeChecker()
checker.import_midas(in_path)
generator = StubsGenerator(checker.types)
module: ast.Module = generator.generate_stubs()
module = ast.fix_missing_locations(module)
output: str = ast.unparse(module)
out_path.write_text(output)
def convert(self, expr: p.Expr) -> ast.expr:
for expr2, alias in self._aliases:
if expr2 == expr:
return alias
assertions = self._typed_ast.assertions.get_assertions_for(expr)
if len(assertions) != 0:
return self._apply_assertions(expr, assertions)
return expr.accept(self)
def visit_binary_expr(self, expr: p.BinaryExpr) -> ast.expr:
return ast.BinOp(
left=self.convert(expr.left),
left=expr.left.accept(self),
op=expr.operator,
right=self.convert(expr.right),
right=expr.right.accept(self),
)
def visit_compare_expr(self, expr: p.CompareExpr) -> ast.expr:
return ast.Compare(
left=self.convert(expr.left),
left=expr.left.accept(self),
ops=[expr.operator],
comparators=[self.convert(expr.right)],
comparators=[expr.right.accept(self)],
)
def visit_unary_expr(self, expr: p.UnaryExpr) -> ast.expr:
return ast.UnaryOp(
op=expr.operator,
operand=self.convert(expr.right),
operand=expr.right.accept(self),
)
def visit_call_expr(self, expr: p.CallExpr) -> ast.expr:
return ast.Call(
func=self.convert(expr.callee),
args=[self.convert(arg) for arg in expr.arguments],
func=expr.callee.accept(self),
args=[arg.accept(self) for arg in expr.arguments],
keywords=[
ast.keyword(arg=name, value=self.convert(arg))
ast.keyword(arg=name, value=arg.accept(self))
for name, arg in expr.keywords.items()
],
)
def visit_get_expr(self, expr: p.GetExpr) -> ast.expr:
return ast.Attribute(
value=self.convert(expr.object),
value=expr.object.accept(self),
attr=expr.name,
)
@@ -188,58 +115,47 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
def visit_logical_expr(self, expr: p.LogicalExpr) -> ast.expr:
return ast.BoolOp(
op=expr.operator,
values=[self.convert(expr.left), self.convert(expr.right)],
values=[expr.left.accept(self), expr.right.accept(self)],
)
def visit_cast_expr(self, expr: p.CastExpr) -> ast.expr:
expr2: ast.expr = self.convert(expr.expr)
if expr in self._typed_ast.evaluated_casts or expr.unsafe:
return expr2
alias: ast.expr = self._make_alias(expr.expr, expr2)
expr2: ast.expr = expr.expr.accept(self)
alias: ast.expr = self._make_alias(expr2)
type: Type = self._get_expr_type(expr)
asserts: list[ast.stmt] = self._make_cast_asserts(expr.location, alias, type)
for assert_ in asserts:
self._add_assert(assert_)
self._make_cast_asserts(expr.location, alias, type)
return alias
def visit_ternary_expr(self, expr: p.TernaryExpr) -> ast.expr:
return ast.IfExp(
test=self.convert(expr.test),
body=self.convert(expr.if_true),
orelse=self.convert(expr.if_false),
test=expr.test.accept(self),
body=expr.if_true.accept(self),
orelse=expr.if_false.accept(self),
)
def visit_list_expr(self, expr: p.ListExpr) -> ast.expr:
return ast.List(
elts=[self.convert(item) for item in expr.items],
elts=[item.accept(self) for item in expr.items],
)
def visit_dict_expr(self, expr: p.DictExpr) -> ast.expr:
return ast.Dict(
keys=[self.convert(key) if key is not None else None for key in expr.keys],
values=[self.convert(value) for value in expr.values],
keys=[key.accept(self) if key is not None else None for key in expr.keys],
values=[value.accept(self) for value in expr.values],
)
def visit_subscript_expr(self, expr: p.SubscriptExpr) -> ast.expr:
return ast.Subscript(
value=self.convert(expr.object),
slice=self.convert(expr.index),
value=expr.object.accept(self),
slice=expr.index.accept(self),
)
def visit_slice_expr(self, expr: p.SliceExpr) -> ast.expr:
return ast.Slice(
lower=self.convert(expr.lower) if expr.lower is not None else None,
upper=self.convert(expr.upper) if expr.upper is not None else None,
step=self.convert(expr.step) if expr.step is not None else None,
)
def visit_tuple_expr(self, expr: p.TupleExpr) -> ast.expr:
return ast.Tuple(
elts=[self.convert(item) for item in expr.items],
lower=expr.lower.accept(self) if expr.lower is not None else None,
upper=expr.upper.accept(self) if expr.upper is not None else None,
step=expr.step.accept(self) if expr.step is not None else None,
)
def visit_raw_expr(self, expr: p.RawExpr) -> ast.expr:
@@ -247,29 +163,28 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
def visit_expression_stmt(self, stmt: p.ExpressionStmt) -> ast.stmt:
return ast.Expr(
value=self.convert(stmt.expr),
)
def make_args(self, params: p.ParamSpec) -> ast.arguments:
return ast.arguments(
posonlyargs=[ast.arg(arg=param.name) for param in params.pos],
args=[ast.arg(arg=param.name) for param in params.mixed],
kwonlyargs=[ast.arg(arg=param.name) for param in params.kw],
defaults=[
self.convert(param.default)
for param in params.pos + params.mixed
if param.default is not None
],
kw_defaults=[
self.convert(param.default) if param.default is not None else None
for param in params.kw
],
value=stmt.expr.accept(self),
)
def visit_function(self, stmt: p.Function) -> ast.stmt:
return ast.FunctionDef(
name=stmt.name,
args=self.make_args(stmt.params),
args=ast.arguments(
posonlyargs=[ast.arg(arg=arg.name) for arg in stmt.posonlyargs],
vararg=None,
args=[ast.arg(arg=arg.name) for arg in stmt.args],
kwonlyargs=[ast.arg(arg=arg.name) for arg in stmt.kwonlyargs],
kwarg=None,
defaults=[
arg.default.accept(self)
for arg in stmt.posonlyargs + stmt.args
if arg.default is not None
],
kw_defaults=[
arg.default.accept(self) if arg.default is not None else None
for arg in stmt.kwonlyargs
],
),
body=self._visit_body(stmt.body),
decorator_list=[],
)
@@ -280,20 +195,20 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
def visit_assign_stmt(self, stmt: p.AssignStmt) -> ast.stmt:
return ast.Assign(
targets=[self.convert(target) for target in stmt.targets],
value=self.convert(stmt.value),
targets=[target.accept(self) for target in stmt.targets],
value=stmt.value.accept(self),
)
def visit_return_stmt(self, stmt: p.ReturnStmt) -> ast.stmt:
return ast.Return(
value=self.convert(stmt.value) if stmt.value is not None else None,
value=stmt.value.accept(self) if stmt.value is not None else None,
)
def visit_if_stmt(self, stmt: p.IfStmt) -> ast.stmt:
return ast.If(
test=self.convert(stmt.test),
test=stmt.test.accept(self),
body=self._visit_body(stmt.body),
orelse=self._visit_body(stmt.orelse, can_be_empty=True),
orelse=self._visit_body(stmt.orelse),
)
def visit_pass(self, stmt: p.Pass) -> ast.stmt:
@@ -301,8 +216,8 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
def visit_for_stmt(self, stmt: p.ForStmt) -> ast.stmt:
return ast.For(
target=self.convert(stmt.target),
iter=self.convert(stmt.iterator),
target=stmt.target.accept(self),
iter=stmt.iterator.accept(self),
body=self._visit_body(stmt.body),
orelse=[],
)
@@ -310,9 +225,7 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
def visit_raw_stmt(self, stmt: p.RawStmt) -> ast.stmt:
return stmt.stmt
def _visit_body(
self, stmts: list[p.Stmt], can_be_empty: bool = False
) -> list[ast.stmt]:
def _visit_body(self, stmts: list[p.Stmt]) -> list[ast.stmt]:
generated: list[ast.stmt] = []
for stmt in stmts:
scope = Scope()
@@ -330,12 +243,10 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
# Remove redundant pass statements
if len(generated) > 1:
generated = [stmt for stmt in generated if not isinstance(stmt, ast.Pass)]
if len(generated) == 0 and not can_be_empty:
generated = [ast.Pass()]
return generated
def _make_alias(self, node: p.Expr, expr: ast.expr) -> ast.expr:
name: str = f"__midas_a{self._alias_count}__"
def _make_alias(self, expr: ast.expr) -> ast.expr:
name: str = f"__midas_alias_{self._alias_count}__"
alias = ast.Name(id=name)
self._alias_count += 1
self._scopes[-1].aliases.append(name)
@@ -345,182 +256,69 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
value=expr,
)
)
self._aliases.append((node, alias))
return alias
def _build_assert(self, expr: ast.expr, message: str | ast.expr) -> ast.stmt:
def _add_assert(self, expr: ast.expr, message: str | ast.expr):
if isinstance(message, str):
message = ast.Constant(value=message)
return ast.Assert(
test=expr,
msg=message,
self._scopes[-1].pre_assertions.append(
ast.Assert(
test=expr,
msg=message,
)
)
def _add_assert(self, assertion: ast.stmt):
self._scopes[-1].pre_assertions.append(assertion)
def _get_expr_type(self, query: p.Expr) -> Type:
for expr, type in self._typed_ast.judgements:
if expr == query:
return type
raise RuntimeError(f"Cannot get type judgement for {query}")
def _make_cast_asserts(
self, src_location: Location, expr: ast.expr, type: Type
) -> list[ast.stmt]:
def _make_cast_asserts(self, src_location: Location, expr: ast.expr, type: Type):
match type:
case UnknownType() | TopType():
return []
case BaseType(name=name):
return [
self._build_assert(
ast.Call(
func=ast.Name(id="isinstance"),
args=[expr, ast.Name(id=name)],
keywords=[],
),
self._make_cast_assert_message(src_location, expr, type),
)
]
self._add_assert(
ast.Call(
func=ast.Name(id="isinstance"),
args=[expr, ast.Name(id=name)],
keywords=[],
),
self._make_cast_assert_message(src_location, expr, type),
)
case DerivedType(type=base):
return self._make_cast_asserts(src_location, expr, base)
case AliasType(type=base):
self._make_cast_asserts(src_location, expr, base)
case UnitType():
return [
self._build_assert(
ast.Compare(
left=expr,
ops=[ast.Is()],
comparators=[
ast.Constant(value=None),
],
),
self._make_cast_assert_message(src_location, expr, type),
self._add_assert(
ast.Compare(
left=expr,
ops=[ast.Is()],
comparators=[
ast.Constant(value=None),
],
),
]
case AppliedType(body=body):
return self._make_cast_asserts(src_location, expr, body)
case ConstraintType(type=base, constraint=constraint):
asserts: list[ast.stmt] = self._make_cast_asserts(
src_location, expr, base
self._make_cast_assert_message(src_location, expr, type),
)
asserts.append(
self._make_constraint_assert(src_location, expr, constraint)
)
return asserts
case TypeVar(bound=bound):
# TODO: check with type from arguments / use call-site context
if bound is None:
return []
return self._make_cast_asserts(src_location, expr, bound)
case TupleType(items=items):
asserts: list[ast.stmt] = [
self._build_assert(
ast.Call(
func=ast.Name(id="isinstance"),
args=[expr, ast.Name(id="tuple")],
keywords=[],
),
self._make_cast_assert_message(src_location, expr, type),
),
]
assert isinstance(expr, ast.Tuple)
for item, item_type in zip(expr.elts, items):
asserts.extend(
self._make_cast_asserts(src_location, item, item_type)
)
return asserts
case DataFrameType(columns=columns):
self.define_is_dataframe = True
asserts: list[ast.stmt] = [
self._build_assert(
ast.Call(
func=ast.Name(id=self.IS_DATAFRAME_FUNC),
args=[expr],
keywords=[],
),
self._make_cast_assert_message(
src_location, expr, type, ": Not a dataframe"
),
),
]
for column in columns:
asserts.append(
self._build_assert(
ast.Compare(
left=ast.Constant(value=column.name),
ops=[ast.In()],
comparators=[expr],
),
self._make_cast_assert_message(
src_location,
expr,
type,
f": Missing column {column.name}",
),
)
)
asserts.extend(
self._make_cast_asserts(
src_location,
ast.Subscript(
value=expr, slice=ast.Constant(value=column.name)
),
column.type,
)
)
return asserts
case ColumnType():
self.define_is_column = True
asserts: list[ast.stmt] = [
self._build_assert(
ast.Call(
func=ast.Name(id=self.IS_COLUMN_FUNC),
args=[expr],
keywords=[],
),
self._make_cast_assert_message(
src_location, expr, type, ": Not a column"
),
),
]
inner_assert: Optional[ast.stmt] = self._make_column_inner_assert(
src_location, expr, type
)
if inner_assert is not None:
asserts.append(inner_assert)
return asserts
case AppliedType():
self._make_cast_asserts(src_location, expr, type.body)
case (
Function()
TopType()
| Function()
| OverloadedFunction()
| ComplexType()
| ExtensionType()
| GenericType()
| FrameGroupBy()
| ColumnGroupBy()
):
self.logger.warning(f"Can't make assertion for type {type}")
return []
raise NotImplementedError(f"Can't make assertion for type {type}")
# Ensure exhaustiveness
case _:
assert_never(type)
case TypeVar():
raise RuntimeError("Unexpected TypeVar")
def _make_cast_assert_message(
self,
location: Location,
expr: ast.expr,
type: Type,
extra: Optional[str] = None,
self, location: Location, expr: ast.expr, type: Type
) -> ast.expr:
loc_str: str = f"{self.rel_src_path}:L{location.lineno}:{location.col_offset+1}"
# f"file.py:L1:1: CastError: Cannot cast {type(expr).__name__} to Type"
@@ -538,153 +336,6 @@ class Generator(p.Stmt.Visitor[ast.stmt], p.Expr.Visitor[ast.expr]):
),
conversion=-1,
),
ast.Constant(f" to {type}{extra or ''}"),
ast.Constant(f" to {type}"),
]
)
def _make_constraint_assert(
self, src_location: Location, expr: ast.expr, constraint: m.Expr
) -> ast.stmt:
test_func: ast.expr = self._get_constraint(constraint)
return self._build_assert(
ast.Call(
func=test_func,
args=[expr],
keywords=[],
),
self._make_constraint_assert_message(src_location, expr, constraint),
)
def _make_constraint_assert_message(
self, location: Location, expr: ast.expr, constraint: m.Expr
) -> ast.expr:
printer = MidasPrinter()
constraint_str: str = printer.print(constraint)
loc_str: str = f"{self.rel_src_path}:L{location.lineno}:{location.col_offset+1}"
# f"file.py:L1:1: ConstraintError: Value does not fit constraint 'v > 0'"
return ast.Constant(
f"{loc_str}: ConstraintError: Value does not fit constraint '{constraint_str}'"
)
def _get_constraint(self, expr: m.Expr) -> ast.expr:
for expr2, constraint in self._constraints:
if expr2 == expr:
return constraint
constraint: ast.expr = self._constraint_generator.generate(expr)
self._constraints.append((expr, constraint))
return constraint
def _is_dataframe_definition(self) -> ast.stmt:
"""
def IS_DATAFRAME_FUNC(obj) -> bool:
import pandas as pd
return isinstance(obj, pd.DataFrame)
"""
return ast.FunctionDef(
name=self.IS_DATAFRAME_FUNC,
args=ast.arguments(
posonlyargs=[ast.arg(arg="obj")],
args=[],
kwonlyargs=[],
defaults=[],
kw_defaults=[],
),
body=[
ast.Import(names=[ast.alias(name="pandas", asname="pd")]),
ast.Return(
value=ast.Call(
func=ast.Name(id="isinstance"),
args=[
ast.Name(id="obj"),
ast.Attribute(
value=ast.Name(id="pd"),
attr="DataFrame",
),
],
keywords=[],
)
),
],
decorator_list=[],
returns=ast.Name(id="bool"),
)
def _is_column_definition(self) -> ast.stmt:
"""
def IS_COLUMN_FUNC(obj) -> bool:
import pandas as pd
return isinstance(obj, pd.Series)
"""
return ast.FunctionDef(
name=self.IS_COLUMN_FUNC,
args=ast.arguments(
posonlyargs=[ast.arg(arg="obj")],
args=[],
kwonlyargs=[],
defaults=[],
kw_defaults=[],
),
body=[
ast.Import(names=[ast.alias(name="pandas", asname="pd")]),
ast.Return(
value=ast.Call(
func=ast.Name(id="isinstance"),
args=[
ast.Name(id="obj"),
ast.Attribute(
value=ast.Name(id="pd"),
attr="Series",
),
],
keywords=[],
)
),
],
decorator_list=[],
returns=ast.Name(id="bool"),
)
def _make_column_inner_assert(
self, src_location: Location, column: ast.expr, type: ColumnType
) -> Optional[ast.stmt]:
# TODO: improve message, maybe chain contexts
col: ast.expr = ast.Name(id="col")
body: list[ast.stmt] = self._make_cast_asserts(src_location, col, type.type)
if len(body) == 0:
return None
return ast.For(
target=col,
iter=column,
body=body,
orelse=[],
)
def _convert_assertion(self, assertion: Assertion) -> ast.stmt:
inputs: list[ast.expr] = []
for input in assertion.inputs:
converted: ast.expr = self.convert(input)
alias: ast.expr = self._make_alias(input, converted)
inputs.append(alias)
test: ast.expr = assertion.builder(*inputs)
location: Location = assertion.bound_expr.location
loc_str: str = f"{self.rel_src_path}:L{location.lineno}:{location.col_offset+1}"
return self._build_assert(
test, f"{loc_str}: AssertionError: {assertion.message}"
)
def _apply_assertions(self, expr: p.Expr, assertions: list[Assertion]) -> ast.expr:
for assertion in assertions:
assert_stmt: ast.stmt
assert_stmt = self._convert_assertion(assertion)
self._add_assert(assert_stmt)
# Mutating list in frozen dataclass
# Not ideal but easiest way to avoid duplicate assertions
self._typed_ast.assertions.remove(assertion)
return expr.accept(self)

197
midas/generator/stubs.py
View File

@@ -1,30 +1,22 @@
import ast
from typing import Optional, assert_never
from typing import Optional
import midas.ast.midas as m
from midas.checker.registry import Member, TypesRegistry
from midas.checker.types import (
AliasType,
AppliedType,
BaseType,
ColumnGroupBy,
ColumnType,
ComplexType,
ConstraintType,
DataFrameType,
DerivedType,
ExtensionType,
FrameGroupBy,
Function,
GenericType,
OverloadedFunction,
ParamSpec,
TopType,
TupleType,
Type,
TypeVar,
UnitType,
UnknownType,
Variance,
substitute_typevars,
)
@@ -36,7 +28,6 @@ class StubsGenerator:
self.types: TypesRegistry = types
self.stubs: list[ast.stmt] = []
self.typing_imports: set[str] = set()
self.import_pandas: bool = False
self.protocol_idx: int = 0
self.stub_idx: int = 0
self.type_var_idx: int = 0
@@ -45,23 +36,10 @@ class StubsGenerator:
def generate_stubs(self) -> ast.Module:
self.stubs = []
self.typing_imports = set()
self.import_pandas = False
for name, type in self.types._types.items():
# Skip builtin types, not just based on name so the user can override
# TODO: check if added members on builtin type
match type:
case BaseType(name=name_) if name == name_:
continue
case GenericType(
name=name1,
body=BaseType(name=name2),
) if (
name == name1 == name2
):
continue
self.generate_stub(name, type)
imports: list[ast.stmt] = [
imports = [
ast.ImportFrom(
module="__future__",
names=[ast.alias(name="annotations")],
@@ -78,37 +56,11 @@ class StubsGenerator:
level=0,
)
)
if self.import_pandas:
imports.append(
ast.Import(
names=[
ast.alias(
name="pandas",
asname="pd",
)
],
)
)
return ast.Module(body=imports + self.stubs, type_ignores=[])
def generate_stub(self, name: str, type: Type):
base_type: Type = type
# TODO: improve
match type:
case DerivedType(name=name_) | GenericType(name=name_) if name_ == name:
pass
case UnitType() if name == "None":
pass
case TopType() if name == "Any":
pass
case _:
alias = ast.Assign(
targets=[ast.Name(id=name)], value=self.dump_type(type)
)
self.add_stub(alias)
return
members: dict[str, Member] = self.types._members.get(name, {})
if isinstance(base_type, (BaseType, TopType, UnitType)) and len(members) == 0:
return
@@ -130,9 +82,8 @@ class StubsGenerator:
def get_bases(self, type: Type) -> tuple[list[ast.expr], dict[str, Type]]:
match type:
case DerivedType(type=base):
case AliasType(type=base):
return [self.dump_type(base)], {}
case GenericType(params=params, body=body):
self.add_typing_import("Generic")
type_vars: ast.expr
@@ -160,13 +111,6 @@ class StubsGenerator:
],
body_subsitutions | substitutions,
)
case ConstraintType(type=base):
return self.get_bases(base)
case TypeVar(bound=bound) if bound is not None:
return [self.dump_type(bound)], {}
case _:
return [], {}
@@ -195,7 +139,7 @@ class StubsGenerator:
def dump_type(self, type: Type) -> ast.expr:
match type:
case DerivedType(name=name) | GenericType(name=name) if (
case AliasType(name=name) | GenericType(name=name) if (
name in self.substitutions
):
type = substitute_typevars(type, self.substitutions[name])
@@ -204,20 +148,15 @@ class StubsGenerator:
case TopType() | UnknownType():
self.add_typing_import("Any")
return ast.Name(id="Any")
case BaseType(name=name):
return ast.Name(id=name)
case DerivedType(name=name):
case AliasType(name=name):
return ast.Name(id=name)
case UnitType():
return ast.Constant(value=None)
case Function():
name: str = self.define_protocol(type)
return ast.Name(id=name)
case OverloadedFunction(overloads=overloads):
if len(overloads) == 1:
return self.dump_type(overloads[0])
@@ -237,7 +176,6 @@ class StubsGenerator:
case TypeVar():
return ast.Name(id=type.name)
case GenericType(name=name):
params: ast.expr
if len(type.params) == 1:
@@ -250,7 +188,6 @@ class StubsGenerator:
value=ast.Name(id=type.name),
slice=params,
)
case AppliedType():
args: ast.expr
if len(type.args) == 1:
@@ -262,63 +199,6 @@ class StubsGenerator:
slice=args,
)
case ConstraintType():
return self.dump_type(type.type)
case TupleType(items=items):
return ast.Subscript(
value=ast.Name(id="tuple"),
slice=ast.Tuple(
elts=[self.dump_type(item) for item in items],
),
)
case ColumnType(type=inner):
self.import_pandas = True
return ast.Subscript(
value=ast.Attribute(
value=ast.Name(id="pd"),
attr="Series",
),
slice=self.dump_type(inner),
)
case DataFrameType():
self.import_pandas = True
return ast.Attribute(
value=ast.Name(id="pd"),
attr="DataFrame",
)
case FrameGroupBy():
self.import_pandas = True
return ast.Attribute(
value=ast.Attribute(
value=ast.Attribute(
value=ast.Name(id="pd"),
attr="api",
),
attr="typing",
),
attr="DataFrameGroupBy",
)
case ColumnGroupBy():
self.import_pandas = True
return ast.Attribute(
value=ast.Attribute(
value=ast.Attribute(
value=ast.Name(id="pd"),
attr="api",
),
attr="typing",
),
attr="SeriesGroupBy",
)
case _:
assert_never(type)
def dump_method(
self, name: str, method: Type, overloaded: bool = False
) -> list[ast.stmt]:
@@ -329,7 +209,7 @@ class StubsGenerator:
return [
ast.FunctionDef(
name=name,
args=self.dump_params(method.params, with_self=True),
args=self.dump_args(method, with_self=True),
returns=self.dump_type(method.returns),
body=[ast.Expr(value=Empty)],
decorator_list=[ast.Name(id="overload")] if overloaded else [],
@@ -349,33 +229,24 @@ class StubsGenerator:
)
]
def dump_params(self, params: ParamSpec, with_self: bool = False) -> ast.arguments:
def dump_args(self, func: Function, with_self: bool = False) -> ast.arguments:
pos: list[ast.arg] = [
ast.arg(
arg=f"_{param.pos}",
annotation=self.dump_type(param.type),
)
for param in params.pos
ast.arg(arg=f"_{arg.pos}", annotation=self.dump_type(arg.type))
for arg in func.pos_args
]
mixed: list[ast.arg] = [
ast.arg(
arg=param.name,
annotation=self.dump_type(param.type),
)
for param in params.mixed
ast.arg(arg=arg.name, annotation=self.dump_type(arg.type))
for arg in func.args
]
kw: list[ast.arg] = [
ast.arg(
arg=param.name,
annotation=self.dump_type(param.type),
)
for param in params.kw
ast.arg(arg=arg.name, annotation=self.dump_type(arg.type))
for arg in func.kw_args
]
defaults: list[ast.expr] = [
Empty for param in params.pos + params.mixed if not param.required
Empty for arg in func.pos_args + func.args if not arg.required
]
kw_defaults: list[Optional[ast.expr]] = [
None if param.required else Empty for param in params.kw
None if arg.required else Empty for arg in func.kw_args
]
if with_self:
arg = ast.arg(arg="self", annotation=None)
@@ -401,7 +272,7 @@ class StubsGenerator:
body=[
ast.FunctionDef(
name="__call__",
args=self.dump_params(func.params, with_self=True),
args=self.dump_args(func, with_self=True),
returns=self.dump_type(func.returns),
body=[ast.Expr(value=Empty)],
decorator_list=[],
@@ -442,29 +313,6 @@ class StubsGenerator:
def define_type_var(self, var: TypeVar) -> TypeVar:
name: str = self.new_type_var_name()
self.add_typing_import("TypeVar")
kwargs: list[ast.keyword] = []
if var.bound is not None:
kwargs.append(
ast.keyword(
arg="bound",
value=self.dump_type(var.bound),
)
)
if var.variance == Variance.COVARIANT:
kwargs.append(
ast.keyword(
arg="covariant",
value=ast.Constant(value=True),
)
)
elif var.variance == Variance.CONTRAVARIANT:
kwargs.append(
ast.keyword(
arg="contravariant",
value=ast.Constant(value=True),
)
)
self.add_stub(
ast.Assign(
targets=[ast.Name(id=name)],
@@ -473,7 +321,16 @@ class StubsGenerator:
args=[
ast.Constant(value=name),
],
keywords=kwargs,
keywords=(
[]
if var.bound is None
else [
ast.keyword(
arg="bound",
value=self.dump_type(var.bound),
)
]
),
),
)
)

5
midas/lexer/base.py
View File

@@ -16,10 +16,9 @@ class Lexer(ABC):
"""An abstract lexer which provides methods to easily extend it into a concrete one
This implementation is based on the [_Crafting Interpreters_][1] book by Robert Nystrom,
more specifically on my [previous Python implementation][2]
more specifically on my [previous Python implementation](https://git.kb28.ch/HEL/pebble)
[1]: https://craftinginterpreters.com/
[2]: https://git.kb28.ch/HEL/pebble
"""
def __init__(self, source: str, file: Optional[str] = None) -> None:
@@ -169,6 +168,6 @@ class Lexer(ABC):
def scan_token(self) -> None:
"""Scan a token
This function should (at least) consume the current character and produce the appropriate token(s), using :func:`add_token`
This function should (at least) consume the current character and produce the appropriate token(s), using `add_token`
"""
pass

35
midas/lexer/midas.py
View File

@@ -46,8 +46,8 @@ class MidasLexer(Lexer):
self.add_token(TokenType.UNDERSCORE)
case "-" if self.match(">"):
self.add_token(TokenType.ARROW)
case "+":
self.add_token(TokenType.PLUS)
# case "+":
# self.add_token(TokenType.PLUS)
case "-":
self.add_token(TokenType.MINUS)
case "*":
@@ -69,8 +69,6 @@ class MidasLexer(Lexer):
):
self.advance()
self.add_token(TokenType.WHITESPACE)
case '"' | "'":
self.scan_string(char)
case _:
if char.isdigit():
self.scan_number()
@@ -80,23 +78,6 @@ class MidasLexer(Lexer):
self.error("Unexpected character")
return None
def scan_string(self, opening: str):
"""Scan the rest of a string and add it as a token
Args:
opening (str): the opening quote or double quote, to be matched
at the end of the string
"""
while self.peek() != opening and not self.is_at_end():
self.advance()
if self.is_at_end():
self.error("Unterminated string")
self.advance()
value: str = self.source[self.start + 1 : self.idx - 1]
self.add_token(TokenType.STRING, value)
def scan_number(self):
"""Scan the rest of number and add it as a token
@@ -153,18 +134,6 @@ class MidasLexer(Lexer):
self.add_token(TokenType.COMMENT)
def is_identifier_char(self, char: str, *, start: bool) -> bool:
"""Check whether a character is a valid as part of an identifier
Identifiers can contain any alphanumerical character or underscore.
They cannot start with a digit.
Args:
char (str): the character to check
start (bool): whether this is the first character of the identifier
Returns:
bool: `True` if the character is valid, `False` otherwise
"""
if char == "_":
return True
if char.isalpha():

14
midas/lexer/token.py
View File

@@ -25,7 +25,7 @@ class TokenType(Enum):
DOT = auto()
# Operators
PLUS = auto()
# PLUS = auto()
MINUS = auto()
STAR = auto()
SLASH = auto()
@@ -43,11 +43,9 @@ class TokenType(Enum):
TRUE = auto()
FALSE = auto()
NONE = auto()
STRING = auto()
# Keywords
TYPE = auto()
ALIAS = auto()
PREDICATE = auto()
EXTEND = auto()
WHERE = auto()
@@ -64,7 +62,6 @@ class TokenType(Enum):
KEYWORDS: dict[str, TokenType] = {
"type": TokenType.TYPE,
"alias": TokenType.ALIAS,
"predicate": TokenType.PREDICATE,
"extend": TokenType.EXTEND,
"where": TokenType.WHERE,
@@ -104,15 +101,6 @@ class Token:
)
def location_to(self, to: Token) -> Location:
"""Create a new :class:`Location` spanning from this token to another
Args:
to (Token): the end token
Returns:
Location: a new :class:`Location` starting at this token and ending
at `to`, both included
"""
return Location.span(self.get_location(), to.get_location())
@property

8
midas/parser/base.py
View File

@@ -16,9 +16,6 @@ class TokenError:
def get_report(self) -> str:
"""Get a detailed error message
The error message is formatted as "(<position>) Error at <token>: <message>".
For example: "(L2:5) Error at '3': Expected ')' after arguments."
Returns:
str: the complete error message
"""
@@ -35,10 +32,9 @@ class Parser(ABC, Generic[T]):
"""An abstract parser which provides methods to easily extend it into a concrete one
This implementation is based on the [_Crafting Interpreters_][1] book by Robert Nystrom,
more specifically on my [previous Python implementation][2]
more specifically on my [previous Python implementation](https://git.kb28.ch/HEL/pebble)
[1]: https://craftinginterpreters.com/
[2]: https://git.kb28.ch/HEL/pebble
"""
IGNORE: set[TokenType] = {
@@ -177,7 +173,7 @@ class Parser(ABC, Generic[T]):
error_msg (str): the error message if the token doesn't match
Raises:
ParsingError: if the current token doesn't match the given type
SyntaxError: if the current token doesn't match the given type
Returns:
Token: the current token which matched the given type

491
midas/parser/midas.py
View File

@@ -2,15 +2,12 @@ from typing import Optional
from midas.ast.location import Location
from midas.ast.midas import (
AliasStmt,
BinaryExpr,
CallExpr,
ComplexType,
ConstraintType,
Expr,
ExtendStmt,
ExtensionType,
FrameType,
FunctionType,
GenericType,
GetExpr,
@@ -20,7 +17,6 @@ from midas.ast.midas import (
MemberKind,
MemberStmt,
NamedType,
ParamSpec,
PredicateStmt,
Stmt,
Type,
@@ -35,11 +31,10 @@ from midas.parser.base import Parser
from midas.parser.errors import ParsingError
class MidasParser(Parser[list[Stmt]]):
class MidasParser(Parser):
"""A simple parser for midas type definitions"""
SYNC_BOUNDARY: set[TokenType] = {
TokenType.ALIAS,
TokenType.TYPE,
TokenType.EXTEND,
TokenType.PREDICATE,
@@ -74,16 +69,14 @@ class MidasParser(Parser[list[Stmt]]):
def declaration(self) -> Optional[Stmt]:
"""Try and parse a declaration
Any parsing error is caught and `None` is returned
Any parsing error is caught and None is returned
Returns:
Optional[Stmt]: the parsed Midas statement, or `None` if a ParsingError was raised
Optional[Stmt]: the parsed Midas statement, or None if a ParsingError was raised
"""
try:
if self.match(TokenType.TYPE):
return self.type_declaration()
if self.match(TokenType.ALIAS):
return self.alias_declaration()
if self.match(TokenType.EXTEND):
return self.extend_declaration()
if self.match(TokenType.PREDICATE):
@@ -96,14 +89,23 @@ class MidasParser(Parser[list[Stmt]]):
def type_declaration(self) -> TypeStmt:
"""Parse a type declaration
A type declaration creates a named subtype of a type expression.
It can have an optional template expression after its name, wrapped in brackets, to handle type parameters.
A type declaration can either be a simple type alias or a new complex type.
In either case, it can have an optional template expression after its name, wrapped in brackets.
A simple type alias is derived from a base type expression, and can have a optional constraint expression preceded by the `where` keyword.
A full simple type alias is thus written:
```
type Name[Template](TypeExpr) where Condition
```
A type statement consists of:
- the `type` keyword
- a name (identifier)
- (optional) type parameters
- a body, a type expression (see :func:`type_expr`)
A new complex type has a set of properties which are named, have a type and an optional constraint expression (also preceded by the `where` keyword).
A full complex type definition is thus written:
```
type Name[Template] {
prop1: TypeExpr1 where Condition1
prop2: TypeExpr2 where Condition2
...
}
```
Returns:
TypeStmt: the parsed type declaration statement
@@ -154,35 +156,11 @@ class MidasParser(Parser[list[Stmt]]):
self.consume(TokenType.RIGHT_BRACKET, "Missing ']' after type parameters")
return params
def alias_declaration(self) -> AliasStmt:
"""Parse an alias declaration
An alias statement consists of:
- the `alias` keyword
- a name (identifier)
- a body, a type expression (see :func:`type_expr`)
Returns:
AliasStmt: the parsed alias declaration statement
"""
keyword: Token = self.previous()
name: Token = self.consume_identifier("Expected alias name")
self.consume(TokenType.EQUAL, "Expected '=' before alias definition")
type: Type = self.type_expr()
return AliasStmt(
location=keyword.location_to(self.previous()),
name=name,
type=type,
)
def type_expr(self) -> Type:
"""Parse a type expression
A type expression can either be a function type (see :func:`function`)
or a constraint type (see :func:`constraint_type`)
A type is an identifier, optionally followed by a template expression.
It can also optionally be followed by a '?' to indicate a nullable type
Returns:
TypeExpr: the parsed type expression
@@ -202,15 +180,6 @@ class MidasParser(Parser[list[Stmt]]):
return base
def constraint_type(self) -> Type:
"""Parse a constraint type expression
A constraint type consists of a base type (see :func:`base_type`),
optionally followed by the `where` keyword and a constraint
expression (see :func:`constraint`)
Returns:
Type: the parsed constraint type expression
"""
type: Type = self.base_type()
if self.match(TokenType.WHERE):
constraint: Expr = self.constraint()
@@ -222,14 +191,6 @@ class MidasParser(Parser[list[Stmt]]):
return type
def base_type(self) -> Type:
"""Parse a base type expression
A base type is either a parenthesized type expression (see :func:`type_expr`)
or a generic type (see :func:`generic_type`)
Returns:
Type: the parsed base type expression
"""
if self.match(TokenType.LEFT_PAREN):
type: Type = self.type_expr()
self.consume(TokenType.RIGHT_PAREN, "Unclosed parenthesis")
@@ -241,21 +202,8 @@ class MidasParser(Parser[list[Stmt]]):
return self.generic_type()
def generic_type(self) -> Type:
"""Parse a generic type expression
A generic type consists of a named type (see :func:`named_type`),
optionally followed by type arguments in brackets.
The special `Frame` type accepts a frame schema instead of type
arguments (see :func:`frame_type`).
Returns:
Type: the parsed generic type
"""
type: NamedType = self.named_type()
type: Type = self.named_type()
if self.check(TokenType.LEFT_BRACKET):
if type.name.lexeme == "Frame":
return self.frame_type()
args: list[Type] = self.type_args()
return GenericType(
location=Location.span(type.location, self.previous().get_location()),
@@ -265,13 +213,6 @@ class MidasParser(Parser[list[Stmt]]):
return type
def type_args(self) -> list[Type]:
"""Parse a list of type arguments
Type arguments are a comma-separated list of type expression wrapped in brackets.
Returns:
list[Type]: the list of type arguments, if any, or an empty list
"""
args: list[Type] = []
self.consume(TokenType.LEFT_BRACKET, "Missing '[' before generic arguments")
while not self.is_at_end() and not self.check(TokenType.RIGHT_BRACKET):
@@ -281,14 +222,7 @@ class MidasParser(Parser[list[Stmt]]):
self.consume(TokenType.RIGHT_BRACKET, "Missing ']' after generic arguments")
return args
def named_type(self) -> NamedType:
"""Parse a named type expression
A named type is an identifier token
Returns:
NamedType: the parsed named type expression
"""
def named_type(self) -> Type:
name: Token = self.consume_identifier("Expected type name")
return NamedType(
location=name.get_location(),
@@ -296,13 +230,13 @@ class MidasParser(Parser[list[Stmt]]):
)
def complex_type(self) -> ComplexType:
"""Parse a complex type expression
"""Parse a type definition body
A complex type consists of zero or more member statements enclosed in
curly braces
A type definition body is a set of whitespace-separated
property statements enclosed in curly braces
Returns:
ComplexType: the parsed complex type expression
ComplexType: the parsed complex type
"""
left: Token = self.consume(
TokenType.LEFT_BRACE, "Expected '{' to start type body"
@@ -323,71 +257,18 @@ class MidasParser(Parser[list[Stmt]]):
members=members,
)
def frame_type(self) -> FrameType:
"""Parse a frame type expression
A frame type consists of:
- the `Frame` identifier
- an opening bracket `[`
- a list of comma-separated column expression consisting of:
- a name (token)
- a colon `:`
- a type expression (see :func:`type_expr`)
- a closing bracket `]`
Returns:
FrameType: the parsed frame type
"""
keyword: Token = self.previous()
self.consume(TokenType.LEFT_BRACKET, "Expected '[' to start frame schema")
columns: list[FrameType.Column] = []
while not self.check(TokenType.RIGHT_BRACKET) and not self.is_at_end():
name: Token = self.advance()
self.consume(TokenType.COLON, "Expected ':' between column name and type")
type: Type = self.type_expr()
columns.append(
FrameType.Column(
location=name.location_to(self.previous()),
name=name,
type=type,
)
)
if not self.match(TokenType.COMMA):
break
self.consume(TokenType.RIGHT_BRACKET, "Unclosed frame schema")
return FrameType(
location=keyword.location_to(self.previous()),
columns=columns,
)
def constraint(self) -> Expr:
"""Parse a constraint expression
"""Parse a constraint
A constraint is an expression (see :func:`expression`)
A constraint is basically a logical predicate
Returns:
Expr: the parsed constraint expression
"""
return self.expression()
def expression(self) -> Expr:
"""Parse an expression
An expression consists of a logical AND expression (see :func:`and_`)
Returns:
Expr: the parsed expression
"""
return self.and_()
def and_(self) -> Expr:
"""Parse a logical AND expression
An AND consists of one or more equality expressions (see :func:`equality`)
separated by logical AND operators (`&`)
"""Parse a logical AND expression or a simpler expression
Returns:
Expr: the parsed expression
@@ -403,10 +284,7 @@ class MidasParser(Parser[list[Stmt]]):
return expr
def equality(self) -> Expr:
"""Parse an equality expression
An equality consists of one or more comparison expressions (see :func:`comparison`)
separated by equality operators (`==`, `!=`)
"""Parse a logical equality expression or a simpler expression
Returns:
Expr: the parsed expression
@@ -422,59 +300,18 @@ class MidasParser(Parser[list[Stmt]]):
return expr
def comparison(self) -> Expr:
"""Parse a comparison expression
A comparison consists of one or more term expressions (see :func:`term`)
separated by comparison operators (`<`, `<=`, `>`, `>=`)
"""Parse a logical comparison expression or a simpler expression
Returns:
Expr: the parsed expression
"""
expr: Expr = self.term()
expr: Expr = self.unary()
while self.match(
TokenType.LESS,
TokenType.LESS_EQUAL,
TokenType.GREATER,
TokenType.GREATER_EQUAL,
):
operator: Token = self.previous()
right: Expr = self.term()
location: Location = Location.span(expr.location, right.location)
expr = BinaryExpr(
location=location, left=expr, operator=operator, right=right
)
return expr
def term(self) -> Expr:
"""Parse a term expression
A term consists of one or more factor expressions (see :func:`factor`)
separated by weak arithmetic operators (`+`, `-`)
Returns:
Expr: the parsed expression
"""
expr: Expr = self.factor()
while self.match(TokenType.PLUS, TokenType.MINUS):
operator: Token = self.previous()
right: Expr = self.factor()
location: Location = Location.span(expr.location, right.location)
expr = BinaryExpr(
location=location, left=expr, operator=operator, right=right
)
return expr
def factor(self) -> Expr:
"""Parse a factor expression
A factor consists of one or more unary expressions (see :func:`unary`)
separated by strong arithmetic operators (`*`, `/`)
Returns:
Expr: the parsed expression
"""
expr: Expr = self.unary()
while self.match(TokenType.STAR, TokenType.SLASH):
operator: Token = self.previous()
right: Expr = self.unary()
location: Location = Location.span(expr.location, right.location)
@@ -484,109 +321,20 @@ class MidasParser(Parser[list[Stmt]]):
return expr
def unary(self) -> Expr:
"""Parse a unary expression
A unary consists of a call expression (see :func:`call`) optionally
preceded by zero or more unary operators (`+`, `-`)
"""Parse a unary expression or a simpler expression
Returns:
Expr: the parsed expression
"""
if self.match(TokenType.PLUS, TokenType.MINUS):
if self.match(TokenType.MINUS):
operator: Token = self.previous()
right: Expr = self.unary()
location: Location = Location.span(operator.get_location(), right.location)
return UnaryExpr(location=location, operator=operator, right=right)
return self.call()
def call(self) -> Expr:
"""Parse a call expression
A call consists of a reference expression (see :func:`reference`)
optionally followed by zero or more argument groups.
Argument groups are parenthesize, comma-separated list of arguments (see :func:`finish_call`)
Returns:
Expr: the parsed expression
"""
expr: Expr = self.reference()
while self.match(TokenType.LEFT_PAREN):
expr = self.finish_call(expr)
return expr
def finish_call(self, callee: Expr) -> Expr:
"""Parse an argument group, i.e. the arguments of a call
Arguments are either passed positionally or by name (keyword argument).
All positional arguments must come before any keyword argument and
vice-versa. Arguments are separated by commas.
A positional argument simply consists of an expression (see :func:`expression`)
A keyword argument consists of and identifier, followed by the equal `=`
token and an expression (see :func:`expression`).
Args:
callee (Expr): the callee expression
Raises:
ParsingError: if a positional argument is passed after a keyword
argument or if a keyword argument's name is invalid (i.e. not
an identifier)
Returns:
Expr: the parsed call expression
"""
pos_args: list[Expr] = []
kw_args: dict[str, Expr] = {}
keywords: bool = False
while not self.check(TokenType.RIGHT_PAREN):
if self.check_identifier() and self.check_next(TokenType.EQUAL):
keywords = True
keyword: Token = self.advance()
self.advance()
value: Expr = self.expression()
name: str = keyword.lexeme
if name in kw_args:
self.error(
self.peek(),
f"Multiple values passed for '{name}', only the last occurrence will be used",
)
kw_args[name] = value
else:
value = self.expression()
if self.check(TokenType.EQUAL):
error_msg: str
if keywords:
error_msg = "Invalid keyword argument name"
else:
error_msg = (
"Cannot pass positional arguments after a keyword argument"
)
raise self.error(self.peek(), error_msg)
pos_args.append(value)
if not self.match(TokenType.COMMA):
break
r_paren: Token = self.consume(
TokenType.RIGHT_PAREN, "Expected ')' after arguments."
)
return CallExpr(
location=Location.span(callee.location, r_paren.get_location()),
callee=callee,
arguments=pos_args,
keywords=kw_args,
)
return self.reference()
def reference(self) -> Expr:
"""Parse a reference expression
A reference consists of a primary expression (see :func:`primary`)
optionally followed by zero or more attribute accesses.
An attribute access consists of a dot `.` token followed by an identifier
"""Parse an attribute access expression or a simpler expression
Returns:
Expr: the parsed expression
@@ -601,12 +349,7 @@ class MidasParser(Parser[list[Stmt]]):
def primary(self) -> Expr:
"""Parse a primary expression
This includes literals (booleans, numbers, etc.), wildcards, identifiers
and grouped expressions
Raises:
ParsingError: if a primary expressions cannot be parsed from the
following tokens
This includes literals (booleans, numbers, etc.), wildcards, identifiers and grouped expressions
Returns:
Expr: the parsed expression
@@ -622,9 +365,6 @@ class MidasParser(Parser[list[Stmt]]):
if self.match(TokenType.NUMBER):
return LiteralExpr(location=token.get_location(), value=token.value)
if self.match(TokenType.STRING):
return LiteralExpr(location=token.get_location(), value=token.value)
if self.match_identifier():
return VariableExpr(location=token.get_location(), name=token)
@@ -639,41 +379,14 @@ class MidasParser(Parser[list[Stmt]]):
raise self.error(self.peek(), "Expected expression")
def consume_identifier(self, message: str = "Expected identifier") -> Token:
"""Consume the current token if it is a valid identifier or raise an error (see :func:`check_identifier`)
If the current token is not a valid identifier, an error is raised
with the provided message
Args:
message (str, optional): the error message. Defaults to "Expected identifier".
Raises:
ParsingError: if the current token is not a valid identifier
Returns:
Token: the current token which is a valid identifier
"""
if not self.match_identifier():
raise self.error(self.peek(), message)
return self.previous()
def match_identifier(self) -> bool:
"""Consume the next token if it is a valid identifier (see :func:`check_identifier`)
Returns:
bool: whether a token was matched and consumed
"""
return self.match(TokenType.IDENTIFIER, *KEYWORDS.values())
def check_identifier(self) -> bool:
"""Check whether the current token is a valid identifier
A valid identifier is either an identifier token or a keyword token.
This function always returns False if the parser is at the EOF token
Returns:
bool: True if the current token is a valid identifier and not EOF
"""
for tt in [TokenType.IDENTIFIER, *KEYWORDS.values()]:
if self.check(tt):
return True
@@ -682,14 +395,7 @@ class MidasParser(Parser[list[Stmt]]):
def member_stmt(self) -> MemberStmt:
"""Parse a member statement
A member statement is written consists of:
- the `prop` (for a property) or `def` (for a method) keyword
- an name (identifier)
- a colon `:`
- a type expression (see :func:`type_expr`)
Raises:
ParsingError: if the first token is neither `prop` nor `def`
A type member statement is written `prop name: Type` or `def name: Type`
Returns:
MemberStmt: the parsed member statement
@@ -716,13 +422,7 @@ class MidasParser(Parser[list[Stmt]]):
def extend_declaration(self) -> ExtendStmt:
"""Parse an extension definition
An extension statement consists of:
- the `extend` keyword
- a type name (identifier)
- (optional) type parameters (see :func:`type_params`)
- an opening brace `{`
- zero or more member statements (see :func:`member_stmt`)
- a closing brace `}`
An extension is written `extend Type { operations }` or `extend[S <: T, U] Type { operations }`
Returns:
ExtendStmt: the parsed extension statement
@@ -747,103 +447,58 @@ class MidasParser(Parser[list[Stmt]]):
def predicate_declaration(self) -> PredicateStmt:
"""Parse a predicate declaration
A predicate statement consists of:
- the `predicate` keyword
- a name (identifier)
- (optional) zero or more parameter specs (see :func:`function_params`)
- an equal sign `=`
- a body, a constraint expression (see :func:`constraint`)
A predicate is written `predicate Name(subject: Type) = constraint_expression`
Returns:
PredicateStmt: the parsed predicate declaration statement
"""
keyword: Token = self.previous()
name: Token = self.consume_identifier("Expected predicate name")
params: list[ParamSpec] = []
while self.check(TokenType.LEFT_PAREN):
params.append(self.function_params())
self.consume(TokenType.LEFT_PAREN, "Expected '(' before predicate subject")
subject: Token = self.consume_identifier("Expected subject name")
self.consume(TokenType.COLON, "Expected ':' after subject name")
type: Type = self.type_expr()
self.consume(TokenType.RIGHT_PAREN, "Expected ')' after predicate subject")
self.consume(TokenType.EQUAL, "Expected '=' after predicate subject")
body: Expr = self.constraint()
condition: Expr = self.constraint()
return PredicateStmt(
location=keyword.location_to(self.previous()),
name=name,
params=params,
body=body,
subject=subject,
type=type,
condition=condition,
)
def function(self) -> FunctionType:
"""Parse a function type expression
A function consists of:
- the `fn` keyword
- a parameter spec (see :func:`function_params`)
- the arrow keyword `->`
- a result type expression (see :func:`type_expr`)
Returns:
FunctionType: the parsed function type expression
"""
params: ParamSpec = self.function_params()
self.consume(TokenType.ARROW, "Expected '->' before result type")
result: Type = self.type_expr()
return FunctionType(
location=params.l_paren.location_to(self.previous()),
params=params,
returns=result,
)
def function_params(self) -> ParamSpec:
"""Parse a parameter spec
A parameter spec consists of zero or more comma-separated parameters,
wrapped in parentheses.
Like in Python, it can contain positional-only, mixed and keyword-only
parameters (separated by `/` and `*`).
Each parameter has a type (see :func:`type_expr`),
preceded by a name (identifier) and a colon `:` (not required for
positional-only parameters).
Returns:
ParamSpec: the parsed parameter spec
"""
l_paren: Token = self.consume(
TokenType.LEFT_PAREN, "Expected '(' before function parameters"
)
pos: list[FunctionType.Parameter] = []
mixed: list[FunctionType.Parameter] = []
kw: list[FunctionType.Parameter] = []
pos_args: list[FunctionType.Argument] = []
args: list[FunctionType.Argument] = []
kw_args: list[FunctionType.Argument] = []
mixed_first_tokens: list[Token] = []
args_first_tokens: list[Token] = []
section: int = 0
while not self.is_at_end() and not self.check(TokenType.RIGHT_PAREN):
match section:
case 0 if self.match(TokenType.SLASH):
pos = mixed
mixed = []
mixed_first_tokens = []
pos_args = args
args = []
args_first_tokens = []
section = 1
case 0 | 1 if self.match(TokenType.STAR):
section = 2
case _:
# Record first token of mixed parameters for errors if unnamed
# Record first token of mixed argument for errors if unnamed
if section != 2:
mixed_first_tokens.append(self.peek())
args_first_tokens.append(self.peek())
name: Optional[Token] = None
if section == 2:
name = self.consume_identifier(
"Expected keyword parameter name"
)
name = self.consume_identifier("Expected keyword argument name")
self.consume(
TokenType.COLON, "Expected ':' after parameter name"
TokenType.COLON, "Expected ':' after argument name"
)
elif self.check_identifier() and self.check_next(TokenType.COLON):
name = self.advance()
@@ -851,24 +506,34 @@ class MidasParser(Parser[list[Stmt]]):
type: Type = self.type_expr()
optional: bool = self.match(TokenType.QMARK)
param = FunctionType.Parameter(
arg = FunctionType.Argument(
location=None,
name=name,
type=type,
required=not optional,
)
if section == 2:
kw.append(param)
kw_args.append(arg)
else:
mixed.append(param)
args.append(arg)
if not self.match(TokenType.COMMA):
break
for param, token in zip(mixed, mixed_first_tokens):
if param.name is None:
for arg, token in zip(args, args_first_tokens):
if arg.name is None:
# Not raised because we can keep parsing
self.error(token, "Unnamed mixed parameter")
self.error(token, "Unnamed mixed argument")
self.consume(TokenType.RIGHT_PAREN, "Expected ')' after function parameters")
return ParamSpec(l_paren=l_paren, pos=pos, mixed=mixed, kw=kw)
self.consume(TokenType.ARROW, "Expected '->' before result type")
result: Type = self.type_expr()
return FunctionType(
location=l_paren.location_to(self.previous()),
pos_args=pos_args,
args=args,
kw_args=kw_args,
returns=result,
)

124
midas/parser/python.py
View File

@@ -23,7 +23,6 @@ from midas.ast.python import (
LiteralExpr,
LogicalExpr,
MidasType,
ParamSpec,
RawExpr,
RawStmt,
ReturnStmt,
@@ -31,7 +30,6 @@ from midas.ast.python import (
Stmt,
SubscriptExpr,
TernaryExpr,
TupleExpr,
TypeAssign,
UnaryExpr,
VariableExpr,
@@ -50,10 +48,7 @@ class UnsupportedSyntaxError(Exception):
class PythonParser:
"""A parser to convert raw Python `ast` nodes in custom IR nodes"""
CAST_FUNCTION = "cast"
UNSAFE_CAST_FUNCTION = "unsafe_cast"
def parse_module(self, node: ast.Module) -> list[Stmt]:
statements: list[Stmt] = []
@@ -215,10 +210,27 @@ class PythonParser:
match node:
case ast.FunctionDef(
name=name,
args=args,
args=ast.arguments(
posonlyargs=posonlyargs,
args=args,
vararg=sink,
kwonlyargs=kwonlyargs,
kwarg=kw_sink,
defaults=defaults,
kw_defaults=kw_defaults,
),
returns=returns,
body=raw_body,
):
def parse_args(
args_list: list[ast.arg], defaults: list[Optional[Expr]]
) -> list[Function.Argument]:
return [
self._parse_function_argument(arg, default)
for arg, default in zip(args_list, defaults)
]
body: list[Stmt] = []
for stmt in raw_body:
stmts = self.parse_stmt(stmt)
@@ -227,58 +239,54 @@ class PythonParser:
elif stmts is not None:
body.extend(stmts)
parsed_defaults: list[Optional[Expr]] = [
self.parse_expr(default) for default in defaults
]
n_posargs: int = len(posonlyargs)
n_args: int = len(args)
n_all_posargs = n_posargs + n_args
parsed_defaults = [
None,
] * (n_all_posargs - len(defaults)) + parsed_defaults
posargs_defaults: list[Optional[Expr]] = parsed_defaults[:n_posargs]
args_defaults: list[Optional[Expr]] = parsed_defaults[n_posargs:]
kwargs_defaults: list[Optional[Expr]] = [
self.parse_expr(default) if default is not None else None
for default in kw_defaults
]
return Function(
location=loc,
name=name,
params=self._parse_param_spec(args),
posonlyargs=parse_args(posonlyargs, posargs_defaults),
args=parse_args(args, args_defaults),
sink=(
self._parse_function_argument(sink, None)
if sink is not None
else None
),
kwonlyargs=parse_args(kwonlyargs, kwargs_defaults),
kw_sink=(
self._parse_function_argument(kw_sink, None)
if kw_sink is not None
else None
),
returns=self._parse_type(returns) if returns is not None else None,
body=body,
)
case _:
print(f"Unsupported function definition: {ast.unparse(node)}")
def _parse_param_spec(self, args: ast.arguments) -> ParamSpec:
def parse_params(
args_list: list[ast.arg], defaults: list[Optional[Expr]]
) -> list[Function.Parameter]:
return [
self._parse_function_parameter(arg, default)
for arg, default in zip(args_list, defaults)
]
defaults: list[ast.expr] = args.defaults
parsed_defaults: list[Optional[Expr]] = [
self.parse_expr(default) for default in defaults
]
n_pos: int = len(args.posonlyargs)
n_mixed: int = len(args.args)
n_all_pos = n_pos + n_mixed
parsed_defaults = [
None,
] * (n_all_pos - len(defaults)) + parsed_defaults
pos_defaults: list[Optional[Expr]] = parsed_defaults[:n_pos]
mixed_defaults: list[Optional[Expr]] = parsed_defaults[n_pos:]
kw_defaults: list[Optional[Expr]] = [
self.parse_expr(default) if default is not None else None
for default in args.kw_defaults
]
return ParamSpec(
pos=parse_params(args.posonlyargs, pos_defaults),
mixed=parse_params(args.args, mixed_defaults),
kw=parse_params(args.kwonlyargs, kw_defaults),
)
def _parse_function_parameter(
def _parse_function_argument(
self, arg: ast.arg, default: Optional[Expr]
) -> Function.Parameter:
) -> Function.Argument:
loc: Location = Location.from_ast(arg)
name: str = arg.arg
type: Optional[MidasType] = None
if arg.annotation is not None:
type = self._parse_type(arg.annotation)
return Function.Parameter(
return Function.Argument(
location=loc,
name=name,
type=type,
@@ -291,28 +299,26 @@ class PythonParser:
case ast.Subscript(value=ast.Name(id="Frame"), slice=schema):
return self._parse_frame_type(schema)
case ast.Subscript(value=ast.Name(id=name), slice=arg):
args: tuple[MidasType, ...] = (
tuple(self._parse_type(a) for a in arg.elts)
if isinstance(arg, ast.Tuple)
else (self._parse_type(arg),)
)
case ast.Subscript(value=ast.Name(id=name), slice=param):
return BaseType(
location=loc,
base=name,
args=args,
param=self._parse_type(param),
)
case ast.Name(id=name):
return BaseType(
location=loc,
base=name,
args=(),
param=None,
)
case ast.BinOp(left=left_expr, op=ast.Add(), right=right_expr):
left = self._parse_type(left_expr)
match left:
case None:
raise InvalidSyntaxError()
# If chained constraints, separate base type and rebuild constraint
case ConstraintType(type=left_type, constraint=left_constraint):
constraint = ast.BinOp(
@@ -338,7 +344,7 @@ class PythonParser:
return BaseType(
location=loc,
base="None",
args=(),
param=None,
)
case _:
@@ -417,9 +423,6 @@ class PythonParser:
case ast.Call(func=ast.Name(id=self.CAST_FUNCTION)):
return self.parse_cast(node)
case ast.Call(func=ast.Name(id=self.UNSAFE_CAST_FUNCTION)):
return self.parse_cast(node)
case ast.Call():
return self.parse_call(node)
@@ -470,12 +473,6 @@ class PythonParser:
step=self.parse_expr(step) if step is not None else None,
)
case ast.Tuple(elts=items):
return TupleExpr(
location=location,
items=tuple(self.parse_expr(item) for item in items),
)
case _:
print(f"Unsupported expression: {ast.unparse(node)}")
return RawExpr(location=location, expr=node)
@@ -530,19 +527,16 @@ class PythonParser:
return expr
def parse_cast(self, node: ast.Call) -> CastExpr:
assert isinstance(node.func, ast.Name)
func: str = node.func.id
match node:
case ast.Call(args=[type, expr], keywords=[]):
return CastExpr(
location=Location.from_ast(node),
type=self._parse_type(type),
expr=self.parse_expr(expr),
unsafe=func == self.UNSAFE_CAST_FUNCTION,
)
case _:
raise InvalidSyntaxError(
f"Invalid call to {func}, expected type and expression"
f"Invalid call to {self.CAST_FUNCTION}, expected type and expression"
)
def parse_call(self, node: ast.Call) -> CallExpr:

52
midas/typing.py
View File

@@ -1,52 +0,0 @@
from typing import Generic, TypeVar
from typing import cast as typing_cast
cast = typing_cast
"""### Midas documentation
Cast a value to a type.
- **Compile-time**: tells the type checker that the return value has the designated type.
- **Run-time**: generates assertions to ensure the value can be interpreted as the given type.
---
<br>
<br>
<br>
_**Internal Python documentation**_
"""
unsafe_cast = typing_cast
"""### Midas documentation
Cast a value to a type.
- **Compile-time**: tells the type checker that the return value has the designated type.
- **Run-time**: -
This operation is unsound, use at your own risk!
---
<br>
<br>
<br>
_**Internal Python documentation**_
"""
T = TypeVar("T")
class Frame(Generic[T]):
"""A `Frame` is the abstract type implemented by `DataFrame`
A frame contains any number of named columns (see :class:`Column`)
"""
class Column(Generic[T]):
"""A `Column` is the abstract type implemented by `Series`
A column contains a any number of values of the same type
"""

3
midas/utils.py
View File

@@ -3,7 +3,6 @@ from typing import Any, Callable, Optional
import midas.ast.python as p
from midas.checker.types import Type
from midas.generator.collector import AssertionCollector
AllowRepeat = Callable[[object], bool]
@@ -63,5 +62,3 @@ class UniversalJSONDumper:
class TypedAST:
stmts: list[p.Stmt]
judgements: list[tuple[p.Expr, Type]]
evaluated_casts: list[p.CastExpr]
assertions: AssertionCollector

6
pyproject.toml
View File

@@ -8,11 +8,7 @@ authors = [
{ name = "Louis Heredero", email = "louis.heredero@students.hevs.ch" },
]
classifiers = ["Programming Language :: Python :: 3"]
dependencies = [
"black>=26.5.1",
"click>=8.4.1",
"watchdog>=6.0.0",
]
dependencies = ["click>=8.4.1"]
[project.urls]
Homepage = "https://git.kbk28.ch/HEL/midas"

43
tests/__main__.py
View File

@@ -1,43 +0,0 @@
from typing import Type
from midas.cli.ansi import Ansi
from tests.base import Tester
from tests.checker import CheckerTester
from tests.generator import GeneratorTester
from tests.midas import MidasTester
from tests.python import PythonTester
def print_banner(name: str):
horizontal: str = "+" + "-" * (len(name) + 2) + "+"
print(horizontal)
print(f"| {name} |")
print(horizontal)
def run_tests(tester_cls: Type[Tester]) -> bool:
print_banner(tester_cls.__name__)
tester: Tester = tester_cls()
success: bool = tester.run_all_tests()
print()
return success
def main():
testers: list[Type[Tester]] = [
PythonTester,
MidasTester,
CheckerTester,
GeneratorTester,
]
success: bool = all(map(run_tests, testers))
if success:
print(Ansi.FG(Ansi.BRIGHT_GREEN) + "All tests passed!" + Ansi.RESET)
else:
print(Ansi.FG(Ansi.BRIGHT_RED) + "Some tests failed!" + Ansi.RESET)
if __name__ == "__main__":
main()

12
tests/base.py
View File

@@ -7,8 +7,6 @@ from abc import ABC, abstractmethod
from pathlib import Path
from typing import Iterator, Protocol
from midas.cli.ansi import Ansi
class CaseResult(Protocol):
def dumps(self) -> str: ...
@@ -46,11 +44,8 @@ class Tester(ABC):
print(rule)
for i, test in enumerate(tests):
path: Path = test.resolve().relative_to(self.CASES_DIR)
print(f"{Ansi.FG(Ansi.BRIGHT_CYAN)}Case {i+1}/{n}: {path}{Ansi.RESET}")
print(Ansi.DIM, end="")
print(f"Case {i+1}/{n}: {test.resolve().relative_to(self.CASES_DIR)}")
success: bool = self._run_test(test)
print(Ansi.RESET, end="")
if success:
successes += 1
else:
@@ -151,9 +146,8 @@ class Tester(ABC):
if not success:
sys.exit(1)
case None:
success: bool = tester.run_all_tests()
if not success:
sys.exit(1)
print("No subcommand provided. Available subcommands: run, update")
sys.exit(1)
case _:
print(f"Unknown subcommand '{args.subcommand}'")
sys.exit(1)

32
tests/cases/checker/01_simple_types.py.ref.json
View File

@@ -4,35 +4,7 @@
"type": "Warning",
"location": {
"start": [
8,
12
],
"end": [
8,
43
]
},
"message": "ConstraintType not yet supported"
},
{
"type": "Warning",
"location": {
"start": [
10,
10
],
"end": [
10,
18
]
},
"message": "Unknown type 'datetime'"
},
{
"type": "Warning",
"location": {
"start": [
13,
6,
4
],
"end": [
@@ -40,7 +12,7 @@
5
]
},
"message": "Unknown type '_'"
"message": "FrameType not yet supported"
}
],
"judgments": []

1080
tests/cases/checker/03_functions.py.ref.json
View File

File diff suppressed because it is too large Load Diff

36
tests/cases/checker/04_custom_types.py.ref.json
View File

@@ -1,19 +1,6 @@
{
"diagnostics": [],
"judgments": [
{
"location": {
"from": "L4:30",
"to": "L4:36"
},
"expr": {
"_type": "LiteralExpr",
"value": 123.45
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L4:18",
@@ -24,13 +11,12 @@
"type": {
"_type": "BaseType",
"base": "Meter",
"args": []
"param": null
},
"expr": {
"_type": "LiteralExpr",
"value": 123.45
},
"unsafe": false
}
},
"type": {
"name": "Meter",
@@ -39,19 +25,6 @@
}
}
},
{
"location": {
"from": "L5:28",
"to": "L5:31"
},
"expr": {
"_type": "LiteralExpr",
"value": 6.7
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L5:15",
@@ -62,13 +35,12 @@
"type": {
"_type": "BaseType",
"base": "Second",
"args": []
"param": null
},
"expr": {
"_type": "LiteralExpr",
"value": 6.7
},
"unsafe": false
}
},
"type": {
"name": "Second",

72
tests/cases/checker/06_subtyping.py.ref.json
View File

@@ -100,6 +100,41 @@
"name": "float"
}
},
{
"location": {
"from": "L11:5",
"to": "L11:12"
},
"expr": {
"_type": "VariableExpr",
"name": "maximum"
},
"type": {
"pos_args": [],
"args": [
{
"pos": 0,
"name": "a",
"type": {
"name": "float"
},
"required": true
},
{
"pos": 1,
"name": "b",
"type": {
"name": "float"
},
"required": true
}
],
"kw_args": [],
"returns": {
"name": "float"
}
}
},
{
"location": {
"from": "L11:13",
@@ -126,43 +161,6 @@
"name": "float"
}
},
{
"location": {
"from": "L11:5",
"to": "L11:12"
},
"expr": {
"_type": "VariableExpr",
"name": "maximum"
},
"type": {
"params": {
"pos": [],
"mixed": [
{
"pos": 0,
"name": "a",
"type": {
"name": "float"
},
"required": true
},
{
"pos": 1,
"name": "b",
"type": {
"name": "float"
},
"required": true
}
],
"kw": []
},
"returns": {
"name": "float"
}
}
},
{
"location": {
"from": "L11:5",

59
tests/cases/checker/07_variance.midas
View File

@@ -1,59 +0,0 @@
// T is invariant (unused)
type Unused[T] = object
// T is covariant
type Covariant[T] = object
// T is contravariant
type Contravariant[T] = object
// T is invariant
type Invariant[T] = object
extend Covariant[T] {
def foo: fn() -> T
}
extend Contravariant[T] {
def foo: fn(T, /) -> None
}
extend Invariant[T] {
def foo: fn(T, /) -> T
}
// T is covariant
type Coco[T] = object
extend Coco[T] {
def foo: fn() -> Covariant[T]
}
// T is contravariant
type Cocontra[T] = object
extend Cocontra[T] {
def foo: fn() -> Contravariant[T]
}
// T is contravariant
type Contraco[T] = object
extend Contraco[T] {
def foo: fn(Covariant[T], /) -> None
}
// T is covariant
type Contracontra[T] = object
extend Contracontra[T] {
def foo: fn(Contravariant[T], /) -> None
}
type T1[T] = object
type T2[T] = object
extend T1[T] {
def foo: fn() -> T2[T]
}
extend T2[T] {
def foo: fn() -> T1[T]
}

52
tests/cases/checker/07_variance.py
View File

@@ -1,52 +0,0 @@
from _ import (
T1,
T2,
Coco,
Cocontra,
Contraco,
Contracontra,
Contravariant,
Covariant,
Invariant,
Unused,
)
unused: Unused
covariant: Covariant
contravariant: Contravariant
invariant: Invariant
coco: Coco
cocontra: Cocontra
contraco: Contraco
contracontra: Contracontra
t1: T1
t2: T2
# Dummy print to prudce judgements for the expressions
print(
unused,
covariant,
contravariant,
invariant,
coco,
cocontra,
contraco,
contracontra,
t1,
t2,
)
cov1: Covariant[float]
cov2: Covariant[int]
cov1 = cov2 # Ok because int <: float => Covariant[int] <: Covariant[float]
cov2 = cov1 # Invalid
contra1: Contravariant[float]
contra2: Contravariant[int]
contra1 = contra2 # Invalid
contra2 = contra1 # Ok because int <: float => Covariant[float] <: Covariant[int]
inv1: Invariant[float]
inv2: Invariant[int]
inv1 = inv2 # Invalid
inv2 = inv1 # Invalid

514
tests/cases/checker/07_variance.py.ref.json
View File

@@ -1,514 +0,0 @@
{
"diagnostics": [
{
"type": "Error",
"location": {
"start": [
28,
4
],
"end": [
28,
13
]
},
"message": "Too many positional arguments"
},
{
"type": "Error",
"location": {
"start": [
42,
0
],
"end": [
42,
11
]
},
"message": "Cannot assign Covariant[float] to variable 'cov2' of type Covariant[int]"
},
{
"type": "Error",
"location": {
"start": [
46,
0
],
"end": [
46,
17
]
},
"message": "Cannot assign Contravariant[int] to variable 'contra1' of type Contravariant[float]"
},
{
"type": "Error",
"location": {
"start": [
51,
0
],
"end": [
51,
11
]
},
"message": "Cannot assign Invariant[int] to variable 'inv1' of type Invariant[float]"
},
{
"type": "Error",
"location": {
"start": [
52,
0
],
"end": [
52,
11
]
},
"message": "Cannot assign Invariant[float] to variable 'inv2' of type Invariant[int]"
}
],
"judgments": [
{
"location": {
"from": "L27:4",
"to": "L27:10"
},
"expr": {
"_type": "VariableExpr",
"name": "unused"
},
"type": {
"name": "Unused",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L28:4",
"to": "L28:13"
},
"expr": {
"_type": "VariableExpr",
"name": "covariant"
},
"type": {
"name": "Covariant",
"params": [
{
"name": "T",
"bound": null,
"variance": "COVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L29:4",
"to": "L29:17"
},
"expr": {
"_type": "VariableExpr",
"name": "contravariant"
},
"type": {
"name": "Contravariant",
"params": [
{
"name": "T",
"bound": null,
"variance": "CONTRAVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L30:4",
"to": "L30:13"
},
"expr": {
"_type": "VariableExpr",
"name": "invariant"
},
"type": {
"name": "Invariant",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L31:4",
"to": "L31:8"
},
"expr": {
"_type": "VariableExpr",
"name": "coco"
},
"type": {
"name": "Coco",
"params": [
{
"name": "T",
"bound": null,
"variance": "COVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L32:4",
"to": "L32:12"
},
"expr": {
"_type": "VariableExpr",
"name": "cocontra"
},
"type": {
"name": "Cocontra",
"params": [
{
"name": "T",
"bound": null,
"variance": "CONTRAVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L33:4",
"to": "L33:12"
},
"expr": {
"_type": "VariableExpr",
"name": "contraco"
},
"type": {
"name": "Contraco",
"params": [
{
"name": "T",
"bound": null,
"variance": "CONTRAVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L34:4",
"to": "L34:16"
},
"expr": {
"_type": "VariableExpr",
"name": "contracontra"
},
"type": {
"name": "Contracontra",
"params": [
{
"name": "T",
"bound": null,
"variance": "COVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L35:4",
"to": "L35:6"
},
"expr": {
"_type": "VariableExpr",
"name": "t1"
},
"type": {
"name": "T1",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L36:4",
"to": "L36:6"
},
"expr": {
"_type": "VariableExpr",
"name": "t2"
},
"type": {
"name": "T2",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L26:0",
"to": "L26:5"
},
"expr": {
"_type": "VariableExpr",
"name": "print"
},
"type": {
"params": {
"pos": [
{
"pos": 0,
"name": "object",
"type": {},
"required": false
}
],
"mixed": [],
"kw": []
},
"returns": {}
}
},
{
"location": {
"from": "L26:0",
"to": "L37:1"
},
"expr": {
"_type": "CallExpr",
"callee": {
"_type": "VariableExpr",
"name": "print"
},
"arguments": [
{
"_type": "VariableExpr",
"name": "unused"
},
{
"_type": "VariableExpr",
"name": "covariant"
},
{
"_type": "VariableExpr",
"name": "contravariant"
},
{
"_type": "VariableExpr",
"name": "invariant"
},
{
"_type": "VariableExpr",
"name": "coco"
},
{
"_type": "VariableExpr",
"name": "cocontra"
},
{
"_type": "VariableExpr",
"name": "contraco"
},
{
"_type": "VariableExpr",
"name": "contracontra"
},
{
"_type": "VariableExpr",
"name": "t1"
},
{
"_type": "VariableExpr",
"name": "t2"
}
],
"keywords": {}
},
"type": {}
},
{
"location": {
"from": "L41:7",
"to": "L41:11"
},
"expr": {
"_type": "VariableExpr",
"name": "cov2"
},
"type": {
"name": "Covariant",
"args": [
{
"name": "int"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L42:7",
"to": "L42:11"
},
"expr": {
"_type": "VariableExpr",
"name": "cov1"
},
"type": {
"name": "Covariant",
"args": [
{
"name": "float"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L46:10",
"to": "L46:17"
},
"expr": {
"_type": "VariableExpr",
"name": "contra2"
},
"type": {
"name": "Contravariant",
"args": [
{
"name": "int"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L47:10",
"to": "L47:17"
},
"expr": {
"_type": "VariableExpr",
"name": "contra1"
},
"type": {
"name": "Contravariant",
"args": [
{
"name": "float"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L51:7",
"to": "L51:11"
},
"expr": {
"_type": "VariableExpr",
"name": "inv2"
},
"type": {
"name": "Invariant",
"args": [
{
"name": "int"
}
],
"body": {
"name": "object"
}
}
},
{
"location": {
"from": "L52:7",
"to": "L52:11"
},
"expr": {
"_type": "VariableExpr",
"name": "inv1"
},
"type": {
"name": "Invariant",
"args": [
{
"name": "float"
}
],
"body": {
"name": "object"
}
}
}
]
}

14
tests/cases/checker/08_unification.py
View File

@@ -1,14 +0,0 @@
def double(value: float) -> float:
return value * 2
def is_odd(value: int) -> bool:
return bool(value % 2)
floats: list[float] = [0.2, 0.5, 0.1, 1.2]
ints: list[int] = [1, 2, 6, -3]
doubled_floats = map(double, floats)
doubled_ints = map(double, ints)
odd_ints = map(is_odd, ints)

894
tests/cases/checker/08_unification.py.ref.json
View File

@@ -1,894 +0,0 @@
{
"diagnostics": [
{
"type": "Error",
"location": {
"start": [
13,
15
],
"end": [
13,
32
]
},
"message": "Could not unify map[T, U]=(transform: (v: T, /) -> U, iterable: list[T], /) -> list[U] with pos=[(value: float) -> float, list[int]] and kw={}"
}
],
"judgments": [
{
"location": {
"from": "L2:11",
"to": "L2:16"
},
"expr": {
"_type": "VariableExpr",
"name": "value"
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L2:19",
"to": "L2:20"
},
"expr": {
"_type": "LiteralExpr",
"value": 2
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L2:11",
"to": "L2:20"
},
"expr": {
"_type": "BinaryExpr",
"left": {
"_type": "VariableExpr",
"name": "value"
},
"operator": "*",
"right": {
"_type": "LiteralExpr",
"value": 2
}
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L6:16",
"to": "L6:21"
},
"expr": {
"_type": "VariableExpr",
"name": "value"
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L6:24",
"to": "L6:25"
},
"expr": {
"_type": "LiteralExpr",
"value": 2
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L6:16",
"to": "L6:25"
},
"expr": {
"_type": "BinaryExpr",
"left": {
"_type": "VariableExpr",
"name": "value"
},
"operator": "%",
"right": {
"_type": "LiteralExpr",
"value": 2
}
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L6:11",
"to": "L6:15"
},
"expr": {
"_type": "VariableExpr",
"name": "bool"
},
"type": {
"params": {
"pos": [
{
"pos": 0,
"name": "object",
"type": {},
"required": false
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "bool"
}
}
},
{
"location": {
"from": "L6:11",
"to": "L6:26"
},
"expr": {
"_type": "CallExpr",
"callee": {
"_type": "VariableExpr",
"name": "bool"
},
"arguments": [
{
"_type": "BinaryExpr",
"left": {
"_type": "VariableExpr",
"name": "value"
},
"operator": "%",
"right": {
"_type": "LiteralExpr",
"value": 2
}
}
],
"keywords": {}
},
"type": {
"name": "bool"
}
},
{
"location": {
"from": "L9:23",
"to": "L9:26"
},
"expr": {
"_type": "LiteralExpr",
"value": 0.2
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L9:28",
"to": "L9:31"
},
"expr": {
"_type": "LiteralExpr",
"value": 0.5
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L9:33",
"to": "L9:36"
},
"expr": {
"_type": "LiteralExpr",
"value": 0.1
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L9:38",
"to": "L9:41"
},
"expr": {
"_type": "LiteralExpr",
"value": 1.2
},
"type": {
"name": "float"
}
},
{
"location": {
"from": "L9:22",
"to": "L9:42"
},
"expr": {
"_type": "ListExpr",
"items": [
{
"_type": "LiteralExpr",
"value": 0.2
},
{
"_type": "LiteralExpr",
"value": 0.5
},
{
"_type": "LiteralExpr",
"value": 0.1
},
{
"_type": "LiteralExpr",
"value": 1.2
}
]
},
"type": {
"name": "list",
"args": [
{
"name": "float"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L10:19",
"to": "L10:20"
},
"expr": {
"_type": "LiteralExpr",
"value": 1
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L10:22",
"to": "L10:23"
},
"expr": {
"_type": "LiteralExpr",
"value": 2
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L10:25",
"to": "L10:26"
},
"expr": {
"_type": "LiteralExpr",
"value": 6
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L10:29",
"to": "L10:30"
},
"expr": {
"_type": "LiteralExpr",
"value": 3
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L10:28",
"to": "L10:30"
},
"expr": {
"_type": "UnaryExpr",
"operator": "-",
"right": {
"_type": "LiteralExpr",
"value": 3
}
},
"type": {
"name": "int"
}
},
{
"location": {
"from": "L10:18",
"to": "L10:31"
},
"expr": {
"_type": "ListExpr",
"items": [
{
"_type": "LiteralExpr",
"value": 1
},
{
"_type": "LiteralExpr",
"value": 2
},
{
"_type": "LiteralExpr",
"value": 6
},
{
"_type": "UnaryExpr",
"operator": "-",
"right": {
"_type": "LiteralExpr",
"value": 3
}
}
]
},
"type": {
"name": "list",
"args": [
{
"name": "int"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L12:21",
"to": "L12:27"
},
"expr": {
"_type": "VariableExpr",
"name": "double"
},
"type": {
"params": {
"pos": [],
"mixed": [
{
"pos": 0,
"name": "value",
"type": {
"name": "float"
},
"required": true
}
],
"kw": []
},
"returns": {
"name": "float"
}
}
},
{
"location": {
"from": "L12:29",
"to": "L12:35"
},
"expr": {
"_type": "VariableExpr",
"name": "floats"
},
"type": {
"name": "list",
"args": [
{
"name": "float"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L12:17",
"to": "L12:20"
},
"expr": {
"_type": "VariableExpr",
"name": "map"
},
"type": {
"name": "map",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"params": {
"pos": [
{
"pos": 0,
"name": "transform",
"type": {
"params": {
"pos": [
{
"pos": 0,
"name": "v",
"type": {
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
},
"required": true
},
{
"pos": 1,
"name": "iterable",
"type": {
"name": "list",
"args": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "list",
"args": [
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
}
}
}
},
{
"location": {
"from": "L12:17",
"to": "L12:36"
},
"expr": {
"_type": "CallExpr",
"callee": {
"_type": "VariableExpr",
"name": "map"
},
"arguments": [
{
"_type": "VariableExpr",
"name": "double"
},
{
"_type": "VariableExpr",
"name": "floats"
}
],
"keywords": {}
},
"type": {
"name": "list",
"args": [
{
"name": "float"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L13:19",
"to": "L13:25"
},
"expr": {
"_type": "VariableExpr",
"name": "double"
},
"type": {
"params": {
"pos": [],
"mixed": [
{
"pos": 0,
"name": "value",
"type": {
"name": "float"
},
"required": true
}
],
"kw": []
},
"returns": {
"name": "float"
}
}
},
{
"location": {
"from": "L13:27",
"to": "L13:31"
},
"expr": {
"_type": "VariableExpr",
"name": "ints"
},
"type": {
"name": "list",
"args": [
{
"name": "int"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L13:15",
"to": "L13:18"
},
"expr": {
"_type": "VariableExpr",
"name": "map"
},
"type": {
"name": "map",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"params": {
"pos": [
{
"pos": 0,
"name": "transform",
"type": {
"params": {
"pos": [
{
"pos": 0,
"name": "v",
"type": {
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
},
"required": true
},
{
"pos": 1,
"name": "iterable",
"type": {
"name": "list",
"args": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "list",
"args": [
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
}
}
}
},
{
"location": {
"from": "L13:15",
"to": "L13:32"
},
"expr": {
"_type": "CallExpr",
"callee": {
"_type": "VariableExpr",
"name": "map"
},
"arguments": [
{
"_type": "VariableExpr",
"name": "double"
},
{
"_type": "VariableExpr",
"name": "ints"
}
],
"keywords": {}
},
"type": {}
},
{
"location": {
"from": "L14:15",
"to": "L14:21"
},
"expr": {
"_type": "VariableExpr",
"name": "is_odd"
},
"type": {
"params": {
"pos": [],
"mixed": [
{
"pos": 0,
"name": "value",
"type": {
"name": "int"
},
"required": true
}
],
"kw": []
},
"returns": {
"name": "bool"
}
}
},
{
"location": {
"from": "L14:23",
"to": "L14:27"
},
"expr": {
"_type": "VariableExpr",
"name": "ints"
},
"type": {
"name": "list",
"args": [
{
"name": "int"
}
],
"body": {
"name": "list"
}
}
},
{
"location": {
"from": "L14:11",
"to": "L14:14"
},
"expr": {
"_type": "VariableExpr",
"name": "map"
},
"type": {
"name": "map",
"params": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"params": {
"pos": [
{
"pos": 0,
"name": "transform",
"type": {
"params": {
"pos": [
{
"pos": 0,
"name": "v",
"type": {
"name": "T",
"bound": null,
"variance": "INVARIANT"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
},
"required": true
},
{
"pos": 1,
"name": "iterable",
"type": {
"name": "list",
"args": [
{
"name": "T",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
},
"required": true
}
],
"mixed": [],
"kw": []
},
"returns": {
"name": "list",
"args": [
{
"name": "U",
"bound": null,
"variance": "INVARIANT"
}
],
"body": {
"name": "list"
}
}
}
}
},
{
"location": {
"from": "L14:11",
"to": "L14:28"
},
"expr": {
"_type": "CallExpr",
"callee": {
"_type": "VariableExpr",
"name": "map"
},
"arguments": [
{
"_type": "VariableExpr",
"name": "is_odd"
},
{
"_type": "VariableExpr",
"name": "ints"
}
],
"keywords": {}
},
"type": {
"name": "list",
"args": [
{
"name": "bool"
}
],
"body": {
"name": "list"
}
}
}
]
}

117
tests/cases/checker/09_frame_ops.py
View File

@@ -1,117 +0,0 @@
# type: ignore
# ruff: disable [F821]
df1: Frame[a:int, b:float]
df2: Frame[a:int, b:float]
_: Any
# Arithmetic
_ = df1 + df2
_ = df1 - df2
_ = df1 * df2
_ = df1 / df2
_ = df1 // df2
_ = df1 % df2
_ = df1**df2
# Comparisons
_ = df1 < df2
_ = df1 > df2
_ = df1 <= df2
_ = df1 >= df2
_ = df1 != df2
_ = df1 == df2
# Aggregate
_ = df1.kurt()
_ = df1.kurtosis()
_ = df1.max()
_ = df1.mean()
_ = df1.median()
_ = df1.min()
_ = df1.mode()
_ = df1.prod()
_ = df1.product()
_ = df1.std()
_ = df1.sum()
_ = df1.var()
# Groupby
df_gb = df1.groupby(by="a")
_ = df_gb.kurt()
_ = df_gb.max()
_ = df_gb.mean()
_ = df_gb.median()
_ = df_gb.min()
_ = df_gb.prod()
_ = df_gb.std()
_ = df_gb.sum()
_ = df_gb.var()
# Columns
col1 = df1["a"]
col2 = df1["a"]
# Arithmetic
_ = col1 + col2
_ = col1 - col2
_ = col1 * col2
_ = col1 / col2
_ = col1 // col2
_ = col1 % col2
_ = col1**col2
# Comparisons
_ = col1 < col2
_ = col1 > col2
_ = col1 <= col2
_ = col1 >= col2
_ = col1 != col2
_ = col1 == col2
# Aggregate
_ = col1.kurt()
_ = col1.kurtosis()
_ = col1.max()
_ = col1.mean()
_ = col1.median()
_ = col1.min()
_ = col1.mode()
_ = col1.prod()
_ = col1.product()
_ = col1.std()
_ = col1.sum()
_ = col1.var()
# Groupby
col_gb = col1.groupby(level=0)
_ = col_gb.kurt()
_ = col_gb.max()
_ = col_gb.mean()
_ = col_gb.median()
_ = col_gb.min()
_ = col_gb.prod()
_ = col_gb.std()
_ = col_gb.sum()
_ = col_gb.var()
# Attributes
_ = df1.ndim # int
_ = df1.size # int
_ = df1.shape # (int, int)
_ = col1.ndim # int
_ = col1.size # int
_ = col1.shape # (int)
_ = col1.T # Column[int]
# Misc
_ = df1.head()
_ = df1.tail()
_ = col1.head()
_ = col1.tail()

4924
tests/cases/checker/09_frame_ops.py.ref.json
View File

File diff suppressed because it is too large Load Diff

58
tests/cases/generator/01_simple_types.py.ref.txt
View File

@@ -9,12 +9,66 @@ Module(
level=0),
Assign(
targets=[
Name(id='distance')],
Name(id='__midas_alias_0__')],
value=Constant(value=123.45)),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_alias_0__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='01_simple_types.py:L3:19: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_alias_0__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assign(
targets=[
Name(id='distance')],
value=Name(id='__midas_alias_0__')),
Delete(
targets=[
Name(id='__midas_alias_0__')]),
Assign(
targets=[
Name(id='__midas_alias_1__')],
value=Constant(value=6.7)),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_alias_1__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='01_simple_types.py:L4:16: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_alias_1__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assign(
targets=[
Name(id='time')],
value=Constant(value=6.7)),
value=Name(id='__midas_alias_1__')),
Delete(
targets=[
Name(id='__midas_alias_1__')]),
Assign(
targets=[
Name(id='speed')],

14
tests/cases/generator/02_constraints.midas
View File

@@ -1,14 +0,0 @@
// Inline
type T1 = float where _ > 0
// Named
predicate is_positive(v: float) = v > 0
type T2 = float where is_positive(_)
// Curried
predicate in_range(mn: float, mx: float)(v: float) = v >= mn & v < mx
type T3 = float where in_range(100, 200)(_)
// Alias
predicate minor = in_range(0, 18)
type T4 = float where minor(_)

8
tests/cases/generator/02_constraints.py
View File

@@ -1,8 +0,0 @@
from midas import T1, T2, T3, T4, cast
t: float = 12.5
t1: T1 = cast(T1, t)
t2: T2 = cast(T2, t)
t3: T3 = cast(T3, t)
t4: T4 = cast(T4, t)

333
tests/cases/generator/02_constraints.py.ref.txt
View File

@@ -1,333 +0,0 @@
Module(
body=[
FunctionDef(
name='__midas_p0__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='_',
annotation=Constant(value='Any'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=Compare(
left=Name(id='_'),
ops=[
Gt()],
comparators=[
Constant(value=0.0)]))],
decorator_list=[],
returns=Constant(value='bool')),
FunctionDef(
name='__midas_is_positive__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='v',
annotation=Constant(value='float'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=Compare(
left=Name(id='v'),
ops=[
Gt()],
comparators=[
Constant(value=0.0)]))],
decorator_list=[],
returns=Constant(value='bool')),
FunctionDef(
name='__midas_p1__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='_',
annotation=Constant(value='Any'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=Call(
func=Name(id='__midas_is_positive__'),
args=[
Name(id='_')],
keywords=[]))],
decorator_list=[],
returns=Constant(value='bool')),
FunctionDef(
name='__midas_in_range__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='mn',
annotation=Constant(value='float')),
arg(
arg='mx',
annotation=Constant(value='float'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
FunctionDef(
name='inner0',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='v',
annotation=Constant(value='float'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=BoolOp(
op=And(),
values=[
Compare(
left=Name(id='v'),
ops=[
GtE()],
comparators=[
Name(id='mn')]),
Compare(
left=Name(id='v'),
ops=[
Lt()],
comparators=[
Name(id='mx')])]))],
decorator_list=[],
returns=Constant(value='bool')),
Return(
value=Name(id='inner0'))],
decorator_list=[],
returns=Constant(value='Callable[[float], bool]')),
FunctionDef(
name='__midas_p2__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='_',
annotation=Constant(value='Any'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=Call(
func=Call(
func=Name(id='__midas_in_range__'),
args=[
Constant(value=100.0),
Constant(value=200.0)],
keywords=[]),
args=[
Name(id='_')],
keywords=[]))],
decorator_list=[],
returns=Constant(value='bool')),
Assign(
targets=[
Name(id='__midas_minor__')],
value=Call(
func=Name(id='__midas_in_range__'),
args=[
Constant(value=0.0),
Constant(value=18.0)],
keywords=[])),
FunctionDef(
name='__midas_p3__',
args=arguments(
posonlyargs=[],
args=[
arg(
arg='_',
annotation=Constant(value='Any'))],
kwonlyargs=[],
kw_defaults=[],
defaults=[]),
body=[
Return(
value=Call(
func=Name(id='__midas_minor__'),
args=[
Name(id='_')],
keywords=[]))],
decorator_list=[],
returns=Constant(value='bool')),
ImportFrom(
module='midas',
names=[
alias(name='T1'),
alias(name='T2'),
alias(name='T3'),
alias(name='T4'),
alias(name='cast')],
level=0),
Assign(
targets=[
Name(id='t')],
value=Constant(value=12.5)),
Assign(
targets=[
Name(id='__midas_a0__')],
value=Name(id='t')),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_a0__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='02_constraints.py:L5:10: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_a0__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assert(
test=Call(
func=Name(id='__midas_p0__'),
args=[
Name(id='__midas_a0__')],
keywords=[]),
msg=Constant(value="02_constraints.py:L5:10: ConstraintError: Value does not fit constraint '_ > 0.0'")),
Assign(
targets=[
Name(id='t1')],
value=Name(id='__midas_a0__')),
Delete(
targets=[
Name(id='__midas_a0__')]),
Assign(
targets=[
Name(id='__midas_a1__')],
value=Name(id='t')),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_a1__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='02_constraints.py:L6:10: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_a1__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assert(
test=Call(
func=Name(id='__midas_p1__'),
args=[
Name(id='__midas_a1__')],
keywords=[]),
msg=Constant(value="02_constraints.py:L6:10: ConstraintError: Value does not fit constraint 'is_positive(_)'")),
Assign(
targets=[
Name(id='t2')],
value=Name(id='__midas_a1__')),
Delete(
targets=[
Name(id='__midas_a1__')]),
Assign(
targets=[
Name(id='__midas_a2__')],
value=Name(id='t')),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_a2__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='02_constraints.py:L7:10: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_a2__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assert(
test=Call(
func=Name(id='__midas_p2__'),
args=[
Name(id='__midas_a2__')],
keywords=[]),
msg=Constant(value="02_constraints.py:L7:10: ConstraintError: Value does not fit constraint 'in_range(100.0, 200.0)(_)'")),
Assign(
targets=[
Name(id='t3')],
value=Name(id='__midas_a2__')),
Delete(
targets=[
Name(id='__midas_a2__')]),
Assign(
targets=[
Name(id='__midas_a3__')],
value=Name(id='t')),
Assert(
test=Call(
func=Name(id='isinstance'),
args=[
Name(id='__midas_a3__'),
Name(id='float')],
keywords=[]),
msg=JoinedStr(
values=[
Constant(value='02_constraints.py:L8:10: CastError: Cannot cast '),
FormattedValue(
value=Attribute(
value=Call(
func=Name(id='type'),
args=[
Name(id='__midas_a3__')],
keywords=[]),
attr='__name__'),
conversion=-1),
Constant(value=' to float')])),
Assert(
test=Call(
func=Name(id='__midas_p3__'),
args=[
Name(id='__midas_a3__')],
keywords=[]),
msg=Constant(value="02_constraints.py:L8:10: ConstraintError: Value does not fit constraint 'minor(_)'")),
Assign(
targets=[
Name(id='t4')],
value=Name(id='__midas_a3__')),
Delete(
targets=[
Name(id='__midas_a3__')])],
type_ignores=[])

177
tests/cases/midas-parser/01_simple_types.midas.ref.json
View File

@@ -2582,21 +2582,18 @@
"name": "__sub__",
"type": {
"_type": "FunctionType",
"params": {
"_type": "ParamSpec",
"pos": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"pos_args": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"required": true
}
],
"args": [],
"kw_args": [],
"returns": {
"_type": "GenericType",
"type": {
@@ -2676,21 +2673,18 @@
"name": "__sub__",
"type": {
"_type": "FunctionType",
"params": {
"_type": "ParamSpec",
"pos": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "Latitude"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"pos_args": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "Latitude"
},
"required": true
}
],
"args": [],
"kw_args": [],
"returns": {
"_type": "GenericType",
"type": {
@@ -2719,21 +2713,18 @@
"name": "__sub__",
"type": {
"_type": "FunctionType",
"params": {
"_type": "ParamSpec",
"pos": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "Longitude"
},
"required": true
}
],
"mixed": [],
"kw": []
},
"pos_args": [
{
"name": null,
"type": {
"_type": "NamedType",
"name": "Longitude"
},
"required": true
}
],
"args": [],
"kw_args": [],
"returns": {
"_type": "GenericType",
"type": {
@@ -2754,24 +2745,12 @@
{
"_type": "PredicateStmt",
"name": "Positive",
"params": [
{
"_type": "ParamSpec",
"pos": [],
"mixed": [
{
"name": "v",
"type": {
"_type": "NamedType",
"name": "float"
},
"required": true
}
],
"kw": []
}
],
"body": {
"subject": "v",
"type": {
"_type": "NamedType",
"name": "float"
},
"condition": {
"_type": "BinaryExpr",
"left": {
"_type": "VariableExpr",
@@ -2787,24 +2766,12 @@
{
"_type": "PredicateStmt",
"name": "StrictlyPositive",
"params": [
{
"_type": "ParamSpec",
"pos": [],
"mixed": [
{
"name": "v",
"type": {
"_type": "NamedType",
"name": "float"
},
"required": true
}
],
"kw": []
}
],
"body": {
"subject": "v",
"type": {
"_type": "NamedType",
"name": "float"
},
"condition": {
"_type": "BinaryExpr",
"left": {
"_type": "VariableExpr",
@@ -2820,24 +2787,12 @@
{
"_type": "PredicateStmt",
"name": "Equatorial",
"params": [
{
"_type": "ParamSpec",
"pos": [],
"mixed": [
{
"name": "loc",
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"required": true
}
],
"kw": []
}
],
"body": {
"subject": "loc",
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"condition": {
"_type": "GroupingExpr",
"expr": {
"_type": "BinaryExpr",
@@ -2872,24 +2827,12 @@
{
"_type": "PredicateStmt",
"name": "Arctic",
"params": [
{
"_type": "ParamSpec",
"pos": [],
"mixed": [
{
"name": "loc",
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"required": true
}
],
"kw": []
}
],
"body": {
"subject": "loc",
"type": {
"_type": "NamedType",
"name": "GeoLocation"
},
"condition": {
"_type": "GroupingExpr",
"expr": {
"_type": "BinaryExpr",

14
tests/cases/python-parser/01_simple_types.py.ref.json
View File

@@ -16,7 +16,7 @@
"type": {
"_type": "BaseType",
"base": "bool",
"args": []
"param": null
}
},
{
@@ -25,7 +25,7 @@
"type": {
"_type": "BaseType",
"base": "int",
"args": []
"param": null
}
},
{
@@ -36,7 +36,7 @@
"type": {
"_type": "BaseType",
"base": "float",
"args": []
"param": null
},
"constraint": "(_ > 0) + (_ < 250)"
}
@@ -47,7 +47,7 @@
"type": {
"_type": "BaseType",
"base": "str",
"args": []
"param": null
}
},
{
@@ -56,7 +56,7 @@
"type": {
"_type": "BaseType",
"base": "datetime",
"args": []
"param": null
}
},
{
@@ -65,7 +65,7 @@
"type": {
"_type": "BaseType",
"base": "float",
"args": []
"param": null
}
},
{
@@ -79,7 +79,7 @@
"type": {
"_type": "BaseType",
"base": "_",
"args": []
"param": null
}
}
]

50
tests/cases/python-parser/02_custom_types.py.ref.json
View File

@@ -16,7 +16,7 @@
"type": {
"_type": "BaseType",
"base": "GeoLocation",
"args": []
"param": null
}
}
]
@@ -28,13 +28,11 @@
"type": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "BaseType",
"base": "GeoLocation",
"args": []
}
]
"param": {
"_type": "BaseType",
"base": "GeoLocation",
"param": null
}
}
},
{
@@ -67,13 +65,11 @@
"type": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "BaseType",
"base": "GeoLocation",
"args": []
}
]
"param": {
"_type": "BaseType",
"base": "GeoLocation",
"param": null
}
}
},
{
@@ -121,7 +117,7 @@
"type": {
"_type": "BaseType",
"base": "Latitude",
"args": []
"param": null
}
},
{
@@ -150,7 +146,7 @@
"type": {
"_type": "BaseType",
"base": "Latitude",
"args": []
"param": null
}
},
{
@@ -179,13 +175,11 @@
"type": {
"_type": "BaseType",
"base": "Difference",
"args": [
{
"_type": "BaseType",
"base": "Latitude",
"args": []
}
]
"param": {
"_type": "BaseType",
"base": "Latitude",
"param": null
}
}
},
{
@@ -223,7 +217,7 @@
"type": {
"_type": "BaseType",
"base": "int",
"args": []
"param": null
},
"constraint": "_ >= 0"
}
@@ -236,7 +230,7 @@
"type": {
"_type": "BaseType",
"base": "float",
"args": []
"param": null
},
"constraint": "_ >= 0"
}
@@ -258,7 +252,7 @@
"type": {
"_type": "BaseType",
"base": "int",
"args": []
"param": null
},
"constraint": "Positive"
}
@@ -271,7 +265,7 @@
"type": {
"_type": "BaseType",
"base": "float",
"args": []
"param": null
},
"constraint": "Positive"
}

194
tests/cases/python-parser/03_functions.py.ref.json
View File

@@ -7,65 +7,58 @@
{
"_type": "Function",
"name": "func",
"params": {
"_type": "ParamSpec",
"pos": [],
"mixed": [
{
"name": "col1",
"type": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"args": []
},
"constraint": "0 <= _ <= 1"
}
]
},
"default": null
"posonlyargs": [],
"args": [
{
"name": "col1",
"type": {
"_type": "BaseType",
"base": "Column",
"param": {
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"param": null
},
"constraint": "0 <= _ <= 1"
}
},
{
"name": "col2",
"type": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"args": []
},
"constraint": "0 <= _ <= 1"
}
]
},
"default": null
}
],
"kw": []
},
"default": null
},
{
"name": "col2",
"type": {
"_type": "BaseType",
"base": "Column",
"param": {
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"param": null
},
"constraint": "0 <= _ <= 1"
}
},
"default": null
}
],
"sink": null,
"kwonlyargs": [],
"kw_sink": null,
"returns": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"args": []
},
"constraint": "0 <= _ <= 2"
}
]
"param": {
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"param": null
},
"constraint": "0 <= _ <= 2"
}
},
"body": [
{
@@ -74,17 +67,15 @@
"type": {
"_type": "BaseType",
"base": "Column",
"args": [
{
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"args": []
},
"constraint": "0 <= _ <= 2"
}
]
"param": {
"_type": "ConstraintType",
"type": {
"_type": "BaseType",
"base": "float",
"param": null
},
"constraint": "0 <= _ <= 2"
}
}
},
{
@@ -120,42 +111,41 @@
{
"_type": "Function",
"name": "func2",
"params": {
"_type": "ParamSpec",
"pos": [
{
"name": "a",
"type": {
"_type": "BaseType",
"base": "int",
"args": []
},
"default": null
}
],
"mixed": [
{
"name": "b",
"type": {
"_type": "BaseType",
"base": "float",
"args": []
},
"default": null
}
],
"kw": [
{
"name": "c",
"type": {
"_type": "BaseType",
"base": "str",
"args": []
},
"default": null
}
]
},
"posonlyargs": [
{
"name": "a",
"type": {
"_type": "BaseType",
"base": "int",
"param": null
},
"default": null
}
],
"args": [
{
"name": "b",
"type": {
"_type": "BaseType",
"base": "float",
"param": null
},
"default": null
}
],
"sink": null,
"kwonlyargs": [
{
"name": "c",
"type": {
"_type": "BaseType",
"base": "str",
"param": null
},
"default": null
}
],
"kw_sink": null,
"returns": null,
"body": []
}

5
tests/generator.py
View File

@@ -45,9 +45,8 @@ class GeneratorTester(Tester):
typed_ast: TypedAST = checker.type_check(path)
if not any(d.type == DiagnosticType.ERROR for d in checker.diagnostics):
generator = Generator(workdir=path.parent, types=checker.types)
generator.set_src_path(path)
result.compiled_ast = generator.generate_ast(typed_ast)
generator = Generator(workdir=path.parent)
result.compiled_ast = generator.generate_ast(typed_ast, path)
return result

56
tests/serializer/midas.py
View File

@@ -1,15 +1,12 @@
from typing import Optional, Sequence
from midas.ast.midas import (
AliasStmt,
BinaryExpr,
CallExpr,
ComplexType,
ConstraintType,
Expr,
ExtendStmt,
ExtensionType,
FrameType,
FunctionType,
GenericType,
GetExpr,
@@ -18,7 +15,6 @@ from midas.ast.midas import (
LogicalExpr,
MemberStmt,
NamedType,
ParamSpec,
PredicateStmt,
Stmt,
Type,
@@ -62,13 +58,6 @@ class MidasAstJsonSerializer(
"bound": self._serialize_optional(param.bound),
}
def visit_alias_stmt(self, stmt: AliasStmt) -> dict:
return {
"_type": "AliasStmt",
"name": stmt.name.lexeme,
"type": stmt.type.accept(self),
}
def visit_member_stmt(self, stmt: MemberStmt) -> dict:
return {
"_type": "MemberStmt",
@@ -89,8 +78,9 @@ class MidasAstJsonSerializer(
return {
"_type": "PredicateStmt",
"name": stmt.name.lexeme,
"params": [self._serialize_param_spec(spec) for spec in stmt.params],
"body": stmt.body.accept(self),
"subject": stmt.subject.lexeme,
"type": stmt.type.accept(self),
"condition": stmt.condition.accept(self),
}
def visit_logical_expr(self, expr: LogicalExpr) -> dict:
@@ -116,14 +106,6 @@ class MidasAstJsonSerializer(
"right": expr.right.accept(self),
}
def visit_call_expr(self, expr: CallExpr) -> dict:
return {
"_type": "CallExpr",
"callee": expr.callee.accept(self),
"arguments": self._serialize_list(expr.arguments),
"keywords": {name: arg.accept(self) for name, arg in expr.keywords.items()},
}
def visit_get_expr(self, expr: GetExpr) -> dict:
return {
"_type": "GetExpr",
@@ -181,23 +163,17 @@ class MidasAstJsonSerializer(
def visit_function_type(self, type: FunctionType) -> dict:
return {
"_type": "FunctionType",
"params": self._serialize_param_spec(type.params),
"pos_args": [self._serialize_func_arg(arg) for arg in type.pos_args],
"args": [self._serialize_func_arg(arg) for arg in type.args],
"kw_args": [self._serialize_func_arg(arg) for arg in type.kw_args],
"returns": type.returns.accept(self),
}
def _serialize_param_spec(self, spec: ParamSpec) -> dict:
def _serialize_func_arg(self, arg: FunctionType.Argument) -> dict:
return {
"_type": "ParamSpec",
"pos": [self._serialize_func_param(arg) for arg in spec.pos],
"mixed": [self._serialize_func_param(arg) for arg in spec.mixed],
"kw": [self._serialize_func_param(arg) for arg in spec.kw],
}
def _serialize_func_param(self, param: FunctionType.Parameter) -> dict:
return {
"name": param.name.lexeme if param.name is not None else None,
"type": param.type.accept(self),
"required": param.required,
"name": arg.name,
"type": arg.type.accept(self),
"required": arg.required,
}
def visit_extension_type(self, type: ExtensionType) -> dict:
@@ -206,15 +182,3 @@ class MidasAstJsonSerializer(
"base": type.base.accept(self),
"extension": type.extension.accept(self),
}
def visit_frame_type(self, type: FrameType) -> dict:
return {
"_type": "FrameType",
"columns": [self._serialize_column(col) for col in type.columns],
}
def _serialize_column(self, column: FrameType.Column):
return {
"name": column.name.lexeme,
"type": column.type.accept(self),
}

45
tests/serializer/python.py
View File

@@ -22,7 +22,6 @@ from midas.ast.python import (
LiteralExpr,
LogicalExpr,
MidasType,
ParamSpec,
Pass,
RawExpr,
RawStmt,
@@ -31,7 +30,6 @@ from midas.ast.python import (
Stmt,
SubscriptExpr,
TernaryExpr,
TupleExpr,
TypeAssign,
UnaryExpr,
VariableExpr,
@@ -100,7 +98,7 @@ class PythonAstJsonSerializer(
return {
"_type": "BaseType",
"base": node.base,
"args": self._serialize_list(node.args),
"param": self._serialize_optional(node.param),
}
def visit_constraint_type(self, node: ConstraintType) -> dict:
@@ -129,30 +127,32 @@ class PythonAstJsonSerializer(
"expr": stmt.expr.accept(self),
}
def _serialize_argument(self, arg: Function.Argument) -> dict:
return {
"name": arg.name,
"type": self._serialize_optional(arg.type),
"default": self._serialize_optional(arg.default),
}
def visit_function(self, stmt: Function) -> dict:
return {
"_type": "Function",
"name": stmt.name,
"params": self._serialize_param_spec(stmt.params),
"posonlyargs": [self._serialize_argument(arg) for arg in stmt.posonlyargs],
"args": [self._serialize_argument(arg) for arg in stmt.args],
"sink": (
self._serialize_argument(stmt.sink) if stmt.sink is not None else None
),
"kwonlyargs": [self._serialize_argument(arg) for arg in stmt.kwonlyargs],
"kw_sink": (
self._serialize_argument(stmt.kw_sink)
if stmt.kw_sink is not None
else None
),
"returns": self._serialize_optional(stmt.returns),
"body": self._serialize_list(stmt.body),
}
def _serialize_param_spec(self, spec: ParamSpec) -> dict:
return {
"_type": "ParamSpec",
"pos": [self._serialize_func_param(arg) for arg in spec.pos],
"mixed": [self._serialize_func_param(arg) for arg in spec.mixed],
"kw": [self._serialize_func_param(arg) for arg in spec.kw],
}
def _serialize_func_param(self, param: Function.Parameter) -> dict:
return {
"name": param.name,
"type": self._serialize_optional(param.type),
"default": self._serialize_optional(param.default),
}
def visit_type_assign(self, stmt: TypeAssign) -> dict:
return {
"_type": "TypeAssign",
@@ -263,7 +263,6 @@ class PythonAstJsonSerializer(
"_type": "CastExpr",
"type": expr.type.accept(self),
"expr": expr.expr.accept(self),
"unsafe": expr.unsafe,
}
def visit_ternary_expr(self, expr: TernaryExpr) -> dict:
@@ -302,12 +301,6 @@ class PythonAstJsonSerializer(
"step": self._serialize_optional(expr.step),
}
def visit_tuple_expr(self, expr: TupleExpr) -> dict:
return {
"_type": "TupleExpr",
"items": [item.accept(self) for item in expr.items],
}
def visit_raw_expr(self, expr: RawExpr) -> dict:
return {
"_type": "RawExpr",