Skip to content

Commit 092fcdc

Browse files
authored
Merge pull request #263 from thisisthepy/feat/tp-numpy
Feat: numpy array parameters, step 1 (#220)
2 parents 2249259 + 07e7b37 commit 092fcdc

9 files changed

Lines changed: 838 additions & 47 deletions

File tree

‎docs/SPEC.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -688,7 +688,8 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
688688
| Array access in bounds, or a runtime bounds check kept | the check stays (never an unchecked access) |
689689
| Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic |
690690
| Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) |
691-
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays are not compiled yet (#220) | anything else: object path |
691+
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path |
692+
| numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. Other dtypes, including bool, are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. The numpy RuntimeWarnings (overflow, invalid value) and numpy's division are not emulated in this step: a pure function deopts on a non-finite float `+ - *` result or an unproved int overflow, so the interpreter emits them; `/`, `//`, `%` on a value derived from an element, and `+ - *` on one in an impure function, are not compiled | anything else: object path |
692693
| A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic |
693694
| OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic |
694695
| Integer overflow checked and promoted (deopt); an op marked `proven` must re-prove its interval fits i64 | not compiled; diagnostic `verify/proven-overflow` |

‎python-multiplatform-ksp/src/main/python/typedpython/cgen.py‎

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -37,12 +37,11 @@
3737
(tp_runtime.h).
3838
3939
Buffer arrays (issue #149, identity-safe part). A parameter annotated `array.array[float]`,
40-
`array.array[int]` is an `ArrayParam` with `buffer` "array". numpy: not supported yet (#220):
41-
a numpy element's arithmetic produces np.float64 / np.int64 / np.bool (compiled code returns float),
42-
numpy scalar operations emit observable RuntimeWarnings and divide by zero without raising (compiled
43-
double arithmetic does neither), and emulating that needs taint tracking of numpy-derived values,
44-
numpy-scalar boxing and FP-exception-to-warning emulation. An `NDArray[...]` parameter is an ordinary
45-
object parameter. `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter`
40+
`array.array[int]` is an `ArrayParam` with `buffer` "array"; `NDArray[np.float64]` / `NDArray[np.int64]`
41+
(#220) is one with `buffer` "numpy" (exact `numpy.ndarray`, an out-of-range index raises numpy's
42+
IndexError text). A numpy element is an np.float64 / np.int64 in the interpreter: `Box.np` boxes
43+
the matching numpy scalar when its run-time flag is true, and `BinOp.np` deopts a non-finite float
44+
`+ - *` result so the interpreter emits numpy's RuntimeWarning (taint.py). `tp_wrap_f` does not copy it: after the scalar guards, `tp_buf_enter`
4645
checks the exact type (captured in `tp_state.buf_types` by the exec slot), takes one `Py_buffer`
4746
export (C-contiguous, 1-D, 8-byte items, format `d` / `q` / `l`, aligned, writable when the function
4847
stores) and any mismatch deopts before an effect. The export is released by `tp_buf_exit`, part of the
@@ -186,7 +185,9 @@ class CGenError(Exception):
186185
_ARRAY_CT = {Type.F64_ARRAY: "tp_f64_array", Type.I64_ARRAY: "tp_i64_array"}
187186
# Buffer arrays (#149): `ArrayParam.buffer` -> the exact type the guard compares against (the index
188187
# into the module state's `buf_types`), and the format strings the export must carry.
189-
_BUFFER_KINDS = {None: None, "array": 0}
188+
_BUFFER_KINDS = {None: None, "array": 0, "numpy": 1}
189+
# buf_types slots 2..4: the numpy scalar types a Box.np makes (np.float64, np.int64, np.bool_)
190+
_NP_SLOT = {Type.F64: 2, Type.I64: 3, Type.BOOL: 4}
190191
_BUFFER_FMT = {Type.F64_ARRAY: ('"d"', "NULL"), Type.I64_ARRAY: ('"q"', '"l"')}
191192
_BUFFER_SFX = {Type.F64_ARRAY: "f64", Type.I64_ARRAY: "i64"}
192193
_CT_ARRAY_REV = {"tp_f64_array": Type.F64_ARRAY, "tp_i64_array": Type.I64_ARRAY}
@@ -672,7 +673,7 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No
672673
w(f" PyObject *ints[{len(mg.int_consts) + 1}]; /* strong: the int constants boxed once (SPEC N-13) */")
673674
w(f" tp_gslot gc[{len(mg.gnames) + 1}]; /* cached global/builtin reads (SPEC N-13) */")
674675
w(" PyObject *builtins; /* strong: the builtins dict the module source ran with; NULL: none */")
675-
w(" PyObject *buf_types[1]; /* strong: array.array (buffer arrays, #149); NULL: not importable */")
676+
w(" PyObject *buf_types[5]; /* strong: array.array, numpy.ndarray, np.float64, np.int64, np.bool_ (buffer arrays #149, #220); NULL: not importable */")
676677
w(" PyObject *mtype; /* strong: the descriptor type of compiled methods; NULL: none */")
677678
w(f" PyObject *mdescrs[{mg.n_methods + 1}]; /* strong: the installed descriptor of each method */")
678679
w("} tp_state;")
@@ -1048,6 +1049,9 @@ def ev_BinOp(self, e: ir.BinOp) -> str:
10481049
t = self.tmp("double")
10491050
if e.op in _F64_INLINE:
10501051
self.emit(f"{t} = {a} {_F64_INLINE[e.op]} {b};")
1052+
if e.np:
1053+
# #220: numpy would warn (overflow / invalid value); the interpreter redoes the call
1054+
self.emit(f"if (!isfinite({t})) {{ tp_rc = 1; goto tp_exit; }}")
10511055
else:
10521056
self.check(f"{_F64_OP[e.op]}({a}, {b}, &{t})")
10531057
return t
@@ -1155,6 +1159,8 @@ def ev_Truth(self, e: ir.Truth) -> str:
11551159
return t
11561160

11571161
def ev_ToFloat(self, e: ir.ToFloat) -> str:
1162+
if e.explicit and e.operand.type is Type.F64:
1163+
return self.ev(e.operand)
11581164
if e.operand.type is not Type.I64:
11591165
raise CGenError(f"{self.f.name}: ToFloat of {e.operand.type}")
11601166
v = self.ev(e.operand)
@@ -1194,6 +1200,12 @@ def ev_Box(self, e: ir.Box) -> str:
11941200
if ot not in _BOX:
11951201
raise CGenError(f"{self.f.name}: Box of {ot}")
11961202
v = self.ev(e.operand)
1203+
if e.np is not None:
1204+
if e.np.type is not Type.BOOL:
1205+
raise CGenError(f"{self.f.name}: Box np flag of {e.np.type}")
1206+
flag = self.ev(e.np)
1207+
return self._own_new(f"({flag} ? tp_box_np(tp_st->buf_types[{_NP_SLOT[ot]}], "
1208+
f"{_BOX[ot]}({v})) : {_BOX[ot]}({v}))")
11971209
return self._own_new(f"{_BOX[ot]}({v})")
11981210

11991211
def ev_Unbox(self, e: ir.Unbox) -> str:
@@ -2108,7 +2120,8 @@ def wrapper(self) -> str:
21082120
for i, p in buffers:
21092121
fmt1, fmt2 = _BUFFER_FMT[p.type]
21102122
kind = _BUFFER_KINDS[p.buffer]
2111-
L.append(f" tp_s = tp_buf_enter(tp_args[{i}], "
2123+
enter = "tp_buf_enter_np" if p.buffer == "numpy" else "tp_buf_enter"
2124+
L.append(f" tp_s = {enter}(tp_args[{i}], "
21122125
f"((tp_state *)PyModule_GetState(tp_module))->buf_types[{kind}], "
21132126
f"{fmt1}, {fmt2}, {int(p.stored)}, &tp_a{i});")
21142127
L.append(" if (tp_s < 0) goto tp_error;")
@@ -2403,7 +2416,7 @@ def _state_functions(mg: _ModGen) -> str:
24032416
" for (i = 0; i < sizeof(tp_st->ints) / sizeof(tp_st->ints[0]); i++) Py_VISIT(tp_st->ints[i]);",
24042417
" Py_VISIT(tp_st->mtype);",
24052418
" for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) Py_VISIT(tp_st->mdescrs[i]);",
2406-
" for (i = 0; i < 1; i++) Py_VISIT(tp_st->buf_types[i]);",
2419+
" for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) Py_VISIT(tp_st->buf_types[i]);",
24072420
" for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) Py_VISIT(tp_st->names[i]);",
24082421
" for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) Py_VISIT(tp_st->kwnames[i]);",
24092422
" for (i = 0; i < sizeof(tp_st->classes) / sizeof(tp_st->classes[0]); i++) Py_VISIT(tp_st->classes[i]);",
@@ -2425,7 +2438,7 @@ def _state_functions(mg: _ModGen) -> str:
24252438
" for (i = 0; i < sizeof(tp_st->gc) / sizeof(tp_st->gc[0]); i++) tp_gcache_clear(&tp_st->gc[i]);",
24262439
" for (i = 0; i < sizeof(tp_st->mdescrs) / sizeof(tp_st->mdescrs[0]); i++) tp_release(&tp_st->mdescrs[i]);",
24272440
" tp_release(&tp_st->mtype);",
2428-
" for (i = 0; i < 1; i++) tp_release(&tp_st->buf_types[i]);",
2441+
" for (i = 0; i < sizeof(tp_st->buf_types) / sizeof(tp_st->buf_types[0]); i++) tp_release(&tp_st->buf_types[i]);",
24292442
" for (i = 0; i < sizeof(tp_st->names) / sizeof(tp_st->names[0]); i++) tp_release(&tp_st->names[i]);",
24302443
" for (i = 0; i < sizeof(tp_st->kwnames) / sizeof(tp_st->kwnames[0]); i++) tp_release(&tp_st->kwnames[i]);",
24312444
" for (i = 0; i < sizeof(tp_st->cls_ok) / sizeof(tp_st->cls_ok[0]); i++) tp_st->cls_ok[i] = 0;",
@@ -2541,6 +2554,9 @@ def _exec_function(mg: _ModGen) -> str:
25412554
# module source ran (so its own imports are done). Not importable: NULL, and every guard deopts.
25422555
if "array" in kinds:
25432556
L.append(' if (tp_import_attr("array", "array", &tp_st->buf_types[0]) < 0) goto tp_done;')
2557+
if "numpy" in kinds:
2558+
for slot, attr in ((1, "ndarray"), (2, "float64"), (3, "int64"), (4, "bool_")):
2559+
L.append(f' if (tp_import_attr("numpy", "{attr}", &tp_st->buf_types[{slot}]) < 0) goto tp_done;')
25442560
L += [
25452561
" tp_table = PyDict_New();",
25462562
" if (tp_table == NULL || PyDict_SetItemString(tp_dict, \"__typedpython_interpreted__\", tp_table) < 0) goto tp_done;",

0 commit comments

Comments
 (0)