Skip to content

Commit 7db8fff

Browse files
authored
Merge pull request #264 from thisisthepy/feat/tp-numpy2
Feat: numpy array parameters, step 2: warnings and division (#220)
2 parents 092fcdc + db70836 commit 7db8fff

8 files changed

Lines changed: 569 additions & 31 deletions

File tree

‎docs/SPEC.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -689,7 +689,7 @@ All in `PM/commonTest`, so they run wherever the interpreter loads.
689689
| Array parameters only indexed, stored, `len`-ed; no aliasing (entry guard) | not compiled; diagnostic |
690690
| Integer ops proven safe by the verifier's intervals (constants, `len`, `range` variables, guards such as `if x < e` / `while lo < hi`, joined local assignments, `//` and `%` by constants) | emitted unchecked (a shift/mask for `//`/`%` by a power of two of a non-negative value); the sanitizer CI run builds with `TP_CHECK_PROVEN` and turns a wrong proof into a SystemError (#148) |
691691
| `array.array[float]` / `array.array[int]` parameters (exact `array.array`, typecode `d` or `q`/`l`, 8-byte items) are read and written in place: the buffer export is held for the call and released on every exit, and every element access is one relaxed atomic 8-byte load or store, so each access is atomic like the interpreter's `a[i]`; no copy-in and no limit on the number of buffers. Open (#149): while the export is held, another thread's resize of that array raises BufferError. numpy arrays: see the next row | anything else: object path |
692-
| numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. Other dtypes, including bool, are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. The numpy RuntimeWarnings (overflow, invalid value) and numpy's division are not emulated in this step: a pure function deopts on a non-finite float `+ - *` result or an unproved int overflow, so the interpreter emits them; `/`, `//`, `%` on a value derived from an element, and `+ - *` on one in an impure function, are not compiled | anything else: object path |
692+
| numpy `npt.NDArray[np.float64]` / `NDArray[np.int64]` parameters (exact `numpy.ndarray`, 1-D, C-contiguous, native byte order, writable when stored to) take the same buffer path (#220, step 1) when the function accesses the array purely by element (`x[i]`, `len(x)`); a function that uses it as a whole object (`np.sum(x)`, `x.T`, passing `x` on, a slice) compiles as before with the parameter an opaque object. Other dtypes, including bool, are opaque objects. Results follow the interpreter: a value derived from an element is a numpy scalar there, so it is boxed as `np.float64` / `np.int64` / `np.bool` when it escapes (return, `Box`), decided per value at run time by a taint flag; a function declared `-> float` / `int` / `bool` that holds a numpy array returns the interpreter's object. `float(v)` of such a value is a plain float. An out-of-range index raises numpy's `IndexError("index i is out of bounds for axis 0 with size n")`. Arithmetic on a value derived from an element follows numpy's scalar semantics (#220, step 2): float `/`, `//`, `%` never raise ZeroDivisionError (inf / nan, with numpy's result bits and sign rules, `//` and `%` as numpy's `divmod`), int64 `//` and `%` by zero give 0, int64 `/` converts both operands to double. numpy's RuntimeWarning (category and text exactly as numpy's, e.g. `overflow encountered in scalar add`, `divide by zero encountered in scalar divide`, `invalid value encountered in scalar floor_divide`) is emitted once per flagged operation, with the source file and line of the operation. A pure function does not emit it itself: on any floating-point exception (overflow, invalid, divide by zero) or int64 wrap or division by zero it deopts, so the interpreter emits it; an impure function cannot deopt after an effect, so the compiled code emits the warning (`PyErr_WarnExplicitObject`, honouring `warnings` filters and the module's `__warningregistry__`). Float `+ - * / // %` on such values is compiled in both kinds of function, including a store of the result into the array (`x[i] -= d * m`); int64 `/ // %` is compiled in a pure function only. Int64 `+ - *` and `// %` of an element in an impure function (a native INT would have to be a numpy scalar) stay on the object path | anything else: object path |
693693
| A function holding an array parameter runs no user code: closed, and no slot read or `C(args)` whose slow path runs user code, in its whole call tree (#203) | not compiled; diagnostic |
694694
| OBJ references follow one ownership rule; no IR form for free, borrowed alias or NULL read | not compiled; diagnostic |
695695
| Integer overflow checked and promoted (deopt); an op marked `proven` must re-prove its interval fits i64 | not compiled; diagnostic `verify/proven-overflow` |

‎python-multiplatform-ksp/src/main/python/typedpython/cgen.py‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,15 @@ class CGenError(Exception):
202202
ir.BinOpKind.MOD: "tp_mod_i64", ir.BinOpKind.TRUEDIV: "tp_truediv_i64"}
203203
_F64_OP = {ir.BinOpKind.TRUEDIV: "tp_truediv_f64", ir.BinOpKind.FLOORDIV: "tp_floordiv_f64",
204204
ir.BinOpKind.MOD: "tp_mod_f64"}
205+
_NP_I64_OP = {ir.BinOpKind.TRUEDIV: "tp_np_truediv_i64", ir.BinOpKind.FLOORDIV: "tp_np_floordiv_i64",
206+
ir.BinOpKind.MOD: "tp_np_mod_i64"}
207+
_NP_F64_CODE = {k: f"TP_NP_{k.name}" for k in (
208+
ir.BinOpKind.ADD, ir.BinOpKind.SUB, ir.BinOpKind.MUL, ir.BinOpKind.TRUEDIV,
209+
ir.BinOpKind.FLOORDIV, ir.BinOpKind.MOD)}
210+
_NP_F64_CODE[ir.BinOpKind.TRUEDIV] = "TP_NP_DIV"
211+
_NP_NAME = {ir.BinOpKind.ADD: "add", ir.BinOpKind.SUB: "subtract", ir.BinOpKind.MUL: "multiply",
212+
ir.BinOpKind.TRUEDIV: "divide", ir.BinOpKind.FLOORDIV: "floor_divide",
213+
ir.BinOpKind.MOD: "remainder"}
205214
_F64_INLINE = {ir.BinOpKind.ADD: "+", ir.BinOpKind.SUB: "-", ir.BinOpKind.MUL: "*"}
206215
_OBJ_OP = {k: f"TP_BINOP_{k.name}" for k in ir.BinOpKind}
207216
# SPEC N-13: the INT operations (tp_runtime.h "Native ints"); TRUEDIV of ints is an OBJ operation
@@ -482,6 +491,7 @@ def visit(name: str) -> int | None:
482491
class _ModGen:
483492
def __init__(self, module: ir.Module):
484493
self.module = module
494+
self.display_path = ""
485495
self.functions = {f.name: f for f in module.functions}
486496
if len(self.functions) != len(module.functions):
487497
raise CGenError("duplicate compiled function names")
@@ -631,6 +641,7 @@ def generate(module: ir.Module, source_path: Path, display_path: str | None = No
631641
display_path = source_path.name if display_path is None else display_path
632642
source = source_path.read_text(encoding="utf-8")
633643
mg = _ModGen(module)
644+
mg.display_path = display_path # the file name of a compiled numpy RuntimeWarning (#220)
634645
for f in module.functions:
635646
_check_function(f, mg.functions)
636647
for f in module.functions:
@@ -1042,16 +1053,28 @@ def ev_BinOp(self, e: ir.BinOp) -> str:
10421053
self.emit(f"{t} = {a} & {_i64_lit((1 << k) - 1)};")
10431054
elif e.proven:
10441055
self.emit(f"tp_proven({e.op.name.lower()}, {a}, {b}, {t});")
1056+
elif e.np and e.op in _NP_I64_OP:
1057+
# #220: numpy's int64 / // % (division by zero, INT64_MIN // -1): a deopt, the
1058+
# interpreter then warns (the verifier allows it in a pure function only)
1059+
self.check(f"{_NP_I64_OP[e.op]}({a}, {b}, &{t})")
10451060
else:
10461061
self.check(f"{_I64_OP[e.op]}({a}, {b}, &{t})")
10471062
return t
1063+
if lt is Type.F64 and e.np:
1064+
t = self.tmp("double")
1065+
self.emit(f"tp_s = tp_np_f64({_NP_F64_CODE[e.op]}, {a}, {b}, &{t});")
1066+
if self.f.pure:
1067+
# #220: numpy would warn: the interpreter redoes the call and emits it
1068+
self.emit("if (tp_s != 0) { tp_rc = 1; goto tp_exit; }")
1069+
else:
1070+
fi = self.mg.name_index(self.mg.display_path)
1071+
self.emit(f"if (tp_s != 0 && tp_np_warn(tp_module, tp_s, \"{_NP_NAME[e.op]}\", "
1072+
f"tp_st->names[{fi}], {e.line}) != 0) {{ tp_rc = -1; goto tp_exit; }}")
1073+
return t
10481074
if lt is Type.F64:
10491075
t = self.tmp("double")
10501076
if e.op in _F64_INLINE:
10511077
self.emit(f"{t} = {a} {_F64_INLINE[e.op]} {b};")
1052-
if e.np:
1053-
# #220: numpy would warn (overflow / invalid value); the interpreter redoes the call
1054-
self.emit(f"if (!isfinite({t})) {{ tp_rc = 1; goto tp_exit; }}")
10551078
else:
10561079
self.check(f"{_F64_OP[e.op]}({a}, {b}, &{t})")
10571080
return t

‎python-multiplatform-ksp/src/main/python/typedpython/frontend.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1825,6 +1825,13 @@ def to_int(self, e: ir.Expr) -> ir.Expr:
18251825

18261826
def arith(self, op: ir.BinOpKind, left: ir.Expr, right: ir.Expr, node: ast.AST,
18271827
inplace: bool = False) -> ir.Expr:
1828+
e = self._arith(op, left, right, node, inplace)
1829+
if isinstance(e, ir.BinOp) and e.type in ir.SCALARS and self.numpy:
1830+
e = dataclasses.replace(e, line=node.lineno) # the warning site (#220)
1831+
return e
1832+
1833+
def _arith(self, op: ir.BinOpKind, left: ir.Expr, right: ir.Expr, node: ast.AST,
1834+
inplace: bool = False) -> ir.Expr:
18281835
lt, rt = left.type, right.type
18291836
if Type.BOOL in (lt, rt) or Type.NONE in (lt, rt):
18301837
raise Skip(node.lineno, f"arithmetic on {TYPE_NAME[Type.BOOL if Type.BOOL in (lt, rt) else Type.NONE]} "

‎python-multiplatform-ksp/src/main/python/typedpython/ir.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,13 @@ class BinOp(Expr):
160160
# F64 ADD/SUB/MUL only (#220): an operand may be a numpy scalar in the interpreter, which warns
161161
# (RuntimeWarning: overflow / invalid value) where a double does not. A non-finite result →
162162
# deopt, so the interpreter redoes the call and emits the warning; legal only in a pure function.
163+
# Step 2 (#220): `np` also covers F64 TRUEDIV / FLOORDIV / MOD (numpy's results: inf / nan, no
164+
# ZeroDivisionError) and unproven I64 TRUEDIV / FLOORDIV / MOD (numpy's int64: division by zero
165+
# gives 0 / inf / nan; both are a deopt, and legal only in a pure function). Any floating-point
166+
# exception (overflow, invalid, divide by zero) is a deopt in a pure function; in an impure F64
167+
# operation the code emits numpy's RuntimeWarning itself, at `line` of the cgen display path.
163168
np: bool = False
169+
line: int = 0 # source line of the operation (the warning site of an impure `np` operation)
164170

165171

166172
class UnaryOpKind(Enum):

‎python-multiplatform-ksp/src/main/python/typedpython/runtime/tp_runtime.h‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
#include <Python.h>
1515
#include <stdint.h>
1616
#include <math.h>
17+
#include <fenv.h>
1718
#include <errno.h>
1819
#include <string.h>
1920
#include <stdlib.h>
@@ -398,6 +399,175 @@ static inline int tp_mod_f64(double a, double b, double *out)
398399
return 0;
399400
}
400401

402+
/* ------------------------------------------------------------------------------------------
403+
* numpy scalar arithmetic (#220, step 2). An operation on a value derived from a numpy element is
404+
* numpy's, not CPython's: no ZeroDivisionError, inf / nan results, int64 division by zero, and a
405+
* RuntimeWarning per floating-point exception. tp_np_f64 returns the exceptions that numpy reports
406+
* (TP_NP_* bits). A pure function deopts on any bit (the interpreter then warns); an impure one calls
407+
* tp_np_warn. The algorithms are numpy's (npymath: npy_divmod / npy_floor_divide / npy_remainder).
408+
* ------------------------------------------------------------------------------------------ */
409+
410+
#define TP_NP_DIVZERO 1
411+
#define TP_NP_OVERFLOW 2
412+
#define TP_NP_INVALID 4
413+
enum { TP_NP_ADD, TP_NP_SUB, TP_NP_MUL, TP_NP_DIV, TP_NP_FLOORDIV, TP_NP_MOD };
414+
415+
#if defined(__GNUC__) || defined(__clang__)
416+
#define TP_NOINLINE __attribute__((noinline, unused))
417+
#else
418+
#define TP_NOINLINE
419+
#endif
420+
421+
static inline double tp_np_divmod(double a, double b, double *modulus)
422+
{
423+
double div, mod, floordiv;
424+
mod = fmod(a, b);
425+
if (!b) {
426+
*modulus = mod;
427+
return a / b;
428+
}
429+
div = (a - mod) / b;
430+
if (mod) {
431+
if ((b < 0) != (mod < 0)) {
432+
mod += b;
433+
div -= 1.0;
434+
}
435+
}
436+
else {
437+
mod = copysign(0.0, b);
438+
}
439+
if (div) {
440+
floordiv = floor(div);
441+
if (div - floordiv > 0.5) {
442+
floordiv += 1.0;
443+
}
444+
}
445+
else {
446+
floordiv = copysign(0.0, a / b);
447+
}
448+
*modulus = mod;
449+
return floordiv;
450+
}
451+
452+
/* The volatile operands and result keep the compiler from moving, folding or sharing the operation
453+
* across the flag clear and test (the function is also noinline). */
454+
static TP_NOINLINE int tp_np_f64(int op, double a, double b, double *out)
455+
{
456+
volatile double va = a, vb = b, vr = 0.0, vd = 0.0;
457+
double mod;
458+
int flags, force = 0, clear = 0;
459+
feclearexcept(FE_ALL_EXCEPT);
460+
switch (op) {
461+
case TP_NP_ADD: vr = va + vb; break;
462+
case TP_NP_SUB: vr = va - vb; break;
463+
case TP_NP_MUL: vr = va * vb; break;
464+
case TP_NP_DIV: vr = va / vb; break;
465+
case TP_NP_FLOORDIV:
466+
if (!vb) {
467+
feclearexcept(FE_ALL_EXCEPT);
468+
vr = va / vb;
469+
if (!va) {
470+
force = TP_NP_INVALID;
471+
}
472+
}
473+
else {
474+
vr = tp_np_divmod(va, vb, &mod);
475+
}
476+
break;
477+
default:
478+
if (!vb) {
479+
vr = fmod(va, vb);
480+
clear = 1;
481+
}
482+
else {
483+
vd = tp_np_divmod(va, vb, &mod); /* numpy computes the quotient too: its flags count */
484+
(void)vd;
485+
vr = mod;
486+
}
487+
break;
488+
}
489+
flags = 0;
490+
if (!clear) {
491+
int f = fetestexcept(FE_DIVBYZERO | FE_OVERFLOW | FE_INVALID);
492+
flags = ((f & FE_DIVBYZERO) ? TP_NP_DIVZERO : 0) | ((f & FE_OVERFLOW) ? TP_NP_OVERFLOW : 0)
493+
| ((f & FE_INVALID) ? TP_NP_INVALID : 0);
494+
}
495+
*out = vr;
496+
return flags | force;
497+
}
498+
499+
/* numpy's RuntimeWarning for each bit of `flags`, in numpy's order, attributed to `file`:`line` (the
500+
* operation's own source position, as the interpreter's frame would give). 0, or -1 with the
501+
* exception set (a warning turned into an error). `op` is numpy's scalar name ("add", "divide"...). */
502+
static TP_NOINLINE int tp_np_warn(PyObject *tp_module, int flags, const char *op, PyObject *file, int line)
503+
{
504+
PyObject *dict = PyModule_GetDict(tp_module); /* borrowed */
505+
PyObject *reg = NULL, *modname = NULL;
506+
static const struct { int bit; const char *text; } kinds[3] = {
507+
{TP_NP_DIVZERO, "divide by zero"}, {TP_NP_OVERFLOW, "overflow"}, {TP_NP_INVALID, "invalid value"}};
508+
int k, rc = 0;
509+
if (PyDict_GetItemStringRef(dict, "__warningregistry__", &reg) < 0) {
510+
return -1;
511+
}
512+
if (reg == NULL) {
513+
PyObject *fresh = PyDict_New(), *key = PyUnicode_FromString("__warningregistry__"), *got = NULL;
514+
int rr = -1;
515+
if (fresh != NULL && key != NULL) {
516+
rr = PyDict_SetDefaultRef(dict, key, fresh, &got);
517+
}
518+
Py_XDECREF(fresh);
519+
Py_XDECREF(key);
520+
if (rr < 0) {
521+
return -1;
522+
}
523+
reg = got;
524+
}
525+
if (PyDict_GetItemStringRef(dict, "__name__", &modname) < 0) {
526+
Py_DECREF(reg);
527+
return -1;
528+
}
529+
for (k = 0; k < 3 && rc == 0; k++) {
530+
if (flags & kinds[k].bit) {
531+
PyObject *msg = PyUnicode_FromFormat("%s encountered in scalar %s", kinds[k].text, op);
532+
if (msg == NULL) {
533+
rc = -1;
534+
break;
535+
}
536+
rc = PyErr_WarnExplicitObject(PyExc_RuntimeWarning, msg, file, line, modname, reg);
537+
Py_DECREF(msg);
538+
}
539+
}
540+
Py_XDECREF(modname);
541+
Py_DECREF(reg);
542+
return rc;
543+
}
544+
545+
/* int64 on the numpy path (pure functions): 0 ok, 1 deopt (the interpreter warns). */
546+
static inline int tp_np_truediv_i64(int64_t a, int64_t b, double *out)
547+
{
548+
if (b == 0) {
549+
return 1;
550+
}
551+
*out = (double)a / (double)b; /* numpy converts both to double; CPython rounds exactly */
552+
return 0;
553+
}
554+
555+
static inline int tp_np_floordiv_i64(int64_t a, int64_t b, int64_t *out)
556+
{
557+
if (b == 0) {
558+
return 1;
559+
}
560+
return tp_floordiv_i64(a, b, out); /* INT64_MIN // -1 is 1 (numpy's overflow warning) */
561+
}
562+
563+
static inline int tp_np_mod_i64(int64_t a, int64_t b, int64_t *out)
564+
{
565+
if (b == 0) {
566+
return 1;
567+
}
568+
return tp_mod_i64(a, b, out);
569+
}
570+
401571
/* ------------------------------------------------------------------------------------------
402572
* Comparisons: exact int/float comparison (float_richcompare), never via (double)int.
403573
* ------------------------------------------------------------------------------------------ */

0 commit comments

Comments
 (0)