diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 8ac3b1c..0144526 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -1,5 +1,5 @@ = Bulk loads and commit size -:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), and a known limitation: a graph batch does not reuse the space of deleted edges. +:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), a known limitation: a graph batch does not reuse the space of deleted edges, and how columnar loads, parallel loads and vertex loads through insert_dataframe and insert_arrow behave, including what a failed parallel load leaves behind. `insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails. Data that comes as a file, a DataFrame or an Arrow table has its own pages: xref:how-to/import-export.adoc[Importing and exporting data] and xref:how-to/dataframes-and-columns.adoc[DataFrames, Arrow and columns]. @@ -20,9 +20,17 @@ | `batch.create_vertices(type_name, rows)` | 1000 rows for each call into the engine, one transaction for each call (the `commit_every` of the batch) | 100000 rows for each call into the engine + +| `db.insert_columns(type_name, columns)` +| `commit_every=10000` +| `commit_every=10000` + +| `db.insert_dataframe(type_name, df)` and `db.insert_arrow(type_name, table)` +| `batch_size=10000`, one transaction for each batch +| `batch_size=10000`, one transaction for each batch |=== -The JVM defaults are the same as before. They stay until someone measures them on the JVM build. +The JVM defaults of `insert_many` and `graph_batch` are the same as before. They stay until someone measures them on the JVM build. A vertex type loaded with `insert_dataframe` or `insert_arrow` now commits every 10,000 rows on both packages: see <>. An explicit value always wins. `commit_every=0` gives one transaction for `insert_many` and one commit for each flush for `graph_batch`, on both packages. In a transaction that you open yourself, `insert_many` commits nothing. `graph_batch` refuses to run in one: see xref:how-to/transactions.adoc#graph-batch[GraphBatch inside a transaction]. The durability that a load gets is on xref:how-to/durability.adoc[Durability settings], and the async executor on xref:how-to/async-executor.adoc[The async executor]. @@ -57,6 +65,46 @@ Because the two packages have different defaults, the same program can keep diff On both packages, `insert_many` keeps rows with `datetime`, `date`, `Decimal`, `bytes`, numpy scalar, or pandas `NA` and `NaT` values on the bulk path. On the `arcadedb` (native) package, a row with a value the bulk text cannot carry (NaN or infinity, an integer beyond 64 bits, a date-time or `Decimal` inside a list or dict, or a `datetime.time`) goes in with its own statement, in the order given. +`insert_many` and `create_vertices` check their rows while they write the JSON text, in one pass over the rows, not in a loop before it. The same rows fail the check as before, and they take the same fallback path as before. The check raises no error to your code. + +[[columnar-loads]] +== Columnar loads + +`insert_columns` takes one column for each property. `insert_arrow` and `insert_dataframe` read an Arrow table or a pandas or Polars DataFrame as columns, in batches of `batch_size` rows. + +On the `arcadedb` (native) package, a batch whose columns the library's buffers can carry crosses in one call, with no Python dict per row and no JSON text. These columns stay columnar: integers, floats, bools, text, date-times (`datetime`, `pandas.Timestamp`, and Arrow timestamps of any unit), dates, `Decimal`, and `bytes` or `bytearray`. The stored values are the ones the row path stores. + +A column the buffers cannot carry makes the batch take the row path, which stores the same values through `insert_many`. These are a column that mixes types, a nested value, an integer beyond 64 bits, a string with a lone surrogate, a non-finite `Decimal`, and, in `insert_columns`, a numpy `datetime64` array or a list that holds pandas `NaT` or `NA`. A DataFrame goes through Arrow first, where a `NaT` or `NA` is a null, so it stays columnar. `ARCADEDB_NATIVE_BULK=0` in the environment turns the columnar path off. + +On the `arcadedb-jvm` package, a column of only `str` and `None` crosses as one UTF-8 buffer with its row offsets, and a mask when a row is null. A column that mixes types, is empty, or holds more than 2 GiB of text crosses value by value. A column with a lone surrogate also crosses value by value, and is refused with the error it always raised, before anything is written. + +[[parallel-inserts]] +== Parallel and async inserts + +`insert_many(parallel=True)` and `insert_columns(parallel=True)` hand the records to the async executor's writers and wait for the writers before they return. `insert_arrow` and `insert_dataframe` do the same for a document type, one batch at a time. A vertex type refuses `parallel=True` there with a `ValueError`. + +A parallel call opens no transaction, so `commit_every` does not apply. Each writer commits every `arcadedb.asyncTxBatchSize` records (10,240 by default, and `async_executor().set_commit_every()` changes it). On the native package, `insert_columns(parallel=True)` sends the columns to the library in one call, and the library hands each record to the writers. A column set the buffers cannot carry goes as one dict per row through `insert_many(parallel=True)`. + +What a failure leaves behind: + +* A record that a writer cannot store, for example a duplicate key, abandons the whole batch that writer had open, up to 10,240 records. The valid records of that batch are not stored either. +* The call waits for all writers, then raises `ArcadeDBError`. The message gives the number of failed records and the first failure. +* Nothing is rolled back. Batches of other writers, and earlier batches of the same call, may have been stored. After such an error, count the stored rows with a query. Do not assume all or none. +* On the native package, an interrupt (`KeyboardInterrupt`) during a parallel call waits for the writers before it propagates, so no writer still runs when the exception reaches you. +* In a transaction you opened, `insert_arrow` and `insert_dataframe` with `parallel=True` do not join it. The writers commit on their own, and a rollback of your transaction does not remove those rows. + +For one record at a time, `db.async_executor().create_record(document, callback=..., error_callback=...)` queues the record. The writers call `error_callback` with the exception when they reject it (a duplicate key under a `UNIQUE` index, or a batch abandoned at a failed commit). Without an `error_callback`, the failure reaches only the executor-wide `on_error` handler, if you set one. Call `wait_completion()` before you read the records back. + +[[vertex-loads]] +== Vertex types in `insert_dataframe` and `insert_arrow` + +On both packages, `insert_dataframe` and `insert_arrow` on a vertex type load through the same columnar insert as a document type. They no longer go through `GraphBatch.create_vertices`. Use `graph_batch()` when edges follow or when you need the RIDs of the vertices. + +* The load runs with the write-ahead log setting of the database. `graph_batch()` has its own `use_wal` argument. +* Each batch of `batch_size` rows (10,000 by default) is one transaction. A failed batch is rolled back, earlier batches stay committed, and the error message says how many rows that is. +* A batch whose save or commit raises a `NeedRetryException` (an MVCC conflict with another writer, a lock timeout) is rolled back and written again from its first row. It is tried up to 10 more times, after a wait that starts at 1 second and doubles up to 10 seconds. After the last try the error is raised unchanged. The columnar insert does this for every type it loads, not only for vertices, when no transaction of yours is open. A call with `parallel=True` does not retry, and on the native package neither does a batch that takes the row path. +* A vertex load inside a transaction you opened is refused with `ArcadeDBError` before anything is written, because the load commits its own batches. A document load joins your transaction. + == Change the commit size [source,python] @@ -85,7 +133,11 @@ The `arcadedb` (native) package does not change the collector. In our measuremen On the `arcadedb-jvm` package, `GraphBatch.create_vertices` and `GraphBatch.new_edges` load these property values through the bulk path (the same tagged JSON text as `insert_many`): a list, a tuple, a dict, a `datetime`, a `date`, a `Decimal` and `bytes` (also `bytearray`). A value is stored as the one-by-one path stores it. A list or tuple of only `int` and `float` keeps the older path, which crosses it as one `double[]`. -A value that the text cannot carry sends work to the older path on its own: a `NaN`, an integer beyond 64 bits, a `datetime` inside a list or a dict, or a dict key that is not a `str`. `create_vertices` falls back for the chunk of at most 100,000 rows that holds the value, and `new_edges` for the whole call. A value of another type (a `set`, a numpy or pandas value, a subclass of the types above) also takes the older path. Nothing is lost on a fallback, and the data is the same. Only the speed changes. +A value that the text cannot carry sends work to the older path on its own. In `create_vertices`, a scalar that the text would change (a `NaN`, an integer beyond 64 bits, a lone surrogate, a subnormal float) sends the whole call to the older path, wherever in the input it sits. So does a property name that is not a `str`. A value inside a list or a dict that the text cannot carry (a `datetime` inside a list, a `NaN` in a list, a dict key that is not a `str`) sends only the chunk of at most 100,000 rows that holds it. `new_edges` falls back for the whole call. A value of another type (a `set`, a numpy or pandas value, a subclass of the types above) also takes the older path. + +Nothing is lost on a fallback and the values are the same, but the stored type of a property that the type does not declare can differ: the older path stores a Python `int` as a `Long`, and the text path as an `Integer`. That is why a bad scalar sends the whole call to the older path. The stored type of a property then does not depend on which chunk held the bad value. + +A property whose value in every row is a `float32` numpy array or a Java `float[]` of one length crosses as one binary matrix for each property and chunk, and is stored as a `float[]`. A declared `ARRAY_OF_FLOATS` property takes the same path for lists and `float64` arrays of numbers too. See xref:how-to/vectors.adoc[Vectors]. The table is one laptop study of the `arcadedb-jvm` package: per-row time of this build against the build before it, 20,000 vertices and 20,000 edges for each shape in chunks of 5,000, one property on every row. Each cell is the median of the pair ratios (below 1 means faster) with its 95 % interval, from 11 to 12 clean ABBA pairs for each figure of `ab.py --noise`, with a clean A/A control. The `arcadedb` (native) package does not change. @@ -172,7 +224,8 @@ This section is about a `graph_batch()` load that stops before `close()` returns === What is in the database -* The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it. +* The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it, with one exception: `create_vertices` on the `arcadedb-jvm` package (next point). +* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. The same probe on the `26.11.1.post1` candidate build stored another count in 2 of 240 cycles, and in both the stored keys were a prefix of the input. It has not been run on `26.11.1.post2`. * The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in `close()`. Until `close()` has run, `outE()`, `inE()`, `out()` and `MATCH` find none of them, but `SELECT count(*) FROM E` counts them. In the example: 70,000 edge records, 0 reachable. * `CHECK DATABASE` reports each of these edges as an `ORPHAN RECORD` and counts them in `unreachableEdgeRecords`. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex's OUT/IN list" for the same state. Read the statement in the {arcadedb-docs-url}/reference/sql/sql-database-admin#sql-check-database[`CHECK DATABASE` reference]; this page does not repeat it. diff --git a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc index 71854a3..03ef5de 100644 --- a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc +++ b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc @@ -249,7 +249,7 @@ A query vector for `vectorNeighbors` can be a numpy array or a list. This sectio `db.query` returns a result set you can iterate. What a row holds, how a Python value is converted and which rows are record objects is on xref:how-to/values-and-records.adoc[Values and records]. Two methods cover the usual needs: * `first()` returns the first row, or `None` when there is none, so `if db.query(...).first():` tests whether anything matched. -* `to_list()` materialises the whole result as a list of dicts. Use it when you need the whole result in memory; iterate when you stop early or the result is large. +* `to_list()` materialises the whole result as a list of dicts, one Python dict per row. Use it for a small result, or when you need the whole result in memory; iterate when you stop early. For a large result, see <>. * `iter_json_batches(batch_size=10_000)` yields the rows as lists of JSON-native dicts, `batch_size` rows per list (the last may be shorter), with the same values as `to_json_list()`. On the native build `batch_size` bounds memory: the result set holds one reply of the cursor (2048 rows) and one batch at a time, and reads the next reply only when the next batch is asked for, so a large result is never held whole. On the JVM build a batch is serialised on the Java side one at a time. `to_list()`, `to_json_list()`, `to_dataframe()`, `to_columns()`, `to_arrow()` and `to_polars()` still hold the whole result, because they return it whole. + One laptop study on the native build, 1,000,000 rows read with `batch_size=10_000`, 10 clean pairs of this change against the previous behaviour; each figure is the ratio of the new value to the old one, so below 1 is lower. Peak RSS: 0.286 (95% interval 0.284 to 0.289). Time to the first batch: 0.010 (95% interval 0.009 to 0.010). Wall time of the whole read: 0.958 (95% interval 0.952 to 0.966). This is one machine and one result shape, not a speed promise. @@ -275,6 +275,55 @@ On the JVM build, `Document.to_dict()`, `Result.to_dict()` and `get_property_nam The control times `json.dumps` of rows already fetched, which uses no query and no JPype call. Its ratios on the four workloads are 0.996, 0.987, 0.993 and 1.000, and the same build measured twice gave ratios whose intervals contain 1. The figures come from one Linux x86_64 laptop with two cores for the process, Python 3.12, the median of 21 timed passes per run, and builds alternated in both orders. They cover the JVM build and these four workloads only, not wider rows, other value types, `to_list()`, or the native build. +[[reading-large-results]] +=== Reading large results + +The row APIs build one Python dict per row: `to_list()`, `to_json_list()`, and a `for` loop over the result set. They suit small results. For a large result, read columns: + +* `to_columns()` returns a dict of column name to a numpy array, or to a Python list for text and for JSON-typed values. +* `to_arrow()` returns a `pyarrow.Table`. +* `to_dataframe()` returns a `pandas.DataFrame`. +* `to_polars()` returns a `polars.DataFrame`, read through `to_arrow()`. + +These four build no dict per row. Both packages read the result as typed column frames and decode them with the same code, so one query gives the same column types and values on `arcadedb` (native) and on `arcadedb-jvm`. On the native package, the first rows of a query (up to 256) are the one exception: they arrive with the query as JSON, because the row APIs read them. A column read puts those rows into the same column rules as the later frames. + +`to_columns()` needs numpy, `to_arrow()` needs numpy and pyarrow, `to_dataframe()` needs pandas, and `to_polars()` needs polars and pyarrow. `to_columns()` and `to_arrow()` return `None` when a library is missing. `to_dataframe(convert_types=False)` on `arcadedb-jvm` builds the frame from `to_list()` instead; on `arcadedb` the argument has no effect. + +`to_columns()`, `to_arrow()` and `to_polars()` take `batch_size`, the most rows in one frame (default 25,000), and `columns`, a list of names to read as a projection would. `to_dataframe()` takes neither and uses 25,000. `batch_size` must be at least 1 (`ValueError` otherwise). A value above 2,147,483,647 (2^31^ - 1) is clamped to 2,147,483,647. On the native package a frame also ends when the values it holds reach about 1 MiB. + +Memory: these calls return the whole result, so it must fit in memory. The frames are decoded as they are read and kept until the end of the result. For a result of several frames, `to_columns()` and `to_dataframe()` then copy them into one array for each column, so the frames and the merged arrays are held together for a moment. `to_arrow()` keeps the frames as the chunks of one column. If the result does not fit, read it in pieces with `iter_json_batches()`. + +==== What the column calls return + +* The type of a column belongs to the whole result, not to where a frame ends. A frame that lacks a property, or holds only nulls for it, takes the type of the frames that have values. `batch_size` never changes a column's type. +* Nulls: in `to_columns()` and `to_dataframe()`, an integer column with a null is `float64` with `NaN` (the pandas rule), a `DATETIME` column with a null holds `NaT`, and text and JSON-typed columns hold `None`. In `to_arrow()` and `to_polars()`, an integer column with a null stays `int64` with a null. A `float64` column cannot hold every integer beyond 2^53^, so `to_columns()` and `to_dataframe()` round such an integer in a column that has a null. `to_arrow()` and `to_polars()` keep it exact. +* An integer beyond 2^53^ next to a floating value has no exact double, so the column is not made a `float64` column. `to_columns()` gives a list of Python values, each an `int` or a `float` as stored. `to_arrow()` and `to_polars()` give a string column with the exact text of each value. `to_dataframe()` gives an `object` column of the same Python values, so the integer stays exact. This holds also when the integer and the floats are in different frames of a large result. Two reads still round the integer, because they build a `float64` column: `to_dataframe(convert_types=False)` on `arcadedb-jvm`, and the fallback of `to_dataframe()` that reads through `to_list()` when the Java bridge classes are missing. Use `to_columns()` or `to_arrow()` when such a value must stay exact on those two reads. +* A vector column is a column whose values are all `float[]`, which is what a property declared `ARRAY_OF_FLOATS` holds. A `LIST` property of numbers is a list column, whatever its values. +* Dates and decimals: `DATETIME` is `datetime64[ms]` in `to_columns()` and `timestamp[ms]` in `to_arrow()`, and `datetime64[us]` in `to_dataframe()`. A `DECIMAL` stays a `Decimal` (an object array in `to_columns()`, a decimal column in Arrow). A `DATE` in `to_dataframe()` is a `datetime.date` in an object column. +* The arrays of `to_columns()` are writable. +* A result with no rows gives `{}` from `to_columns()`. With `columns=[...]`, it gives those columns as empty lists. `to_arrow()` gives an empty table, or with `columns=[...]` those columns empty and of the null type. + +A vector cell has one form for each call: + +[cols="2,3"] +|=== +| Call | Vector column (`float[]`) + +| `to_columns()` +| One `(count, dim)` `float32` matrix. A null vector is a row of `NaN`. + +| `to_dataframe()` +| One `float32` numpy array in each cell, and `None` for a null. + +| `to_arrow()` +| A fixed-size list of `float32`. A null vector is a null. + +| `to_polars()` +| `Array(Float32, n)`. +|=== + +A `double[]` property gives `float64` in each of these. If the vectors of a result do not all have one length, the column is a list of variable length: `list` in Arrow, a list of lists in `to_columns()`. + [[result-set-single-pass]] === A result set is read once @@ -295,6 +344,8 @@ A record id inside a row value (`SELECT @rid AS r`, a `LINK` property, a list of On the native build the record wrappers send their record id to the engine as a parameter, not in the statement text. The engine keeps a parsed statement in a cache keyed by its text (`arcadedb.sqlStatementCache`, 300 entries by default) and leaves a record id written in a `FROM` target as part of that text, so each record would be a cache entry of its own. `get_out_edges()`, `get_in_edges()`, `get_both_edges()`, `new_edge()`, `Edge.get_in()`, `Edge.get_out()`, `save()` and `delete()` send the id as a parameter, so every record shares one text. `db.lookup_by_rid()` keeps the id in the statement text: measured, the parameter path was 2 % slower for a lookup, so it stays as it is. +On the native build, `save()` of a new record and `Vertex.new_edge` read the record they create from the reply of the statement that creates it, so a create is one call into the library. `save()` of a saved record makes three calls: the transaction check, the `SELECT` of the stale-record check, and the `UPDATE`. The records, their values and the errors are the same as before. + `Vertex.new_edge()` works on a vertex object that you read in an earlier transaction, on both builds. The first edge of a vertex rewrites its record, so an object read before that commit is out of date. The native build goes by record id. The JVM build looks up the current source vertex by its record id inside the active transaction and passes the target as its record id, so the engine loads that one too. The vertex object you hold is not refreshed: call `db.lookup_by_rid()` again to see its current properties. [source,python,subs="attributes+"] diff --git a/docs/modules/ROOT/pages/how-to/vectors.adoc b/docs/modules/ROOT/pages/how-to/vectors.adoc index 23df5dc..2eef4df 100644 --- a/docs/modules/ROOT/pages/how-to/vectors.adoc +++ b/docs/modules/ROOT/pages/how-to/vectors.adoc @@ -114,6 +114,8 @@ print(db.count_type("Chunk")) On the `arcadedb-jvm` package, `create_vertices` sends the values of a property declared `ARRAY_OF_FLOATS` as one binary float32 matrix when every row holds a list, a numpy array or a Java `float[]` or `double[]` of numbers of one length. A property that is not declared takes the same matrix when every row holds a `float32` numpy array or a Java `float[]` (what `to_java_float_array` returns on this package) of one length, and the stored value is a `float[]`, as before. A list or a `float64` array on an undeclared property keeps its doubles and the older path. So `float32` arrays and `to_java_float_array` values take the binary path on both packages, declared or not. +To read many vectors, use `to_columns()`, `to_dataframe()`, `to_arrow()` or `to_polars()`. `to_list()` builds one Python list of floats for each vector. A column is a vector column when its values are `float[]`, which is what a property declared `ARRAY_OF_FLOATS` holds; a `LIST` property of numbers is a list column. `to_columns()` returns a `(count, dim)` `float32` matrix, with a row of `NaN` for a null vector. `to_dataframe()` returns one `float32` numpy array in each cell, and `None` for a null. `to_arrow()` returns a fixed-size list of `float32`, and `to_polars()` an `Array(Float32, n)`. Vectors of different lengths in one result make a variable-length list column (`list` in Arrow). The rules for nulls, `batch_size` and memory are in xref:how-to/databases-and-queries.adoc#reading-large-results[Reading large results]. + == Search === SQL diff --git a/docs/modules/ROOT/pages/installation.adoc b/docs/modules/ROOT/pages/installation.adoc index 0f2e67d..3e64546 100644 --- a/docs/modules/ROOT/pages/installation.adoc +++ b/docs/modules/ROOT/pages/installation.adoc @@ -160,7 +160,7 @@ Each Python process starts its own JVM, and each JVM sees the whole container li [[jvm-cds]] === Class-data sharing -The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. +The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. It is a file in the bundled JRE (about 15 MB, about 3.9 MB inside the wheel), so each JVM wheel is larger by that much. The native package has no JVM and no archive. [[jvm-aot]] === Faster start-up with the AOT cache diff --git a/tests/test_column_frames.py b/tests/test_column_frames.py index 98de315..c5a6d13 100644 --- a/tests/test_column_frames.py +++ b/tests/test_column_frames.py @@ -601,6 +601,36 @@ def test_to_dataframe_integers_up_to_2_53_next_to_floats_stay_float64(temp_db): assert df["x"].tolist() == [1.0, 1.5, 2.0**53, -(2.0**53)] +def test_to_dataframe_keeps_the_integer_when_the_floats_are_in_a_later_frame(temp_db): + # docs, "What the column calls return": the integer stays exact also when it and the floats are in different frames of a + # large result. to_dataframe() reads 25,000 rows in one frame, so rows 0 to 24,999 are integers and the last 10 are floats + big = 2**60 + 1 + want = [big + i for i in range(25_000)] + [0.5 + i for i in range(10)] + df = _mix_frame(temp_db, want) + assert df["x"].dtype == object + assert _same_values_and_types(df["x"].tolist(), want) + # the same column in to_columns() is a list of exact values, not a float64 array + columns = temp_db.query("sql", "SELECT i, x FROM Mix ORDER BY i").to_columns() + assert isinstance(columns["x"], list) + assert _same_values_and_types(columns["x"], want) + + +def test_an_integer_beyond_2_53_in_a_column_with_a_null_is_rounded_but_in_arrow( + temp_db, +): + # docs, "What the column calls return": the limit that stays. An integer column with a null is float64 with NaN in + # to_columns() and to_dataframe() (the pandas rule), which cannot hold 2**60 + 1; to_arrow() keeps int64 with a null + big = 2**60 + 1 + df = _mix_frame(temp_db, [big, None]) + assert df["x"].dtype == np.float64 and np.isnan(df["x"][1]) + assert int(df["x"][0]) == 2**60 # rounded to the nearest double + query = "SELECT i, x FROM Mix ORDER BY i" + columns = temp_db.query("sql", query).to_columns() + assert columns["x"].dtype == np.float64 and int(columns["x"][0]) == 2**60 + arrow = temp_db.query("sql", query).to_arrow()["x"] + assert arrow.type == pa.int64() and arrow.to_pylist() == [big, None] + + def test_to_dataframe_of_an_empty_result_is_an_empty_frame(temp_db): _mix_frame(temp_db, [2**60 + 1, 1.5]) df = temp_db.query("sql", "SELECT i, x FROM Mix WHERE i < 0").to_dataframe()