diff --git a/docs/source/python/index.rst b/docs/source/python/index.rst index cef8998ed4cd..e6e8a62cee95 100644 --- a/docs/source/python/index.rst +++ b/docs/source/python/index.rst @@ -52,17 +52,17 @@ files into Arrow structures. filesystems numpy pandas - interchange_protocol - dlpack timestamps - orc + parquet/index + dataset csv + orc feather json - parquet - dataset flight extending_types + interchange_protocol + dlpack integration env_vars api diff --git a/docs/source/python/parquet.rst b/docs/source/python/parquet.rst deleted file mode 100644 index 4fc1368fb270..000000000000 --- a/docs/source/python/parquet.rst +++ /dev/null @@ -1,987 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one -.. or more contributor license agreements. See the NOTICE file -.. distributed with this work for additional information -.. regarding copyright ownership. The ASF licenses this file -.. to you under the Apache License, Version 2.0 (the -.. "License"); you may not use this file except in compliance -.. with the License. You may obtain a copy of the License at - -.. http://www.apache.org/licenses/LICENSE-2.0 - -.. Unless required by applicable law or agreed to in writing, -.. software distributed under the License is distributed on an -.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -.. KIND, either express or implied. See the License for the -.. specific language governing permissions and limitations -.. under the License. - -.. currentmodule:: pyarrow -.. _parquet: - -Reading and Writing the Apache Parquet Format -============================================= - -The `Apache Parquet `_ project provides a -standardized open-source columnar storage format for use in data analysis -systems. It was created originally for use in `Apache Hadoop -`_ with systems like `Apache Drill -`_, `Apache Hive `_, `Apache -Impala `_, and `Apache Spark -`_ adopting it as a shared standard for high -performance data IO. - -Apache Arrow is an ideal in-memory transport layer for data that is being read -or written with Parquet files. We have been concurrently developing the `C++ -implementation of -Apache Parquet `_, -which includes a native, multithreaded C++ adapter to and from in-memory Arrow -data. PyArrow includes Python bindings to this code, which thus enables reading -and writing Parquet files with pandas as well. - -Obtaining pyarrow with Parquet Support --------------------------------------- - -If you installed ``pyarrow`` with pip or conda, it should be built with Parquet -support bundled: - -.. code-block:: python - - >>> import pyarrow.parquet as pq - -If you are building ``pyarrow`` from source, you must use ``-DARROW_PARQUET=ON`` -when compiling the C++ libraries and enable the Parquet extensions when -building ``pyarrow``. If you want to use Parquet Encryption, then you must -use ``-DPARQUET_REQUIRE_ENCRYPTION=ON`` too when compiling the C++ libraries. -See the :ref:`Python Development ` page for more details. - -Reading and Writing Single Files --------------------------------- - -The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` -read and write the :ref:`pyarrow.Table ` object, respectively. - -Let's look at a simple table: - -.. code-block:: python - - >>> import numpy as np - >>> import pandas as pd - >>> import pyarrow as pa - >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], - ... 'two': ['foo', 'bar', 'baz'], - ... 'three': [True, False, True]}, - ... index=list('abc')) - >>> table = pa.Table.from_pandas(df) - -We write this to Parquet format with ``write_table``: - -.. code-block:: python - - >>> import pyarrow.parquet as pq - >>> pq.write_table(table, 'example.parquet') - -This creates a single Parquet file. In practice, a Parquet dataset may consist -of many files in many directories. We can read a single file back with -``read_table``: - -.. code-block:: python - - >>> table2 = pq.read_table('example.parquet') - >>> table2.to_pandas() - one two three - a -1.0 foo True - b NaN bar False - c 2.5 baz True - -You can pass a subset of columns to read, which can be much faster than reading -the whole file (due to the columnar layout): - -.. code-block:: python - - >>> pq.read_table('example.parquet', columns=['one', 'three']) - pyarrow.Table - one: double - three: bool - ---- - one: [[-1,null,2.5]] - three: [[true,false,true]] - -When reading a subset of columns from a file that used a Pandas dataframe as the -source, we use ``read_pandas`` to maintain any additional index column data: - -.. code-block:: python - - >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas() - two - a foo - b bar - c baz - -We do not need to use a string to specify the origin of the file. It can be any of: - -* A file path as a string -* A :ref:`NativeFile ` from PyArrow -* A Python file object - -In general, a Python file object will have the worst read performance, while a -string file path or an instance of :class:`~.NativeFile` (especially memory -maps) will perform the best. - -.. _parquet_mmap: - -Reading Parquet and Memory Mapping -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Because Parquet data needs to be decoded from the Parquet format -and compression, it can't be directly mapped from disk. -Thus the ``memory_map`` option might perform better on some systems -but won't help much with resident memory consumption. - -.. code-block:: python - - >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP - >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP - RSS: 4299MB - - >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP - >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP - RSS: 4299MB - -If you need to deal with Parquet data bigger than memory, -the :ref:`dataset` and partitioning is probably what you are looking for. - -Parquet file writing options -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:func:`~pyarrow.parquet.write_table()` has a number of options to -control various settings when writing a Parquet file. - -* ``version``, the Parquet format version to use. ``'1.0'`` ensures - compatibility with older readers, while ``'2.4'`` and greater values - enable more Parquet types and encodings. -* ``data_page_size``, to control the approximate size of encoded data - pages within a column chunk. This currently defaults to 1MB. -* ``flavor``, to set compatibility options particular to a Parquet - consumer like ``'spark'`` for Apache Spark. - -See the :func:`~pyarrow.parquet.write_table()` docstring for more details. - -There are some additional data type handling-specific options -described below. - -Omitting the DataFrame index -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When using ``pa.Table.from_pandas`` to convert to an Arrow table, by default -one or more special columns are added to keep track of the index (row -labels). Storing the index takes extra space, so if your index is not valuable, -you may choose to omit it by passing ``preserve_index=False`` - -.. code-block:: python - - >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], - ... 'two': ['foo', 'bar', 'baz'], - ... 'three': [True, False, True]}, - ... index=list('abc')) - >>> table = pa.Table.from_pandas(df, preserve_index=False) - -Then we have: - -.. code-block:: python - - >>> pq.write_table(table, 'example_noindex.parquet') - >>> t = pq.read_table('example_noindex.parquet') - >>> t.to_pandas() - one two three - 0 -1.0 foo True - 1 NaN bar False - 2 2.5 baz True - -Here you see the index did not survive the round trip. - -Finer-grained Reading and Writing ---------------------------------- - -``read_table`` uses the :class:`~.ParquetFile` class, which has other features: - -.. code-block:: python - - >>> parquet_file = pq.ParquetFile('example.parquet') - >>> parquet_file.metadata - - created_by: parquet-cpp-arrow version ... - num_columns: 4 - num_rows: 3 - num_row_groups: 1 - format_version: 2.6 - serialized_size: ... - >>> parquet_file.schema - - required group field_id=-1 schema { - optional double field_id=-1 one; - optional binary field_id=-1 two (String); - optional boolean field_id=-1 three; - optional binary field_id=-1 __index_level_0__ (String); - } - - -As you can learn more in the `Apache Parquet format -`_, a Parquet file consists of -multiple row groups. ``read_table`` will read all of the row groups and -concatenate them into a single table. You can read individual row groups with -``read_row_group``: - -.. code-block:: python - - >>> parquet_file.num_row_groups - 1 - >>> parquet_file.read_row_group(0) - pyarrow.Table - one: double - two: large_string - three: bool - __index_level_0__: large_string - ---- - one: [[-1,null,2.5]] - two: [["foo","bar","baz"]] - three: [[true,false,true]] - __index_level_0__: [["a","b","c"]] - -We can similarly write a Parquet file with multiple row groups by using -``ParquetWriter``: - -.. code-block:: python - - >>> with pq.ParquetWriter('example2.parquet', table.schema) as writer: - ... for i in range(3): - ... writer.write_table(table) - >>> pf2 = pq.ParquetFile('example2.parquet') - >>> pf2.num_row_groups - 3 - -Inspecting the Parquet File Metadata ------------------------------------- - -The ``FileMetaData`` of a Parquet file can be accessed through -:class:`~.ParquetFile` as shown above: - -.. code-block:: python - - >>> parquet_file = pq.ParquetFile('example.parquet') - >>> metadata = parquet_file.metadata - >>> metadata - - created_by: parquet-cpp-arrow version ... - num_columns: 4 - num_rows: 3 - num_row_groups: 1 - format_version: 2.6 - serialized_size: ... - -or can also be read directly using :func:`~parquet.read_metadata`: - -.. code-block:: python - - >>> metadata = pq.read_metadata('example.parquet') - >>> metadata - - created_by: parquet-cpp-arrow version ... - num_columns: 4 - num_rows: 3 - num_row_groups: 1 - format_version: 2.6 - serialized_size: ... - -The returned ``FileMetaData`` object allows to inspect the -`Parquet file metadata `__, -such as the row groups and column chunk metadata and statistics: - -.. code-block:: python - - >>> metadata.row_group(0) - - num_columns: 4 - num_rows: 3 - total_byte_size: 290 - sorting_columns: () - >>> metadata.row_group(0).column(0) - - file_offset: 0 - file_path:... - physical_type: DOUBLE - num_values: 3 - path_in_schema: one - is_stats_set: True - statistics: - - has_min_max: True - min: -1.0 - max: 2.5 - null_count: 1 - distinct_count: None - num_values: 2 - physical_type: DOUBLE - logical_type: None - converted_type (legacy): NONE - geo_statistics: - None - compression: SNAPPY - encodings: ('PLAIN', 'RLE', 'RLE_DICTIONARY') - has_dictionary_page: True - dictionary_page_offset: 4 - data_page_offset: 36 - total_compressed_size: 106 - total_uncompressed_size: 102 - bloom_filter_offset: None - bloom_filter_length: None - -Data Type Handling ------------------- - -Reading types as DictionaryArray -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will -cause columns to be read as ``DictionaryArray``, which will become -``pandas.Categorical`` when converted to pandas. This option is only valid for -string and binary column types, and it can yield significantly lower memory use -and improved performance for columns with many repeated string values. - -.. code-block:: python - - >>> pq.read_table('example.parquet', read_dictionary=['two']) - pyarrow.Table - one: double - two: dictionary - three: bool - __index_level_0__: large_string - ---- - one: [[-1,null,2.5]] - two: [ -- dictionary: - ["foo","bar","baz"] -- indices: - [0,1,2]] - three: [[true,false,true]] - __index_level_0__: [["a","b","c"]] - -Storing timestamps -~~~~~~~~~~~~~~~~~~ - -Some Parquet readers may only support timestamps stored in millisecond -(``'ms'``) or microsecond (``'us'``) resolution. Since pandas uses nanoseconds -to represent timestamps, this can occasionally be a nuisance. When writing -older ``version='1.0'`` or ``version='2.4'`` Parquet files, the nanoseconds -will be cast to microseconds (``'us'``). - -In addition, We provide the ``coerce_timestamps`` option to allow you to select -the desired resolution: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', coerce_timestamps='ms') - -If a cast to a lower resolution value may result in a loss of data, by default -an exception will be raised. This can be suppressed by passing -``allow_truncated_timestamps=True``: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', coerce_timestamps='ms', - ... allow_truncated_timestamps=True) - -Timestamps with nanoseconds can be stored without casting when using -Parquet format version 2.6, which is the default: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', version='2.6') - -However, some Parquet readers may not yet support this newer format version. -When compatibility across different processing frameworks is required, the -older ``version='1.0'`` or ``version='2.4'`` remain available. - -Older Parquet implementations use ``INT96`` based storage of -timestamps, but this is now deprecated. This includes some older -versions of Apache Impala and Apache Spark. To write timestamps in -this format, set the ``use_deprecated_int96_timestamps`` option to -``True`` in ``write_table``. - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', use_deprecated_int96_timestamps=True) - -Compression, Encoding, and File Compatibility ---------------------------------------------- - -The most commonly used Parquet implementations use dictionary encoding when -writing files; if the dictionaries grow too large, then they "fall back" to -plain encoding. Whether dictionary encoding is used can be toggled using the -``use_dictionary`` option: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', use_dictionary=False) - -The data pages within a column in a row group can be compressed after the -encoding passes (dictionary, RLE encoding). In PyArrow we use Snappy -compression by default, but Brotli, Gzip, ZSTD, LZ4, and uncompressed are -also supported: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', compression='snappy') - >>> pq.write_table(table, 'example.parquet', compression='gzip') - >>> pq.write_table(table, 'example.parquet', compression='brotli') - >>> pq.write_table(table, 'example.parquet', compression='zstd') - >>> pq.write_table(table, 'example.parquet', compression='lz4') - >>> pq.write_table(table, 'example.parquet', compression='none') - -Snappy generally results in better performance, while Gzip may yield smaller -files. - -``'lz4_raw'`` is also accepted as an alias for ``'lz4'``. Both use the -LZ4_RAW codec as defined in the Parquet specification. - -These settings can also be set on a per-column basis: - -.. code-block:: python - - >>> pq.write_table(table, 'example.parquet', compression={'one': 'snappy', 'two': 'gzip'}, - ... use_dictionary=['one', 'two']) - -Partitioned Datasets (Multiple Files) ------------------------------------------------- - -Multiple Parquet files constitute a Parquet *dataset*. These may present in a -number of ways: - -* A list of Parquet absolute file paths -* A directory name containing nested directories defining a partitioned dataset - -A dataset partitioned by year and month may look like on disk: - -.. code-block:: text - - dataset_name/ - year=2007/ - month=01/ - 0.parq - 1.parq - ... - month=02/ - 0.parq - 1.parq - ... - month=03/ - ... - year=2008/ - month=01/ - ... - ... - -Writing to Partitioned Datasets -------------------------------- - -You can write a partitioned dataset for any ``pyarrow`` file system that is a -file-store (e.g. local, HDFS, S3). The default behaviour when no filesystem is -added is to use the local filesystem. - -.. code-block:: python - - >>> # Local dataset write - >>> pq.write_to_dataset(table, root_path='dataset_name', - ... partition_cols=['one', 'two']) - -The root path in this case specifies the parent directory to which data will be -saved. The partition columns are the column names by which to partition the -dataset. Columns are partitioned in the order they are given. The partition -splits are determined by the unique values in the partition columns. - -To use another filesystem you only need to add the filesystem parameter, the -individual table writes are wrapped using ``with`` statements so the -``pq.write_to_dataset`` function does not need to be. - -.. code-block:: python - - >>> # Remote file-system example - >>> from pyarrow.fs import HadoopFileSystem # doctest: +SKIP - >>> fs = HadoopFileSystem(host, port, user=user, kerb_ticket=ticket_cache_path) # doctest: +SKIP - >>> pq.write_to_dataset(table, root_path='dataset_name', # doctest: +SKIP - ... partition_cols=['one', 'two'], filesystem=fs) - -Compatibility Note: if using ``pq.write_to_dataset`` to create a table that -will then be used by HIVE then partition column values must be compatible with -the allowed character set of the HIVE version you are running. - -Writing ``_metadata`` and ``_common_metadata`` files -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Some processing frameworks such as Spark or Dask (optionally) use ``_metadata`` -and ``_common_metadata`` files with partitioned datasets. - -Those files include information about the schema of the full dataset (for -``_common_metadata``) and potentially all row group metadata of all files in the -partitioned dataset as well (for ``_metadata``). The actual files are -metadata-only Parquet files. Note this is not a Parquet standard, but a -convention set in practice by those frameworks. - -Using those files can give a more efficient creation of a parquet Dataset, -since it can use the stored schema and file paths of all row groups, -instead of inferring the schema and crawling the directories for all Parquet -files (this is especially the case for filesystems where accessing files -is expensive). - -The :func:`~pyarrow.parquet.write_to_dataset` function does not automatically -write such metadata files, but you can use it to gather the metadata and -combine and write them manually: - -.. code-block:: python - - >>> # Write a dataset and collect metadata information of all written files - >>> metadata_collector = [] - >>> root_path = "dataset_name_1" - >>> pq.write_to_dataset(table, root_path, metadata_collector=metadata_collector) - - >>> # Write the ``_common_metadata`` parquet file without row groups statistics - >>> pq.write_metadata(table.schema, root_path + '/_common_metadata') - - >>> # Write the ``_metadata`` parquet file with row groups statistics of all files - >>> pq.write_metadata( - ... table.schema, root_path + '/_metadata', - ... metadata_collector=metadata_collector - ... ) - -When not using the :func:`~pyarrow.parquet.write_to_dataset` function, but -writing the individual files of the partitioned dataset using -:func:`~pyarrow.parquet.write_table` or :class:`~pyarrow.parquet.ParquetWriter`, -the ``metadata_collector`` keyword can also be used to collect the FileMetaData -of the written files. In this case, you need to ensure to set the file path -contained in the row group metadata yourself before combining the metadata, and -the schemas of all different files and collected FileMetaData objects should be -the same: - -.. code-block:: python - - >>> import os - >>> os.mkdir("year=2017") - - >>> metadata_collector = [] - >>> pq.write_table( - ... table, "year=2017/data1.parquet", - ... metadata_collector=metadata_collector - ... ) - - >>> # set the file path relative to the root of the partitioned dataset - >>> metadata_collector[-1].set_file_path("year=2017/data1.parquet") - - >>> # combine and write the metadata - >>> metadata = metadata_collector[0] - >>> for _meta in metadata_collector[1:]: - ... metadata.append_row_groups(_meta) - >>> metadata.write_metadata_file("_metadata") - - >>> # or use pq.write_metadata to combine and write in a single step - >>> pq.write_metadata( - ... table.schema, "_metadata", - ... metadata_collector=metadata_collector - ... ) - - >>> pq.read_metadata("_metadata") - - created_by: parquet-cpp-arrow version ... - num_columns: 3 - num_rows: 3 - num_row_groups: 1 - format_version: 2.6 - serialized_size: ... - -Reading from Partitioned Datasets ------------------------------------------------- - -The :class:`~.ParquetDataset` class accepts either a directory name or a list -of file paths, and can discover and infer some common partition structures, -such as those produced by Hive: - -.. code-block:: python - - >>> dataset = pq.ParquetDataset('dataset_name/') - >>> table = dataset.read() - >>> table - pyarrow.Table - three: bool - one: dictionary - two: dictionary - ---- - three: [[true],[true],[false]] - one: [ -- dictionary: - ["-1","2.5"] -- indices: - [0], -- dictionary: - ["-1","2.5"] -- indices: - [1], -- dictionary: - [null] -- indices: - [0]] - two: [ -- dictionary: - ["foo","baz","bar"] -- indices: - [0], -- dictionary: - ["foo","baz","bar"] -- indices: - [1], -- dictionary: - ["foo","baz","bar"] -- indices: - [2]] - -You can also use the convenience function ``read_table`` exposed by -``pyarrow.parquet`` that avoids the need for an additional Dataset object -creation step. - -.. code-block:: python - - >>> table = pq.read_table('dataset_name') - -Note: the partition columns in the original table will have their types -converted to Arrow dictionary types (pandas categorical) on load. Ordering of -partition columns is not preserved through the save/load process. If reading -from a remote filesystem into a pandas dataframe you may need to run -``sort_index`` to maintain row ordering (as long as the ``preserve_index`` -option was enabled on write). - -Other features: - -- Filtering on all columns (using row group statistics) instead of only on - the partition keys. -- Fine-grained partitioning: support for a directory partitioning scheme - in addition to the Hive-like partitioning (e.g. "/2019/11/15/" instead of - "/year=2019/month=11/day=15/"), and the ability to specify a schema for - the partition keys. - -Note: - -- The partition keys need to be explicitly included in the ``columns`` - keyword when you want to include them in the result while reading a - subset of the columns - - -Using with Spark ----------------- - -Spark places some constraints on the types of Parquet files it will read. The -option ``flavor='spark'`` will set these options automatically and also -sanitize field characters unsupported by Spark SQL. - -Multithreaded Reads -------------------- - -Each of the reading functions by default use multi-threading for reading -columns in parallel. Depending on the speed of IO -and how expensive it is to decode the columns in a particular file -(particularly with GZIP compression), this can yield significantly higher data -throughput. - -This can be disabled by specifying ``use_threads=False``. - -.. note:: - The number of threads to use concurrently is automatically inferred by Arrow - and can be inspected using the :func:`~pyarrow.cpu_count()` function. - -Reading from cloud storage --------------------------- - -In addition to local files, pyarrow supports other filesystems, such as cloud -filesystems, through the ``filesystem`` keyword: - -.. code-block:: python - - >>> from pyarrow import fs - - >>> s3 = fs.S3FileSystem(region="us-east-2") # doctest: +SKIP - >>> table = pq.read_table("bucket/object/key/prefix", filesystem=s3) # doctest: +SKIP - -Currently, :class:`HDFS ` and -:class:`Amazon S3-compatible storage ` are -supported. See the :ref:`filesystem` docs for more details. For those -built-in filesystems, the filesystem can also be inferred from the file path, -if specified as a URI: - -.. code-block:: python - - >>> table = pq.read_table("s3://bucket/object/key/prefix") # doctest: +SKIP - -Other filesystems can still be supported if there is an -`fsspec `__-compatible -implementation available. See :ref:`filesystem-fsspec` for more details. -One example is Azure Blob storage, which can be interfaced through the -`adlfs `__ package. - -.. code-block:: python - - >>> from adlfs import AzureBlobFileSystem # doctest: +SKIP - >>> abfs = AzureBlobFileSystem(account_name="XXXX", account_key="XXXX", container_name="XXXX") # doctest: +SKIP - >>> table = pq.read_table("file.parquet", filesystem=abfs) # doctest: +SKIP - -Parquet Modular Encryption (Columnar Encryption) ------------------------------------------------- - -Columnar encryption is supported for Parquet files in C++ starting from -Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. - -Parquet uses the envelope encryption practice, where file parts are encrypted -with "data encryption keys" (DEKs), and the DEKs are encrypted with "master -encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each -encrypted file/column. The MEKs are generated, stored and managed in a Key -Management Service (KMS) of user’s choice. - -Reading and writing encrypted Parquet files involves passing file encryption -and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to -:class:`~.ParquetFile`, respectively. - -Writing an encrypted Parquet file: - -.. code-block:: python - - >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP - ... kms_connection_config, encryption_config) - >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP - ... encryption_properties=encryption_properties) as writer: - ... writer.write_table(table) - -Reading an encrypted Parquet file: - -.. code-block:: python - - >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP - ... kms_connection_config) - >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP - ... decryption_properties=decryption_properties) - - -In order to create the encryption and decryption properties, a -:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and -initialized with KMS Client details, as described below. - - -KMS Client -~~~~~~~~~~ - -The master encryption keys should be kept and managed in a production-grade -Key Management System (KMS), deployed in the user's organization. Using Parquet -encryption requires implementation of a client class for the KMS server. -Any KmsClient implementation should implement the informal interface -defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: - -.. code-block:: python - - >>> import pyarrow.parquet.encryption as pe - >>> class MyKmsClient(pe.KmsClient): - ... - ... """An example KmsClient implementation skeleton""" - ... def __init__(self, kms_connection_configuration): - ... pe.KmsClient.__init__(self) - ... # Any KMS-specific initialization based on - ... # kms_connection_configuration comes here - ... - ... def wrap_key(self, key_bytes, master_key_identifier): - ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by - ... # master_key_identifier - ... return wrapped_key - ... - ... def unwrap_key(self, wrapped_key, master_key_identifier): - ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by - ... # master_key_identifier - ... return key_bytes - -The concrete implementation will be loaded at runtime by a factory function -provided by the user. This factory function will be used to initialize the -:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption -and decryption properties. - -For example, in order to use the ``MyKmsClient`` defined above: - -.. code-block:: python - - >>> def kms_client_factory(kms_connection_configuration): - ... return MyKmsClient(kms_connection_configuration) - - >>> crypto_factory = pe.CryptoFactory(kms_client_factory) - -An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` -of such a class for an open source -`KMS `_ can be found in the Apache -Arrow GitHub repository. The production KMS client should be designed in -cooperation with an organization's security administrators, and built by -developers with experience in access control management. Once such a class is -created, it can be passed to applications via a factory method and leveraged -by general PyArrow users as shown in the encrypted parquet write/read sample -above. - -KMS connection configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig` -used when creating file encryption and decryption properties) includes the -following options: - -* ``kms_instance_url``, URL of the KMS instance. -* ``kms_instance_id``, ID of the KMS instance that will be used for encryption - (if multiple KMS instances are available). -* ``key_access_token``, authorization token that will be passed to KMS. -* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration. - -Encryption configuration -~~~~~~~~~~~~~~~~~~~~~~~~ - -:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when -creating file encryption properties) includes the following options: - -* ``footer_key``, the ID of the master key for footer encryption/signing. -* ``column_keys``, which columns to encrypt with which key. Dictionary with - master key IDs as the keys, and column name lists as the values, - e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below. -* ``encryption_algorithm``, the Parquet encryption algorithm. - Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``. -* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted). -* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs) - are encrypted with key encryption keys (KEKs), which in turn are encrypted - with master encryption keys (MEKs). If set to ``false``, single wrapping is - used - where DEKs are encrypted directly with MEKs. -* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, - local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. -* ``internal_key_material``, whether to store key material inside Parquet file footers; - this mode doesn’t produce additional files. If set to ``false``, key material is - stored in separate files in the same folder, which enables key rotation for - immutable Parquet files. -* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly - generated by Parquet key management tools. Can be 128, 192 or 256 bits. - -.. note:: - When ``double_wrapping`` is true, Parquet implements a "double envelope - encryption" mode that minimizes the interaction of the program with a KMS - server. In this mode, the DEKs are encrypted with "key encryption keys" - (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master - encryption keys" (MEKs) in the KMS; the result and the KEK itself are - cached in the process memory. - -An example encryption configuration: - -.. code-block:: python - - >>> encryption_config = pe.EncryptionConfiguration( - ... footer_key="footer_key_name", - ... column_keys={ - ... "column_key_name": ["Column1", "Column2"], - ... }, - ... ) - -.. note:: - - Columns with nested fields (struct or map data types) can be encrypted as a whole, or only - individual fields. Configure an encryption key for the root column name to encrypt all nested - fields with this key, or configure a key for individual leaf nested fields. - - Conventionally, the key and value fields of a map column ``m`` have the names - ``m.key_value.key`` and ``m.key_value.value``, respectively. - An inner field ``f`` of a struct column ``s`` has the name ``s.f``. - - With above example, *all* inner fields are encrypted with the same key by configuring that key - for column ``m`` and ``s``, respectively. - -An example encryption configuration for columns with nested fields, where -all columns are encrypted with the same key identified by ``column_key_id``: - -.. code-block:: python - - >>> import pyarrow.parquet.encryption as pe - - schema = pa.schema([ - ("MapColumn", pa.map_(pa.string(), pa.int32())), - ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), - ]) - - encryption_config = pe.EncryptionConfiguration( - footer_key="footer_key_name", - column_keys={ - "column_key_id": [ "MapColumn", "StructColumn" ], - }, - ) - -An example encryption configuration for columns with nested fields, where -some inner fields are encrypted with the same key identified by ``column_key_id``: - -.. code-block:: python - - >>> import pyarrow.parquet.encryption as pe - - >>> schema = pa.schema([ - ... ("MapColumn", pa.map_(pa.string(), pa.int32())), - ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), - ... ]) - - >>> encryption_config = pe.EncryptionConfiguration( - ... footer_key="footer_key_name", - ... column_keys={ - ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ], - ... }, - ... ) - -Decryption configuration -~~~~~~~~~~~~~~~~~~~~~~~~ - -:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating -file decryption properties) is optional and it includes the following options: - -* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local - wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. - - -Content-Defined Chunking ------------------------- - -.. note:: - This feature is experimental and may change in future releases. - -PyArrow introduces an experimental feature for optimizing Parquet files for content -addressable storage (CAS) systems using content-defined chunking (CDC). This feature -enables efficient deduplication of data across files, improving network transfers and -storage efficiency. - -When enabled, data pages are written according to content-defined chunk boundaries, -determined by a rolling hash algorithm that identifies chunk boundaries based on the -actual content of the data. When data in a column is modified (e.g., inserted, deleted, -or updated), this approach minimizes the number of changed data pages. - -The feature can be enabled by setting the ``use_content_defined_chunking`` parameter in -the Parquet writer. It accepts either a boolean or a dictionary for configuration: - -- ``True``: Uses the default configuration with: - - Minimum chunk size: 256 KiB - - Maximum chunk size: 1024 KiB - - Normalization level: 0 - -- ``dict``: Allows customization of the chunking parameters: - - ``min_chunk_size``: Minimum chunk size in bytes (default: 256 KiB). - - ``max_chunk_size``: Maximum chunk size in bytes (default: 1024 KiB). - - ``norm_level``: Normalization level to adjust chunk size distribution (default: 0). - -Note that the chunk size is calculated on the logical values before applying any encoding -or compression. The actual size of the data pages may vary based on the encoding and -compression used. - -.. note:: - To make the most of this feature, you should ensure that Parquet write options - remain consistent across writes and files. - Using different write options (like compression, encoding, or row group size) - for different files may prevent proper deduplication and lead to suboptimal - storage efficiency. - -.. code-block:: python - - >>> table = pa.Table.from_pandas(df) - - >>> # Enable content-defined chunking with default settings - >>> pq.write_table(table, 'example.parquet', use_content_defined_chunking=True) - - >>> # Enable content-defined chunking with custom settings - >>> pq.write_table( - ... table, - ... 'example_custom.parquet', - ... use_content_defined_chunking={ - ... 'min_chunk_size': 128 * 1024, # 128 KiB - ... 'max_chunk_size': 512 * 1024, # 512 KiB - ... } - ... ) diff --git a/docs/source/python/parquet/index.rst b/docs/source/python/parquet/index.rst new file mode 100644 index 000000000000..1da5f8f00ce3 --- /dev/null +++ b/docs/source/python/parquet/index.rst @@ -0,0 +1,89 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet: + +Reading and Writing the Apache Parquet Format +============================================= + +The `Apache Parquet `_ project provides a +standardized open-source columnar storage format for use in data analysis +systems. It was created originally for use in `Apache Hadoop +`_ with systems like `Apache Drill +`_, `Apache Hive `_, `Apache +Impala `_, and `Apache Spark +`_ adopting it as a shared standard for high +performance data IO. + +Apache Arrow is an ideal in-memory transport layer for data that is being read +or written with Parquet files. We have been concurrently developing the `C++ +implementation of +Apache Parquet `_, +which includes a native, multithreaded C++ adapter to and from in-memory Arrow +data. PyArrow includes Python bindings to this code, which thus enables reading +and writing Parquet files with pandas as well. + +Obtaining pyarrow with Parquet Support +-------------------------------------- + +If you installed ``pyarrow`` with pip or conda, it should be built with Parquet +support bundled: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + +If you are building ``pyarrow`` from source, you must use ``-DARROW_PARQUET=ON`` +when compiling the C++ libraries and enable the Parquet extensions when +building ``pyarrow``. If you want to use Parquet Encryption, then you must +use ``-DPARQUET_REQUIRE_ENCRYPTION=ON`` too when compiling the C++ libraries. +See the :ref:`Python Development ` page for more details. + + +Reading and Writing Single Files +-------------------------------- + +.. toctree:: + :maxdepth: 2 + + parquet + +Data Type Handling +------------------ + +.. toctree:: + :maxdepth: 2 + + parquet_type_handling + + +Partitioned Datasets (Multiple Files) +------------------------------------- + +.. toctree:: + :maxdepth: 2 + + parquet_datasets + +Parquet Modular Encryption +-------------------------- + +.. toctree:: + :maxdepth: 2 + + parquet_encryption diff --git a/docs/source/python/parquet/parquet.rst b/docs/source/python/parquet/parquet.rst new file mode 100644 index 000000000000..d3c0ab294479 --- /dev/null +++ b/docs/source/python/parquet/parquet.rst @@ -0,0 +1,605 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_single_file: + + +Reading and Writing Single Files +================================ + +The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` +read and write the :ref:`pyarrow.Table ` object, respectively. + +Let's look at a simple table: + +.. code-block:: python + + >>> import numpy as np + >>> import pandas as pd + >>> import pyarrow as pa + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df) + +We write this to Parquet format with ``write_table``: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> pq.write_table(table, 'example.parquet') + +This creates a single Parquet file. In practice, a Parquet dataset may consist +of many files in many directories. We can read a single file back with +``read_table``: + +.. code-block:: python + + >>> table2 = pq.read_table('example.parquet') + >>> table2.to_pandas() + one two three + a -1.0 foo True + b NaN bar False + c 2.5 baz True + +You can pass a subset of columns to read, which can be much faster than reading +the whole file (due to the columnar layout): + +.. code-block:: python + + >>> pq.read_table('example.parquet', columns=['one', 'three']) + pyarrow.Table + one: double + three: bool + ---- + one: [[-1,null,2.5]] + three: [[true,false,true]] + +When reading a subset of columns from a file that used a Pandas dataframe as the +source, we use ``read_pandas`` to maintain any additional index column data: + +.. code-block:: python + + >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas() + two + a foo + b bar + c baz + +We do not need to use a string to specify the origin of the file. It can be any of: + +* A file path as a string +* A :ref:`NativeFile ` from PyArrow +* A Python file object + +In general, a Python file object will have the worst read performance, while a +string file path or an instance of :class:`~.NativeFile` (especially memory +maps) will perform the best. + +.. _parquet_mmap: + +Reading Parquet and Memory Mapping +---------------------------------- + +Because Parquet data needs to be decoded from the Parquet format +and compression, it can't be directly mapped from disk. +Thus the ``memory_map`` option might perform better on some systems +but won't help much with resident memory consumption. + +.. code-block:: python + + >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + + >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + +If you need to deal with Parquet data bigger than memory, +the :ref:`dataset` and partitioning is probably what you are looking for. + +Parquet file writing options +---------------------------- + +:func:`~pyarrow.parquet.write_table()` has a number of options to +control various settings when writing a Parquet file. + +* ``version``, the Parquet format version to use. ``'1.0'`` ensures + compatibility with older readers, while ``'2.4'`` and greater values + enable more Parquet types and encodings. +* ``data_page_size``, to control the approximate size of encoded data + pages within a column chunk. This currently defaults to 1MB. +* ``max_rows_per_page``, to cap the number of rows per data page within + a column chunk (default 20000). Smaller values reduce memory usage + during reads at the cost of more page metadata. +* ``flavor``, to set compatibility options particular to a Parquet + consumer like ``'spark'`` for Apache Spark. +* ``store_decimal_as_integer``, to store decimals with precision 1–18 + as ``int32`` or ``int64`` instead of ``fixed_len_byte_array``. + This produces more compact files but may not be supported by all readers. +* ``write_time_adjusted_to_utc``, to mark ``TIME`` columns as + adjusted to UTC (``isAdjustedToUTC=True``). When ``False`` (the default), + the time is treated as local/unknown timezone. +* ``write_page_index``, to write statistics to the page index + instead of writing it to each data page header. + Note that PyArrow does not yet use the page index on the read side. +* ``write_page_checksum``, to write a page checksum. Use with + ``page_checksum_verification=True`` on read to detect data corruption. +* ``sorting_columns``, to record the sort order of the data in each + row group's metadata. The writer does not sort the data nor does it verify + that the data is sorted. Readers can use this metadata to optimize queries. + + Sort order is expressed as a sequence of :class:`~pyarrow.parquet.SortingColumn` + objects. + +See the :func:`~pyarrow.parquet.write_table()` docstring for more details. + +There are some additional data type handling-specific options +described below. + +Omitting the DataFrame index +---------------------------- + +When using ``pa.Table.from_pandas`` to convert to an Arrow table, by default +one or more special columns are added to keep track of the index (row +labels). Storing the index takes extra space, so if your index is not valuable, +you may choose to omit it by passing ``preserve_index=False`` + +.. code-block:: python + + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df, preserve_index=False) + +Then we have: + +.. code-block:: python + + >>> pq.write_table(table, 'example_noindex.parquet') + >>> t = pq.read_table('example_noindex.parquet') + >>> t.to_pandas() + one two three + 0 -1.0 foo True + 1 NaN bar False + 2 2.5 baz True + +Here you see the index did not survive the round trip. + +Finer-grained Reading and Writing +--------------------------------- + +``read_table`` uses the :class:`~.ParquetFile` class, which has other features: + +.. code-block:: python + + >>> parquet_file = pq.ParquetFile('example.parquet') + >>> parquet_file.metadata + + created_by: parquet-cpp-arrow version ... + num_columns: 4 + num_rows: 3 + num_row_groups: 1 + format_version: 2.6 + serialized_size: ... + >>> parquet_file.schema + + required group field_id=-1 schema { + optional double field_id=-1 one; + optional binary field_id=-1 two (String); + optional boolean field_id=-1 three; + optional binary field_id=-1 __index_level_0__ (String); + } + + +As you can learn more in the `Apache Parquet format +`_, a Parquet file consists of +multiple row groups. ``read_table`` will read all of the row groups and +concatenate them into a single table. You can read individual row groups with +``read_row_group``: + +.. code-block:: python + + >>> parquet_file.num_row_groups + 1 + >>> parquet_file.read_row_group(0) + pyarrow.Table + one: double + two: large_string + three: bool + __index_level_0__: large_string + ---- + one: [[-1,null,2.5]] + two: [["foo","bar","baz"]] + three: [[true,false,true]] + __index_level_0__: [["a","b","c"]] + +We can similarly write a Parquet file with multiple row groups by using +``ParquetWriter``: + +.. code-block:: python + + >>> with pq.ParquetWriter('example2.parquet', table.schema) as writer: + ... for i in range(3): + ... writer.write_table(table) + >>> pf2 = pq.ParquetFile('example2.parquet') + >>> pf2.num_row_groups + 3 + +For memory-efficient reads of large files, :meth:`~.ParquetFile.iter_batches` +streams the file as a sequence of :class:`~pyarrow.RecordBatch` objects rather +than loading the entire file into a single table: + +.. code-block:: python + + >>> parquet_file = pq.ParquetFile('example.parquet') + >>> for batch in parquet_file.iter_batches(batch_size=2): + ... print(batch.num_rows) + 2 + 1 + +The ``batch_size`` parameter controls the maximum number of rows per +batch. The ``row_groups`` and ``columns`` parameters allow limiting +which row groups and columns are read. + +.. _inspecting_parquet_file_metadata: + +Inspecting the Parquet File Metadata +------------------------------------ + +The ``FileMetaData`` of a Parquet file can be accessed through +:class:`~.ParquetFile` as shown above: + +.. code-block:: python + + >>> parquet_file = pq.ParquetFile('example.parquet') + >>> metadata = parquet_file.metadata + >>> metadata + + created_by: parquet-cpp-arrow version ... + num_columns: 4 + num_rows: 3 + num_row_groups: 1 + format_version: 2.6 + serialized_size: ... + +or can also be read directly using :func:`~parquet.read_metadata`: + +.. code-block:: python + + >>> metadata = pq.read_metadata('example.parquet') + >>> metadata + + created_by: parquet-cpp-arrow version ... + num_columns: 4 + num_rows: 3 + num_row_groups: 1 + format_version: 2.6 + serialized_size: ... + +The returned ``FileMetaData`` object allows to inspect the +`Parquet file metadata `__, +such as the row groups and column chunk metadata and statistics: + +.. code-block:: python + + >>> metadata.row_group(0) + + num_columns: 4 + num_rows: 3 + total_byte_size: 290 + sorting_columns: () + >>> metadata.row_group(0).column(0) + + file_offset: 0 + file_path:... + physical_type: DOUBLE + num_values: 3 + path_in_schema: one + is_stats_set: True + statistics: + + has_min_max: True + min: -1.0 + max: 2.5 + null_count: 1 + distinct_count: None + num_values: 2 + physical_type: DOUBLE + logical_type: None + converted_type (legacy): NONE + geo_statistics: + None + compression: SNAPPY + encodings: ('PLAIN', 'RLE', 'RLE_DICTIONARY') + has_dictionary_page: True + dictionary_page_offset: 4 + data_page_offset: 36 + total_compressed_size: 106 + total_uncompressed_size: 102 + bloom_filter_offset: None + bloom_filter_length: None + +.. _parquet_filtering: + +Filtering / Predicate Pushdown +------------------------------ + +The ``filters`` parameter of :func:`~pyarrow.parquet.read_table` pushes +predicates into the reader, skipping entire row groups based on their +column statistics (min/max values stored in the row group metadata). +For large files with many row groups this can dramatically reduce the +data read. + +Predicates can be expressed as a :class:`~pyarrow.compute.Expression`: + +.. code-block:: python + + >>> import pyarrow.compute as pc + >>> pq.read_table('example.parquet', filters=pc.field('three') == True) + pyarrow.Table + one: double + two: large_string + three: bool + __index_level_0__: large_string + ---- + one: [[-1,2.5]] + two: [["foo","baz"]] + three: [[true,true]] + __index_level_0__: [["a","c"]] + +Or in disjunctive normal form (DNF) as a list of ``(column, operator, +value)`` tuples: + +.. code-block:: python + + >>> pq.read_table('example.parquet', filters=[('three', '==', True)]) + pyarrow.Table + one: double + two: large_string + three: bool + __index_level_0__: large_string + ---- + one: [[-1,2.5]] + two: [["foo","baz"]] + three: [[true,true]] + __index_level_0__: [["a","c"]] + +Supported operators are ``=`` / ``==``, ``!=``, ``<``, ``>``, ``<=``, +``>=``, ``in``, and ``not in``. Multiple predicates in the inner list are +combined as AND; the outer list combines predicate groups as OR: + +.. code-block:: python + + >>> filters = [ + ... [('one', '>', 0), ('three', '==', True)], + ... [('two', 'in', ['foo', 'bar'])], + ... ] + >>> pq.read_table('example.parquet', filters=filters) + pyarrow.Table + one: double + two: large_string + three: bool + __index_level_0__: large_string + ---- + one: [[-1,null,2.5]] + two: [["foo","bar","baz"]] + three: [[true,false,true]] + __index_level_0__: [["a","b","c"]] + +Row group skipping based on column statistics is automatic when filters are +applied. + +.. _parquet_bloom_filters: + +Writing Bloom Filters +--------------------- + +Bloom filters are a probabilistic data structure stored per column per row +group that can answer, for a given value, either that value is +“definitely not present” or “probably present”. This makes them useful for +readers that support Bloom filter-based row group skipping. + +The probability of false positives is configurable. + +.. note:: + + PyArrow reader does not currently use Bloom filters during + filtering on read. + +Bloom filters are enabled per column via the ``bloom_filter_options`` parameter: + +.. code-block:: python + + >>> pq.write_table(table, 'bloom.parquet', + ... bloom_filter_options={'two': True}) + +Passing ``True`` uses the defaults: ``ndv=1048576`` (number of distinct values) +and ``fpp=0.05`` (false-positive probability). + +Both can also be configured explicitly via a dictionary: + +.. code-block:: python + + >>> pq.write_table(table, 'bloom.parquet', + ... bloom_filter_options={ + ... 'two': {'ndv': 100, 'fpp': 0.01}, + ... }) + +Recommended value for ``ndv`` is the number of rows. Lower values of ``fpp`` +reduce false positives but require more space (space grows roughly proportional +to ``log(1/FPP)``). Recommended values are 0.1, 0.05, or 0.01. + +Whether a Bloom filter was written can be confirmed with observing column chunk +metadata, see: :ref:`inspecting_parquet_file_metadata`. + +Multithreaded Reads +------------------- + +Each of the reading functions by default use multi-threading for reading +columns in parallel. Depending on the speed of IO +and how expensive it is to decode the columns in a particular file +(particularly with GZIP compression), this can yield significantly higher data +throughput. + +This can be disabled by specifying ``use_threads=False``. + +.. note:: + The number of threads to use concurrently is automatically inferred by Arrow + and can be inspected using the :func:`~pyarrow.cpu_count()` function. + +Compression, Encoding, and File Compatibility +--------------------------------------------- + +The most commonly used Parquet implementations use dictionary encoding when +writing files; if the dictionaries grow too large, then they "fall back" to +plain encoding. Whether dictionary encoding is used can be toggled using the +``use_dictionary`` option: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', use_dictionary=False) + +The data pages within a column in a row group can be compressed after the +encoding passes (dictionary, RLE encoding). In PyArrow we use Snappy +compression by default, but Brotli, Gzip, ZSTD, LZ4, and uncompressed are +also supported: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', compression='snappy') + >>> pq.write_table(table, 'example.parquet', compression='gzip') + >>> pq.write_table(table, 'example.parquet', compression='brotli') + >>> pq.write_table(table, 'example.parquet', compression='zstd') + >>> pq.write_table(table, 'example.parquet', compression='lz4') + >>> pq.write_table(table, 'example.parquet', compression='none') + +Snappy generally results in better performance, while Gzip may yield smaller +files. + +``'lz4_raw'`` is also accepted as an alias for ``'lz4'``. Both use the +LZ4_RAW codec as defined in the Parquet specification. + +These settings can also be set on a per-column basis: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', compression={'one': 'snappy', 'two': 'gzip'}, + ... use_dictionary=['one', 'two']) + +Using with Spark +---------------- + +Spark places some constraints on the types of Parquet files it will read. The +option ``flavor='spark'`` will set these options automatically and also +sanitize field characters unsupported by Spark SQL. + +Reading from cloud storage +-------------------------- + +In addition to local files, pyarrow supports other filesystems, such as cloud +filesystems, through the ``filesystem`` keyword: + +.. code-block:: python + + >>> from pyarrow import fs + + >>> s3 = fs.S3FileSystem(region="us-east-2") # doctest: +SKIP + >>> table = pq.read_table("bucket/object/key/prefix", filesystem=s3) # doctest: +SKIP + +Currently, :class:`HDFS ` and +:class:`Amazon S3-compatible storage ` are +supported. See the :ref:`filesystem` docs for more details. For those +built-in filesystems, the filesystem can also be inferred from the file path, +if specified as a URI: + +.. code-block:: python + + >>> table = pq.read_table("s3://bucket/object/key/prefix") # doctest: +SKIP + +Other filesystems can still be supported if there is an +`fsspec `__-compatible +implementation available. See :ref:`filesystem-fsspec` for more details. +One example is Azure Blob storage, which can be interfaced through the +`adlfs `__ package. + +.. code-block:: python + + >>> from adlfs import AzureBlobFileSystem # doctest: +SKIP + >>> abfs = AzureBlobFileSystem(account_name="XXXX", account_key="XXXX", container_name="XXXX") # doctest: +SKIP + >>> table = pq.read_table("file.parquet", filesystem=abfs) # doctest: +SKIP + +Content-Defined Chunking +------------------------ + +.. note:: + This feature is experimental and may change in future releases. + +PyArrow introduces an experimental feature for optimizing Parquet files for content +addressable storage (CAS) systems using content-defined chunking (CDC). This feature +enables efficient deduplication of data across files, improving network transfers and +storage efficiency. + +When enabled, data pages are written according to content-defined chunk boundaries, +determined by a rolling hash algorithm that identifies chunk boundaries based on the +actual content of the data. When data in a column is modified (e.g., inserted, deleted, +or updated), this approach minimizes the number of changed data pages. + +The feature can be enabled by setting the ``use_content_defined_chunking`` parameter in +the Parquet writer. It accepts either a boolean or a dictionary for configuration: + +- ``True``: Uses the default configuration with: + - Minimum chunk size: 256 KiB + - Maximum chunk size: 1024 KiB + - Normalization level: 0 + +- ``dict``: Allows customization of the chunking parameters: + - ``min_chunk_size``: Minimum chunk size in bytes (default: 256 KiB). + - ``max_chunk_size``: Maximum chunk size in bytes (default: 1024 KiB). + - ``norm_level``: Normalization level to adjust chunk size distribution (default: 0). + +Note that the chunk size is calculated on the logical values before applying any encoding +or compression. The actual size of the data pages may vary based on the encoding and +compression used. + +.. note:: + To make the most of this feature, you should ensure that Parquet write options + remain consistent across writes and files. + Using different write options (like compression, encoding, or row group size) + for different files may prevent proper deduplication and lead to suboptimal + storage efficiency. + +.. code-block:: python + + >>> table = pa.Table.from_pandas(df) + + >>> # Enable content-defined chunking with default settings + >>> pq.write_table(table, 'example.parquet', use_content_defined_chunking=True) + + >>> # Enable content-defined chunking with custom settings + >>> pq.write_table( + ... table, + ... 'example_custom.parquet', + ... use_content_defined_chunking={ + ... 'min_chunk_size': 128 * 1024, # 128 KiB + ... 'max_chunk_size': 512 * 1024, # 512 KiB + ... } + ... ) diff --git a/docs/source/python/parquet/parquet_datasets.rst b/docs/source/python/parquet/parquet_datasets.rst new file mode 100644 index 000000000000..f7833fe0cb66 --- /dev/null +++ b/docs/source/python/parquet/parquet_datasets.rst @@ -0,0 +1,251 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_datasets: + + +Partitioned Datasets (Multiple Files) +===================================== + +Multiple Parquet files constitute a Parquet *dataset*. These may present in a +number of ways: + +* A list of Parquet absolute file paths +* A directory name containing nested directories defining a partitioned dataset + +A dataset partitioned by year and month may look like on disk: + +.. code-block:: text + + dataset_name/ + year=2007/ + month=01/ + 0.parq + 1.parq + ... + month=02/ + 0.parq + 1.parq + ... + month=03/ + ... + year=2008/ + month=01/ + ... + ... + +Writing to Partitioned Datasets +------------------------------- + +You can write a partitioned dataset for any ``pyarrow`` file system that is a +file-store (e.g. local, HDFS, S3). The default behaviour when no filesystem is +added is to use the local filesystem. + +.. code-block:: python + + >>> import pyarrow as pa + >>> import pyarrow.parquet as pq + + >>> table = pa.table({'one': [-1, None, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}) + ... + + >>> # Local dataset write + >>> pq.write_to_dataset(table, root_path='dataset_name', + ... partition_cols=['one', 'two']) + +The root path in this case specifies the parent directory to which data will be +saved. The partition columns are the column names by which to partition the +dataset. Columns are partitioned in the order they are given. The partition +splits are determined by the unique values in the partition columns. + +To use another filesystem you only need to add the filesystem parameter, the +individual table writes are wrapped using ``with`` statements so the +``pq.write_to_dataset`` function does not need to be. + +.. code-block:: python + + >>> # Remote file-system example + >>> from pyarrow.fs import HadoopFileSystem # doctest: +SKIP + >>> fs = HadoopFileSystem(host, port, user=user, kerb_ticket=ticket_cache_path) # doctest: +SKIP + >>> pq.write_to_dataset(table, root_path='dataset_name', # doctest: +SKIP + ... partition_cols=['one', 'two'], filesystem=fs) + +Compatibility Note: if using ``pq.write_to_dataset`` to create a table that +will then be used by HIVE then partition column values must be compatible with +the allowed character set of the HIVE version you are running. + +Writing ``_metadata`` and ``_common_metadata`` files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some processing frameworks such as Spark or Dask (optionally) use ``_metadata`` +and ``_common_metadata`` files with partitioned datasets. + +Those files include information about the schema of the full dataset (for +``_common_metadata``) and potentially all row group metadata of all files in the +partitioned dataset as well (for ``_metadata``). The actual files are +metadata-only Parquet files. Note this is not a Parquet standard, but a +convention set in practice by those frameworks. + +Using those files can give a more efficient creation of a parquet Dataset, +since it can use the stored schema and file paths of all row groups, +instead of inferring the schema and crawling the directories for all Parquet +files (this is especially the case for filesystems where accessing files +is expensive). + +The :func:`~pyarrow.parquet.write_to_dataset` function does not automatically +write such metadata files, but you can use it to gather the metadata and +combine and write them manually: + +.. code-block:: python + + >>> # Write a dataset and collect metadata information of all written files + >>> metadata_collector = [] + >>> root_path = "dataset_name_1" + >>> pq.write_to_dataset(table, root_path, metadata_collector=metadata_collector) + + >>> # Write the ``_common_metadata`` parquet file without row groups statistics + >>> pq.write_metadata(table.schema, root_path + '/_common_metadata') + + >>> # Write the ``_metadata`` parquet file with row groups statistics of all files + >>> pq.write_metadata( + ... table.schema, root_path + '/_metadata', + ... metadata_collector=metadata_collector + ... ) + +When not using the :func:`~pyarrow.parquet.write_to_dataset` function, but +writing the individual files of the partitioned dataset using +:func:`~pyarrow.parquet.write_table` or :class:`~pyarrow.parquet.ParquetWriter`, +the ``metadata_collector`` keyword can also be used to collect the FileMetaData +of the written files. In this case, you need to ensure to set the file path +contained in the row group metadata yourself before combining the metadata, and +the schemas of all different files and collected FileMetaData objects should be +the same: + +.. code-block:: python + + >>> import os + >>> os.mkdir("year=2017") + + >>> metadata_collector = [] + >>> pq.write_table( + ... table, "year=2017/data1.parquet", + ... metadata_collector=metadata_collector + ... ) + + >>> # set the file path relative to the root of the partitioned dataset + >>> metadata_collector[-1].set_file_path("year=2017/data1.parquet") + + >>> # combine and write the metadata + >>> metadata = metadata_collector[0] + >>> for _meta in metadata_collector[1:]: + ... metadata.append_row_groups(_meta) + >>> metadata.write_metadata_file("_metadata") + + >>> # or use pq.write_metadata to combine and write in a single step + >>> pq.write_metadata( + ... table.schema, "_metadata", + ... metadata_collector=metadata_collector + ... ) + + >>> pq.read_metadata("_metadata") + + created_by: parquet-cpp-arrow version ... + num_columns: 3 + num_rows: 3 + num_row_groups: 1 + format_version: 2.6 + serialized_size: ... + +Reading from Partitioned Datasets +--------------------------------- + +The :class:`~.ParquetDataset` class accepts either a directory name or a list +of file paths, and can discover and infer some common partition structures, +such as those produced by Hive: + +.. code-block:: python + + >>> dataset = pq.ParquetDataset('dataset_name/') + >>> table = dataset.read() + >>> table + pyarrow.Table + three: bool + one: dictionary + two: dictionary + ---- + three: [[true],[true],[false]] + one: [ -- dictionary: + ["-1","2.5"] -- indices: + [0], -- dictionary: + ["-1","2.5"] -- indices: + [1], -- dictionary: + [null] -- indices: + [0]] + two: [ -- dictionary: + ["foo","baz","bar"] -- indices: + [0], -- dictionary: + ["foo","baz","bar"] -- indices: + [1], -- dictionary: + ["foo","baz","bar"] -- indices: + [2]] + +You can also use the convenience function ``read_table`` exposed by +``pyarrow.parquet`` that avoids the need for an additional Dataset object +creation step. + +.. code-block:: python + + >>> table = pq.read_table('dataset_name') + +Note: the partition columns in the original table will have their types +converted to Arrow dictionary types (pandas categorical) on load. Ordering of +partition columns is not preserved through the save/load process. If reading +from a remote filesystem into a pandas dataframe you may need to run +``sort_index`` to maintain row ordering (as long as the ``preserve_index`` +option was enabled on write). + +Other features: + +- Filtering on all columns (using row group statistics) instead of only on + the partition keys. +- Fine-grained partitioning: support for a directory partitioning scheme + in addition to the Hive-like partitioning (e.g. "/2019/11/15/" instead of + "/year=2019/month=11/day=15/"), and the ability to specify a schema for + the partition keys. + +.. note:: + The partition keys need to be explicitly included in the ``columns`` + keyword when you want to include them in the result while reading a + subset of the columns. + +.. note:: + + When passing a single file path to :func:`~pyarrow.parquet.read_table` + or :class:`~pyarrow.parquet.ParquetDataset`, partition columns are not + inferred from the file path, even if the path contains Hive-like segments. + To get partition columns, pass the parent directory instead: + + .. code-block:: python + + >>> # Doesn't include 'year' as a column + >>> pq.read_table('dataset_name/year=2017/data1.parquet') # doctest: +SKIP + + >>> # Includes 'year' as a partition column + >>> pq.read_table('dataset_name/') # doctest: +SKIP diff --git a/docs/source/python/parquet/parquet_encryption.rst b/docs/source/python/parquet/parquet_encryption.rst new file mode 100644 index 000000000000..7e9ad781683b --- /dev/null +++ b/docs/source/python/parquet/parquet_encryption.rst @@ -0,0 +1,278 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: + +.. code-block:: python + + >>> import pyarrow.parquet.encryption as pe + >>> class MyKmsClient(pe.KmsClient): + ... + ... """An example KmsClient implementation skeleton""" + ... def __init__(self, kms_connection_configuration): + ... pe.KmsClient.__init__(self) + ... # Any KMS-specific initialization based on + ... # kms_connection_configuration comes here + ... + ... def wrap_key(self, key_bytes, master_key_identifier): + ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by + ... # master_key_identifier + ... return wrapped_key + ... + ... def unwrap_key(self, wrapped_key, master_key_identifier): + ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by + ... # master_key_identifier + ... return key_bytes + +The concrete implementation will be loaded at runtime by a factory function +provided by the user. This factory function will be used to initialize the +:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption +and decryption properties. + +For example, in order to use the ``MyKmsClient`` defined above: + +.. code-block:: python + + >>> def kms_client_factory(kms_connection_configuration): + ... return MyKmsClient(kms_connection_configuration) + + >>> crypto_factory = pe.CryptoFactory(kms_client_factory) + +An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` +of such a class for an open source +`KMS `_ can be found in the Apache +Arrow GitHub repository. The production KMS client should be designed in +cooperation with an organization's security administrators, and built by +developers with experience in access control management. Once such a class is +created, it can be passed to applications via a factory method and leveraged +by general PyArrow users as shown in the encrypted parquet write/read sample +above. + +KMS connection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig` +used when creating file encryption and decryption properties) includes the +following options: + +* ``kms_instance_url``, URL of the KMS instance. +* ``kms_instance_id``, ID of the KMS instance that will be used for encryption + (if multiple KMS instances are available). +* ``key_access_token``, authorization token that will be passed to KMS. +* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration. + +Encryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when +creating file encryption properties) includes the following options: + +* ``footer_key``, the ID of the master key for footer encryption/signing. +* ``column_keys``, which columns to encrypt with which key. Dictionary with + master key IDs as the keys, and column name lists as the values, + e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below. +* ``uniform_encryption``, whether to encrypt the footer and all columns with + the same ``footer_key``, instead of specifying ``column_keys`` + individually. Cannot be used together with ``column_keys``. +* ``encryption_algorithm``, the Parquet encryption algorithm. + Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``. +* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted). +* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs) + are encrypted with key encryption keys (KEKs), which in turn are encrypted + with master encryption keys (MEKs). If set to ``false``, single wrapping is + used - where DEKs are encrypted directly with MEKs. +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, + local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. +* ``internal_key_material``, whether to store key material inside Parquet file footers; + this mode doesn’t produce additional files. If set to ``false``, key material is + stored in separate files in the same folder, which enables key rotation for + immutable Parquet files. +* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly + generated by Parquet key management tools. Can be 128, 192 or 256 bits. + +.. note:: + When ``double_wrapping`` is true, Parquet implements a "double envelope + encryption" mode that minimizes the interaction of the program with a KMS + server. In this mode, the DEKs are encrypted with "key encryption keys" + (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master + encryption keys" (MEKs) in the KMS; the result and the KEK itself are + cached in the process memory. + +An example encryption configuration: + +.. code-block:: python + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_name": ["Column1", "Column2"], + ... }, + ... ) + +.. note:: + + Columns with nested fields (struct or map data types) can be encrypted as a whole, or only + individual fields. Configure an encryption key for the root column name to encrypt all nested + fields with this key, or configure a key for individual leaf nested fields. + + Conventionally, the key and value fields of a map column ``m`` have the names + ``m.key_value.key`` and ``m.key_value.value``, respectively. + An inner field ``f`` of a struct column ``s`` has the name ``s.f``. + + With above example, *all* inner fields are encrypted with the same key by configuring that key + for column ``m`` and ``s``, respectively. + +An example encryption configuration for columns with nested fields, where +all columns are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + schema = pa.schema([ + ("MapColumn", pa.map_(pa.string(), pa.int32())), + ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ]) + + encryption_config = pe.EncryptionConfiguration( + footer_key="footer_key_name", + column_keys={ + "column_key_id": [ "MapColumn", "StructColumn" ], + }, + ) + +An example encryption configuration for columns with nested fields, where +some inner fields are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ], + ... }, + ... ) + +Decryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating +file decryption properties) is optional and it includes the following options: + +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local + wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. + +External key material and key rotation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``internal_key_material=False`` is set on ``EncryptionConfiguration``, +key material is stored in a separate file next to the Parquet file instead of +in its footer. + +Storing key material externally is what enables key rotation: +``crypto_factory.rotate_master_keys()`` re-wraps the data encryption keys of +a file that uses external key material under new master keys, without rewriting +the file itself: + +.. code-block:: python + + >>> crypto_factory.rotate_master_keys( # doctest: +SKIP + ... kms_connection_config, parquet_file_path="table.parquet", + ... ) + +Direct Key Encryption (without KMS) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:func:`pyarrow.parquet.encryption.create_encryption_properties` and +:func:`pyarrow.parquet.encryption.create_decryption_properties` build +encryption/decryption properties directly from a plaintext key, bypassing +:class:`~pyarrow.parquet.encryption.CryptoFactory` and the KMS-based flow. + +.. note:: + Only uniform encryption (a single key for the footer and all columns) is + supported by these functions. For per-column keys, use the + ``CryptoFactory``/``EncryptionConfiguration`` flow described above. + +.. code-block:: python + + >>> props = pe.create_encryption_properties( # doctest: +SKIP + ... footer_key=b'0123456789abcdef', + ... aad_prefix=b'table_id', + ... store_aad_prefix=False, + ... ) + >>> pq.write_table(table, 'encrypted.parquet', encryption_properties=props) # doctest: +SKIP + + >>> decryption_props = pe.create_decryption_properties( # doctest: +SKIP + ... footer_key=b'0123456789abcdef', + ... aad_prefix=b'table_id', + ... ) + >>> pq.read_table('encrypted.parquet', decryption_properties=decryption_props) # doctest: +SKIP diff --git a/docs/source/python/parquet/parquet_type_handling.rst b/docs/source/python/parquet/parquet_type_handling.rst new file mode 100644 index 000000000000..de36c2cca714 --- /dev/null +++ b/docs/source/python/parquet/parquet_type_handling.rst @@ -0,0 +1,156 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_type_handling: + + +Data Type Handling +================== + +Reading types as DictionaryArray +-------------------------------- + +The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will +cause columns to be read as ``DictionaryArray``, which will become +``pandas.Categorical`` when converted to pandas. This option is only valid for +string and binary column types, and it can yield significantly lower memory use +and improved performance for columns with many repeated string values. + +.. code-block:: python + + >>> import pyarrow as pa + >>> import pyarrow.parquet as pq + + >>> table = pa.table({'one': [-1, None, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}) + ... + >>> pq.write_table(table, 'example.parquet') + + >>> pq.read_table('example.parquet', read_dictionary=['two']) + pyarrow.Table + one: double + two: dictionary + three: bool + ---- + one: [[-1,null,2.5]] + two: [ -- dictionary: + ["foo","bar","baz"] -- indices: + [0,1,2]] + three: [[true,false,true]] + +Reading binary and list columns +------------------------------- + +By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type +and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use +32-bit offsets. For very large datasets this can overflow. The ``binary_type`` +and ``list_type`` parameters let you choose a different Arrow type on read. + +``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or +``pa.binary_view()``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())}) + >>> pq.write_table(table, 'binary.parquet', store_schema=False) + >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema + data: large_binary + +``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``: + +.. code-block:: python + + >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))}) + >>> pq.write_table(table, 'lists.parquet', store_schema=False) + >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema + lists: large_list + child 0, element: int32 + +.. note:: + Both settings are ignored when a serialized Arrow schema is present in the + Parquet file metadata (i.e. when the parquet file was written with + ``store_schema=True``). + +Read in Arrow Extension Types +----------------------------- + +Certain Parquet logical types (JSON, UUID, Geometry, Geography) are supported +and read as Arrow extension types by default (``arrow.json``, ``arrow.uuid`` +and ``geoarrow.wkb`` respectively). This support is enabled via the +``arrow_extensions_enabled`` parameter and is used in :func:`~pyarrow.parquet.read_table`, +:class:`~pyarrow.parquet.ParquetFile`, :class:`~pyarrow.parquet.ParquetDataset`, and +:func:`~pyarrow.parquet.read_schema`. + +To read these Parquet logical types as storage types (default behavior +until PyArrow version ``21.0.0``), set ``arrow_extensions_enabled=False``. + +.. note:: + Reading GEOMETRY/GEOGRAPHY columns as ``geoarrow.wkb`` additionally + requires the ``geoarrow.wkb`` extension type to be registered. For that + you can install Python bindings for GeoArrow + (`geoarrow-pyarrow `_) + and import ``geoarrow.pyarrow`` module. + +Storing timestamps +------------------ + +Some Parquet readers may only support timestamps stored in millisecond +(``'ms'``) or microsecond (``'us'``) resolution. Since pandas uses nanoseconds +to represent timestamps, this can occasionally be a nuisance. By default +(when writing version 1.0 Parquet files), the nanoseconds will be cast to +microseconds ('us'). + +In addition, We provide the ``coerce_timestamps`` option to allow you to select +the desired resolution: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', coerce_timestamps='ms') + +If a cast to a lower resolution value may result in a loss of data, by default +an exception will be raised. This can be suppressed by passing +``allow_truncated_timestamps=True``: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', coerce_timestamps='ms', + ... allow_truncated_timestamps=True) + +Timestamps with nanoseconds can be stored without casting when using the +more recent Parquet format version 2.6: + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', version='2.6') + +However, many Parquet readers do not yet support this newer format version, and +therefore the default is to write version 1.0 files. When compatibility across +different processing frameworks is required, it is recommended to use the +default version 1.0. + +Older Parquet implementations use ``INT96`` based storage of +timestamps, but this is now deprecated. This includes some older +versions of Apache Impala and Apache Spark. To write timestamps in +this format, set the ``use_deprecated_int96_timestamps`` option to +``True`` in ``write_table``. + +.. code-block:: python + + >>> pq.write_table(table, 'example.parquet', use_deprecated_int96_timestamps=True)