ArrowDataset
ArrowDataset
背景
-
- 高效的多文件、列式数据格式.
- 由 Apache 软件基金会开发.
Import 与 Export
- Import["dir","ArrowDataset"] 将 ArrowDataset 目录导入为 Tabular 对象.
- Import["dir",{"ArrowDataset",elem,…}] 导入指定的元素.
- Import["dir",{"ArrowDataset",elem,subelem1,…}] 导入子元素 subelemi，适用于部分数据导入.
- Export["dir",expr,"ArrowDataset"] 从 expr 创建一个 ArrowDataset 目录.
- 支持的表达式 expr 包括：
-
{v1,v2,…} 单列数据 {{v11,v12,…},{v21,v22,…},…} 数据行列表 array 数组，例如 SparseArray、QuantityArray 等 tseries TimeSeries、EventSeries 或 TemporalData 对象 dataset Dataset 或 Tabular 对象 - 有关完整的通用信息，请参阅以下参考页面：
-
Import, Export 从文件导入或导出至文件 CloudImport, CloudExport 从云对象导入或导出至云对象 ImportString, ExportString 从字符串导入或导出为字符串 ImportByteArray, ExportByteArray 从字节数组导入或导出为字节数组
参数
- Import 的通用参数：
-
"Elements" 此文件中可用的元素和选项列表 "Summary" 文件摘要 "Rules" 所有可用元素的规则列表 - 数据表示元素：
-
"Data" 二维数组 "Dataset" 作为 Dataset 的表格数据 "EventSeries" 作为 EventSeries 的表格数据 "Tabular" Tabular 对象 "TimeSeries" 作为 TimeSeries 的表格数据 - 其他元素可以根据 "Format" 选项指定. 有关元素的详细描述，请参阅 "Parquet"、"ArrowIPC"、"ORC"、"CSV" 或 "TSV".
- Import 默认使用 "Tabular" 元素.
- "Tabular" 元素的部分数据导入的子元素可以采用 {"Tabular",rows,cols} 形式的行和列规范，其中 rows 和 cols 可以是以下任一种：
-
n 第 n 行或列 -n 从最后开始计数 n;;m 从 n 到 m n;;m;;s 从 n 到 m，步长为 s {n1,n2,…} 特定行或列 ni - 列规范也可以是以下任意形式：
-
"col" 单列 "col" {col1,col2,…} 列名列表 coli - 数据描述符元素：
-
"ColumnLabels" 列名 "ColumnTypes" 与每列数据类型的关联 "Schema" TabularSchema 对象
选项
- Import 的通用选项：
-
"Format" Automatic 要使用的底层格式 "Partitioning" None 分区方案 - Export 的通用选项：
-
"Format" "Parquet" 要使用的底层格式 "MaxPartitions" 4096 分区的最大数量 "MaxRowsPerFile" Infinity 每个文件的最大行数 "NameTemplate" "part{i}" 文件名模板 "Partitioning" "Hive" 分区方案 "SplitColumns" Automatic 用于分区的列 - Import 支持以下 "Partitioning" 设置：
-
None 无分区 "Hive" Hive 分区 {col1,col2,…} 使用分区键进行目录分区 {"Directory", {col1,col2,…}} 使用分区键进行目录分区 - Export 支持以下 "Partitioning" 设置：
-
"Directory" 目录分区 "Hive" Hive 分区 - 其他选项可以根据 "Format" 选项指定. 有关选项的详细描述，请参阅。有关详细选项描述，请参阅 "Parquet"、"ArrowIPC"、"ORC"、"CSV" 或 "TSV".
范例
打开所有单元 关闭所有单元基本范例 (2)
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];tabular["ColumnKeys"]Export["tabular_dataset", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", OverwriteTarget -> True]Import["tabular_dataset", "ArrowDataset", "Partitioning" -> "Hive"]范围 (3)
导入 (3)
Import["tabular_dataset", {"ArrowDataset", "Elements"}]默认情况下，返回 Tabular 对象：
Import["tabular_dataset", "ArrowDataset", "Partitioning" -> "Hive"]//TabularQImport["tabular_dataset", {"ArrowDataset", "ColumnTypes"}, "Partitioning" -> "Hive"]导入元素 (19)
"ColumnCount" (1)
"ColumnLabels" (1)
"ColumnTypes" (1)
"Data" (3)
Import["tabular_dataset", {"ArrowDataset", "Data"}, "Partitioning" -> "Hive"]//ShallowImport["tabular_dataset", {"ArrowDataset", "Data", 1 ;; 3}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Data", All, {1, 3}}, "Partitioning" -> "Hive"]//ShallowImport["tabular_dataset", {"ArrowDataset", "Data", All, {"Tree Species", "Diameter at Breast Height (in Feet)"}}, "Partitioning" -> "Hive"]//Shallow"Dataset" (3)
以 Dataset 形式获取数据：
Import["tabular_dataset", {"ArrowDataset", "Dataset"}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Dataset", 1 ;; 3}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Dataset", All, {1, 3}}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Dataset", All, {"Tree Species", "Diameter at Breast Height (in Feet)"}}, "Partitioning" -> "Hive"]"Dimensions" (1)
"EventSeries" (1)
将 Tabular 对象导出为 ArrowDataset：
Export["sales_dataset", ResourceData["Sample Tabular Data: Sales Data"], "ArrowDataset", "SplitColumns" -> "Product", OverwriteTarget -> True]将 ArrowDataset 导入为 EventSeries：
Import["sales_dataset", {"ArrowDataset", "EventSeries"}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "EventSeries", 5}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "EventSeries", {1, 5, 7}}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "EventSeries", 1 ;; 10}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "EventSeries", All, {"Product", "Date", "Quantity"}}, "Partitioning" -> "Hive"]"MetaInformation" (1)
"RowCount" (1)
"Schema" (1)
获取 TabularSchema 对象：
Import["tabular_dataset", {"ArrowDataset", "Schema"}, "Partitioning" -> "Hive"]"Summary" (1)
"Tabular" (3)
以 Tabular 对象的形式从文件中获取数据：
Import["tabular_dataset", {"ArrowDataset", "Tabular"}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Tabular", 1 ;; 5}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Tabular", All, {1, 3}}, "Partitioning" -> "Hive"]Import["tabular_dataset", {"ArrowDataset", "Tabular", All, {"Tree Species", "Diameter at Breast Height (in Feet)"}}, "Partitioning" -> "Hive"]"TimeSeries" (1)
将 Tabular 对象导出为 ArrowDataset：
Export["sales_dataset", ResourceData["Sample Tabular Data: Sales Data"], "ArrowDataset", "SplitColumns" -> "Product", OverwriteTarget -> True]将 ArrowDataset 导入为 TimeSeries：
Import["sales_dataset", {"ArrowDataset", "TimeSeries"}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "TimeSeries", 5}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "TimeSeries", {1, 5, 7}}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "TimeSeries", 1 ;; 10}, "Partitioning" -> "Hive"]Import["sales_dataset", {"ArrowDataset", "TimeSeries", All, {"Product", "Date", "Quantity"}}, "Partitioning" -> "Hive"]导入的选项 (3)
"Format" (1)
"Partitioning" (1)
默认情况下，使用 "Partitioning"None. 注意，用于分区的列未被导入：
Import["tabular_dataset", "ArrowDataset"]//Dimensions使用具有正确设置的 "Partitioning" 选项来获取所有列：
Import["tabular_dataset", "ArrowDataset", "Partitioning" -> "Hive"]//Dimensions"TimeColumn" (1)
将 Tabular 对象导出为 ArrowDataset：
Export["date_dataset", Tabular[Association["RawSchema" -> Association["ColumnProperties" ->
Association["Date" -> Association["ElementType" -> TypeSpecifier["Date"]["Integer32", "Day",
"Gregorian", None]], "Value" -> Association["ElementType" -> "Real32"]],
"KeyColumns" -> None, "Backend" -> "WolframKernel"], "Options" -> {},
"BackendData" -> Association["ColumnData" -> DataStructure["ColumnTable",
{{TabularColumn[Association["Data" -> {5, {{NumericArray[{13150, 13151, 13152, 13153, 13156},
"Integer32"], {}, None}}, None}, "ElementType" -> "Date"["Integer32", "Day",
"Gregorian", None]]], TabularColumn[Association[
"Data" -> {NumericArray[{11.819999694824219, 12.039999961853027, 12.09000015258789,
11.880000114440918, 12.430000305175781}, "Real32"], {}, None},
"ElementType" -> "Real32"]]}}]]]], "ArrowDataset", "SplitColumns" -> "Date", OverwriteTarget -> True]默认情况下，对于 "TimeSeries" 和 "EventSeries" 元素，时间列会自动选择：
Import["date_dataset", {"ArrowDataset", "TimeSeries"}, "Partitioning" -> "Hive"]Import["date_dataset", {"ArrowDataset", "TimeSeries"}, "Partitioning" -> "Hive", "TimeColumn" -> "Value"]导出的选项 (6)
"Format" (1)
FileSystemMap[FileSize, "tabular_dataset", Infinity]tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset_arrowipc", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", "Format" -> "ArrowIPC"]FileSystemMap[FileSize, "tabular_dataset_arrowipc", Infinity]"MaxPartitions" (1)
当拆分列中唯一元素的数量大于 "MaxPartitions" 选项的默认值时，Export 将失败：
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset_location", tabular, "ArrowDataset", "SplitColumns" -> "Location"]
Export["tabular_dataset_location", tabular, "ArrowDataset", "SplitColumns" -> "Location", "MaxPartitions" -> 27000, OverwriteTarget -> True]"MaxRowsPerFile" (1)
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", OverwriteTarget -> True]FileNames["*.*", "tabular_dataset", Infinity]//LengthExport["tabular_dataset_limit", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", "MaxRowsPerFile" -> 800, OverwriteTarget -> True]FileNames["*.*", "tabular_dataset_limit", Infinity]//Length"NameTemplate" (1)
默认情况下，"part{i}" 用作 ArrowDataset 文件的名称模板：
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset_limit", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", "MaxRowsPerFile" -> 5000, OverwriteTarget -> True]FileSystemMap[FileSize, "tabular_dataset_limit", Infinity]Export["tabular_dataset_myname", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", "MaxRowsPerFile" -> 5000, "NameTemplate" -> "myname{i}", OverwriteTarget -> True]FileSystemMap[FileSize, "tabular_dataset_myname", Infinity]"Partitioning" (1)
默认情况下，Export 使用 "Hive" 分区：
FileSystemMap[FileSize, "tabular_dataset", Infinity]tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset_directory", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", "Partitioning" -> "Directory"]FileSystemMap[FileSize, "tabular_dataset_directory", Infinity]"SplitColumns" (1)
Export 需要 "SplitColumns" 选项：
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset", tabular, "ArrowDataset", OverwriteTarget -> True]
只有 Tabular 对象的列键才可以作为 "SplitColumns" 选项的值：
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];tabular["ColumnKeys"]Export["tabular_dataset_split", tabular, "ArrowDataset", "SplitColumns" -> {"a", "b"}, OverwriteTarget -> True]
Export["tabular_dataset_split", tabular, "ArrowDataset", "SplitColumns" -> {"Number of Trunks", "Location Type"}, OverwriteTarget -> True]可能存在的问题 (1)
Export 需要 "SplitColumns" 选项：
tabular = Import["ExampleData/TreesOwnedByTheCityOfChampaign.csv", "Tabular"];Export["tabular_dataset", tabular, "ArrowDataset", OverwriteTarget -> True]
Export["tabular_dataset", tabular, "ArrowDataset", "SplitColumns" -> "Number of Trunks", OverwriteTarget -> True]