The matrix-arff module provides reading and writing of ARFF (Attribute-Relation File Format) files. ARFF is a standard format used by machine learning tools, particularly Weka, making this module essential for ML workflows and data interchange between different ML tools.
ARFF (Attribute-Relation File Format) is a text-based format developed for the Weka machine learning toolkit. It's widely used in the machine learning community because it:
- Explicitly defines data types for each column
- Supports both numeric and categorical (nominal) data
- Includes metadata about the dataset (relation name, attribute definitions)
- Is human-readable and easy to inspect
An ARFF file consists of three main sections:
- Header comments - Lines starting with
%containing metadata - Relation and attribute declarations -
@RELATIONand@ATTRIBUTEdirectives - Data section -
@DATAfollowed by comma-separated values
To use the matrix-arff module, add it as a dependency to your project.
implementation 'org.apache.groovy:groovy:6.0.0'
implementation "se.alipsa.matrix:matrix-core:3.9.0"
implementation "se.alipsa.matrix:matrix-arff:0.3.0"<dependencies>
<dependency>
<groupId>org.apache.groovy</groupId>
<artifactId>groovy</artifactId>
<version>6.0.0</version>
</dependency>
<dependency>
<groupId>se.alipsa.matrix</groupId>
<artifactId>matrix-core</artifactId>
<version>3.7.1</version>
</dependency>
<dependency>
<groupId>se.alipsa.matrix</groupId>
<artifactId>matrix-arff</artifactId>
<version>0.3.0</version>
</dependency>
</dependencies>The MatrixArffReader class provides several methods to read ARFF files from different sources.
import se.alipsa.matrix.arff.ArffReadOptions
import se.alipsa.matrix.arff.ArffDateMode
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.core.Matrix
ArffReadOptions readOptions = new ArffReadOptions()
.fallbackMatrixName('fallback-name')
.instanceWeightColumn('weight')
.dateMode(ArffDateMode.WEKA)
Matrix iris = MatrixArffReader.read(new File("iris.arff"), readOptions)
Matrix fromPath = MatrixArffReader.read(Paths.get("data/dataset.arff"), readOptions)
Matrix fromString = MatrixArffReader.readString(arffContent, readOptions)
println "Dataset: ${iris.matrixName}"
println "Rows: ${iris.rowCount()}"
println "Columns: ${iris.columnNames()}"Output:
Dataset: iris
Rows: 150
Columns: [sepallength, sepalwidth, petallength, petalwidth, class]
import se.alipsa.matrix.arff.MatrixArffReader
import java.nio.file.Paths
// Read from a Path without typed options
Matrix data1 = MatrixArffReader.read(Paths.get("data/dataset.arff"))
// Read from a file path string
Matrix data1b = MatrixArffReader.readFile("data/dataset.arff")
// Read from a URL
Matrix data2 = MatrixArffReader.read(new URL("https://example.com/dataset.arff"))
// Read from a URL string
Matrix data2b = MatrixArffReader.readUrl("https://example.com/dataset.arff")
// Read from an InputStream (useful for classpath resources)
InputStream stream = getClass().getResourceAsStream("/data/iris.arff")
Matrix data3 = MatrixArffReader.read(stream, "iris")
// Read from a String containing ARFF content
String arffContent = "... ARFF ..."
Matrix data4 = MatrixArffReader.readString(arffContent)Sparse ARFF data rows are also supported:
@RELATION sparse_metrics
@ATTRIBUTE score NUMERIC
@ATTRIBUTE note STRING
@ATTRIBUTE status {ok,warning,error}
@DATA
{0 1.5,2 ok}
{1 'late sample',2 warning,0 ?}
{}
Matrix sparse = MatrixArffReader.readString(arffContent)
assert sparse[0, 'score'] == 1.5
assert sparse[0, 'note'] == 'late sample'
assert sparse[0, 'status'] == 'ok'
assert sparse[2, 'score'] == 0Supported sparse semantics in matrix-arff (the ARFF specification as implemented by Weka; before 0.3.0 omitted
attributes were read as null):
- an attribute that is omitted from a sparse row has the value
0:0for NUMERIC/INTEGER, the first declared value for nominal attributes and1970-01-01T00:00:00Zfor DATE - an omitted STRING attribute takes the first explicit value that appears in that column anywhere in the data (Weka's
string dictionary index 0); when the column has no explicit value at all the cell is
null(Weka has no value there either) unlessArffReadOptions.omittedStringFallbackis set —omittedStringFallback('0')gives the raw0Weka holds internally and the"0"liac-arff returns - a value that is missing must be written explicitly as
?(for example{0 1.5, 2 ?}) and reads asnull - quoted string and nominal values are supported
- duplicate or out-of-range attribute indices are rejected with an
IllegalArgumentException
The reader stays lenient by default so existing imports keep working:
- unknown attribute types fall back to
STRING - dense rows with missing trailing values are padded with
null - dense rows with extra trailing values ignore the extras
When you want stricter schema validation, use ArffReadOptions:
import se.alipsa.matrix.arff.ArffReadOptions
import se.alipsa.matrix.arff.MatrixArffReader
ArffReadOptions options = new ArffReadOptions()
.strict(true)
.fallbackMatrixName('fallback-name')
Matrix strictMatrix = MatrixArffReader.read(new File('dataset.arff'), options)Useful validation flags:
strict(value)enables strict unknown-type, dense-row-length and undeclared-nominal validation; default:falsefailOnUnknownAttributeType(value)rejects unsupported@ATTRIBUTEtypes instead of treating them asSTRING; by default it followsstrictfailOnRowLengthMismatch(value)rejects dense rows whose value count does not match the declared schema; by default it followsstrictfailOnUndeclaredNominalValue(value)rejects nominal values missing from the attribute declaration; by default it followsstrictfallbackMatrixName(name)supplies the matrix name when no@RELATIONexists; default: the source nameomittedStringFallback(value)sets a sparse-omitted STRING value only when the attribute has no dictionary entry; default:null('0'reproduces Weka's raw value and liac-arff's string)instanceWeightColumn(name)keeps{w}instance weights in aBigDecimalcolumn, using1when a row omits its weight; default:null, which parses and discards weightsdateMode(ArffDateMode.UTC)parses strictly withLocale.ROOTin UTC (default);dateMode(ArffDateMode.WEKA)uses the JVM's locale and zone and ignores trailing date text like Weka
Syntax errors such as unterminated quoted values are always rejected, and the parser now includes the offending line number and line content in those exceptions.
Since 0.3.0, matrix-arff supports the ARFF features used by Weka: comments and escaping, numeric, nominal, STRING,
DATE and relational attributes, comma-, tab-, or space-delimited dense rows, sparse rows, missing values and instance
weights. Use ArffDateMode.WEKA on both read and write options when files must produce the same Date instants as Weka
on the same machine. The default UTC mode is machine-independent instead.
Important interoperability limitations:
- sparse input is always written back as dense rows
- UTC date mode requires the entire value to match and uses
Locale.ROOT; Weka uses the JVM defaults and accepts trailing date text. Weka mode opts into those Weka rules - trailing text after a DATE format declaration is rejected even though Weka ignores it
\a,\b,\f,\vand octal escapes are not decoded exactly like Weka'sStreamTokenizer; every escape Weka's writer emits is supported- empty comma fields read as missing, while Weka rejects them
- INTEGER values use
Integerby default; selectintegerType(Long)for large integers orintegerType(BigDecimal)for fractional input. Weka stores every numeric type asdouble - a sparse-omitted STRING takes dictionary value 0, which can be the first explicit value from another row. Use an
explicit
?for missing data and consider a deliberate dummy dictionary value - sparse-omitted STRING values inside relational cells are resolved per nested
Matrix; Weka appears to share the dictionary across all cells of the relational attribute - undeclared nominal values are rejected only when strict or explicit nominal validation is enabled
- weights are retained only when
instanceWeightColumnis configured; Weka attribute weights are parsed and discarded
| Option | Applies to | Default | Accepted values |
|---|---|---|---|
charset |
Read/write File, Path, URL input and byte streams | UTF-8 | A Charset (encoding-capable for writes), or its name in SPI maps; null resets to UTF-8 |
integerType |
Read INTEGER attributes, including nested and sparse values | Integer |
Integer, Long, BigDecimal; SPI also accepts case-insensitive simple or fully-qualified names; null resets to Integer |
Byte input is decoded strictly. Select the source encoding explicitly when it is not UTF-8; malformed bytes produce
an error mentioning charset. A caller-supplied Reader/Writer or String entry point owns its encoding and skips the
byte-encoding checks. Files and streams validate every emitted declaration and relevant cell before writing bytes.
A genuine I/O error during printing (for example a full disk) can still leave a partial file and is reported to the
caller. A supplied PrintWriter is flushed and checked before and after writing; an existing error state is rejected.
To exchange a Latin-1 file with large INTEGER values, create the Matrix, write it, and read it with matching options:
import se.alipsa.matrix.core.Matrix
import se.alipsa.matrix.arff.ArffReadOptions
import se.alipsa.matrix.arff.ArffWriteOptions
import se.alipsa.matrix.arff.ArffTypeDecl
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.arff.MatrixArffWriter
import java.nio.charset.StandardCharsets
Matrix data = Matrix.builder('cities')
.data(id: [3000000000L], city: ['Malmö'])
.types(Long, String).build()
File file = new File('cities.arff')
MatrixArffWriter.write(data, file, new ArffWriteOptions()
.charset(StandardCharsets.ISO_8859_1)
.attributeTypesByColumn([id: ArffTypeDecl.INTEGER]))
Matrix restored = MatrixArffReader.read(file, new ArffReadOptions()
.charset(StandardCharsets.ISO_8859_1).integerType(Long))
assert restored[0, 'id'] == 3000000000L
assert restored[0, 'city'] == 'Malmö'
// The same options through the Matrix SPI (map first):
Matrix fromSpi = Matrix.read([charset: 'ISO-8859-1', integerType: 'Long'], file)
data.write([charset: 'ISO-8859-1', attributeTypesByColumn: [id: 'INTEGER']], file)integerType(BigDecimal) also accepts fractional INTEGER input. With the default Integer, values outside the
int range or with a fractional part fail; values written outside that range require Long or BigDecimal to read
back. NaN and infinite numeric cells continue to write as ?.
- Header matching is locale-independent. An initial BOM is ignored. Missing
@DATA, unknown header keywords and stray pre-data lines fail even in lenient mode; empty/comment-only input still yields an empty Matrix. Keywords accept whitespace, quotes or{as boundaries, so@relationshipis rejected. - Nominal declarations split on whitespace and commas:
{a b}declares two values,{a,b,}declares two,{'',a}preserves an empty value, and{a,a}fails with line context.@attribute c{a,b}is accepted. - Writer validation rejects duplicate column names, out-of-mapping nominal cells, invalid numeric cells, fractional
INTEGER cells and DATE strings that do not fully match the selected format, including WEKA mode. Parseable numeric
and date strings are still accepted. Numeric strings must use ASCII without surrounding whitespace:
'', blank strings,'?',' 1 'and non-ASCII digits are errors. Usenullfor missing numeric values. - Boolean columns default to nominal
{false,true}and read back as'false'/'true'; explicit type andstringColumnsoverrides still apply. Explicit Boolean nominal columns need no mappings. - Lenient dense row mismatches still pad/drop values and now produce one warning per read, including nested rows.
Set
failOnRowLengthMismatch(true)to fail instead. - File validation failures preserve existing contents; I/O errors propagate. Encoding validation includes nested names, declarations of all-null DATE columns, and nested cell values. Unpaired surrogates are rejected on byte output.
- Path overloads support NIO providers such as zip filesystems and null output paths fail with
IllegalArgumentException.
ArffWriteOptions is the primary configuration surface for direct writes:
import se.alipsa.matrix.arff.ArffTypeDecl
import se.alipsa.matrix.arff.ArffDateMode
import se.alipsa.matrix.arff.ArffWriteOptions
import se.alipsa.matrix.arff.MatrixArffWriter
ArffWriteOptions writeOptions = new ArffWriteOptions()
.inferNominals(false)
.attributeTypesByColumn([category: ArffTypeDecl.STRING])
.dateFormatsByColumn([created: 'yyyy-MM-dd'])
.instanceWeightColumn('weight')
.dateMode(ArffDateMode.WEKA)
MatrixArffWriter.write(matrix, new File('configured.arff'), writeOptions)
String arff = MatrixArffWriter.writeString(matrix, writeOptions)Convenience overloads without ArffWriteOptions still exist for default behavior.
Write-option parameters and defaults:
inferNominals(value): infer nominal String/Object columns; defaulttruenominalThreshold(n): maximum distinct values for nominal inference; default50nominalColumns(names): columns forced to nominal; default emptystringColumns(names): columns forced to STRING; default emptyattributeTypesByColumn(types): per-columnSTRING,NOMINAL,DATE,NUMERIC,REAL,INTEGERorRELATIONAL; default emptynominalMappings(values): ordered nominal declarations by column; default emptydateFormat(pattern): global DATE format; defaultyyyy-MM-dd'T'HH:mm:ssdateFormatsByColumn(patterns): per-column DATE formats; default emptyinstanceWeightColumn(name): numeric column emitted as{w}at every relational depth; defaultnulldateMode(mode):UTCfor machine-independent output (default), orWEKAfor the JVM locale and zone
A Matrix-typed column is written as an ARFF relational attribute. Every non-null nested matrix in that column must
have identical column names and types. The following example writes two bags, reads them back, and accesses a nested
value end to end:
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.arff.MatrixArffWriter
import se.alipsa.matrix.core.Matrix
Matrix bagA = Matrix.builder('bag')
.columns(item: [1, 2], label: ['a', 'b'])
.types([Integer, String])
.build()
Matrix bagB = Matrix.builder('bag')
.columns(item: [3], label: ['c'])
.types([Integer, String])
.build()
Matrix source = Matrix.builder('bags')
.columns(id: [10, 20], observations: [bagA, bagB])
.types([Integer, Matrix])
.build()
String arff = MatrixArffWriter.writeString(source)
Matrix restored = MatrixArffReader.readString(arff)
Matrix firstBag = restored[0, 'observations'] as Matrix
assert firstBag.columnNames() == ['item', 'label']
assert firstBag[1, 'label'] == 'b'String sparseArff = """
@RELATION sparse_metrics
@ATTRIBUTE score NUMERIC
@ATTRIBUTE note STRING
@ATTRIBUTE status {ok,warning,error}
@DATA
{0 1.5,2 ok}
{1 'late sample',2 warning,0 ?}
{}
""".trim()
Matrix sparse = MatrixArffReader.readString(sparseArff)
assert sparse[0, 'score'] == 1.5
assert sparse[0, 'note'] == 'late sample'
assert sparse[1, 'status'] == 'warning'
assert sparse[2, 'score'] == 0import se.alipsa.matrix.arff.ArffTypeDecl
import se.alipsa.matrix.arff.ArffWriteOptions
import se.alipsa.matrix.arff.MatrixArffWriter
ArffWriteOptions options = new ArffWriteOptions()
.inferNominals(false)
.attributeTypesByColumn([
created : ArffTypeDecl.DATE,
category: ArffTypeDecl.STRING
])
.dateFormatsByColumn([created: 'yyyy-MM-dd'])
MatrixArffWriter.write(matrix, new File('configured.arff'), options)import se.alipsa.matrix.arff.ArffReadOptions
import se.alipsa.matrix.arff.MatrixArffReader
Matrix validated = MatrixArffReader.read(
new File('incoming.arff'),
new ArffReadOptions()
.strict(true)
.fallbackMatrixName('fallback-name')
)Strict mode is useful when you want malformed rows or unsupported attribute declarations to fail immediately instead of being interpreted leniently.
import se.alipsa.matrix.core.Matrix
Matrix source = Matrix.read([fallbackMatrixName: 'fallback'], new File('input.arff'))
source.write([
inferNominals : false,
attributeTypesByColumn: [category: 'STRING']
], new File('output.arff'))
println Matrix.listReadOptions('arff')
println Matrix.listWriteOptions('arff')This SPI path uses the same typed ARFF read/write behavior as the direct MatrixArffReader and MatrixArffWriter APIs.
import se.alipsa.matrix.arff.MatrixArffReader
Matrix iris = MatrixArffReader.read(new File("iris.arff"))
// Check column types
iris.columnNames().each { colName ->
println "${colName}: ${iris.type(colName)}"
}
// View first few rows
println iris.head(5)Output:
sepallength: class java.math.BigDecimal
sepalwidth: class java.math.BigDecimal
petallength: class java.math.BigDecimal
petalwidth: class java.math.BigDecimal
class: class java.lang.String
Matrix (iris, 5 x 5)
sepallength sepalwidth petallength petalwidth class
5.1 3.5 1.4 0.2 Iris-setosa
4.9 3.0 1.4 0.2 Iris-setosa
4.7 3.2 1.3 0.2 Iris-setosa
4.6 3.1 1.5 0.2 Iris-setosa
5.0 3.6 1.4 0.2 Iris-setosa
The ARFF format supports several attribute types, which are automatically converted to appropriate Java/Groovy types:
| ARFF Type | Description | Java Type |
|---|---|---|
NUMERIC |
Floating-point numbers | BigDecimal |
REAL |
Floating-point numbers | BigDecimal |
INTEGER |
Whole numbers (fractional input with integerType(BigDecimal)) |
Integer (default), Long, or BigDecimal |
STRING |
Text strings | String |
DATE |
Date/time values | Date |
RELATIONAL |
Nested ARFF rows | Matrix |
{val1,val2,...} |
Nominal (categorical) | String |
Note: When writing a Matrix, Long and BigInteger columns are stored as NUMERIC and are read back as BigDecimal.
@RELATION employees
@ATTRIBUTE name STRING
@ATTRIBUTE age INTEGER
@ATTRIBUTE salary REAL
@ATTRIBUTE hired DATE 'yyyy-MM-dd'
@ATTRIBUTE department {Engineering,Sales,HR,Marketing}
@ATTRIBUTE active {true,false}
@DATA
'John Doe',35,75000.50,'2020-01-15',Engineering,true
'Jane Smith',28,65000.00,'2021-06-01',Sales,true
'Bob Wilson',42,90000.00,'2015-03-20',Engineering,false
import se.alipsa.matrix.arff.MatrixArffReader
// Assuming the above ARFF content is in employees.arff
Matrix employees = MatrixArffReader.read(new File("employees.arff"))
// Access data with proper types
String name = employees[0, 'name'] // "John Doe"
Integer age = employees[0, 'age'] // 35
BigDecimal salary = employees[0, 'salary'] // 75000.50
Date hired = employees[0, 'hired'] // Date object
String dept = employees[0, 'department'] // "Engineering"
// Filter by department
def engineers = employees.subset { row ->
row['department'] == 'Engineering'
}
println "Engineers: ${engineers.rowCount()}"The MatrixArffWriter class converts Matrix objects to ARFF format.
import se.alipsa.matrix.arff.MatrixArffWriter
import se.alipsa.matrix.core.Matrix
// Create a Matrix
Matrix employees = Matrix.builder('employees')
.data(
name: ['Alice', 'Bob', 'Charlie'],
age: [25, 30, 35],
salary: [50000.0, 60000.0, 75000.0],
department: ['Engineering', 'Sales', 'Engineering']
)
.types([String, Integer, BigDecimal, String])
.build()
// Write to file
MatrixArffWriter.write(employees, new File("employees.arff"))The generated ARFF file will look like:
@RELATION employees
@ATTRIBUTE name STRING
@ATTRIBUTE age INTEGER
@ATTRIBUTE salary NUMERIC
@ATTRIBUTE department {Engineering,Sales}
@DATA
'Alice',25,50000.0,Engineering
'Bob',30,60000.0,Sales
'Charlie',35,75000.0,Engineering
import se.alipsa.matrix.arff.MatrixArffWriter
import java.nio.file.Paths
// Write to a Path
MatrixArffWriter.write(matrix, Paths.get("output/data.arff"))
// Write to an OutputStream
FileOutputStream fos = new FileOutputStream("output.arff")
MatrixArffWriter.write(matrix, fos)
fos.close()
// Write to a Writer (for custom handling)
StringWriter sw = new StringWriter()
MatrixArffWriter.write(matrix, sw)
String arffContent = sw.toString()Null values in the Matrix are automatically converted to the ARFF missing value indicator ?:
Matrix data = Matrix.builder('data')
.data(
name: ['Alice', 'Bob', null],
age: [25, null, 35],
salary: [50000.0, 60000.0, null]
)
.types([String, Integer, BigDecimal])
.build()
MatrixArffWriter.write(data, new File("data_with_nulls.arff"))Output in ARFF:
@DATA
'Alice',25,50000.0
'Bob',?,60000.0
?,35,?
By default, the writer automatically detects categorical columns based on the number of unique values. You can also explicitly define which columns should be treated as nominal and what their allowed values are.
The writer uses these heuristics to determine if a String column should be nominal:
- The column has unique values
- The number of unique values is 50 or fewer
- The unique values represent less than 10% of the total rows (or the dataset has fewer than 10 rows)
import se.alipsa.matrix.arff.MatrixArffWriter
Matrix data = Matrix.builder('survey')
.data(
respondent: ['R001', 'R002', 'R003'],
rating: ['Good', 'Excellent', 'Good'],
category: ['A', 'B', 'A']
)
.types([String, String, String])
.build()
// Define explicit nominal values (including values not in current data)
Map<String, List<String>> nominalMappings = [
rating: ['Poor', 'Fair', 'Good', 'Excellent'],
category: ['A', 'B', 'C', 'D']
]
// Write with custom mappings - respondent will be STRING, others NOMINAL
MatrixArffWriter.write(data, new File("survey.arff"), nominalMappings)Generated ARFF:
@RELATION survey
@ATTRIBUTE respondent STRING
@ATTRIBUTE rating {Poor,Fair,Good,Excellent}
@ATTRIBUTE category {A,B,C,D}
@DATA
'R001',Good,A
'R002',Excellent,B
'R003',Good,A
This is useful when:
- You need to ensure consistent category encoding across different datasets
- Your current data doesn't contain all possible values
- You want to override the automatic detection
The matrix-arff module supports date handling with customizable formats.
The default date format is ISO 8601: yyyy-MM-dd'T'HH:mm:ss
// ARFF file with dates:
// @ATTRIBUTE event_date DATE 'yyyy-MM-dd'
// @DATA
// '2024-01-15'
Matrix events = MatrixArffReader.read(new File("events.arff"))
Date eventDate = events[0, 'event_date']
println eventDate // Mon Jan 15 00:00:00 CET 2024The writer supports various Java date types:
import java.time.LocalDate
import java.time.LocalDateTime
Matrix events = Matrix.builder('events')
.data(
event: ['Conference', 'Meeting', 'Workshop'],
date: [new Date(), LocalDate.now(), LocalDateTime.now()]
)
.types([String, Date])
.build()
MatrixArffWriter.write(events, new File("events.arff"))Supported date types:
java.util.Datejava.sql.Datejava.sql.Timestampjava.time.LocalDatejava.time.LocalDateTimejava.time.Instant
The matrix-arff module is particularly useful for machine learning workflows, enabling data interchange between Matrix and ML tools.
import se.alipsa.matrix.arff.MatrixArffWriter
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.core.Matrix
import se.alipsa.matrix.stats.Normalize
// Load and preprocess data
Matrix raw = MatrixArffReader.read(new File("raw_data.arff"))
// Normalize numeric columns
def numericCols = ['feature1', 'feature2', 'feature3']
Matrix processed = raw
numericCols.each { col ->
processed = processed.apply(col, BigDecimal) { row, val ->
if (val == null) return null
// Min-max normalization to 0-1 range
def colData = raw[col] as List<BigDecimal>
def min = colData.findAll { it != null }.min()
def max = colData.findAll { it != null }.max()
if (max == min) return 0.5
(val - min) / (max - min)
}
}
// Export for Weka
MatrixArffWriter.write(processed, new File("normalized_data.arff"))import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.arff.MatrixArffWriter
import se.alipsa.matrix.smile.SmileUtil
import smile.data.DataFrame
// Read ARFF data
Matrix trainingData = MatrixArffReader.read(new File("train.arff"))
// Convert to Smile DataFrame for ML operations
DataFrame df = SmileUtil.toDataFrame(trainingData)
// Perform ML operations with Smile...
// (training, prediction, etc.)
// Convert results back to Matrix
Matrix results = SmileUtil.toMatrix(df)
// Export as ARFF for further processing or sharing
MatrixArffWriter.write(results, new File("results.arff"))You can read an ARFF file, manipulate it with Matrix operations, and write it back:
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.arff.MatrixArffWriter
// Read iris dataset
Matrix iris = MatrixArffReader.read(new File("iris.arff"))
// Add a new calculated column
iris = iris.addColumn('sepal_ratio', BigDecimal) { row ->
row['sepallength'] / row['sepalwidth']
}
// Filter to specific species
Matrix setosa = iris.subset { row ->
row['class'] == 'Iris-setosa'
}
// Write the filtered and enhanced dataset
MatrixArffWriter.write(setosa, new File("iris_setosa_enhanced.arff"))Here's a comprehensive example that demonstrates reading, processing, and writing ARFF data:
import se.alipsa.matrix.arff.MatrixArffReader
import se.alipsa.matrix.arff.MatrixArffWriter
import se.alipsa.matrix.core.Matrix
import se.alipsa.matrix.core.Stat
// 1. Read the iris dataset
Matrix iris = MatrixArffReader.read(new File("iris.arff"))
println "Loaded ${iris.rowCount()} samples with ${iris.columnCount()} features"
// 2. Explore the data
println "\nColumn types:"
iris.columnNames().each { col ->
println " ${col}: ${iris.type(col).simpleName}"
}
// 3. Calculate statistics for numeric columns
println "\nStatistics by species:"
def species = iris['class'].unique()
species.each { sp ->
def subset = iris.subset { it['class'] == sp }
println "\n${sp}:"
println " Count: ${subset.rowCount()}"
println " Avg sepal length: ${Stat.mean(subset['sepallength']).round(2)}"
println " Avg petal length: ${Stat.mean(subset['petallength']).round(2)}"
}
// 4. Create a summary dataset
Matrix summary = Matrix.builder('iris_summary')
.data(
species: species,
count: species.collect { sp ->
iris.subset { it['class'] == sp }.rowCount()
},
avg_sepal_length: species.collect { sp ->
def subset = iris.subset { it['class'] == sp }
Stat.mean(subset['sepallength']).round(2)
},
avg_petal_length: species.collect { sp ->
def subset = iris.subset { it['class'] == sp }
Stat.mean(subset['petallength']).round(2)
}
)
.types([String, Integer, BigDecimal, BigDecimal])
.build()
println "\nSummary:"
println summary
// 5. Write the summary to ARFF
MatrixArffWriter.write(summary, new File("iris_summary.arff"))
println "\nSummary written to iris_summary.arff"Output:
Loaded 150 samples with 5 features
Column types:
sepallength: BigDecimal
sepalwidth: BigDecimal
petallength: BigDecimal
petalwidth: BigDecimal
class: String
Statistics by species:
Iris-setosa:
Count: 50
Avg sepal length: 5.01
Avg petal length: 1.46
Iris-versicolor:
Count: 50
Avg sepal length: 5.94
Avg petal length: 4.26
Iris-virginica:
Count: 50
Avg sepal length: 6.59
Avg petal length: 5.55
Summary:
Matrix (iris_summary, 3 x 4)
species count avg_sepal_length avg_petal_length
Iris-setosa 50 5.01 1.46
Iris-versicolor 50 5.94 4.26
Iris-virginica 50 6.59 5.55
Summary written to iris_summary.arff
The matrix-arff module provides seamless integration between the Matrix library and the ARFF file format. This enables:
- Easy loading of standard ML datasets
- Data interchange with Weka and other ML tools
- Preprocessing data with Matrix operations before ML workflows
- Exporting results in a format compatible with other tools
Combined with matrix-smile for machine learning operations, matrix-arff forms a complete pipeline for loading, processing, training, and saving ML data.
In the next section, we'll explore the matrix-smile module which provides integration with the Smile machine learning library for classification, regression, clustering, and more.
Go to previous section | Go to next section | Back to outline