<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Avro test files for `arrow-avro`

This directory contains small Avro Object Container Files (OCF) used by
`arrow-avro` tests to validate the `Reader` implementation. These files are generated from 
a set of python scripts and will gradually be removed as they are merged into `arrow-testing`.

## Decimal Files

This directory contains OCF files used to exercise decoding of Avro’s `decimal` logical type 
across both `bytes` and `fixed` encodings, and to cover Arrow decimal widths ranging
from `Decimal32` up through `Decimal256`. The files were generated from a
script (see **How these files were created** below).

> **Avro decimal recap.** Avro’s `decimal` logical type annotates either a
> `bytes` or `fixed` primitive and stores the **two’s‑complement big‑endian
> representation of the unscaled integer** (value × 10^scale). Implementations
> should reject invalid combinations such as `scale > precision`.

> **Arrow decimal recap.** Arrow defines `Decimal32`, `Decimal64`, `Decimal128`,
> and `Decimal256` types with maximum precisions of 9, 18, 38, and 76 digits,
> respectively. Tests here validate that the Avro reader selects compatible
> Arrow decimal widths given the Avro decimal’s precision and storage.

---

All files are one‑column Avro OCFs with a field named `value`. Each contains 24
rows with the sequence `1 … 24` rendered at the file’s declared `scale`
(i.e., at scale 10: `1.0000000000`, `2.0000000000`).

| File | Avro storage | Decimal (precision, scale) | Intended Arrow width |
|---|---|---|---|
| `int256_decimal.avro` | `bytes` + `logicalType: decimal` | (76, 10) | `Decimal256` |
| `fixed256_decimal.avro` | `fixed[32]` + `logicalType: decimal` | (76, 10) | `Decimal256` |
| `fixed_length_decimal_legacy_32.avro` | `fixed[4]` + `logicalType: decimal` | (9, 2) | `Decimal32` (legacy fixed‑width path) |
| `int128_decimal.avro` | `bytes` + `logicalType: decimal` | (38, 2) | `Decimal128` |

### Schemas (for reference)

#### int256_decimal.avro

```json
{
  "type": "record",
  "name": "OneColDecimal256Bytes",
  "fields": [{
    "name": "value",
    "type": { "type": "bytes", "logicalType": "decimal", "precision": 76, "scale": 10 }
  }]
}
```

#### fixed256_decimal.avro

```json
{
  "type": "record",
  "name": "OneColDecimal256Fixed",
  "fields": [{
    "name": "value",
    "type": {
      "type": "fixed", "name": "Decimal256Fixed", "size": 32,
      "logicalType": "decimal", "precision": 76, "scale": 10
    }
  }]
}
```

#### fixed_length_decimal_legacy_32.avro

```json
{
  "type": "record",
  "name": "OneColDecimal32FixedLegacy",
  "fields": [{
    "name": "value",
    "type": {
      "type": "fixed", "name": "Decimal32FixedLegacy", "size": 4,
      "logicalType": "decimal", "precision": 9, "scale": 2
    }
  }]
}
```

#### int128_decimal.avro

```json
{
  "type": "record",
  "name": "OneColDecimal128Bytes",
  "fields": [{
    "name": "value",
    "type": { "type": "bytes", "logicalType": "decimal", "precision": 38, "scale": 2 }
  }]
}
```

### How these files were created

All four files were generated by the Python script
`create_avro_decimal_files.py` authored for this purpose. The script uses
`fastavro` to write OCFs and encodes decimal values as required by the Avro
spec (two’s‑complement big‑endian of the unscaled integer).

#### Re‑generation

From the repository root (defaults write into arrow-avro/test/data):

```bash
# 1) Ensure Python 3 is available, then install fastavro
python -m pip install --upgrade fastavro

# 2) Fetch the script
curl -L -o create_avro_decimal_files.py \
https://gist.githubusercontent.com/jecsand838/3890349bdb33082a3e8fdcae3257eef7/raw/create_avro_decimal_files.py

# 3) Generate the files (prints a verification dump by default)
python create_avro_decimal_files.py -o arrow-avro/test/data
```

Options:
* --num-rows (default 24) — number of rows to emit per file
* --scale (default 10) — the decimal scale used for the 256 files
* --no-verify — skip reading the files back for printed verification

## Duration Logical Types File

This directory contains an OCF file used to test the decoding of Avro long types annotated with custom logicalType values. This is used to map directly to Arrow Duration types with different time units.

#### duration_logical_types.avro

```json
{
   "type": "record",
   "name": "DurationLogicalTypes",
   "fields": [
      {
         "name": "duration_time_nanos",
         "type": {
            "type": "long",
            "logicalType": "arrow.duration-nanos"
         }
      },
      {
         "name": "duration_time_micros",
         "type": {
            "type": "long",
            "logicalType": "arrow.duration-micros"
         }
      },
      {
         "name": "duration_time_millis",
         "type": {
            "type": "long",
            "logicalType": "arrow.duration-millis"
         }
      },
      {
         "name": "duration_time_seconds",
         "type": {
            "type": "long",
            "logicalType": "arrow.duration-seconds"
         }
      }
   ]
}
```

This file contains 24 rows of random long values across four fields, each annotated with a different custom logical type corresponding to an Arrow Duration unit.


#### How this file was created

The file was generated by the Python script generate_duration_avro.py. The script uses fastavro to write an OCF with the schema and random data described above.

#### Re‑generation
From the repository root (defaults write into arrow-avro/test/data):

```Bash

# 1) Ensure Python 3 is available, then install fastavro
python3 -m pip install --upgrade fastavro

# 2) Fetch the script
curl -L -o generate_duration_avro.py \
https://gist.githubusercontent.com/nathaniel-d-ef/c253cb180b041023e3ccfe9df20ccef7/raw/06c8ca1321efcd8e1c8746fd65aa013e1a566944/generate_duration_avro.py

# 3) Run the generation script
python3 generate_duration_avro.py -o arrow-avro/test/data
```

Options:

* --num-rows (default 24) — number of rows to emit

* --no-verify — skip reading the file back for printed verification

## Union File

**Purpose:** Exercise a wide variety of Avro **union** shapes (including nullable unions, unions of ambiguous scalar types, unions of named types, and unions inside arrays, maps, and nested records) to validate `arrow-avro` union decoding and schema‑resolution paths.

**Format:** Avro Object Container File (OCF) written by `fastavro.writer` with embedded writer schema.

**Record count:** four rows. Each row selects different branches across the unions to ensure coverage (i.e., toggling between bytes vs. string, fixed vs. duration vs. decimal, enum vs. record alternatives, etc.).

**How this file was created:**

1. Script: [`create_avro_union_file.py`](https://gist.github.com/jecsand838/f4bf85ad597ab34575219df515156444)  
   Runs with Python 3 and uses **fastavro** to emit `union_fields.avro` in the working directory.
2. Quick reproduce:
   ```bash
   pip install fastavro
   python3 create_avro_union_file.py
   # Outputs: ./union_fields.avro
   ```

> Note: Avro OCF files include a *sync marker*; `fastavro.writer` generates a random one if not provided, so byte‑for‑byte output may vary between runs even with the same data. This does not affect the embedded schema or logical content.

**Writer schema (overview):** The record is named `UnionTypesRecord` and defines the following fields:

| Field                             | Union branches / details                                                                                                                                                                                                                                                                                                                                                                                                                           |
|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `nullable_int_nullfirst`          | `["null","int"]` (tests null‑first ordering)                                                                                                                                                                                                                                                                                                                                                                                                       |
| `nullable_string_nullsecond`      | `["string","null"]` (tests null‑second ordering; in Avro, a union field’s default must match the *first* branch)                                                                                                                                                                                                                                                                                                                                   |
| `union_prim`                      | `["boolean","int","long","float","double","bytes","string"]`                                                                                                                                                                                                                                                                                                                                                                                       |
| `union_bytes_vs_string`           | `["bytes","string"]` (ambiguous scalar union; script uses fastavro’s tuple notation to disambiguate)                                                                                                                                                                                                                                                                                                                                               |
| `union_fixed_dur_decfix`          | `["Fx8","Dur12","DecFix16"]` where:<br>• `Fx8` = `fixed`(size=8)<br>• `Dur12` = `fixed`(size=12, `logicalType`=`duration`)<br>• `DecFix16` = `fixed`(size=16, `logicalType`=`decimal`, precision=10, scale=2)<br>**Notes:** Avro `duration` is a `fixed[12]` storing **months, days, millis** as three **little‑endian** 32‑bit integers; Avro `decimal` on `bytes`/`fixed` uses **two’s‑complement big‑endian** encoding of the unscaled integer. |
| `union_enum_records_array_map`    | `[ColorU, RecA, RecB, array<long>, map<string>]` where:<br>• `ColorU` = `enum` {`RED`,`GREEN`,`BLUE`}<br>• `RecA` = `record` {`a:int`, `b:string`}<br>• `RecB` = `record` {`x:long`, `y:bytes`}                                                                                                                                                                                                                                                    |
| `union_date_or_fixed4`            | `[int (logicalType=`date`), Fx4]` where `Fx4` = `fixed`(size=4)                                                                                                                                                                                                                                                                                                                                                                                    |
| `union_time_millis_or_enum`       | `[int (logicalType=`time-millis`), OnOff]` where `OnOff` = `enum` {`ON`,`OFF`}                                                                                                                                                                                                                                                                                                                                                                     |
| `union_time_micros_or_string`     | `[long (logicalType=`time-micros`), string]`                                                                                                                                                                                                                                                                                                                                                                                                       |
| `union_ts_millis_utc_or_array`    | `[long (logicalType=`timestamp-millis`), array<int>]`                                                                                                                                                                                                                                                                                                                                                                                              |
| `union_ts_micros_local_or_bytes`  | `[long (logicalType=`local-timestamp-micros`), bytes]`                                                                                                                                                                                                                                                                                                                                                                                             |
| `union_uuid_or_fixed10`           | `[string (logicalType=`uuid`), Fx10]` where `Fx10` = `fixed`(size=10)                                                                                                                                                                                                                                                                                                                                                                              |
| `union_dec_bytes_or_dec_fixed`    | `[bytes (decimal p=10 s=2), DecFix20]` where `DecFix20` = `fixed`(size=20, decimal p=20 s=4) — decimal encoding is big‑endian two’s‑complement.                                                                                                                                                                                                                                                                                                    |
| `union_null_bytes_string`         | `["null","bytes","string"]`                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `array_of_union`                  | `array<["long","string"]>`                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `map_of_union`                    | `map<["null","double"]>`                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `record_with_union_field`         | `HasUnion` = `record` {`id:int`, `u:["int","string"]`}                                                                                                                                                                                                                                                                                                                                                                                             |
| `union_ts_micros_utc_or_map`      | `[long (logicalType=`timestamp-micros`), map<long>]`                                                                                                                                                                                                                                                                                                                                                                                               |
| `union_ts_millis_local_or_string` | `[long (logicalType=`local-timestamp-millis`), string]`                                                                                                                                                                                                                                                                                                                                                                                            |
| `union_bool_or_string`            | `["boolean","string"]`                                                                                                                                                                                                                                                                                                                                                                                                                             |

**Implementation notes (generation):**

* The script uses **fastavro’s tuple notation** `(branch_name, value)` to select branches in ambiguous unions (e.g., bytes vs. string, multiple named records). See *“Using the tuple notation to specify which branch of a union to take”* in the fastavro docs.
* Decimal values are pre‑encoded to the required **big‑endian two’s‑complement** byte sequence before writing (for both `bytes` and `fixed` decimal logical types).
* The `duration` logical type payloads are 12‑byte triples: **months / days / milliseconds**, little‑endian each.

**Source / Repro script:**
`create_avro_union_file.py` (Gist): contains the full writer schema, record builders covering four rows, and the `fastavro.writer` call which emits `union_fields.avro`.

## Comprehensive E2E Coverage File

**Purpose:** A single OCF that exercises **all decoder paths** used by `arrow-avro` with both **nested and non‑nested** shapes, including **dense unions** (null‑first, null‑second, multi‑branch), **aliases** (type and field), **default values**, **docs** and **namespaces**, and combinations thereof. It’s intended to validate the final `Reader` implementation and to stress schema‑resolution behavior in the tests under `arrow-avro/src/reader/mod.rs`.

**File:** `comprehensive_e2e.avro`
**Top‑level record (writer schema):** `org.apache.arrow.avrotests.v1.E2eComprehensive`
**Record count:** four rows (each row selects different union branches and nested shapes)

**Coverage summary (by Arrow / Avro mapping):**

* Primitives: **boolean, int, long, float, double**
* Binary / Text: **bytes**, **string (UTF‑8)**
* Logical types: **date**, **time‑millis**, **time‑micros**, **timestamp‑millis (UTC)**, **timestamp‑micros (UTC)**, **local‑timestamp‑millis**, **local‑timestamp‑micros**, **uuid (string)**, **decimal** on **bytes** and **fixed**, **duration** on **fixed(12)**
* Named types: **fixed**, **enum**, **record**
* Collections: **array**, **map**
* Unions: **nullable unions**, **ambiguous scalar unions**, **unions of named types**, and **unions nested inside arrays/maps/records**
* Schema‑evolution hooks: **type aliases**, **field aliases**, **defaults** (including union defaults on the first branch), **docs**, and **namespaces**

**Writer schema (overview of fields):**

| Field                         | Type / details                                                                                          |
|-------------------------------|---------------------------------------------------------------------------------------------------------|
| `id`                          | `long`                                                                                                  |
| `flag`                        | `boolean` (default `true`)                                                                              |
| `ratio_f32`                   | `float` (default `0.0`)                                                                                 |
| `ratio_f64`                   | `double` (default `0.0`)                                                                                |
| `count_i32`                   | `int` (default `0`)                                                                                     |
| `count_i64`                   | `long` (default `0`)                                                                                    |
| `opt_i32_nullfirst`           | `["null","int"]` (default `null`)                                                                       |
| `opt_str_nullsecond`          | `["string","null"]` (default `""`, alias: `old_opt_str`)                                                |
| `tri_union_prim`              | `["int","string","boolean"]` (default `0`)                                                              |
| `str_utf8`                    | `string` (default `"default"`)                                                                          |
| `raw_bytes`                   | `bytes` (default `""`)                                                                                  |
| `fx16_plain`                  | `fixed` `types.Fx16` (size 16, alias `Fixed16Old`)                                                      |
| `dec_bytes_s10_2`             | `bytes` + `logicalType: decimal` (precision 10, scale 2)                                                |
| `dec_fix_s20_4`               | `fixed` `types.DecFix20` (size 20) + `logicalType: decimal` (precision 20, scale 4)                     |
| `uuid_str`                    | `string` + `logicalType: uuid`                                                                          |
| `d_date`                      | `int` + `logicalType: date`                                                                             |
| `t_millis`                    | `int` + `logicalType: time-millis`                                                                      |
| `t_micros`                    | `long` + `logicalType: time-micros`                                                                     |
| `ts_millis_utc`               | `long` + `logicalType: timestamp-millis`                                                                |
| `ts_micros_utc`               | `long` + `logicalType: timestamp-micros`                                                                |
| `ts_millis_local`             | `long` + `logicalType: local-timestamp-millis`                                                          |
| `ts_micros_local`             | `long` + `logicalType: local-timestamp-micros`                                                          |
| `interval_mdn`                | `fixed` `types.Dur12` (size 12) + `logicalType: duration`                                               |
| `status`                      | `enum` `types.Status` = {`UNKNOWN`,`NEW`,`PROCESSING`,`DONE`} (alias: `State`)                          |
| `arr_union`                   | `array<["long","string","null"]>`                                                                       |
| `map_union`                   | `map<["null","double","string"]>`                                                                       |
| `address`                     | `record` `types.Address` {`street` (alias: `street_name`), `zip:int`, `country:string`}                 |
| `maybe_auth`                  | `record` `types.MaybeAuth` {`user:string`, `token:["null","bytes"]` (default `null`)}                   |
| `union_enum_record_array_map` | `[types.Color enum, types.RecA record, types.RecB record, array<long>, map<string>]`                    |
| `union_date_or_fixed4`        | `[int (logicalType=date), fixed Fx4 size 4]`                                                            |
| `union_interval_or_string`    | `[fixed Dur12U size 12 (logicalType=duration), string]`                                                 |
| `union_uuid_or_fixed10`       | `[string (logicalType=uuid), fixed Fx10 size 10]`                                                       |
| `array_records_with_union`    | `array<record types.KV { key:string, val:["null","int","long"] }>`                                      |
| `union_map_or_array_int`      | `[map<int>, array<int>]`                                                                                |
| `renamed_with_default`        | `int` (default `42`, alias: `old_count`)                                                                |
| `person`                      | `record` `com.example.v2.PersonV2` (alias: `com.example.Person`) `{ name:string, age:int (default 0) }` |

**How this file was created**

* Script: [`create_comprehensive_avro_file.py`](https://gist.github.com/jecsand838/26f9666da8de22651027d485bd83f4a3)
  Uses **fastavro** to write `comprehensive_e2e.avro` with the schema above and four records that intentionally vary union branches and nested shapes.

**Re‑generation**

From the repository root:

```bash
# 1) Ensure Python 3 is available, then install fastavro
python -m pip install --upgrade fastavro

# 2) Run the generator (writes ./comprehensive_e2e.avro by default)
python create_comprehensive_avro_file.py

# 3) Move or copy the file into this directory if needed
mv comprehensive_e2e.avro arrow-avro/test/data/
```

**Notes / tips for tests**

* For **unions of named types** (record/enum/fixed), the generator uses fastavro’s **tuple notation** to select the union branch and, where needed, supplies the **fully‑qualified name (FQN)** to avoid ambiguity when namespaces apply.
* The file contains many **defaults** and **aliases** (type and field) to exercise **schema resolution** code paths.
* As with all OCFs, a random **sync marker** is embedded in the file header; byte‑for‑byte output may vary across runs without affecting the schema or logical content.

## Other Files

This directory contains other small OCF files used by `arrow-avro` tests. Details on these will be added in
follow-up PRs.