diff --git a/README.en.md b/README.en.md index a47c121..4611f35 100644 --- a/README.en.md +++ b/README.en.md @@ -22,7 +22,7 @@ This is research code, not a complete, standards-compliant, or production-grade or `9.1.6` (`4.0` is the quad layout, a front and a back pair; see [Speaker layouts and channel order](#speaker-layouts-and-channel-order)). - Run public SOFA binaural rendering directly from `pcm16 + ID11/OAMD`, without a temporary ADM BWF. -- Keep the binaural DSP in float64/complex128, including 961-sample latency compensation, cross-frame state, and the room tail. +- Keep the binaural DSP in float64/complex128, including the selected filterbank's latency compensation (961 or 448 samples), cross-frame state, and the room tail. - Use a shared float32/PCM24 WAV writer and explicit PCM24 clipping policy for direct outputs. - Implement everything in pure Python on NumPy; - Read or write metadata sidecars and produce metadata, timing, and output reports. @@ -105,6 +105,8 @@ python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa ` --binaural-mode near python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa ` --hrtf-cache-policy disk +python main.py input.m4a --binaural --personalized-headphone ` + --rosella-filterbank 2 python main.py input.m4a --binaural --binaural-output output.binaural.wav ``` @@ -223,7 +225,21 @@ sample. ### Binaural calculation -See [Binaural Rendering Mathematics](docs/binaural.en.md) for QMF, hybrid processing, direction fields, distance, ITD, room processing, the 512-sample parameter updates above, and 961-sample latency compensation. +See [Binaural Rendering Mathematics](docs/binaural.en.md) for QMF, hybrid processing, direction fields, distance, ITD, room processing, the 512-sample parameter updates above, and the selected filterbank's latency compensation (961 by default, 448 with `--rosella-filterbank 2`). + +### Rosella filterbank + +`--rosella-filterbank 1|2` selects the filterbank of the `.personalized_headphone` +path: + +- `1` (default): the public bank, 13-tap hybrid / 6-slot high-band delay, + 961 samples of latency; +- `2`: the Rosella bank captured from the reference renderer, 11-tap hybrid / + 3-slot high-band delay, 448 samples of latency. + +The option only applies to the Rosella model; the SOFA and `.jochrtf` paths +reject it. See [Rosella filterbank](docs/rosella_filterbank.en.md) for the +parameters, buffers, latency and validation of both banks. For all options: @@ -321,3 +337,4 @@ See the [mathematical notes](docs/math.en.md) for the equations used by the deco - [Mathematical notes](docs/math.en.md) · [中文](docs/math.md) - [Binaural rendering](docs/binaural.en.md) · [中文](docs/binaural.md) +- [Rosella filterbank](docs/rosella_filterbank.en.md) · [中文](docs/rosella_filterbank.md) diff --git a/README.md b/README.md index 152ced2..218cee1 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ - 直接渲染 `2.0`、`3.1`、`4.0`、`5.1`、`7.1`、`5.1.2`、`5.1.4`、`7.1.2`、`7.1.4`、`9.1.4`、 `9.1.6`(`4.0` 是 Quad 布局,前后各一对;声道排列见[扬声器布局与声道顺序](#扬声器布局与声道顺序)); - 从 `pcm16 + ID11/OAMD` 直接运行公开 SOFA 双耳渲染,不生成临时 ADM BWF; -- 双耳 DSP 全程使用 float64/complex128,并保留 961-sample latency compensation、跨帧状态和 room 尾声; +- 双耳 DSP 全程使用 float64/complex128,并保留所选滤波器组的延迟补偿(公开版 961、捕获版 448 samples)、跨帧状态和 room 尾声; - 直接输出统一支持 float32 或 PCM24 WAV,并在 PCM24 削波前提供明确处理策略; - 全程使用 NumPy 的纯 Python 实现; - 读取或写入 metadata sidecar,并生成元数据、运行时间和输出摘要。 @@ -103,6 +103,8 @@ python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa ` --binaural-mode near python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa ` --hrtf-cache-policy disk +python main.py input.m4a --binaural --personalized-headphone ` + --rosella-filterbank 2 python main.py input.m4a --binaural --binaural-output output.binaural.wav ``` @@ -192,7 +194,18 @@ outer subpayload offset 与 block offset 落到绝对 sample timeline,再加 ### 双耳计算 -双耳路径的 QMF、hybrid、方向场、距离、ITD、room、上述 512-sample 参数更新和 961-sample 延迟补偿见[双耳渲染数学](docs/binaural.md)。 +双耳路径的 QMF、hybrid、方向场、距离、ITD、room、上述 512-sample 参数更新和所选滤波器组的延迟补偿(默认 961,`--rosella-filterbank 2` 为 448 samples)见[双耳渲染数学](docs/binaural.md)。 + +### Rosella 滤波器组 + +`--rosella-filterbank 1|2` 选择 `.personalized_headphone` 路径使用的滤波器组: + +- `1`(默认):公开滤波器组,13-tap hybrid / 6-slot 高频延迟,961 samples 延迟; +- `2`:从参考渲染器捕获的 Rosella 滤波器组,11-tap hybrid / 3-slot 高频延迟, + 448 samples 延迟。 + +该选项只作用于 Rosella 模型;SOFA 与 `.jochrtf` 路径给出时报错。两套实现的参数、 +缓冲、延迟与验证见 [Rosella 滤波器组](docs/rosella_filterbank.md)。 更多参数可查看: @@ -283,3 +296,4 @@ JustOneCacophony/ - [数学说明](docs/math.md) · [English](docs/math.en.md) - [双耳渲染](docs/binaural.md) · [English](docs/binaural.en.md) +- [Rosella 滤波器组](docs/rosella_filterbank.md) · [English](docs/rosella_filterbank.en.md) diff --git a/data/README.en.md b/data/README.en.md index 4703c35..a3612bf 100644 --- a/data/README.en.md +++ b/data/README.en.md @@ -72,6 +72,25 @@ $$Y_p = \sum_{q\in C_p}\Bigl(\operatorname{Re}X_q + j\,s_q\,\operatorname{Im}X_q The same values also appear in other public implementations of these standards (for example FFmpeg's `aacps_tablegen.h` and `aacsbrdata.h`). +### Captured Rosella filterbank + +`rosella_captured_kernels.npz` holds the bank the reference headphone renderer +actually runs (`--rosella-filterbank 2`): + +```text +format_version little-endian int32[1] +qmf_prototype float32[64,10] +rotation complex64[64] +hybrid complex64[22] +``` + +The table values are promoted to float64/complex128 on load. The full +analysis/synthesis latency is 448 samples (961 for the public bank). These +values come from runtime buffer captures of the reference renderer, not from the +public standards above; the C++ core compiles the same data in as +`src/hrtf/rosella_filterbank_tables.h`. This file also contains no personalized +HRTF or room tuning. + Public availability of a standard does not by itself grant permission to practice related patent claims. diff --git a/data/README.md b/data/README.md index 14a223a..ecd02d0 100644 --- a/data/README.md +++ b/data/README.md @@ -64,6 +64,23 @@ $$Y_p = \sum_{q\in C_p}\Bigl(\operatorname{Re}X_q + j\,s_q\,\operatorname{Im}X_q 相同数值可在 FFmpeg(`aacps_tablegen.h`、`aacsbrdata.h`)等公开实现中查到。 +### Rosella 捕获版滤波器组 + +`rosella_captured_kernels.npz` 保存参考头戴渲染器实际运行的那套滤波器组 +(`--rosella-filterbank 2`): + +```text +format_version little-endian int32[1] +qmf_prototype float32[64,10] +rotation complex64[64] +hybrid complex64[22] +``` + +表值载入后提升为 float64/complex128;全链 latency 为 448 samples(公开版为 961)。 +这些数值来自参考渲染器的运行时缓冲捕获,不是上述公开标准的推导值;C++ 内核以 +`src/hrtf/rosella_filterbank_tables.h` 的形式内置同一组数据。文件同样不含个性化 +HRTF 或 room 调参。 + 标准可公开获取不等于获准实施相关专利。 `.sofa` 是用户可见的 source of truth;`.jochrtf` 是可删除、可从 SOFA 重建的 diff --git a/data/rosella_captured_kernels.npz b/data/rosella_captured_kernels.npz new file mode 100644 index 0000000..69dcc8b Binary files /dev/null and b/data/rosella_captured_kernels.npz differ diff --git a/docs/binaural.en.md b/docs/binaural.en.md index 51a8b31..28c9632 100644 --- a/docs/binaural.en.md +++ b/docs/binaural.en.md @@ -153,6 +153,13 @@ The runtime is fixed at: - fifth order, 36 terms, ACN/N3D real spherical harmonics; - float64 PCM, delay, SH, and room state; complex128 band transfers and spectra. +Those are the SOFA / `.jochrtf` path and the public filterbank. The Rosella +`.personalized_headphone` path selects its bank: the public bank above is the +default (961 samples), and `--rosella-filterbank 2` switches to the bank captured +from the reference renderer (11 taps / 3 slots, 448 samples). Parameters, +buffers, latency and validation of both are in +[Rosella filterbank](rosella_filterbank.en.md). + Real and imaginary unit gains for every hybrid band pass through the same analysis/synthesis chain to form a 154-real-parameter impulse dictionary. The compiler does not sample 77 FFT bins. Defaults are `1e-3` projection ridge and diff --git a/docs/binaural.md b/docs/binaural.md index 663a2cf..06c5d5f 100644 --- a/docs/binaural.md +++ b/docs/binaural.md @@ -139,6 +139,12 @@ convention adapter,不能只通过 reshape 进入核心 importer。 - 五阶、36 项、ACN/N3D real spherical harmonics; - PCM、delay、SH、room state 为 float64;频带传递和频域状态为 complex128。 +上面是 SOFA / `.jochrtf` 路径与公开滤波器组的固定参数。Rosella +`.personalized_headphone` 路径的滤波器组可选:默认仍是这一套公开滤波器组(961 +samples),`--rosella-filterbank 2` 换成从参考渲染器捕获的版本(11 taps / 3 slots、 +448 samples)。两套实现的参数、缓冲、延迟与验证见 +[Rosella 滤波器组](rosella_filterbank.md)。 + 每个 hybrid band 的 real/imaginary 单位增益都通过同一套 analysis/synthesis 链生成 脉冲字典,共 154 个实参数;编译不是直接读取 77 个 FFT bin。默认 projection ridge 为 `1e-3`,SH ridge 为 `1e-5`。同方向 measurement 先合并,再用球面 Voronoi diff --git a/docs/rosella_filterbank.en.md b/docs/rosella_filterbank.en.md new file mode 100644 index 0000000..a439ecf --- /dev/null +++ b/docs/rosella_filterbank.en.md @@ -0,0 +1,89 @@ +# Rosella filterbank + +[中文](rosella_filterbank.md) · [Back to README](../README.en.md) + +The Rosella `.personalized_headphone` path can run either of two 64-QMF / +77-hybrid filterbanks: the shared public bank (the default) or the Rosella bank +captured from the reference renderer. The SOFA and `.jochrtf` paths always use +the public bank and never consult the selector. + +## Which bank runs + +`--rosella-filterbank 1|2` selects the bank of the Rosella path; the library +form is `RosellaBinauralRenderer(..., filterbank=...)`, where omitting the +argument means `1`. + +| Value | Implementation | Low-band hybrid | High-band delay | Latency | +| --- | --- | --- | --- | ---: | +| `1` (default) | `rosella_filterbank.py` | 13 taps | 6 slots | 961 samples | +| `2` | `rosella_captured_filterbank.py` | 11 taps | 3 slots | 448 samples | + +Both banks expose the same four stateful half-steps (QMF analysis, hybrid +analysis, hybrid synthesis, QMF synthesis), so the choice switches the +coefficients, the high-band delay, the history layout, the startup skip and the +flush capacity together. The analysis and synthesis stages keep their own state +and never share it. Any other value is rejected, and the option is an error when +the binaural input is not a Rosella model: a SOFA or `.jochrtf` render must not +silently ignore it. + +## Why the banks differ + +The reference headphone renderer selects its analysis and synthesis callbacks +during preparation, after initialization. Capturing only the initial callbacks +identified the wrong filterbank variant. Runtime buffer captures gave these +parameters: + +| Parameter | Captured Rosella bank | Public bank | +| --- | --- | --- | +| QMF prototype | 640 asymmetric coefficients | polyphase reorder of the public 640-tap prototype | +| Low-band hybrid filter | 11 taps | 13 taps | +| High-band hybrid delay | 3 slots | 6 slots | +| Analysis/synthesis latency | 448 samples | 961 samples | + +The wrong variant produced a frequency-dependent room-tail error. With the +captured room send as input, the existing room recursion already matched the +reference, so the fix changes only the filterbank: not the room coefficients, +the direct HRTF calculation, the phase mapping, or the output EQ. The public +bank stayed as bank `1` so historical renders remain reproducible; the captured +bank is bank `2`. + +## Implementation + +`rosella_captured_filterbank.py` reproduces the four half-steps of the captured +bank in float64/complex128: + +- QMF analysis keeps a 9-slot ring history, reads `(position + tap + 1) mod 10`, + and writes `spectrum[2b] = F[b]/64`, `spectrum[2b+1] = conj(F[63-b])/64`; +- hybrid analysis keeps a 10-slot history, transforms the three low bands with + 8/4/4-point transforms, reorders them with a fixed map, conjugates bands 1 and + 3, and passes the high bands through delayed by 3 slots; +- QMF synthesis keeps a 9-slot history and accumulates 10 prototype-matched + lags, taking the imaginary part on odd lags. + +`data/rosella_captured_kernels.npz` stores the captured prototype +(`float32[64,10]`), rotation (`complex64[64]`) and hybrid coefficients +(`complex64[22]`); they are promoted to float64/complex128 on load. The values +are the ones the C++ core compiles in as `src/hrtf/rosella_filterbank_tables.h` +and contain no personalized HRTF or room tuning. + +The startup skip and the flush capacity both come from the selected bank (961 or +448 samples). `.report.json` records the actual choice in +`binaural_renderer_backend`, as `filterbank`, `filterbank_name` and +`latency_compensated_samples`. + +## Validation + +The local regression script lives in `tests/`, which this repository does not +track (like the rest of its local checks). It covers: + +- reference vectors for the four captured half-steps, the same samples the C++ + core checks, generated by an independent NumPy implementation, to within + `1e-10` per sample; +- an impulse roundtrip peak at 961 and 448 samples for the two banks; +- chunk invariance (including zero-length and unequal chunks) and `reset()`; +- independence: processing one bank never changes the other bank's state; +- rejection of out-of-range values. + +The public filterbank formulas and `.jochrtf` direction field described in +`docs/binaural.md` are unchanged; the captured bank only affects the Rosella +path. diff --git a/docs/rosella_filterbank.md b/docs/rosella_filterbank.md new file mode 100644 index 0000000..325edeb --- /dev/null +++ b/docs/rosella_filterbank.md @@ -0,0 +1,71 @@ +# Rosella 滤波器组 + +[English](rosella_filterbank.en.md) · [返回 README](../README.md) + +Rosella `.personalized_headphone` 路径可以在两套 64-QMF / 77-hybrid 滤波器组之间选择: +公开滤波器组(默认)与从参考渲染器捕获的 Rosella 滤波器组。SOFA 与 `.jochrtf` 路径 +始终使用公开滤波器组,不经过该选择。 + +## 选择哪一套 + +`--rosella-filterbank 1|2` 选择 Rosella 路径的滤波器组;库接口是 +`RosellaBinauralRenderer(..., filterbank=...)`,省略参数等于 `1`。 + +| 取值 | 实现 | 低频 hybrid | 高频 slot delay | 延迟 | +| --- | --- | --- | --- | ---: | +| `1`(默认) | `rosella_filterbank.py` | 13 taps | 6 slots | 961 samples | +| `2` | `rosella_captured_filterbank.py` | 11 taps | 3 slots | 448 samples | + +两套实现暴露同一组四个有状态的 half-step(QMF 分析、hybrid 分析、hybrid 合成、 +QMF 合成),因此切换会同时改变系数、高频延迟、历史缓冲布局、起始延迟跳过量和 +flush 容量;分析与合成两级各自持有状态,互不共享。其它取值直接报错;非 Rosella +输入(SOFA、`.jochrtf`)给出该选项也报错,不会静默忽略。 + +## 为什么两套不同 + +参考头戴渲染器在初始化之后的准备阶段才选定 analysis/synthesis 回调,只捕获初始化 +时的回调会拿到错误的滤波器组变体。运行时缓冲捕获得到的参数是: + +| 参数 | Rosella 捕获版 | 公开版 | +| --- | --- | --- | +| QMF prototype | 640 个非对称系数 | 公开 640-tap prototype 的多相重排 | +| 低频 hybrid 滤波 | 11 taps | 13 taps | +| 高频 hybrid delay | 3 slots | 6 slots | +| analysis/synthesis 延迟 | 448 samples | 961 samples | + +用错变体会在 room 尾音上引入随频率变化的误差;把捕获到的 room send 作为输入时, +现有 room 递推已经能对上参考,因此修复只更换滤波器组,不改 room 系数、direct HRTF、 +相位映射或输出 EQ。公开版是 `1`,捕获版是 `2`;默认保留公开版,便于与历史输出对齐。 + +## 实现 + +`rosella_captured_filterbank.py` 用 float64/complex128 复现捕获版的四个 half-step: + +- QMF 分析保留 9 slot 环形历史,按 `(position + tap + 1) mod 10` 读取,输出 + `spectrum[2b] = F[b]/64`、`spectrum[2b+1] = conj(F[63-b])/64`; +- hybrid 分析保留 10 slot 历史,低频三个 band 分别做 8/4/4 点变换后按固定顺序重排, + band 1 与 band 3 取共轭;高频带原样延迟 3 slot; +- QMF 合成保留 9 slot 历史,用 10 个 lag 的 prototype 匹配累加,奇数 lag 取虚部。 + +`data/rosella_captured_kernels.npz` 保存捕获版的 prototype(`float32[64,10]`)、 +rotation(`complex64[64]`)与 hybrid 系数(`complex64[22]`);载入后提升为 +float64/complex128。这些数值与 C++ 内核编译进去的 +`src/hrtf/rosella_filterbank_tables.h` 相同,不包含个性化 HRTF 或 room 调参。 + +运行时的起始延迟跳过与 flush 容量都取自所选滤波器组:`961` 或 `448` samples, +`.report.json` 的 `binaural_renderer_backend` 用 `filterbank`、`filterbank_name` 与 +`latency_compensated_samples` 记录实际选择。 + +## 验证 + +本地回归脚本放在 `tests/`(该目录不随仓库跟踪,与其余本地验证脚本一致),覆盖: + +- 捕获版四个 half-step 的参考向量(与 C++ 内核 `tests/rosella_filterbank_vectors.h` + 同一组、由独立 NumPy 实现生成),逐样本误差小于 `1e-10`; +- 两套滤波器组的脉冲往返延迟分别落在 `961` 与 `448` samples; +- 分块处理(含 0 长度与不等长切分)与 `reset()` 不变性; +- 两套滤波器组互不干扰:交叉处理一个 bank 不改变另一个 bank 的状态; +- 越界取值被拒绝。 + +`docs/binaural.md` 描述的公开滤波器组公式与 `.jochrtf` 方向和公开版一致;捕获版只 +作用于 Rosella 路径。 diff --git a/main.py b/main.py index a1cbc54..d69a6c3 100644 --- a/main.py +++ b/main.py @@ -38,10 +38,10 @@ from binaural_renderer import ( from rosella_binaural_renderer import ( DEFAULT_PERSONALIZED_HEADPHONE, ROSSELLA_BLOCK_SAMPLES, - ROSSELLA_LATENCY_SAMPLES, RosellaBinauralRenderer, resolve_personalized_headphone, ) +from rosella_bank import DEFAULT_FILTERBANK, FILTERBANK_CHOICES, FILTERBANK_NAMES from sofa_hrtf_field import DEFAULT_HRTF_CACHE_DIR from speaker_backend import create_speaker_renderer from speaker_layouts import (SPEAKER_LAYOUT_CHOICES, get_speaker_layout, @@ -566,6 +566,10 @@ def build_parser(): const=DEFAULT_PERSONALIZED_HEADPHONE, help="Rosella .personalized_headphone 模型;不带路径时默认 " "HRTF/binaural.personalized_headphone") + parser.add_argument( + "--rosella-filterbank", type=int, choices=FILTERBANK_CHOICES, default=None, + help="Rosella 模型的滤波器组:1=13 taps/6 slots(默认)," + "2=11 taps/3 slots;仅作用于个性化模型") parser.add_argument( "--hrtf-cache-policy", choices=("none", "memory", "disk"), default=None, help="SOFA 编译缓存;默认 memory,disk 写入可删除的 .jochrtf") @@ -653,6 +657,10 @@ def main(argv=None): )) if hrtf_options_used and not binaural_mode: raise ValueError("SOFA/HRTF 选项仅与 --binaural 一起使用") + if args.rosella_filterbank is not None and not binaural_mode: + raise ValueError("--rosella-filterbank 仅与 --binaural 一起使用") + rosella_filterbank = (DEFAULT_FILTERBANK if args.rosella_filterbank is None + else args.rosella_filterbank) if (not math.isfinite(args.binaural_tail_seconds) or args.binaural_tail_seconds < 0): raise ValueError("binaural-tail-seconds 必须是非负有限值") @@ -686,6 +694,11 @@ def main(argv=None): raise ValueError("eac3-target-level 必须在 -31..0 之间") binaural_hrtf_input = resolve_binaural_hrtf_input( args, required=binaural_mode and not args.metadata_only) + if (args.rosella_filterbank is not None and binaural_hrtf_input is not None + and binaural_hrtf_input["kind"] != "rosella"): + raise ValueError( + "--rosella-filterbank 只作用于 Rosella .personalized_headphone 输入;" + "当前 SOFA/.jochrtf 路径不使用该滤波器组") ffmpeg = executable(args.ffmpeg, "FFmpeg") decode_options = () decode_option_info = None @@ -828,6 +841,7 @@ def main(argv=None): hrtf_source["path"], bed_channels=getattr(index, "bed_channels", ()), mode=binaural_render_mode, + filterbank=rosella_filterbank, object_delay_samples=args.object_delay_samples, tail_seconds=args.binaural_tail_seconds, output_gain=gain_float64, @@ -844,9 +858,12 @@ def main(argv=None): f"precision=float64/complex128 " f"hrtf={hrtf_source['kind']}:{hrtf_source['path']}", flush=True) if hrtf_source["kind"] == "rosella": + print( + f"[binaural] filterbank={FILTERBANK_NAMES[binaural_decoder.filterbank]}" + f" latency={binaural_decoder.latency_samples}", flush=True) flush_samples = math.ceil( (args.binaural_tail_seconds * RATE - + ROSSELLA_LATENCY_SAMPLES + ROSSELLA_BLOCK_SAMPLES) + + binaural_decoder.latency_samples + ROSSELLA_BLOCK_SAMPLES) / ROSSELLA_BLOCK_SAMPLES) * ROSSELLA_BLOCK_SAMPLES spool_capacity = frame_count * FRAME_SAMPLES + flush_samples else: diff --git a/src/rosella_bank.py b/src/rosella_bank.py new file mode 100644 index 0000000..a1cf4a6 --- /dev/null +++ b/src/rosella_bank.py @@ -0,0 +1,101 @@ +"""Selectable Rosella 64-QMF / 77-hybrid filterbank. + +The Rosella renderer can run either bank: + +``PUBLIC_FILTERBANK`` (1) + The shared public bank of ``rosella_filterbank.py``: a 13-tap hybrid + analysis, a 6-slot high-band delay and 961 samples of analysis/synthesis + latency. This is the bank the reference implementation has always used. + +``CAPTURED_FILTERBANK`` (2) + The bank the reference headphone renderer actually runs, captured from it: + an 11-tap hybrid analysis, a 3-slot high-band delay and 448 samples of + latency. + +Both banks expose the same four stateful stages, so the choice changes the +coefficients, the high-band delay, the history layout and the latency together. +Nothing is shared between them. +""" +from __future__ import annotations + +from pathlib import Path + +from rosella_captured_filterbank import ( + DEFAULT_CAPTURED_KERNEL_DATA, + HybridAnalysis as CapturedHybridAnalysis, + HybridSynthesis as CapturedHybridSynthesis, + QmfAnalysis as CapturedQmfAnalysis, + QmfSynthesis as CapturedQmfSynthesis, +) +from rosella_filterbank import ( + DEFAULT_KERNEL_DATA, + HybridAnalysis as PublicHybridAnalysis, + HybridSynthesis as PublicHybridSynthesis, + QmfAnalysis as PublicQmfAnalysis, + QmfSynthesis as PublicQmfSynthesis, +) + +PUBLIC_FILTERBANK = 1 +CAPTURED_FILTERBANK = 2 +FILTERBANK_CHOICES = (PUBLIC_FILTERBANK, CAPTURED_FILTERBANK) +DEFAULT_FILTERBANK = PUBLIC_FILTERBANK + +FILTERBANK_NAMES = { + PUBLIC_FILTERBANK: "public", + CAPTURED_FILTERBANK: "captured", +} + +# analysis/synthesis latency of each bank, in samples at 48 kHz +FILTERBANK_LATENCY_SAMPLES = { + PUBLIC_FILTERBANK: 961, + CAPTURED_FILTERBANK: 448, +} + + +def latency_samples(filterbank: int) -> int: + """Length of the startup transient the runtime removes for ``filterbank``.""" + try: + return FILTERBANK_LATENCY_SAMPLES[int(filterbank)] + except (KeyError, TypeError, ValueError): + raise ValueError( + f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}" + ) from None + + +class RosellaBank: + """The four stages of one bank, each with its own state.""" + + def __init__(self, filterbank: int = DEFAULT_FILTERBANK, *, + source_channels: int, output_channels: int, + kernel_data: str | Path | None = None): + if filterbank not in FILTERBANK_CHOICES: + raise ValueError( + f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}") + kind = int(filterbank) + if source_channels <= 0 or output_channels <= 0: + raise ValueError("channel counts must be positive") + self.filterbank = kind + self.name = FILTERBANK_NAMES[kind] + self.latency_samples = latency_samples(kind) + if kind == PUBLIC_FILTERBANK: + self.table_path = Path( + DEFAULT_KERNEL_DATA if kernel_data is None else kernel_data + ).expanduser().resolve() + self.qmf_analysis = PublicQmfAnalysis(source_channels, self.table_path) + self.hybrid_analysis = PublicHybridAnalysis(source_channels, self.table_path) + self.hybrid_synthesis = PublicHybridSynthesis(output_channels, self.table_path) + self.qmf_synthesis = PublicQmfSynthesis(output_channels, self.table_path) + else: + self.table_path = Path( + DEFAULT_CAPTURED_KERNEL_DATA if kernel_data is None else kernel_data + ).expanduser().resolve() + self.qmf_analysis = CapturedQmfAnalysis(source_channels, self.table_path) + self.hybrid_analysis = CapturedHybridAnalysis(source_channels, self.table_path) + self.hybrid_synthesis = CapturedHybridSynthesis(output_channels, self.table_path) + self.qmf_synthesis = CapturedQmfSynthesis(output_channels, self.table_path) + + def reset(self): + self.qmf_analysis.reset() + self.hybrid_analysis.reset() + self.hybrid_synthesis.reset() + self.qmf_synthesis.reset() diff --git a/src/rosella_binaural_renderer.py b/src/rosella_binaural_renderer.py index c1964f2..dd6b668 100644 --- a/src/rosella_binaural_renderer.py +++ b/src/rosella_binaural_renderer.py @@ -16,22 +16,16 @@ import numpy as np from bed_layout import bed_object_positions from binaural_metadata import OamdPositionTimeline +from rosella_bank import (DEFAULT_FILTERBANK, FILTERBANK_CHOICES, FILTERBANK_NAMES, + RosellaBank) from rosella_core import RosellaRenderer from rosella_direct import BINAURAL_PROFILE_NAMES -from rosella_filterbank import ( - DEFAULT_KERNEL_DATA, - HybridAnalysis, - HybridSynthesis, - QmfAnalysis, - QmfSynthesis, -) from rosella_model import RosellaModel, load_personalized_headphone SAMPLE_RATE = 48000 FRAME_SAMPLES = 1536 ROSSELLA_BLOCK_SAMPLES = 512 QMF_HOP_SAMPLES = 64 -ROSSELLA_LATENCY_SAMPLES = 961 SOURCE_CHANNELS = 16 OUTPUT_CHANNELS = 2 PROJECT_DIR = Path(__file__).resolve().parent.parent @@ -67,7 +61,8 @@ class RosellaBinauralRenderer: personalized_headphone: str | Path | RosellaModel, *, mode: str = "mid", - kernel_data: str | Path = DEFAULT_KERNEL_DATA, + filterbank: int = DEFAULT_FILTERBANK, + kernel_data: str | Path | None = None, object_delay_samples: int = 1473, tail_seconds: float = 5.0, output_gain: float = 1.0, @@ -76,6 +71,9 @@ class RosellaBinauralRenderer: bed_channels=None): if mode not in BINAURAL_PROFILE_NAMES: raise ValueError("binaural mode must be near, mid, or far") + if filterbank not in FILTERBANK_CHOICES: + raise ValueError( + f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}") if int(object_delay_samples) < 0: raise ValueError("object_delay_samples must be non-negative") if float(tail_seconds) < 0.0: @@ -97,8 +95,6 @@ class RosellaBinauralRenderer: self.mode = mode self.profile_index = BINAURAL_PROFILE_NAMES[mode] - self.kernel_data = Path(kernel_data).expanduser().resolve() - self.kernel_data_sha256 = _sha256_file(self.kernel_data) self.object_delay_samples = int(object_delay_samples) self.tail_seconds = float(tail_seconds) self.output_gain = np.float64(output_gain) @@ -106,13 +102,21 @@ class RosellaBinauralRenderer: self.chunk_samples = self.chunk_frames * FRAME_SAMPLES self.dsp_backend = "python" - self.qmf_analysis = QmfAnalysis(SOURCE_CHANNELS, self.kernel_data) - self.hybrid_analysis = HybridAnalysis(SOURCE_CHANNELS, self.kernel_data) + self.bank = RosellaBank( + filterbank, source_channels=SOURCE_CHANNELS, + output_channels=OUTPUT_CHANNELS, kernel_data=kernel_data) + self.filterbank = self.bank.filterbank + self.filterbank_name = FILTERBANK_NAMES[self.filterbank] + self.latency_samples = self.bank.latency_samples + self.kernel_data = self.bank.table_path + self.kernel_data_sha256 = _sha256_file(self.kernel_data) + self.qmf_analysis = self.bank.qmf_analysis + self.hybrid_analysis = self.bank.hybrid_analysis self.core = RosellaRenderer( self.model, SOURCE_CHANNELS, room_impulse_slots=room_impulse_slots) - self.hybrid_synthesis = HybridSynthesis(OUTPUT_CHANNELS, self.kernel_data) - self.qmf_synthesis = QmfSynthesis(OUTPUT_CHANNELS, self.kernel_data) + self.hybrid_synthesis = self.bank.hybrid_synthesis + self.qmf_synthesis = self.bank.qmf_synthesis self.timeline = OamdPositionTimeline( 15, bed_positions=bed_object_positions(bed_channels or ())) @@ -213,7 +217,7 @@ class RosellaBinauralRenderer: stereo *= self.output_gain skip = max(0, min( - len(stereo), ROSSELLA_LATENCY_SAMPLES - self.raw_output_samples)) + len(stereo), self.latency_samples - self.raw_output_samples)) self.raw_output_samples += len(stereo) self.processed_input_samples += len(values) output = stereo[skip:] @@ -231,7 +235,7 @@ class RosellaBinauralRenderer: self._buffer_used = 0 flush_samples = math.ceil( (self.tail_seconds * SAMPLE_RATE - + ROSSELLA_LATENCY_SAMPLES + ROSSELLA_BLOCK_SAMPLES) + + self.latency_samples + ROSSELLA_BLOCK_SAMPLES) / ROSSELLA_BLOCK_SAMPLES) * ROSSELLA_BLOCK_SAMPLES while flush_samples: count = min(flush_samples, self.chunk_samples) @@ -259,7 +263,9 @@ class RosellaBinauralRenderer: "kernel_data": str(self.kernel_data), "kernel_data_sha256": self.kernel_data_sha256, "mode": self.mode, - "latency_compensated_samples": ROSSELLA_LATENCY_SAMPLES, + "filterbank": self.filterbank, + "filterbank_name": self.filterbank_name, + "latency_compensated_samples": self.latency_samples, "object_delay_samples": self.object_delay_samples, "tail_seconds": self.tail_seconds, "metadata_payloads": self.timeline.payload_count, diff --git a/src/rosella_captured_filterbank.py b/src/rosella_captured_filterbank.py new file mode 100644 index 0000000..590da9b --- /dev/null +++ b/src/rosella_captured_filterbank.py @@ -0,0 +1,218 @@ +"""Captured Rosella 64-QMF / 77-hybrid filterbank. + +The Rosella headphone renderer selects its analysis and synthesis callbacks +during preparation, after initialization. The callbacks it actually runs are +this bank: a 640-coefficient QMF prototype, an 11-tap hybrid analysis with a +3-slot high-band delay, and a 448-sample analysis/synthesis latency. The shared +public bank lives in ``rosella_filterbank.py`` and differs in every one of those +parameters (13-tap hybrid, 6-slot delay, 961-sample latency). + +The tables are the float32 values the C++ core compiles in as +``src/hrtf/rosella_filterbank_tables.h``; ``data/rosella_captured_kernels.npz`` +holds the same numbers. State is float64/complex128, like the rest of the +reference implementation. +""" +from __future__ import annotations + +from functools import lru_cache +from pathlib import Path + +import numpy as np + +PROJECT_DIR = Path(__file__).resolve().parent.parent +DEFAULT_CAPTURED_KERNEL_DATA = PROJECT_DIR / "data" / "rosella_captured_kernels.npz" + +QMF_BANDS = 64 +HYBRID_BANDS = 77 +QMF_TAPS = 10 +HYBRID_TAPS = 11 +HYBRID_DELAY = 3 +HYBRID_LOW_BANDS = 16 +ANALYSIS_SYNTHESIS_LATENCY_SAMPLES = 448 + +# The captured renderer's hybrid band order: destination[k] = low[HYBRID_ORDER[k]]. +HYBRID_ORDER = (0, 7, 1, 6, 2, 9, 3, 4, 10, 5, 11, 15, 8, 12, 13, 14) + +# Bands 1 and 3 are conjugated after the permutation. +_CONJUGATED_BANDS = (1, 3) + + +def _tap_sign(tap: int) -> float: + return 1.0 if (tap // 2) % 2 == 0 else -1.0 + + +@lru_cache(maxsize=4) +def _load_tables(path_string: str) -> dict[str, np.ndarray]: + path = Path(path_string) + if not path.is_file(): + raise FileNotFoundError(f"Rosella captured kernel data not found: {path}") + with np.load(path, allow_pickle=False) as archive: + version = archive["format_version"] + if version.shape != (1,) or int(version[0]) != 1: + raise ValueError( + f"unsupported Rosella captured kernel data version in {path}") + prototype = np.asarray(archive["qmf_prototype"], dtype=np.float64) + rotation = np.asarray(archive["rotation"], dtype=np.complex128) + hybrid = np.asarray(archive["hybrid"], dtype=np.complex128) + if prototype.shape != (QMF_BANDS, QMF_TAPS): + raise ValueError("invalid qmf_prototype shape") + if rotation.shape != (QMF_BANDS,): + raise ValueError("invalid rotation shape") + if hybrid.shape != (2 * HYBRID_TAPS,): + raise ValueError("invalid hybrid shape") + return {"qmf_prototype": prototype, "rotation": rotation, "hybrid": hybrid} + + +def load_kernel_tables(path: str | Path = DEFAULT_CAPTURED_KERNEL_DATA + ) -> dict[str, np.ndarray]: + """Load and cache the captured Rosella kernel tables.""" + return _load_tables(str(Path(path).expanduser().resolve())) + + +class QmfAnalysis: + """64-band analysis: 640-coefficient prototype, 9-slot chronological history.""" + + def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA): + if channels <= 0: + raise ValueError("channels must be positive") + tables = load_kernel_tables(kernel_data) + self.prototype = tables["qmf_prototype"] + self.rotation = tables["rotation"] + self.channels = int(channels) + self.history = np.zeros((QMF_TAPS - 1, self.channels, QMF_BANDS), + dtype=np.float64) + + def reset(self): + self.history.fill(0.0) + + def process_chunk(self, hops) -> np.ndarray: + values = np.asarray(hops, dtype=np.float64) + if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS): + raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}") + count = values.shape[0] + joined = np.concatenate((self.history, values), axis=0) + even = np.zeros_like(values) + odd = np.zeros_like(values) + for tap in range(QMF_TAPS): + # The window of output i is joined[i:i+10], oldest tap first. + source = joined[tap:tap + count] + target = even if tap % 2 == 0 else odd + target += source * ( + 0.5 * self.prototype[:, tap] * _tap_sign(tap))[None, None, :] + self.history[:] = joined[-(QMF_TAPS - 1):] if count else self.history + + spectrum = (odd + 1j * even) * self.rotation[None, None, :] + transformed = np.fft.fft(spectrum, axis=-1) + output = np.empty(values.shape, dtype=np.complex128) + output[..., 0::2] = transformed[..., :QMF_BANDS // 2] / QMF_BANDS + output[..., 1::2] = np.conj( + transformed[..., QMF_BANDS // 2:][..., ::-1]) / QMF_BANDS + return output + + +class HybridAnalysis: + """Sparse 64-QMF to 77-hybrid analysis: 11 taps, 3-slot high-band delay.""" + + def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA): + if channels <= 0: + raise ValueError("channels must be positive") + tables = load_kernel_tables(kernel_data) + self.low_kernel = tables["hybrid"] + self.channels = int(channels) + self.history = np.zeros((HYBRID_TAPS - 1, self.channels, QMF_BANDS), + dtype=np.complex128) + + def reset(self): + self.history.fill(0.0) + + def process_chunk(self, qmf) -> np.ndarray: + values = np.asarray(qmf, dtype=np.complex128) + if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS): + raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}") + count = values.shape[0] + joined = np.concatenate((self.history, values), axis=0) + output = np.zeros((count, self.channels, HYBRID_BANDS), dtype=np.complex128) + low = np.zeros((count, self.channels, HYBRID_LOW_BANDS), dtype=np.complex128) + for band, size in ((0, 8), (1, 4), (2, 4)): + offset = 0 if band == 0 else HYBRID_TAPS + spectrum = np.zeros((count, self.channels, size), dtype=np.complex128) + for tap in range(HYBRID_TAPS): + source = joined[tap:tap + count, :, band] + spectrum[:, :, (tap + 1) % size] += source * self.low_kernel[offset + tap] + base = 0 if band == 0 else 4 + band * 4 + low[:, :, base:base + size] = np.fft.fft(spectrum, axis=-1) + output[:, :, :HYBRID_LOW_BANDS] = low[:, :, list(HYBRID_ORDER)] + for band in _CONJUGATED_BANDS: + output[:, :, band] = np.conj(output[:, :, band]) + # The high bands pass through unchanged, delayed by HYBRID_DELAY slots. + delayed = joined[HYBRID_TAPS - 1 - HYBRID_DELAY: + HYBRID_TAPS - 1 - HYBRID_DELAY + count, :, 3:] + output[:, :, HYBRID_LOW_BANDS:] = delayed + self.history[:] = joined[-(HYBRID_TAPS - 1):] if count else self.history + return output + + +class HybridSynthesis: + """Instantaneous 77-hybrid to 64-QMF recombination.""" + + def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA): + if channels <= 0: + raise ValueError("channels must be positive") + self.channels = int(channels) + + def reset(self): + return None + + def process_chunk(self, hybrid) -> np.ndarray: + values = np.asarray(hybrid, dtype=np.complex128) + if values.ndim != 3 or values.shape[1:] != (self.channels, HYBRID_BANDS): + raise ValueError(f"expected [slots,{self.channels},77], got {values.shape}") + output = np.empty((values.shape[0], self.channels, QMF_BANDS), + dtype=np.complex128) + output[..., 0] = (values[..., 0] + values[..., 2] + values[..., 4] + + values[..., 6] + values[..., 7] + values[..., 9] + + np.conj(values[..., 3]) + np.conj(values[..., 1])) + output[..., 1] = (values[..., 12] + values[..., 5] + values[..., 8] + + values[..., 10]) + output[..., 2] = (values[..., 13] + values[..., 14] + values[..., 15] + + values[..., 11]) + output[..., 3:] = values[..., HYBRID_LOW_BANDS:] + return output + + +class QmfSynthesis: + """64-band synthesis: prototype-matched 10-tap overlap with a 9-slot history.""" + + def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA): + if channels <= 0: + raise ValueError("channels must be positive") + tables = load_kernel_tables(kernel_data) + self.prototype = tables["qmf_prototype"] + self.rotation = tables["rotation"] + self.channels = int(channels) + self.history = np.zeros((QMF_TAPS - 1, self.channels, QMF_BANDS), + dtype=np.complex128) + + def reset(self): + self.history.fill(0.0) + + def process_chunk(self, qmf) -> np.ndarray: + values = np.asarray(qmf, dtype=np.complex128) + if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS): + raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}") + count = values.shape[0] + spectrum = np.empty((count, self.channels, QMF_BANDS), dtype=np.complex128) + spectrum[..., :QMF_BANDS // 2] = np.conj(values[..., 0::2]) + spectrum[..., QMF_BANDS // 2:] = values[..., QMF_BANDS - 1::-2] + current = np.fft.fft(spectrum, axis=-1) * self.rotation[None, None, :] + joined = np.concatenate((self.history, current), axis=0) + output = np.zeros((count, self.channels, QMF_BANDS), dtype=np.float64) + reversed_bands = QMF_BANDS - 1 - np.arange(QMF_BANDS) + for lag in range(QMF_TAPS): + source = joined[QMF_TAPS - 1 - lag:QMF_TAPS - 1 - lag + count] + coefficient = (self.prototype[reversed_bands, QMF_TAPS - 1 - lag] * + _tap_sign(lag) * 2.0) + component = source.real if lag % 2 == 0 else source.imag + output += component * coefficient[None, None, :] + self.history[:] = joined[-(QMF_TAPS - 1):] if count else self.history + return output