Add the captured Rosella filterbank and select it per bank

The Rosella path now chooses its filterbank. Bank 1 is the public bank
the reference implementation has always used (13 taps / 6 slots,
961 samples); bank 2 is the captured bank the reference headphone
renderer actually runs (11 taps / 3 slots, 448 samples).

- src/rosella_captured_filterbank.py implements the four half-steps in
  float64/complex128; the captured prototype, rotation and hybrid
  coefficients live in data/rosella_captured_kernels.npz
- src/rosella_bank.py owns the choice: both banks expose the same four
  stages, each with its own coefficients, history and latency
- RosellaBinauralRenderer takes filterbank= and derives the startup
  skip, the flush capacity and the report fields from it
- --rosella-filterbank 1|2 selects the bank, default 1; it is rejected
  for the SOFA and .jochrtf inputs instead of being ignored
- document both banks in docs/rosella_filterbank.{md,en.md}, and point
  the binaural docs, the data README and the options tables at it

Checked: the captured stages reproduce the reference vectors the C++
core checks (within 1e-10), an impulse returns at 961 and 448 samples,
and chunking, reset and bank independence hold for both.
This commit is contained in:
TheM14 committed 2026-10-31 02:26:55 +08:00
1 parent 247e2a90b6
commit 0e98e91762
13 files changed
+606 -24

No files matched your search

+19 -2
View File
@@ -22,7 +22,7 @@ This is research code, not a complete, standards-compliant, or production-grade
or `9.1.6` (`4.0` is the quad layout, a front and a back pair; see
[Speaker layouts and channel order](#speaker-layouts-and-channel-order)).
- Run public SOFA binaural rendering directly from `pcm16 + ID11/OAMD`, without a temporary ADM BWF.
- Keep the binaural DSP in float64/complex128, including 961-sample latency compensation, cross-frame state, and the room tail.
- Keep the binaural DSP in float64/complex128, including the selected filterbank's latency compensation (961 or 448 samples), cross-frame state, and the room tail.
- Use a shared float32/PCM24 WAV writer and explicit PCM24 clipping policy for direct outputs.
- Implement everything in pure Python on NumPy;
- Read or write metadata sidecars and produce metadata, timing, and output reports.
@@ -105,6 +105,8 @@ python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa `
--binaural-mode near
python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa `
--hrtf-cache-policy disk
python main.py input.m4a --binaural --personalized-headphone `
--rosella-filterbank 2
python main.py input.m4a --binaural --binaural-output output.binaural.wav
```
@@ -223,7 +225,21 @@ sample.
### Binaural calculation
See [Binaural Rendering Mathematics](docs/binaural.en.md) for QMF, hybrid processing, direction fields, distance, ITD, room processing, the 512-sample parameter updates above, and 961-sample latency compensation.
See [Binaural Rendering Mathematics](docs/binaural.en.md) for QMF, hybrid processing, direction fields, distance, ITD, room processing, the 512-sample parameter updates above, and the selected filterbank's latency compensation (961 by default, 448 with `--rosella-filterbank 2`).
### Rosella filterbank
`--rosella-filterbank 1|2` selects the filterbank of the `.personalized_headphone`
path:
- `1` (default): the public bank, 13-tap hybrid / 6-slot high-band delay,
961 samples of latency;
- `2`: the Rosella bank captured from the reference renderer, 11-tap hybrid /
3-slot high-band delay, 448 samples of latency.
The option only applies to the Rosella model; the SOFA and `.jochrtf` paths
reject it. See [Rosella filterbank](docs/rosella_filterbank.en.md) for the
parameters, buffers, latency and validation of both banks.
For all options:
@@ -321,3 +337,4 @@ See the [mathematical notes](docs/math.en.md) for the equations used by the deco
- [Mathematical notes](docs/math.en.md) · [中文](docs/math.md)
- [Binaural rendering](docs/binaural.en.md) · [中文](docs/binaural.md)
- [Rosella filterbank](docs/rosella_filterbank.en.md) · [中文](docs/rosella_filterbank.md)
+16 -2
View File
@@ -20,7 +20,7 @@
- 直接渲染 `2.0`、`3.1`、`4.0`、`5.1`、`7.1`、`5.1.2`、`5.1.4`、`7.1.2`、`7.1.4`、`9.1.4`、
`9.1.6`(`4.0` 是 Quad 布局,前后各一对;声道排列见[扬声器布局与声道顺序](#扬声器布局与声道顺序));
- 从 `pcm16 + ID11/OAMD` 直接运行公开 SOFA 双耳渲染,不生成临时 ADM BWF;
- 双耳 DSP 全程使用 float64/complex128,并保留 961-sample latency compensation、跨帧状态和 room 尾声;
- 双耳 DSP 全程使用 float64/complex128,并保留所选滤波器组的延迟补偿(公开版 961、捕获版 448 samples)、跨帧状态和 room 尾声;
- 直接输出统一支持 float32 或 PCM24 WAV,并在 PCM24 削波前提供明确处理策略;
- 全程使用 NumPy 的纯 Python 实现;
- 读取或写入 metadata sidecar,并生成元数据、运行时间和输出摘要。
@@ -103,6 +103,8 @@ python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa `
--binaural-mode near
python main.py input.m4a --binaural --sofa-hrtf C:\HRTF\subject.sofa `
--hrtf-cache-policy disk
python main.py input.m4a --binaural --personalized-headphone `
--rosella-filterbank 2
python main.py input.m4a --binaural --binaural-output output.binaural.wav
```
@@ -192,7 +194,18 @@ outer subpayload offset 与 block offset 落到绝对 sample timeline,再加
### 双耳计算
双耳路径的 QMF、hybrid、方向场、距离、ITD、room、上述 512-sample 参数更新和 961-sample 延迟补偿见[双耳渲染数学](docs/binaural.md)。
双耳路径的 QMF、hybrid、方向场、距离、ITD、room、上述 512-sample 参数更新和所选滤波器组的延迟补偿(默认 961,`--rosella-filterbank 2` 为 448 samples)见[双耳渲染数学](docs/binaural.md)。
### Rosella 滤波器组
`--rosella-filterbank 1|2` 选择 `.personalized_headphone` 路径使用的滤波器组:
- `1`(默认):公开滤波器组,13-tap hybrid / 6-slot 高频延迟,961 samples 延迟;
- `2`:从参考渲染器捕获的 Rosella 滤波器组,11-tap hybrid / 3-slot 高频延迟,
448 samples 延迟。
该选项只作用于 Rosella 模型;SOFA 与 `.jochrtf` 路径给出时报错。两套实现的参数、
缓冲、延迟与验证见 [Rosella 滤波器组](docs/rosella_filterbank.md)。
更多参数可查看:
@@ -283,3 +296,4 @@ JustOneCacophony/
- [数学说明](docs/math.md) · [English](docs/math.en.md)
- [双耳渲染](docs/binaural.md) · [English](docs/binaural.en.md)
- [Rosella 滤波器组](docs/rosella_filterbank.md) · [English](docs/rosella_filterbank.en.md)
+19
View File
@@ -72,6 +72,25 @@ $$Y_p = \sum_{q\in C_p}\Bigl(\operatorname{Re}X_q + j\,s_q\,\operatorname{Im}X_q
The same values also appear in other public implementations of these standards
(for example FFmpeg's `aacps_tablegen.h` and `aacsbrdata.h`).
### Captured Rosella filterbank
`rosella_captured_kernels.npz` holds the bank the reference headphone renderer
actually runs (`--rosella-filterbank 2`):
```text
format_version little-endian int32[1]
qmf_prototype float32[64,10]
rotation complex64[64]
hybrid complex64[22]
```
The table values are promoted to float64/complex128 on load. The full
analysis/synthesis latency is 448 samples (961 for the public bank). These
values come from runtime buffer captures of the reference renderer, not from the
public standards above; the C++ core compiles the same data in as
`src/hrtf/rosella_filterbank_tables.h`. This file also contains no personalized
HRTF or room tuning.
Public availability of a standard does not by itself grant permission to
practice related patent claims.
+17
View File
@@ -64,6 +64,23 @@ $$Y_p = \sum_{q\in C_p}\Bigl(\operatorname{Re}X_q + j\,s_q\,\operatorname{Im}X_q
相同数值可在 FFmpeg(`aacps_tablegen.h`、`aacsbrdata.h`)等公开实现中查到。
### Rosella 捕获版滤波器组
`rosella_captured_kernels.npz` 保存参考头戴渲染器实际运行的那套滤波器组
(`--rosella-filterbank 2`):
```text
format_version little-endian int32[1]
qmf_prototype float32[64,10]
rotation complex64[64]
hybrid complex64[22]
```
表值载入后提升为 float64/complex128;全链 latency 为 448 samples(公开版为 961)。
这些数值来自参考渲染器的运行时缓冲捕获,不是上述公开标准的推导值;C++ 内核以
`src/hrtf/rosella_filterbank_tables.h` 的形式内置同一组数据。文件同样不含个性化
HRTF 或 room 调参。
标准可公开获取不等于获准实施相关专利。
`.sofa` 是用户可见的 source of truth;`.jochrtf` 是可删除、可从 SOFA 重建的
Binary file not shown.
+7
View File
@@ -153,6 +153,13 @@ The runtime is fixed at:
- fifth order, 36 terms, ACN/N3D real spherical harmonics;
- float64 PCM, delay, SH, and room state; complex128 band transfers and spectra.
Those are the SOFA / `.jochrtf` path and the public filterbank. The Rosella
`.personalized_headphone` path selects its bank: the public bank above is the
default (961 samples), and `--rosella-filterbank 2` switches to the bank captured
from the reference renderer (11 taps / 3 slots, 448 samples). Parameters,
buffers, latency and validation of both are in
[Rosella filterbank](rosella_filterbank.en.md).
Real and imaginary unit gains for every hybrid band pass through the same
analysis/synthesis chain to form a 154-real-parameter impulse dictionary. The
compiler does not sample 77 FFT bins. Defaults are `1e-3` projection ridge and
+6
View File
@@ -139,6 +139,12 @@ convention adapter,不能只通过 reshape 进入核心 importer。
- 五阶、36 项、ACN/N3D real spherical harmonics;
- PCM、delay、SH、room state 为 float64;频带传递和频域状态为 complex128。
上面是 SOFA / `.jochrtf` 路径与公开滤波器组的固定参数。Rosella
`.personalized_headphone` 路径的滤波器组可选:默认仍是这一套公开滤波器组(961
samples),`--rosella-filterbank 2` 换成从参考渲染器捕获的版本(11 taps / 3 slots、
448 samples)。两套实现的参数、缓冲、延迟与验证见
[Rosella 滤波器组](rosella_filterbank.md)。
每个 hybrid band 的 real/imaginary 单位增益都通过同一套 analysis/synthesis 链生成
脉冲字典,共 154 个实参数;编译不是直接读取 77 个 FFT bin。默认 projection
ridge 为 `1e-3`,SH ridge 为 `1e-5`。同方向 measurement 先合并,再用球面 Voronoi
+89
View File
@@ -0,0 +1,89 @@
# Rosella filterbank
[中文](rosella_filterbank.md) · [Back to README](../README.en.md)
The Rosella `.personalized_headphone` path can run either of two 64-QMF /
77-hybrid filterbanks: the shared public bank (the default) or the Rosella bank
captured from the reference renderer. The SOFA and `.jochrtf` paths always use
the public bank and never consult the selector.
## Which bank runs
`--rosella-filterbank 1|2` selects the bank of the Rosella path; the library
form is `RosellaBinauralRenderer(..., filterbank=...)`, where omitting the
argument means `1`.
| Value | Implementation | Low-band hybrid | High-band delay | Latency |
| --- | --- | --- | --- | ---: |
| `1` (default) | `rosella_filterbank.py` | 13 taps | 6 slots | 961 samples |
| `2` | `rosella_captured_filterbank.py` | 11 taps | 3 slots | 448 samples |
Both banks expose the same four stateful half-steps (QMF analysis, hybrid
analysis, hybrid synthesis, QMF synthesis), so the choice switches the
coefficients, the high-band delay, the history layout, the startup skip and the
flush capacity together. The analysis and synthesis stages keep their own state
and never share it. Any other value is rejected, and the option is an error when
the binaural input is not a Rosella model: a SOFA or `.jochrtf` render must not
silently ignore it.
## Why the banks differ
The reference headphone renderer selects its analysis and synthesis callbacks
during preparation, after initialization. Capturing only the initial callbacks
identified the wrong filterbank variant. Runtime buffer captures gave these
parameters:
| Parameter | Captured Rosella bank | Public bank |
| --- | --- | --- |
| QMF prototype | 640 asymmetric coefficients | polyphase reorder of the public 640-tap prototype |
| Low-band hybrid filter | 11 taps | 13 taps |
| High-band hybrid delay | 3 slots | 6 slots |
| Analysis/synthesis latency | 448 samples | 961 samples |
The wrong variant produced a frequency-dependent room-tail error. With the
captured room send as input, the existing room recursion already matched the
reference, so the fix changes only the filterbank: not the room coefficients,
the direct HRTF calculation, the phase mapping, or the output EQ. The public
bank stayed as bank `1` so historical renders remain reproducible; the captured
bank is bank `2`.
## Implementation
`rosella_captured_filterbank.py` reproduces the four half-steps of the captured
bank in float64/complex128:
- QMF analysis keeps a 9-slot ring history, reads `(position + tap + 1) mod 10`,
and writes `spectrum[2b] = F[b]/64`, `spectrum[2b+1] = conj(F[63-b])/64`;
- hybrid analysis keeps a 10-slot history, transforms the three low bands with
8/4/4-point transforms, reorders them with a fixed map, conjugates bands 1 and
3, and passes the high bands through delayed by 3 slots;
- QMF synthesis keeps a 9-slot history and accumulates 10 prototype-matched
lags, taking the imaginary part on odd lags.
`data/rosella_captured_kernels.npz` stores the captured prototype
(`float32[64,10]`), rotation (`complex64[64]`) and hybrid coefficients
(`complex64[22]`); they are promoted to float64/complex128 on load. The values
are the ones the C++ core compiles in as `src/hrtf/rosella_filterbank_tables.h`
and contain no personalized HRTF or room tuning.
The startup skip and the flush capacity both come from the selected bank (961 or
448 samples). `.report.json` records the actual choice in
`binaural_renderer_backend`, as `filterbank`, `filterbank_name` and
`latency_compensated_samples`.
## Validation
The local regression script lives in `tests/`, which this repository does not
track (like the rest of its local checks). It covers:
- reference vectors for the four captured half-steps, the same samples the C++
core checks, generated by an independent NumPy implementation, to within
`1e-10` per sample;
- an impulse roundtrip peak at 961 and 448 samples for the two banks;
- chunk invariance (including zero-length and unequal chunks) and `reset()`;
- independence: processing one bank never changes the other bank's state;
- rejection of out-of-range values.
The public filterbank formulas and `.jochrtf` direction field described in
`docs/binaural.md` are unchanged; the captured bank only affects the Rosella
path.
+71
View File
@@ -0,0 +1,71 @@
# Rosella 滤波器组
[English](rosella_filterbank.en.md) · [返回 README](../README.md)
Rosella `.personalized_headphone` 路径可以在两套 64-QMF / 77-hybrid 滤波器组之间选择:
公开滤波器组(默认)与从参考渲染器捕获的 Rosella 滤波器组。SOFA 与 `.jochrtf` 路径
始终使用公开滤波器组,不经过该选择。
## 选择哪一套
`--rosella-filterbank 1|2` 选择 Rosella 路径的滤波器组;库接口是
`RosellaBinauralRenderer(..., filterbank=...)`,省略参数等于 `1`。
| 取值 | 实现 | 低频 hybrid | 高频 slot delay | 延迟 |
| --- | --- | --- | --- | ---: |
| `1`(默认) | `rosella_filterbank.py` | 13 taps | 6 slots | 961 samples |
| `2` | `rosella_captured_filterbank.py` | 11 taps | 3 slots | 448 samples |
两套实现暴露同一组四个有状态的 half-step(QMF 分析、hybrid 分析、hybrid 合成、
QMF 合成),因此切换会同时改变系数、高频延迟、历史缓冲布局、起始延迟跳过量和
flush 容量;分析与合成两级各自持有状态,互不共享。其它取值直接报错;非 Rosella
输入(SOFA、`.jochrtf`)给出该选项也报错,不会静默忽略。
## 为什么两套不同
参考头戴渲染器在初始化之后的准备阶段才选定 analysis/synthesis 回调,只捕获初始化
时的回调会拿到错误的滤波器组变体。运行时缓冲捕获得到的参数是:
| 参数 | Rosella 捕获版 | 公开版 |
| --- | --- | --- |
| QMF prototype | 640 个非对称系数 | 公开 640-tap prototype 的多相重排 |
| 低频 hybrid 滤波 | 11 taps | 13 taps |
| 高频 hybrid delay | 3 slots | 6 slots |
| analysis/synthesis 延迟 | 448 samples | 961 samples |
用错变体会在 room 尾音上引入随频率变化的误差;把捕获到的 room send 作为输入时,
现有 room 递推已经能对上参考,因此修复只更换滤波器组,不改 room 系数、direct HRTF、
相位映射或输出 EQ。公开版是 `1`,捕获版是 `2`;默认保留公开版,便于与历史输出对齐。
## 实现
`rosella_captured_filterbank.py` 用 float64/complex128 复现捕获版的四个 half-step:
- QMF 分析保留 9 slot 环形历史,按 `(position + tap + 1) mod 10` 读取,输出
`spectrum[2b] = F[b]/64`、`spectrum[2b+1] = conj(F[63-b])/64`;
- hybrid 分析保留 10 slot 历史,低频三个 band 分别做 8/4/4 点变换后按固定顺序重排,
band 1 与 band 3 取共轭;高频带原样延迟 3 slot;
- QMF 合成保留 9 slot 历史,用 10 个 lag 的 prototype 匹配累加,奇数 lag 取虚部。
`data/rosella_captured_kernels.npz` 保存捕获版的 prototype(`float32[64,10]`)、
rotation(`complex64[64]`)与 hybrid 系数(`complex64[22]`);载入后提升为
float64/complex128。这些数值与 C++ 内核编译进去的
`src/hrtf/rosella_filterbank_tables.h` 相同,不包含个性化 HRTF 或 room 调参。
运行时的起始延迟跳过与 flush 容量都取自所选滤波器组:`961` 或 `448` samples,
`.report.json` 的 `binaural_renderer_backend` 用 `filterbank`、`filterbank_name` 与
`latency_compensated_samples` 记录实际选择。
## 验证
本地回归脚本放在 `tests/`(该目录不随仓库跟踪,与其余本地验证脚本一致),覆盖:
- 捕获版四个 half-step 的参考向量(与 C++ 内核 `tests/rosella_filterbank_vectors.h`
同一组、由独立 NumPy 实现生成),逐样本误差小于 `1e-10`;
- 两套滤波器组的脉冲往返延迟分别落在 `961` 与 `448` samples;
- 分块处理(含 0 长度与不等长切分)与 `reset()` 不变性;
- 两套滤波器组互不干扰:交叉处理一个 bank 不改变另一个 bank 的状态;
- 越界取值被拒绝。
`docs/binaural.md` 描述的公开滤波器组公式与 `.jochrtf` 方向和公开版一致;捕获版只
作用于 Rosella 路径。
+19 -2
View File
@@ -38,10 +38,10 @@ from binaural_renderer import (
from rosella_binaural_renderer import (
DEFAULT_PERSONALIZED_HEADPHONE,
ROSSELLA_BLOCK_SAMPLES,
ROSSELLA_LATENCY_SAMPLES,
RosellaBinauralRenderer,
resolve_personalized_headphone,
)
from rosella_bank import DEFAULT_FILTERBANK, FILTERBANK_CHOICES, FILTERBANK_NAMES
from sofa_hrtf_field import DEFAULT_HRTF_CACHE_DIR
from speaker_backend import create_speaker_renderer
from speaker_layouts import (SPEAKER_LAYOUT_CHOICES, get_speaker_layout,
@@ -566,6 +566,10 @@ def build_parser():
const=DEFAULT_PERSONALIZED_HEADPHONE,
help="Rosella .personalized_headphone 模型;不带路径时默认 "
"HRTF/binaural.personalized_headphone")
parser.add_argument(
"--rosella-filterbank", type=int, choices=FILTERBANK_CHOICES, default=None,
help="Rosella 模型的滤波器组:1=13 taps/6 slots(默认),"
"2=11 taps/3 slots;仅作用于个性化模型")
parser.add_argument(
"--hrtf-cache-policy", choices=("none", "memory", "disk"), default=None,
help="SOFA 编译缓存;默认 memory,disk 写入可删除的 .jochrtf")
@@ -653,6 +657,10 @@ def main(argv=None):
))
if hrtf_options_used and not binaural_mode:
raise ValueError("SOFA/HRTF 选项仅与 --binaural 一起使用")
if args.rosella_filterbank is not None and not binaural_mode:
raise ValueError("--rosella-filterbank 仅与 --binaural 一起使用")
rosella_filterbank = (DEFAULT_FILTERBANK if args.rosella_filterbank is None
else args.rosella_filterbank)
if (not math.isfinite(args.binaural_tail_seconds)
or args.binaural_tail_seconds < 0):
raise ValueError("binaural-tail-seconds 必须是非负有限值")
@@ -686,6 +694,11 @@ def main(argv=None):
raise ValueError("eac3-target-level 必须在 -31..0 之间")
binaural_hrtf_input = resolve_binaural_hrtf_input(
args, required=binaural_mode and not args.metadata_only)
if (args.rosella_filterbank is not None and binaural_hrtf_input is not None
and binaural_hrtf_input["kind"] != "rosella"):
raise ValueError(
"--rosella-filterbank 只作用于 Rosella .personalized_headphone 输入;"
"当前 SOFA/.jochrtf 路径不使用该滤波器组")
ffmpeg = executable(args.ffmpeg, "FFmpeg")
decode_options = ()
decode_option_info = None
@@ -828,6 +841,7 @@ def main(argv=None):
hrtf_source["path"],
bed_channels=getattr(index, "bed_channels", ()),
mode=binaural_render_mode,
filterbank=rosella_filterbank,
object_delay_samples=args.object_delay_samples,
tail_seconds=args.binaural_tail_seconds,
output_gain=gain_float64,
@@ -844,9 +858,12 @@ def main(argv=None):
f"precision=float64/complex128 "
f"hrtf={hrtf_source['kind']}:{hrtf_source['path']}", flush=True)
if hrtf_source["kind"] == "rosella":
print(
f"[binaural] filterbank={FILTERBANK_NAMES[binaural_decoder.filterbank]}"
f" latency={binaural_decoder.latency_samples}", flush=True)
flush_samples = math.ceil(
(args.binaural_tail_seconds * RATE
+ ROSSELLA_LATENCY_SAMPLES + ROSSELLA_BLOCK_SAMPLES)
+ binaural_decoder.latency_samples + ROSSELLA_BLOCK_SAMPLES)
/ ROSSELLA_BLOCK_SAMPLES) * ROSSELLA_BLOCK_SAMPLES
spool_capacity = frame_count * FRAME_SAMPLES + flush_samples
else:
+101
View File
@@ -0,0 +1,101 @@
"""Selectable Rosella 64-QMF / 77-hybrid filterbank.
The Rosella renderer can run either bank:
``PUBLIC_FILTERBANK`` (1)
The shared public bank of ``rosella_filterbank.py``: a 13-tap hybrid
analysis, a 6-slot high-band delay and 961 samples of analysis/synthesis
latency. This is the bank the reference implementation has always used.
``CAPTURED_FILTERBANK`` (2)
The bank the reference headphone renderer actually runs, captured from it:
an 11-tap hybrid analysis, a 3-slot high-band delay and 448 samples of
latency.
Both banks expose the same four stateful stages, so the choice changes the
coefficients, the high-band delay, the history layout and the latency together.
Nothing is shared between them.
"""
from __future__ import annotations
from pathlib import Path
from rosella_captured_filterbank import (
DEFAULT_CAPTURED_KERNEL_DATA,
HybridAnalysis as CapturedHybridAnalysis,
HybridSynthesis as CapturedHybridSynthesis,
QmfAnalysis as CapturedQmfAnalysis,
QmfSynthesis as CapturedQmfSynthesis,
)
from rosella_filterbank import (
DEFAULT_KERNEL_DATA,
HybridAnalysis as PublicHybridAnalysis,
HybridSynthesis as PublicHybridSynthesis,
QmfAnalysis as PublicQmfAnalysis,
QmfSynthesis as PublicQmfSynthesis,
)
PUBLIC_FILTERBANK = 1
CAPTURED_FILTERBANK = 2
FILTERBANK_CHOICES = (PUBLIC_FILTERBANK, CAPTURED_FILTERBANK)
DEFAULT_FILTERBANK = PUBLIC_FILTERBANK
FILTERBANK_NAMES = {
PUBLIC_FILTERBANK: "public",
CAPTURED_FILTERBANK: "captured",
}
# analysis/synthesis latency of each bank, in samples at 48 kHz
FILTERBANK_LATENCY_SAMPLES = {
PUBLIC_FILTERBANK: 961,
CAPTURED_FILTERBANK: 448,
}
def latency_samples(filterbank: int) -> int:
"""Length of the startup transient the runtime removes for ``filterbank``."""
try:
return FILTERBANK_LATENCY_SAMPLES[int(filterbank)]
except (KeyError, TypeError, ValueError):
raise ValueError(
f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}"
) from None
class RosellaBank:
"""The four stages of one bank, each with its own state."""
def __init__(self, filterbank: int = DEFAULT_FILTERBANK, *,
source_channels: int, output_channels: int,
kernel_data: str | Path | None = None):
if filterbank not in FILTERBANK_CHOICES:
raise ValueError(
f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}")
kind = int(filterbank)
if source_channels <= 0 or output_channels <= 0:
raise ValueError("channel counts must be positive")
self.filterbank = kind
self.name = FILTERBANK_NAMES[kind]
self.latency_samples = latency_samples(kind)
if kind == PUBLIC_FILTERBANK:
self.table_path = Path(
DEFAULT_KERNEL_DATA if kernel_data is None else kernel_data
).expanduser().resolve()
self.qmf_analysis = PublicQmfAnalysis(source_channels, self.table_path)
self.hybrid_analysis = PublicHybridAnalysis(source_channels, self.table_path)
self.hybrid_synthesis = PublicHybridSynthesis(output_channels, self.table_path)
self.qmf_synthesis = PublicQmfSynthesis(output_channels, self.table_path)
else:
self.table_path = Path(
DEFAULT_CAPTURED_KERNEL_DATA if kernel_data is None else kernel_data
).expanduser().resolve()
self.qmf_analysis = CapturedQmfAnalysis(source_channels, self.table_path)
self.hybrid_analysis = CapturedHybridAnalysis(source_channels, self.table_path)
self.hybrid_synthesis = CapturedHybridSynthesis(output_channels, self.table_path)
self.qmf_synthesis = CapturedQmfSynthesis(output_channels, self.table_path)
def reset(self):
self.qmf_analysis.reset()
self.hybrid_analysis.reset()
self.hybrid_synthesis.reset()
self.qmf_synthesis.reset()
+24 -18
View File
@@ -16,22 +16,16 @@ import numpy as np
from bed_layout import bed_object_positions
from binaural_metadata import OamdPositionTimeline
from rosella_bank import (DEFAULT_FILTERBANK, FILTERBANK_CHOICES, FILTERBANK_NAMES,
RosellaBank)
from rosella_core import RosellaRenderer
from rosella_direct import BINAURAL_PROFILE_NAMES
from rosella_filterbank import (
DEFAULT_KERNEL_DATA,
HybridAnalysis,
HybridSynthesis,
QmfAnalysis,
QmfSynthesis,
)
from rosella_model import RosellaModel, load_personalized_headphone
SAMPLE_RATE = 48000
FRAME_SAMPLES = 1536
ROSSELLA_BLOCK_SAMPLES = 512
QMF_HOP_SAMPLES = 64
ROSSELLA_LATENCY_SAMPLES = 961
SOURCE_CHANNELS = 16
OUTPUT_CHANNELS = 2
PROJECT_DIR = Path(__file__).resolve().parent.parent
@@ -67,7 +61,8 @@ class RosellaBinauralRenderer:
personalized_headphone: str | Path | RosellaModel,
*,
mode: str = "mid",
kernel_data: str | Path = DEFAULT_KERNEL_DATA,
filterbank: int = DEFAULT_FILTERBANK,
kernel_data: str | Path | None = None,
object_delay_samples: int = 1473,
tail_seconds: float = 5.0,
output_gain: float = 1.0,
@@ -76,6 +71,9 @@ class RosellaBinauralRenderer:
bed_channels=None):
if mode not in BINAURAL_PROFILE_NAMES:
raise ValueError("binaural mode must be near, mid, or far")
if filterbank not in FILTERBANK_CHOICES:
raise ValueError(
f"Rosella filterbank must be 1 (public) or 2 (captured), got {filterbank!r}")
if int(object_delay_samples) < 0:
raise ValueError("object_delay_samples must be non-negative")
if float(tail_seconds) < 0.0:
@@ -97,8 +95,6 @@ class RosellaBinauralRenderer:
self.mode = mode
self.profile_index = BINAURAL_PROFILE_NAMES[mode]
self.kernel_data = Path(kernel_data).expanduser().resolve()
self.kernel_data_sha256 = _sha256_file(self.kernel_data)
self.object_delay_samples = int(object_delay_samples)
self.tail_seconds = float(tail_seconds)
self.output_gain = np.float64(output_gain)
@@ -106,13 +102,21 @@ class RosellaBinauralRenderer:
self.chunk_samples = self.chunk_frames * FRAME_SAMPLES
self.dsp_backend = "python"
self.qmf_analysis = QmfAnalysis(SOURCE_CHANNELS, self.kernel_data)
self.hybrid_analysis = HybridAnalysis(SOURCE_CHANNELS, self.kernel_data)
self.bank = RosellaBank(
filterbank, source_channels=SOURCE_CHANNELS,
output_channels=OUTPUT_CHANNELS, kernel_data=kernel_data)
self.filterbank = self.bank.filterbank
self.filterbank_name = FILTERBANK_NAMES[self.filterbank]
self.latency_samples = self.bank.latency_samples
self.kernel_data = self.bank.table_path
self.kernel_data_sha256 = _sha256_file(self.kernel_data)
self.qmf_analysis = self.bank.qmf_analysis
self.hybrid_analysis = self.bank.hybrid_analysis
self.core = RosellaRenderer(
self.model, SOURCE_CHANNELS,
room_impulse_slots=room_impulse_slots)
self.hybrid_synthesis = HybridSynthesis(OUTPUT_CHANNELS, self.kernel_data)
self.qmf_synthesis = QmfSynthesis(OUTPUT_CHANNELS, self.kernel_data)
self.hybrid_synthesis = self.bank.hybrid_synthesis
self.qmf_synthesis = self.bank.qmf_synthesis
self.timeline = OamdPositionTimeline(
15, bed_positions=bed_object_positions(bed_channels or ()))
@@ -213,7 +217,7 @@ class RosellaBinauralRenderer:
stereo *= self.output_gain
skip = max(0, min(
len(stereo), ROSSELLA_LATENCY_SAMPLES - self.raw_output_samples))
len(stereo), self.latency_samples - self.raw_output_samples))
self.raw_output_samples += len(stereo)
self.processed_input_samples += len(values)
output = stereo[skip:]
@@ -231,7 +235,7 @@ class RosellaBinauralRenderer:
self._buffer_used = 0
flush_samples = math.ceil(
(self.tail_seconds * SAMPLE_RATE
+ ROSSELLA_LATENCY_SAMPLES + ROSSELLA_BLOCK_SAMPLES)
+ self.latency_samples + ROSSELLA_BLOCK_SAMPLES)
/ ROSSELLA_BLOCK_SAMPLES) * ROSSELLA_BLOCK_SAMPLES
while flush_samples:
count = min(flush_samples, self.chunk_samples)
@@ -259,7 +263,9 @@ class RosellaBinauralRenderer:
"kernel_data": str(self.kernel_data),
"kernel_data_sha256": self.kernel_data_sha256,
"mode": self.mode,
"latency_compensated_samples": ROSSELLA_LATENCY_SAMPLES,
"filterbank": self.filterbank,
"filterbank_name": self.filterbank_name,
"latency_compensated_samples": self.latency_samples,
"object_delay_samples": self.object_delay_samples,
"tail_seconds": self.tail_seconds,
"metadata_payloads": self.timeline.payload_count,
+218
View File
@@ -0,0 +1,218 @@
"""Captured Rosella 64-QMF / 77-hybrid filterbank.
The Rosella headphone renderer selects its analysis and synthesis callbacks
during preparation, after initialization. The callbacks it actually runs are
this bank: a 640-coefficient QMF prototype, an 11-tap hybrid analysis with a
3-slot high-band delay, and a 448-sample analysis/synthesis latency. The shared
public bank lives in ``rosella_filterbank.py`` and differs in every one of those
parameters (13-tap hybrid, 6-slot delay, 961-sample latency).
The tables are the float32 values the C++ core compiles in as
``src/hrtf/rosella_filterbank_tables.h``; ``data/rosella_captured_kernels.npz``
holds the same numbers. State is float64/complex128, like the rest of the
reference implementation.
"""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
import numpy as np
PROJECT_DIR = Path(__file__).resolve().parent.parent
DEFAULT_CAPTURED_KERNEL_DATA = PROJECT_DIR / "data" / "rosella_captured_kernels.npz"
QMF_BANDS = 64
HYBRID_BANDS = 77
QMF_TAPS = 10
HYBRID_TAPS = 11
HYBRID_DELAY = 3
HYBRID_LOW_BANDS = 16
ANALYSIS_SYNTHESIS_LATENCY_SAMPLES = 448
# The captured renderer's hybrid band order: destination[k] = low[HYBRID_ORDER[k]].
HYBRID_ORDER = (0, 7, 1, 6, 2, 9, 3, 4, 10, 5, 11, 15, 8, 12, 13, 14)
# Bands 1 and 3 are conjugated after the permutation.
_CONJUGATED_BANDS = (1, 3)
def _tap_sign(tap: int) -> float:
return 1.0 if (tap // 2) % 2 == 0 else -1.0
@lru_cache(maxsize=4)
def _load_tables(path_string: str) -> dict[str, np.ndarray]:
path = Path(path_string)
if not path.is_file():
raise FileNotFoundError(f"Rosella captured kernel data not found: {path}")
with np.load(path, allow_pickle=False) as archive:
version = archive["format_version"]
if version.shape != (1,) or int(version[0]) != 1:
raise ValueError(
f"unsupported Rosella captured kernel data version in {path}")
prototype = np.asarray(archive["qmf_prototype"], dtype=np.float64)
rotation = np.asarray(archive["rotation"], dtype=np.complex128)
hybrid = np.asarray(archive["hybrid"], dtype=np.complex128)
if prototype.shape != (QMF_BANDS, QMF_TAPS):
raise ValueError("invalid qmf_prototype shape")
if rotation.shape != (QMF_BANDS,):
raise ValueError("invalid rotation shape")
if hybrid.shape != (2 * HYBRID_TAPS,):
raise ValueError("invalid hybrid shape")
return {"qmf_prototype": prototype, "rotation": rotation, "hybrid": hybrid}
def load_kernel_tables(path: str | Path = DEFAULT_CAPTURED_KERNEL_DATA
) -> dict[str, np.ndarray]:
"""Load and cache the captured Rosella kernel tables."""
return _load_tables(str(Path(path).expanduser().resolve()))
class QmfAnalysis:
"""64-band analysis: 640-coefficient prototype, 9-slot chronological history."""
def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA):
if channels <= 0:
raise ValueError("channels must be positive")
tables = load_kernel_tables(kernel_data)
self.prototype = tables["qmf_prototype"]
self.rotation = tables["rotation"]
self.channels = int(channels)
self.history = np.zeros((QMF_TAPS - 1, self.channels, QMF_BANDS),
dtype=np.float64)
def reset(self):
self.history.fill(0.0)
def process_chunk(self, hops) -> np.ndarray:
values = np.asarray(hops, dtype=np.float64)
if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS):
raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}")
count = values.shape[0]
joined = np.concatenate((self.history, values), axis=0)
even = np.zeros_like(values)
odd = np.zeros_like(values)
for tap in range(QMF_TAPS):
# The window of output i is joined[i:i+10], oldest tap first.
source = joined[tap:tap + count]
target = even if tap % 2 == 0 else odd
target += source * (
0.5 * self.prototype[:, tap] * _tap_sign(tap))[None, None, :]
self.history[:] = joined[-(QMF_TAPS - 1):] if count else self.history
spectrum = (odd + 1j * even) * self.rotation[None, None, :]
transformed = np.fft.fft(spectrum, axis=-1)
output = np.empty(values.shape, dtype=np.complex128)
output[..., 0::2] = transformed[..., :QMF_BANDS // 2] / QMF_BANDS
output[..., 1::2] = np.conj(
transformed[..., QMF_BANDS // 2:][..., ::-1]) / QMF_BANDS
return output
class HybridAnalysis:
"""Sparse 64-QMF to 77-hybrid analysis: 11 taps, 3-slot high-band delay."""
def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA):
if channels <= 0:
raise ValueError("channels must be positive")
tables = load_kernel_tables(kernel_data)
self.low_kernel = tables["hybrid"]
self.channels = int(channels)
self.history = np.zeros((HYBRID_TAPS - 1, self.channels, QMF_BANDS),
dtype=np.complex128)
def reset(self):
self.history.fill(0.0)
def process_chunk(self, qmf) -> np.ndarray:
values = np.asarray(qmf, dtype=np.complex128)
if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS):
raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}")
count = values.shape[0]
joined = np.concatenate((self.history, values), axis=0)
output = np.zeros((count, self.channels, HYBRID_BANDS), dtype=np.complex128)
low = np.zeros((count, self.channels, HYBRID_LOW_BANDS), dtype=np.complex128)
for band, size in ((0, 8), (1, 4), (2, 4)):
offset = 0 if band == 0 else HYBRID_TAPS
spectrum = np.zeros((count, self.channels, size), dtype=np.complex128)
for tap in range(HYBRID_TAPS):
source = joined[tap:tap + count, :, band]
spectrum[:, :, (tap + 1) % size] += source * self.low_kernel[offset + tap]
base = 0 if band == 0 else 4 + band * 4
low[:, :, base:base + size] = np.fft.fft(spectrum, axis=-1)
output[:, :, :HYBRID_LOW_BANDS] = low[:, :, list(HYBRID_ORDER)]
for band in _CONJUGATED_BANDS:
output[:, :, band] = np.conj(output[:, :, band])
# The high bands pass through unchanged, delayed by HYBRID_DELAY slots.
delayed = joined[HYBRID_TAPS - 1 - HYBRID_DELAY:
HYBRID_TAPS - 1 - HYBRID_DELAY + count, :, 3:]
output[:, :, HYBRID_LOW_BANDS:] = delayed
self.history[:] = joined[-(HYBRID_TAPS - 1):] if count else self.history
return output
class HybridSynthesis:
"""Instantaneous 77-hybrid to 64-QMF recombination."""
def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA):
if channels <= 0:
raise ValueError("channels must be positive")
self.channels = int(channels)
def reset(self):
return None
def process_chunk(self, hybrid) -> np.ndarray:
values = np.asarray(hybrid, dtype=np.complex128)
if values.ndim != 3 or values.shape[1:] != (self.channels, HYBRID_BANDS):
raise ValueError(f"expected [slots,{self.channels},77], got {values.shape}")
output = np.empty((values.shape[0], self.channels, QMF_BANDS),
dtype=np.complex128)
output[..., 0] = (values[..., 0] + values[..., 2] + values[..., 4] +
values[..., 6] + values[..., 7] + values[..., 9] +
np.conj(values[..., 3]) + np.conj(values[..., 1]))
output[..., 1] = (values[..., 12] + values[..., 5] + values[..., 8] +
values[..., 10])
output[..., 2] = (values[..., 13] + values[..., 14] + values[..., 15] +
values[..., 11])
output[..., 3:] = values[..., HYBRID_LOW_BANDS:]
return output
class QmfSynthesis:
"""64-band synthesis: prototype-matched 10-tap overlap with a 9-slot history."""
def __init__(self, channels: int, kernel_data: str | Path = DEFAULT_CAPTURED_KERNEL_DATA):
if channels <= 0:
raise ValueError("channels must be positive")
tables = load_kernel_tables(kernel_data)
self.prototype = tables["qmf_prototype"]
self.rotation = tables["rotation"]
self.channels = int(channels)
self.history = np.zeros((QMF_TAPS - 1, self.channels, QMF_BANDS),
dtype=np.complex128)
def reset(self):
self.history.fill(0.0)
def process_chunk(self, qmf) -> np.ndarray:
values = np.asarray(qmf, dtype=np.complex128)
if values.ndim != 3 or values.shape[1:] != (self.channels, QMF_BANDS):
raise ValueError(f"expected [slots,{self.channels},64], got {values.shape}")
count = values.shape[0]
spectrum = np.empty((count, self.channels, QMF_BANDS), dtype=np.complex128)
spectrum[..., :QMF_BANDS // 2] = np.conj(values[..., 0::2])
spectrum[..., QMF_BANDS // 2:] = values[..., QMF_BANDS - 1::-2]
current = np.fft.fft(spectrum, axis=-1) * self.rotation[None, None, :]
joined = np.concatenate((self.history, current), axis=0)
output = np.zeros((count, self.channels, QMF_BANDS), dtype=np.float64)
reversed_bands = QMF_BANDS - 1 - np.arange(QMF_BANDS)
for lag in range(QMF_TAPS):
source = joined[QMF_TAPS - 1 - lag:QMF_TAPS - 1 - lag + count]
coefficient = (self.prototype[reversed_bands, QMF_TAPS - 1 - lag] *
_tap_sign(lag) * 2.0)
component = source.real if lag % 2 == 0 else source.imag
output += component * coefficient[None, None, :]
self.history[:] = joined[-(QMF_TAPS - 1):] if count else self.history
return output