Add JOC downmix configurations 0..4, cross-layer checks, and drop the native core

This commit is contained in:
TheM14 committed 2026-10-16 19:46:24 +08:00
1 parent 428772eb87
commit a1853f2dc5
37 files changed
+926 -6420

No files matched your search

+3 -10
View File
@@ -254,16 +254,9 @@ late sends, the unitary FDN, the 120–180 Hz cosine-squared LFE low-pass, and r
calibration are JOC project-defined behavior, not constants published by SOFA or
Dolby.
The public SOFA binaural renderer defaults to the C++20 native core under
`--backend auto/native` (`ejoc_sofa_binaural_*` in `lib/eac3joc_core.dll`): the
filterbank, the SH direction-field evaluation, the per-object direct/early
histories and the shared FDN all run natively, while Python only compiles the
SOFA source and issues the per-512-sample metadata updates. When the native
library is unavailable the renderer falls back to the Python/NumPy reference
implementation; the two agree to better than 1e-9. `--backend python` forces
the Python backend.
`--backend` still selects native/Python JOC reconstruction and speaker rendering;
native acceleration for the public binaural DSP is outside the current API.
The filterbank, the SH direction-field evaluation, the per-object direct/early
histories and the shared FDN all run in Python/NumPy, with the binaural DSP kept
in float64/complex128 throughout.
## Technical references and rights boundary
+2 -5
View File
@@ -224,11 +224,8 @@ Near/Mid/Far、equal-power direct level、六面 shoebox 一阶 image source、l
unitary FDN、LFE 120–180 Hz cosine-squared 低通及 room calibration 都是 JOC
项目定义行为,不是 SOFA 或 Dolby 公布常数。
公开 SOFA 双耳渲染在 `--backend auto/native` 下默认走 C++20 原生核
(`lib/eac3joc_core.dll` 的 `ejoc_sofa_binaural_*` 接口:filterbank、SH 方向场求值、
逐对象 early/direct 历史与共享 FDN 全部在原生侧执行,Python 只做 SOFA 编译与每
512-sample 的元数据更新);原生库不可用时自动回退 Python/NumPy 参考实现,两者
逐值一致(差异 < 1e-9)。`--backend python` 强制使用 Python 后端。
filterbank、SH 方向场求值、逐对象 early/direct 历史与共享 FDN 全部在 Python/NumPy
一侧执行,双耳 DSP 全程 float64/complex128。
## 技术引用与权利边界
+20 -20
View File
@@ -150,7 +150,12 @@ G_{\mathrm{clip}}=
1+\frac{y}{32}2^{x-4}.
$$
It is applied to object PCM after inverse QMF and does not apply to LFE.
It is folded into the analysis-side scalar $S=\frac{1}{16}G_{\mathrm{clip}}$, which multiplies
every core-bed sample before the QMF analysis; the $\times16$ after the object inverse QMF cancels the
$1/16$, so the net effect is a multiply by $G_{\mathrm{clip}}$ and the $[-1,1]$ clamp sits **after** the
gain. The scalar keeps two per-frame slots: the first 256 samples of a frame ramp linearly from the
previous frame's value to this frame's value and the rest stays constant, and the first frame after a
reset takes the new value directly. LFE bypasses this scalar.
## 3. Parameter-band expansion and time interpolation
@@ -231,36 +236,31 @@ This consists of the analysis window/polyphase stage, modulation, a 64-point FFT
## 5. QMF-domain processing of core channels
L, R, and C are delayed by ten QMF slots before entering the object matrix:
$k$ selects the core channel count $N\in\{5,7\}$ ($k=0\ldots4$ maps to $N=5,7,7,5,7$) and the
phase switch $\sigma(k)=1\iff k\in\{3,4\}$. Every core channel is delayed by ten QMF slots
before entering the object matrix; the $-j$ rotation ($b>0$) and the band-0 21-tap complex FIR
are both gated by that same switch and applied to Ls and Rs only:
$$
\widehat X_{c,b,t}=X_{c,b,t-10},
\qquad c\in\{L,R,C\}.
$$
Ls and Rs use the same ten-slot delay and a $-j$ rotation for $b>0$:
$$
\widehat X_{c,b,t}=-jX_{c,b,t-10},
\qquad c\in\{Ls,Rs\},\ b>0.
$$
Band 0 of each surround channel additionally passes through a 21-tap complex FIR:
$$
\widehat X_{c,0,t}=
\sum_{k=0}^{20}h_kX_{c,0,t-k}.
\widehat X_{c,b,t}=
\begin{cases}
-j\,X_{c,b,t-10}, & \sigma(k)=1,\ c\in\{Ls,Rs\},\ b>0,\\[1mm]
\sum_{m=0}^{20}h_mX_{c,0,t-m}, & \sigma(k)=1,\ c\in\{Ls,Rs\},\ b=0,\\[1mm]
X_{c,b,t-10}, & \text{otherwise}.
\end{cases}
$$
For $N=7$ the sixth and seventh core channels (Lb/Rb or Tfl/Tfr) behave like the front channels
and only take the ten-slot delay: channel identity affects metadata and rendering, not the DSP.
These delays and filter histories are decoder state and cannot be reset independently for every frame.
## 6. Object matrix
For each object $o$, subband $b$, and slot $t$, the object's frequency-domain value is a linear combination of the five core channels:
For each object $o$, subband $b$, and slot $t$, the object's frequency-domain value is a linear combination of the $N$ core channels:
$$
Z_{o,b,t}=
\sum_{c=0}^{4}
\sum_{c=0}^{N-1}
M_{o,c,b,t}\widehat X_{c,b,t}.
$$
+18 -21
View File
@@ -150,7 +150,10 @@ G_{\mathrm{clip}}=
1+\frac{y}{32}2^{x-4}.
$$
它在对象 inverse QMF 之后作用于对象 PCM,不作用于 LFE。
它并入分析端标量 $S=\frac{1}{16}G_{\mathrm{clip}}$,乘在核心 bed 的每个采样上(QMF 分析之前);
对象 inverse QMF 之后的 $\times16$ 抵消 $1/16$,净效果是乘 $G_{\mathrm{clip}}$,因此钳位 $[-1,1]$
处在增益**之后**。标量按帧保留两个槽,本帧前 256 个采样从上一帧值线性过渡到本帧值、之后为常数,
复位后的第一帧直接取新值。LFE 走旁通通路,不参与该标量。
## 3. 参数带展开与时间插值
@@ -231,36 +234,30 @@ $$
## 5. 核心声道的 QMF 域处理
L、R、C 在进入对象矩阵前延迟 10 个 QMF 时槽:
$k$ 决定核心通道数 $N\in\{5,7\}$($k=0\ldots4$ 对应 $N=5,7,7,5,7$)与相位开关
$\sigma(k)=1\iff k\in\{3,4\}$。所有核心声道在进入对象矩阵前延迟 10 个 QMF 时槽;
$-j$ 旋转($b>0$)与 band-0 的 21-tap 复 FIR **由同一个开关**只作用于 Ls、Rs:
$$
\widehat X_{c,b,t}=X_{c,b,t-10},
\qquad c\in\{L,R,C\}.
\widehat X_{c,b,t}=
\begin{cases}
-j\,X_{c,b,t-10}, & \sigma(k)=1,\ c\in\{Ls,Rs\},\ b>0,\\[1mm]
\sum_{m=0}^{20}h_mX_{c,0,t-m}, & \sigma(k)=1,\ c\in\{Ls,Rs\},\ b=0,\\[1mm]
X_{c,b,t-10}, & \text{其他情况}.
\end{cases}
$$
Ls、Rs 同样延迟 10 个时槽,并在 $b>0$ 时作 $-j$ 旋转:
$$
\widehat X_{c,b,t}=-jX_{c,b,t-10},
\qquad c\in\{Ls,Rs\},\ b>0.
$$
环绕声道的 band 0 还经过 21-tap 复 FIR:
$$
\widehat X_{c,0,t}=
\sum_{k=0}^{20}h_kX_{c,0,t-k}.
$$
这些延迟和滤波历史属于解码状态,不能按帧独立清零。
$N=7$ 时第 6、7 个核心通道(Lb/Rb 或 Tfl/Tfr)与前级相同,只做 10 时槽延迟:
通道身份只影响元数据与渲染,不进入 DSP。这些延迟和滤波历史属于解码状态,
不能按帧独立清零。
## 6. 对象矩阵
对每个对象 $o$、子带 $b$ 和时槽 $t$,对象频域值为五个核心声道的线性组合:
对每个对象 $o$、子带 $b$ 和时槽 $t$,对象频域值为 $N$ 个核心声道的线性组合:
$$
Z_{o,b,t}=
\sum_{c=0}^{4}
\sum_{c=0}^{N-1}
M_{o,c,b,t}\widehat X_{c,b,t}.
$$
-178
View File
@@ -1,178 +0,0 @@
# JustOneCacophony native-core notes
[中文](native.md) · [Back to README](../README.en.md)
## 1. Responsibility boundary
`native/` contains only the state-heavy, frequently called DSP and speaker-rendering kernels. High-level EMDF/JOC/OAMD parsing, error reporting, ADM assembly, and the CLI remain in Python.
Python calls a C ABI through the standard-library `ctypes` module. The native core does not use pybind11, Cython, FFTW, MKL, or OpenMP. It is an optional acceleration path and does not expand the set of supported stream variants.
Main files:
```text
native/include/eac3joc_core.h C ABI
native/src/eac3joc_core.cpp JOC/QMF object reconstruction
native/src/speaker_renderer.cpp object-to-speaker rendering
native/src/qmf_tables.h QMF tables
native/src/speaker_layouts.h layout tables
native/src/joc_huffman_tables.h JOC Huffman tables
src/native_renderer.py JOC ctypes bridge
src/speaker_native_renderer.py speaker ctypes bridge
```
## 2. JOC rendering ABI
An opaque renderer owns all cross-frame state. Its main call is:
```c
int ejoc_renderer_process(
ejoc_renderer_handle handle,
const float* bed5_planar, /* [5][1536] */
const float* lfe, /* [1536] or NULL */
uint32_t object_mask,
const uint8_t* n_bands, /* [15] */
const uint8_t* n_dpoints, /* [15] */
const uint8_t* slope_idx, /* [15] */
const uint8_t* offset_ts, /* [15][2] */
const double* dq, /* [15][2][5][23] */
double clipgain,
float phase_new,
float output_scale,
float* output16_planar); /* [16][1536] */
```
Python performs JOC Huffman decoding, differential reconstruction, and dequantization before the call, with the dense and sparse syntaxes sharing one entry point. The native core consumes the already dequantized `dq` in double precision, and both syntaxes have the same layout at that ABI.
Thread control is exposed as:
```c
int ejoc_renderer_set_threads(ejoc_renderer_handle handle, uint32_t total_threads);
uint32_t ejoc_renderer_thread_count(ejoc_renderer_handle handle);
```
`total_threads` includes the calling thread. Frames must be submitted sequentially to one renderer instance; the instance may parallelize work across objects and analysis channels.
## 3. Cross-frame state
Each JOC renderer stores:
- analysis FIFO: `double[5][9][64]`;
- L/R/C analysis delay: `float[3][10][64]`;
- Ls/Rs QMF delay: `complex<double>[2][10][64]`;
- Ls/Rs band-0 FIR history: `complex<double>[2][20]`;
- previous matrix interpolation values: `double[15][5][64]`;
- inverse-QMF state: `double[15][640]`;
- LFE delay: `double[1217]`.
This state belongs to the renderer instance. Processing cannot be arbitrarily segmented or reordered without a corresponding state checkpoint.
## 4. FFT, QMF, and precision
The native core contains a fixed 64-point radix-2 complex FFT:
- analysis QMF uses a forward FFT followed by division by 64;
- inverse QMF uses the fixed reorder, rotation, and 640-value active-window state;
- no external FFT library is called.
The JOC path uses:
- float32 core-PCM input;
- double matrices, complex QMF, FFT, FIR, and cross-frame state;
- float32 phase and final gain;
- float32 16-channel object output.
## 5. Speaker-rendering ABI
The same shared library exports object-to-speaker rendering:
```c
uint32_t ejoc_speaker_layout_channel_count(uint32_t speaker_bitfield);
ejoc_speaker_renderer_handle
ejoc_speaker_renderer_create(uint32_t speaker_bitfield);
int ejoc_speaker_renderer_process(
ejoc_speaker_renderer_handle handle,
const float* objects16_interleaved,
uint32_t sample_count,
uint32_t metadata_count,
const uint32_t* metadata_offsets,
const uint32_t* ramp_durations,
const uint16_t* positions_q15,
const uint8_t* region_indices,
const uint8_t* height_enabled,
const double* object_gains,
double* output_interleaved);
```
Input channel 0 is LFE and channels 1–15 are objects. Each metadata entry is an object-state snapshot. `sample_count` must be a multiple of 32; unfinished gain ramps remain in the handle and continue across calls.
The speaker path uses float32 object input, double coordinates/gains/accumulation, and interleaved double output. Quantization to float32 or PCM24 happens when the WAV is written.
Supported layouts:
```text
2.0 3.1 5.1 7.1 5.1.2 5.1.4 7.1.2 7.1.4 9.1.4 9.1.6
```
## 6. Binaural-rendering ABI
The shared library provides a 512-sample float64 binaural DSP interface:
```c
ejoc_binaural_renderer_handle ejoc_binaural_renderer_create(void);
int ejoc_binaural_renderer_configure_kernels(...);
int ejoc_binaural_renderer_configure_room(...);
int ejoc_binaural_renderer_process(
ejoc_binaural_renderer_handle handle,
const double* input16_interleaved, /* [512][16] */
const double* gains_complex, /* [16][2][77][2] */
const double* room_sends, /* [16] */
double output_gain,
double* output_stereo_interleaved); /* [512][2] */
```
Python parses the model, evaluates the OAMD timeline, and supplies complex gains and room sends every 512 samples. The C++ handle owns QMF, hybrid, recursive-room, and QMF-synthesis state. Inputs, state, accumulation, and output are double/complex double.
## 7. Building
The CMake definition is `native/CMakeLists.txt`. Run from the repository root:
```powershell
cmake -S native -B build/cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$PWD/lib"
cmake --build build/cmake --config Release
cmake --install build/cmake --config Release
```
Platform runtime names:
```text
Windows lib/eac3joc_core.dll
Linux lib/libeac3joc_core.so
macOS lib/libeac3joc_core.dylib
```
The MSVC configuration uses the static CRT. Other runtime dependencies depend on the platform and toolchain and should be checked independently before publishing a prebuilt library.
The repository does not include native binaries by default. A prebuilt Release runtime or a locally built runtime can be placed directly under `lib/`.
## 8. Runtime lookup and fallback
Lookup order:
1. explicit `--native-library`;
2. `EAC3JOC_NATIVE_LIBRARY`;
3. the standard platform filename under `lib/`.
`--backend auto` falls back to NumPy when loading fails, and `--backend python` skips native discovery. The current CLI also prints the failure and falls back for `--backend native`; this existing behavior should not be read as successful native execution.
## 9. Implementation boundaries
- The native layer accepts only dense-JOC data already parsed by Python.
- The ABI fixes a 1536-sample JOC frame, at most 15 objects, at most 23 parameter bands, and at most 2 data points.
- The shared library and Python bridge must report the same ABI version.
- Only the ABI and data types are specified across platforms; bit-identical float64 results are not guaranteed.
- Private table headers under `native/src/` serve the native side only. The current repository does not include the scripts that generated those headers.
See the [mathematical notes](math.en.md) for the related formulas.
-191
View File
@@ -1,191 +0,0 @@
# JustOneCacophony 原生核说明
[English](native.en.md) · [返回 README](../README.md)
## 1. 职责边界
`native/` 只承载状态密集、调用频繁的 DSP 与扬声器渲染核。EMDF/JOC/OAMD 高层解析、错误报告、ADM 组装和 CLI 保留在 Python 中。
Python 通过标准库 `ctypes` 调用 C ABI;原生核不使用 pybind11、Cython、FFTW、MKL 或 OpenMP。它是可选加速路径,不扩大项目所支持的码流范围。
主要文件:
```text
native/include/eac3joc_core.h C ABI
native/src/eac3joc_core.cpp JOC/QMF 对象重建
native/src/speaker_renderer.cpp 对象到扬声器渲染
native/src/qmf_tables.h QMF 表
native/src/speaker_layouts.h 布局表
native/src/joc_huffman_tables.h JOC Huffman 表
src/native_renderer.py JOC ctypes 桥
src/speaker_native_renderer.py 扬声器 ctypes 桥
```
## 2. JOC 渲染 ABI
一个 opaque renderer 保存所有跨帧状态。主要调用为:
```c
int ejoc_renderer_process(
ejoc_renderer_handle handle,
const float* bed5_planar, /* [5][1536] */
const float* lfe, /* [1536] or NULL */
uint32_t object_mask,
const uint8_t* n_bands, /* [15] */
const uint8_t* n_dpoints, /* [15] */
const uint8_t* slope_idx, /* [15] */
const uint8_t* offset_ts, /* [15][2] */
const double* dq, /* [15][2][5][23] */
double clipgain,
float phase_new,
float output_scale,
float* output16_planar); /* [16][1536] */
```
JOC 的 Huffman 解码、差分还原和去量化先在 Python 中完成,dense 与 sparse 两条语法共用同一条入口。原生核心消费已去量化的 `dq`(double),两条语法在该 ABI 上布局一致。
线程接口为:
```c
int ejoc_renderer_set_threads(ejoc_renderer_handle handle, uint32_t total_threads);
uint32_t ejoc_renderer_thread_count(ejoc_renderer_handle handle);
```
`total_threads` 包含调用线程。单个 renderer 实例必须顺序提交帧;实例内部可以按对象和 analysis channel 并行。
## 3. 跨帧状态
每个 JOC renderer 独立保存:
- analysis FIFO:`double[5][9][64]`;
- L/R/C analysis delay:`float[3][10][64]`;
- Ls/Rs QMF delay:`complex<double>[2][10][64]`;
- Ls/Rs band-0 FIR history:`complex<double>[2][20]`;
- 矩阵插值 previous:`double[15][5][64]`;
- inverse-QMF state:`double[15][640]`;
- LFE delay:`double[1217]`。
这些状态属于 renderer 实例,不能在无 checkpoint 的情况下任意分段或乱序处理。
## 4. FFT、QMF 与精度
原生核包含固定 64 点 radix-2 complex FFT:
- analysis QMF 使用 forward FFT 后除以 64;
- inverse QMF 使用固定重排、旋转和 640 项有效窗状态;
- 不调用外部 FFT 库。
JOC 路径的数值类型为:
- 核心 PCM 输入:float32;
- 矩阵、复 QMF、FFT、FIR 和跨帧状态:double;
- phase 与最终 gain:float32;
- 16 声道对象输出:float32。
## 5. 扬声器渲染 ABI
同一个共享库还导出对象到扬声器布局的渲染接口:
```c
uint32_t ejoc_speaker_layout_channel_count(uint32_t speaker_bitfield);
ejoc_speaker_renderer_handle
ejoc_speaker_renderer_create(uint32_t speaker_bitfield);
int ejoc_speaker_renderer_process(
ejoc_speaker_renderer_handle handle,
const float* objects16_interleaved,
uint32_t sample_count,
uint32_t metadata_count,
const uint32_t* metadata_offsets,
const uint32_t* ramp_durations,
const uint16_t* positions_q15,
const uint8_t* region_indices,
const uint8_t* height_enabled,
const double* object_gains,
double* output_interleaved);
```
输入声道 0 为 LFE,1–15 为对象。每个 metadata entry 是一份对象状态快照。`sample_count` 必须是 32 的倍数;未完成的增益斜坡保存在 handle 中并跨调用继续。
扬声器路径使用 float32 对象输入、double 坐标/增益/累加与 interleaved double 输出;写 WAV 时才量化为 float32 或 PCM24。
支持的布局为:
```text
2.0 3.1 5.1 7.1 5.1.2 5.1.4 7.1.2 7.1.4 9.1.4 9.1.6
```
## 6. 双耳渲染 ABI
共享库提供 512-sample float64 双耳 DSP:
```c
ejoc_binaural_renderer_handle ejoc_binaural_renderer_create(void);
int ejoc_binaural_renderer_configure_kernels(...);
int ejoc_binaural_renderer_configure_room(...);
int ejoc_binaural_renderer_process(
ejoc_binaural_renderer_handle handle,
const double* input16_interleaved, /* [512][16] */
const double* gains_complex, /* [16][2][77][2] */
const double* room_sends, /* [16] */
double output_gain,
double* output_stereo_interleaved); /* [512][2] */
```
Python 负责模型解析、OAMD 时间轴和每 512 samples 的 complex gains/room sends。C++ handle 保存 QMF、hybrid、递归 room 和 QMF synthesis 状态。全部输入、状态、乘加和输出均为 double/complex double。
## 6.1 公开 SOFA 双耳渲染 ABI
共享库同时提供完整的原生 SOFA 双耳渲染器(`ejoc_sofa_binaural_*`),它镜像
Python `SofaBinauralBackend` 的全部数学:64-QMF/77-hybrid analysis/synthesis、
五阶 ACN/N3D 实球谐方向场求值、whole-QMF-slot 逐对象 delay 历史、六面一阶
image-source early reflections、共享 unitary FDN late room、LFE 120–180 Hz
低通与 961-sample latency 语义。kernel 表、编译好的 HRTF 场与房间常数通过
`configure_kernels/configure_field/configure_room` 一次上传;每 512-sample
block 先 `set_source` 更新 16 个 source,再 `process` 输入 PCM;`process` 返回
裁剪后的 stereo 样本数(首个 961 samples 被丢弃)。`finish` 以 64-sample 对齐的
块排空尾音。Python 桥位于 `src/sofa_native_backend.py`,与 Python 参考实现逐值
一致(差异 < 1e-9);原生库缺失时 `main.py` 自动回退 Python。
## 7. 构建
CMake 定义位于 `native/CMakeLists.txt`。从仓库根目录运行:
```powershell
cmake -S native -B build/cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="$PWD/lib"
cmake --build build/cmake --config Release
cmake --install build/cmake --config Release
```
平台运行库文件名:
```text
Windows lib/eac3joc_core.dll
Linux lib/libeac3joc_core.so
macOS lib/libeac3joc_core.dylib
```
MSVC 配置使用静态 CRT。其他运行时依赖由平台和工具链决定,发布预构建库前应对产物独立检查。
仓库默认不附带原生二进制。预构建的 Release 运行库或自行构建的运行库均可直接放入 `lib/`。
## 8. 运行时查找与回退
查找顺序为:
1. 显式 `--native-library`;
2. `EAC3JOC_NATIVE_LIBRARY`;
3. `lib/` 下当前平台的标准文件名。
`--backend auto` 在加载失败时回退到 NumPy;`--backend python` 跳过原生探测。`--backend native` 当前也会打印失败原因后回退,这是现有 CLI 行为,不应理解为原生库已成功使用。
## 9. 实现边界
- 原生层只接收 Python 已解析的 dense JOC 数据。
- ABI 固定了 1536-sample JOC 帧、最多 15 个对象、最多 23 个参数带和最多 2 个数据点。
- 共享库与 Python 桥需要 ABI version 一致。
- 跨平台只约定 ABI 与数据类型,不保证 float64 结果逐位一致。
- `native/src/` 中的私有表头只服务于原生侧;当前仓库不包含重新生成这些头文件的脚本。
相关公式见[数学说明](math.md)。