Validation: Eddy3D against the measurements

Each case is solved through Eddy3D’s production case writer (build-case → run → probe). The probe sits at the tunnel’s own measurement coordinates.

Cases

Each case’s Validation page has its measurement locations, every engine’s results and a leaderboard of its runs.

  • Case A: 1:1:2 building
  • Case B: 1:4:4 slab
  • Case C: 3 × 3 block array
  • Case D: high-rise among city blocks
  • Case E: Niigata district
  • Case F: Shinjuku district
  • Case G: flow behind a tree belt
  • Case H: 1:1:2 building with a floor point source
  • Case I: cube with a roof point source
  • Case J: 1:1:2 building, non-isothermal
  • Case K: 9 x 9 cube array, point source
  • Case L: 9 x 14 cube array, line source, non-isothermal
  • Case M: TPU Atsugi campus

Cases C and D by configuration

Correlation and error against the wind tunnel for every configuration of Cases C and D, OpenFOAM and ML surrogate
Figure 1. Cases C and D, one row per configuration: the correlation coefficient C.C. (higher is better) and the root-mean-square difference S.E. (lower is better) between the tunnel and each model over the points of the row. The AIJ gives no formula for S.E.; the RMS difference is this repository’s reading. Speeds are over \(U_H\) in Case D and over the approach speed at the measuring height (\(z = 0.02\thinspace\mathrm{m}\)) in Case C, as in the AIJ benchmark. Filled blue: OpenFOAM scalar speed \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) against the thermistor’s \(V_{\mathrm{s}}\); in the split-fibre row, \(\lvert\mathbf{U}\rvert\) against the measured \(\lvert\mathbf{U}\rvert\). Hollow blue, Case C: OpenFOAM \(\lvert\mathbf{U}\rvert\) without the \(2k\) term, the quantity of the AIJ table. Green: ML surrogate, \(\lvert\mathbf{U}\rvert\). Grey: range of the CFD codes the AIJ ran on the row (steady RANS; four codes in Case C, five runs at 0° and three at 22.5° and 45°; three codes in Case D), from Table 3-1-3 and Figs. 3-2-10 to 3-2-16 of the AIJ benchmark book (2016). Rows: Case C, the centre block (0D absent, 1D, 2D) and the wind direction; Case D, the wind direction.

Every case and engine

Case OpenFOAM 12 OpenLB FluidX3D ML surrogate
Case A scored
R +0.48 … +0.96
FAC2 0.35 … 0.82
scored
R +0.16 … +0.80
FAC2 0.00 … 0.65
scored
R +0.59 … +0.85
FAC2 0.22 … 0.60
scored
R -0.13 … +0.43
FAC2 0.26 … 0.62
Case B scored
R +0.40 … +0.96
FAC2 0.28 … 0.70
scored
R +0.45 … +0.93
FAC2 0.05 … 0.61
scored
R +0.85 … +0.95
FAC2 0.48 … 0.64
scored
R -0.24 … +0.01
FAC2 0.36 … 0.46
Case C scored
R +0.74 … +0.91
FAC2 0.55 … 0.98
not comparable pending scored
R +0.07 … +0.21
FAC2 0.44 … 0.88
Case D scored
R +0.78 … +0.95
FAC2 0.48 … 0.92
not comparable pending scored
R +0.22 … +0.59
FAC2 0.34 … 0.64
Case E scored
R +0.69 … +0.85
FAC2 0.74 … 0.95
not comparable pending scored
R +0.23 … +0.43
FAC2 0.22 … 0.84
Case F
compared with the wind tunnel and the field
scored
R +0.79 … +1.00
FAC2 0.85 … 1.00
pending pending pending
Case G
compared with the field
scored
R +0.21 … +0.50
FAC2 0.68 … 1.00
pending pending pending
Case H scored
R +0.11 … +0.95
FAC2 0.31 … 0.69
pending pending pending
Case I scored
R +0.90 … +1.00
FAC2 0.36 … 0.92
pending pending pending
Case J scored
R +0.32 … +0.92
FAC2 0.29 … 1.00
pending pending pending
Case K scored
R -0.15 … +0.96
FAC2 0.04 … 0.80
pending pending pending
Case L scored
R -0.04 … +0.94
FAC2 0.18 … 1.00
pending pending pending
Case M
compared with the wind tunnel and the field
scored
R +0.18 … +0.73
FAC2 0.07 … 0.73
pending pending pending
Nomenclature: scores

Scores

  • \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
  • \(R\) — Pearson correlation coefficient of the pairs
  • \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
  • \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
  • \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
  • \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component

Quantities scored

  • \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
  • \(k\) (turbulent kinetic energy) over \(U_H^2\)
  • the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
  • \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
  • \(\theta\) (floor-referenced temperature)

R, FAC2: range over the engine’s headline scores, negative controls excluded. The method defines every metric.

Status Meaning
scored A validation run exists and its scores are in cases/{id}/validation.json.
running The solve is in progress; no scores yet.
pending Not run yet.
not comparable The engine cannot produce the measured quantity at the measured location: OpenLB at 2 m on Cases C to E, where its empty-domain control misses the bar. The measurement behind it is on the OpenLB and FluidX3D page.
  • A dash in the matrix: the engine does not apply to the case.
  • OpenFOAM 12 on Cases J and L is urbanMicroclimateFoam (UMF), the OpenFOAM 12 solver Eddy3D uses for non-isothermal flow; the isothermal solver was not run on them.
  • API engine ids: openfoam12-realizableKE (also openfoam12-kEpsilon and openfoam12-kOmegaSST, Case I only), umf-realizableKE (also umf-buoyantKEpsilon, Case J only), openlb, fluidx3d and ml-surrogate.

In this section

  • Method and metrics - how each case is scored, the metrics, the inflow, the two systematic differences
  • Inlet contract - the approach flow a submission should use, per case: the measured profile, and the inlet this repository recommends
  • OpenLB and FluidX3D - the two lattice-Boltzmann engines: how they are run, and why they are not scored on C to E
  • ML surrogate - how the Wind Predictor is run on Cases A to E, and what its scores grade