Method and metrics
Scores: Validation.
How the scores are computed
- Each case is solved through Eddy3D’s production case writer (
build-case → run → probe). The probe sits at the tunnel’s own measurement coordinates; no interpolation enters the comparison. - The measured side of every pair is the cleaned table (the legacy workbook at full precision). Eddy3D’s fixtures score against the published CSVs, which round the workbooks; their scores differ from the scores on these pages in the third digit.
- Normalisation (the benchmark’s convention): velocities by \(U_H\), \(k\) by \(U_H^2\).
- Cases A and B are solved at 1:100 with the tunnel’s own velocities. The log-law inlet is fitted to each case’s own approach-flow file over \(z = 0.02\thinspace\ldots\thinspace 2H\) (
u*,z₀below). - Cases C–E measure an omnidirectional scalar speed with thermistors; the model is scored as \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) and as \(\lvert\mathbf{U}\rvert\). Case F (thermistor and field, scalar speed over \(U(500\text{ m})\)) is scored the same way.
- Cases H, I, J, K, L and M add a tracer concentration. The measured \(C/c_0\) is scored against the model’s \(c^\ast = c\thinspace U_H H^2/Q\) (Case M: \(c\thinspace U_R H_R^2/Q\)) at three turbulent Schmidt numbers, 0.3, 0.5 and 0.7. See the Validation pages of Cases H, I, J, K, L and M.
- Cases J and L add a temperature, scored as the floor-referenced fraction \(\theta = (T_f - T)/(T_f - T_H)\) (the workbook’s normalisation; 0 at the floor, 1 in the approach air at the building height). They are solved at a Froude scale on Eddy3D’s buoyant engine.
- Case G is field data with an unknown canopy drag, solved as a sweep of arms that are each their own run.
- Case F is solved for one wind direction in two vertical frames that are each their own run. Case M (tracer over a real campus) is solved at one heading with the three Schmidt numbers as three runs on one flow.
- Case B’s lateral velocity is sign-corrected before scoring (see Known data issues).
- The reattachment length \(L_r\) behind the building is read from a wake line at ground level.
- Fixture structure checks (ordering of regions, plume decay, symmetry): passed for every run.
Metrics
- \(n\), the number of pairs scored: the points at which the measured and the modelled value both exist; a pair is the measured value \(o_i\) and the modelled value \(p_i\) at one point; \(\overline{o}\) and \(\overline{p}\) are the means over the pairs.
- \(R = \dfrac{\sum_i (o_i-\overline{o})(p_i-\overline{p})}{\sqrt{\sum_i (o_i-\overline{o})^2 \sum_i (p_i-\overline{p})^2}}\), Pearson’s correlation coefficient of the pairs.
- \(R^2\), the square of \(R\): the share of the variance of the measured values that a straight-line fit to the modelled values explains, between 0 and 1. It drops the sign of \(R\), so an anti-correlated model can score a high \(R^2\); read it beside \(R\).
- \(\mathrm{FAC2}\), the fraction of pairs with \(0.5 \le p_i/o_i \le 2\). The ratio keeps its sign, so a model of the other sign than the measurement is a miss; for a concentration, a pair with both values at or below the detection limit also counts.
- \(\mathrm{FB} = \dfrac{\overline{o}-\overline{p}}{0.5\thinspace(\overline{o}+\overline{p})}\), the fractional bias, positive when the model under-predicts. It is undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive; \(R\), \(\mathrm{FAC2}\), the hit rate and \(n\) stay.
- \(\mathrm{HR}\), the hit rate: the fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), with \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}}\) equal to \(0.05\thinspace U_H\) for a velocity component, to the detection limit for a concentration, and absent for \(k\). VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings.
- NMSE, NAD, MG and VG are in
validation/results/*.jsonand in the API’svalidation.json. NMSE and NAD divide by the means like \(\mathrm{FB}\), so they are undefined for a signed component unless the measured and the modelled mean are both positive. - \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\), the normalised mean square error.
- \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\), the normalised absolute difference.
- \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\), the geometric mean bias.
- \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\), the geometric variance.
- The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
- The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.
Controls
A control is scored with the same metrics as the run.
- Frame: the rotation sense mirrored (Cases C and D); upwind and downwind swapped (Case C); the lateral sign of \(V\) flipped (Cases L and M, OpenLB, FluidX3D); the points turned 180° about the source or mirrored across the plume axis (Case M); the tunnel’s column for the opposite wind (Case F, SSE); \(U\) scrambled across the points (OpenLB).
- Ceiling: one measurement scored as a model of another: the tunnel’s before as a model of its after (Case E); the measurement against the mean of each mirror pair (Cases K and L); the tunnel’s WD 175, WD 185 and their mean as models of its WD 180 (Case M); the tunnel’s WD 180 or 9 m column as a model of the field (Cases M and F).
- Reference runs: no belt (Case G); the approach column alone (Case L); the empty-domain flow (FluidX3D); the 9 m plane over the imposed \(U(500\text{ m})\) (Case F); the reversed wind (ML surrogate).
How the scores were checked against the solvers’ own output
- Case C: a re-scoring from the probe files and the original
RS_caseC.csvreproduces every number (largest difference \(9 \times 10^{-16}\)). - Case D: the numbers were reproduced from the solver’s probe output against the cleaned table, with the frame mapped independently and separate metric code.
- OpenLB: reproduced from the solver’s probe output against the original Zenodo CSVs (largest difference \(1 \times 10^{-15}\)).
- FluidX3D: reproduced from the solver’s probe output against the cleaned tables, with the comparable set derived independently (largest difference \(3 \times 10^{-12}\)).
- Cases F, G, H, I, J, K, L and M: the scores served are recomputed from the per-point pairs by
validation/dispersion_metrics.py, a transcription of Eddy3D’sMetaFOAM.DispersionMetrics. Fed the published CSVs and the probes of Eddy3D’s test, it gives that test’s own scores to \(10^{-9}\) (24 scores of Case F, 43 of Case K, five per arm of Case G, 80 per run of Case J, 48 of Case L, 84 of Case M). The SHA-256 of every file a snapshot was read from is in its results file undersources, and the script fails if the check is not met.
Interpretation: Scoring choices
- The scalar-speed pair \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) and \(\lvert\mathbf{U}\rvert\) are the two ends of the bracket the thermistor measurement has to fall in.
- The profile shape of a log law is scale-invariant, so the \(U/U_H\) values of the 1:100 solves of Cases A and B carry over unchanged.
Two systematic differences between the tunnels and the solvers
These shape the validations of Cases C, D and E.
- Scalar vs vector speed. C, D and E measure with thermistor anemometers, which report a mean omnidirectional scalar speed; steady RANS gives the magnitude of the mean velocity. Case D is the only case where both instruments overlap: over its 33 shared points the thermistor reads 1.34× \(\lvert\mathbf{U}\rvert\) (ratio of means), and \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) from the split-fibre data matches the thermistor to FB +0.058 where \(\lvert\mathbf{U}\rvert\) alone gives +0.292. A Case C/D/E score is reported at both ends of the bracket \(\lvert\mathbf{U}\rvert\) ≤ \(V_{\mathrm{s}}\) ≤ \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\).
- Inflow profile. The inlet is a log law. Fitted power-law exponents of the tunnels’ profiles are 0.19 (C, D) and 0.16–0.20 (E) against the papers’ 0.28 / 0.25. A log-law fit misses the single measurement height by −4 … −11 % depending on the fitting window, and sets the inlet \(k\) 26–48 % below the tunnel’s (Case D).
Interpretation: The two differences
- Steady RANS gives a mean velocity magnitude that is lower than the thermistor’s scalar speed by construction.
- Both differences lower the model against Cases C, D and E relative to Cases A and B.
- The tunnels’ profiles of Cases C, D and E are closer to a power law than a log law, and a single near-ground height is where a log-law inlet is least right.
Inflow
Fitted to each case’s own approach-flow file over \(z = 0.02\thinspace\ldots\thinspace 2H\):
| Case | \(u_\ast\) (m/s) | \(z_0\) (mm) | \(U_H\) measured (m/s) | log-law error |
|---|---|---|---|---|
| A | 0.343 | 0.668 | 4.491 | +2.0 % |
| B | 0.399 | 0.882 | 5.133 | +2.9 % |
Nomenclature: symbols, columns and scores
Geometry, reference scales and release
- \(H\), \(b\), \(D\) (m) — height of the building, cube, tower or tree belt; width of the building; edge of the cubes of Case C: the lengths the cases are drawn and normalised by
- \(U_H\) (m/s) — reference speed: the mean approach-flow speed at the height \(H\) (each Data page gives the value used)
- \(U_R\), \(H_R\), \(Z_R\) (m/s, m) — reference speed and height of a workbook: Case E, \(U_R\) at \(Z_R\) = 250 m full scale (1 m in the model); Case M, \(U_R\) and \(H_R\) at the reference point of the tunnel and of each field run
- \(U_{\mathrm{ref}}\), \(L_{\mathrm{ref}}\) (m/s, m) — reference speed and length of a case’s normalisation: \(U_H\) and \(H\) (\(U_R\) and \(H_R\) in Case M); for the velocity ratios the approach speed at 15.9 m (Case E) or at 500 m (Case F)
- \(U_b\) (m/s) — Case G: mean approach-flow speed at 9 m, the normalisation of its approach-flow table
- \(q\), \(c_{\mathrm{gas}}\) (m³/s, ppm) — tracer release of the experiment: volume flow of the released gas, and its tracer content (\(10^6\) ppm when pure); \(Q\) is the source strength given to a solve
- \(R_b\) (dimensionless) — bulk Richardson number of the heated-floor cases J and L (negative: unstable stratification)
- \(\mathrm{Re_H}\), \(\mathrm{Re_D}\) (dimensionless) — Reynolds numbers \(U_H H/\nu\) and \(U_H D/\nu\) (\(\nu\): kinematic viscosity of air)
- \(u_\ast\), \(z_0\), \(\kappa\) (m/s, mm, dimensionless) — friction velocity and roughness length of the log law fitted to the approach flow, and the von Kármán constant (0.41)
- \(\mathrm{WD}\) (column
wind_direction_deg; °) — wind direction (each frame says how it is counted); \(\mathrm{WD_R}\) is the mean direction of a field run at its reference point (Case M) - \(x\), \(y\), \(z\) (columns
x,y,z; m) — coordinates in the case’s frame (\(+x\), \(+y\) and \(+z\) are the positive directions; each Data page says which way \(+x\) points), \(z\) up from the floor of the tunnel or from the ground - \(\mathrm{d}x\) (m) — lattice spacing of the OpenLB and FluidX3D runs, in metres or as a fraction of \(H\) (\(H/64\))
- \(L_r\) (m) — reattachment length behind the building, read from a wake line at ground level; reported as \(L_r/H\)
Velocity and turbulence
- \(U\), \(V\), \(W\) (columns
U,V,W; m/s) — mean velocity along \(+x\), \(+y\), \(+z\); \(V\) is positive toward \(+y\) in every case - \(\lvert\mathbf{U}\rvert\) (m/s) — magnitude of the mean velocity vector, \(\sqrt{U^2+V^2+W^2}\)
- \(V_{\mathrm{s}}\) (column
Vs; m/s) — mean omnidirectional scalar speed of a thermistor (never a vector magnitude) - \(\overline{u'u'}\), \(\overline{v'v'}\), \(\overline{w'w'}\) (columns
uu,vv,ww; m²/s²) — velocity variances (overbar: time average, prime: fluctuation about it) - \(\overline{u'v'}\), \(\overline{u'w'}\), \(\overline{v'w'}\) (columns
uv,uw,vw; m²/s²) — Reynolds shear stresses - \(k = \tfrac{1}{2}\left(\overline{u'u'}+\overline{v'v'}+\overline{w'w'}\right)\) (column
k; m²/s²) — turbulent kinetic energy - \(U/U_{\mathrm{ref}}\) (column
velocity_ratio; dimensionless) — mean speed over the case’s reference speed (\(V_{\mathrm{s}}/U_{\mathrm{ref}}\) for a thermistor) - \(\sigma_{U/U_{\mathrm{ref}}}\) (column
velocity_ratio_sd; dimensionless) — standard deviation ofvelocity_ratioover the observations behind a field value (Case F) - \(k/U_{\mathrm{ref}}^2\) (column
k_ratio; dimensionless) — turbulent kinetic energy over the square of the reference speed, where only normalised values are published (Case G)
Concentration and temperature
- \(C\) (column
C; ppm) — mean tracer-gas concentration as published (volume fraction \(\times 10^6\) of the released gas diluted in air) - \(c_{\mathrm{rms}}\) (column
c_rms; ppm) — root mean square of the concentration fluctuation - \(c_0\) (ppm) — reference concentration, defined for each case on its Data page: \(c_0 = \dfrac{c_{\mathrm{gas}}\thinspace q}{U_H H^2}\) (\(U_R H_R^2\) in the denominator in Case M)
- \(c^\ast\) (column
concentration_ratio; dimensionless) — dimensionless concentration \(c^\ast = C/c_0\); the value of a solve is \(c\thinspace U_H H^2/Q\) for its mean concentration \(c\) and its source strength \(Q\) - \(\mathrm{Sc_t}\) (dimensionless) — turbulent Schmidt number of a solve (0.3, 0.5 and 0.7)
- \(T\), \(T_{\mathrm{rms}}\) (columns
T,T_rms; °C, K) — mean air temperature as published (Cases J and L) and the root mean square of its fluctuation - \(\overline{u'c'}\), \(\overline{v'c'}\), \(\overline{w'c'}\) (columns
uc,vc,wc; ppm·m/s) — turbulent concentration fluxes - \(\overline{u'\theta'}\), \(\overline{v'\theta'}\), \(\overline{w'\theta'}\) (columns
uT,vT,wT; K·m/s) — turbulent heat fluxes (\(\theta'\): the temperature fluctuation) - \(\theta = \dfrac{T_f - T}{T_f - T_H}\) (dimensionless) — floor-referenced temperature the solves of Cases J and L are scored on: 0 at the floor, 1 in the approach air at \(H\); \(T_f\) is the floor temperature and \(T_H\) the temperature of the approach flow at \(H\)
Scores
- \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
- \(R\) — Pearson correlation coefficient of the pairs
- \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
- \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
- \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
- \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component
- \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\) — normalised mean square error
- \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\) — normalised absolute difference
- \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\) — geometric mean bias
- \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\) — geometric variance
- The ratio of \(\mathrm{FAC2}\) keeps its sign, so a model of the other sign than the measurement is a miss; \(\mathrm{FB}\), \(\mathrm{NMSE}\) and \(\mathrm{NAD}\) divide by the means, so they are undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive
- VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings (\(q\) the tracer release rate, \(D\) the block size of Case C, \(W\) the vertical velocity)
- The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
- The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.
Quantities scored
- \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
- \(k\) (turbulent kinetic energy) over \(U_H^2\)
- the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
- \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
- \(\theta\) (floor-referenced temperature)
- Cases C, D and E carry their own fitted profiles on their pages. All three score at a single near-ground height.
- The fits are in
/api/cases/{id}.jsonunderinflow_fit.
Grid uncertainty
Cases A and E, OpenFOAM: four meshes, edge ratio \(\sqrt{2}\). Band per point, second mesh: Roache’s GCI (Celik et al. 2008, safety factor 1.25, foamgci) where the meshes converge monotonically, half the spread where they oscillate, none where they diverge. Run with the eddy-viscosity cap lifted, so shown in its own section, not on the served run. No other run has a grid study.
Reproducing the figures
validation/plot_cases_ab_openfoam.py (Cases A and B), validation/plot_case_c_openfoam.py, validation/plot_cases_hi_openfoam.py (Cases H and I), validation/plot_cases_gk_openfoam.py (Cases G and K), validation/plot_cases_jl_umf.py (Cases J and L), validation/plot_cases_fm_openfoam.py (Cases F and M) and validation/plot_ml_surrogate.py (UltraPlot) build the figures, validation/plot_grid_openfoam.py the grid figures (snapshot: validation/snapshot_grid.py); validation/plot_layout.py draws where each case was measured, from the groups of lines and planes in validation/layout.py, which also colour the dots of the agreement figures (validation/test_layout.py pins them). The Eddy3D fixtures are TestAijCaseAContainerE2E, TestAijCaseBContainerE2E, TestAijCaseC{0D,1D,2D}ContainerE2E, TestAijCaseFContainerE2E, TestAijCaseGContainerE2E, TestAijCaseHContainerE2E, TestAijCubeContainerE2E, TestAijCaseKContainerE2E, TestAijCaseJUmfContainerE2E, TestAijCaseLUmfContainerE2E, TestAijCaseMContainerE2E and Outdoor.ML.Tests/TestAijSurrogateE2E. The scores they wrote are snapshotted in validation/results/ with the per-point pairs beside them, and served unchanged by /api/cases/{id}/validation.json.