Case I: validation
Contents
Engine status: OpenFOAM 12 scored · OpenFOAM 12 k-ε scored · OpenFOAM 12 k-ω SST scored · OpenLB pending · FluidX3D pending · ML surrogate pending
Nomenclature: symbols, columns and scores
Geometry, reference scales and release
- \(H\), \(b\), \(D\) (m) — height of the building, cube, tower or tree belt; width of the building; edge of the cubes of Case C: the lengths the cases are drawn and normalised by
- \(U_H\) (m/s) — reference speed: the mean approach-flow speed at the height \(H\) (each Data page gives the value used)
- \(U_R\), \(H_R\), \(Z_R\) (m/s, m) — reference speed and height of a workbook: Case E, \(U_R\) at \(Z_R\) = 250 m full scale (1 m in the model); Case M, \(U_R\) and \(H_R\) at the reference point of the tunnel and of each field run
- \(U_{\mathrm{ref}}\), \(L_{\mathrm{ref}}\) (m/s, m) — reference speed and length of a case’s normalisation: \(U_H\) and \(H\) (\(U_R\) and \(H_R\) in Case M); for the velocity ratios the approach speed at 15.9 m (Case E) or at 500 m (Case F)
- \(U_b\) (m/s) — Case G: mean approach-flow speed at 9 m, the normalisation of its approach-flow table
- \(q\), \(c_{\mathrm{gas}}\) (m³/s, ppm) — tracer release of the experiment: volume flow of the released gas, and its tracer content (\(10^6\) ppm when pure); \(Q\) is the source strength given to a solve
- \(R_b\) (dimensionless) — bulk Richardson number of the heated-floor cases J and L (negative: unstable stratification)
- \(\mathrm{Re_H}\), \(\mathrm{Re_D}\) (dimensionless) — Reynolds numbers \(U_H H/\nu\) and \(U_H D/\nu\) (\(\nu\): kinematic viscosity of air)
- \(u_\ast\), \(z_0\), \(\kappa\) (m/s, mm, dimensionless) — friction velocity and roughness length of the log law fitted to the approach flow, and the von Kármán constant (0.41)
- \(\mathrm{WD}\) (column
wind_direction_deg; °) — wind direction (each frame says how it is counted); \(\mathrm{WD_R}\) is the mean direction of a field run at its reference point (Case M) - \(x\), \(y\), \(z\) (columns
x,y,z; m) — coordinates in the case’s frame (\(+x\), \(+y\) and \(+z\) are the positive directions; each Data page says which way \(+x\) points), \(z\) up from the floor of the tunnel or from the ground - \(\mathrm{d}x\) (m) — lattice spacing of the OpenLB and FluidX3D runs, in metres or as a fraction of \(H\) (\(H/64\))
- \(L_r\) (m) — reattachment length behind the building, read from a wake line at ground level; reported as \(L_r/H\)
Velocity and turbulence
- \(U\), \(V\), \(W\) (columns
U,V,W; m/s) — mean velocity along \(+x\), \(+y\), \(+z\); \(V\) is positive toward \(+y\) in every case - \(\lvert\mathbf{U}\rvert\) (m/s) — magnitude of the mean velocity vector, \(\sqrt{U^2+V^2+W^2}\)
- \(V_{\mathrm{s}}\) (column
Vs; m/s) — mean omnidirectional scalar speed of a thermistor (never a vector magnitude) - \(\overline{u'u'}\), \(\overline{v'v'}\), \(\overline{w'w'}\) (columns
uu,vv,ww; m²/s²) — velocity variances (overbar: time average, prime: fluctuation about it) - \(\overline{u'v'}\), \(\overline{u'w'}\), \(\overline{v'w'}\) (columns
uv,uw,vw; m²/s²) — Reynolds shear stresses - \(k = \tfrac{1}{2}\left(\overline{u'u'}+\overline{v'v'}+\overline{w'w'}\right)\) (column
k; m²/s²) — turbulent kinetic energy - \(U/U_{\mathrm{ref}}\) (column
velocity_ratio; dimensionless) — mean speed over the case’s reference speed (\(V_{\mathrm{s}}/U_{\mathrm{ref}}\) for a thermistor) - \(\sigma_{U/U_{\mathrm{ref}}}\) (column
velocity_ratio_sd; dimensionless) — standard deviation ofvelocity_ratioover the observations behind a field value (Case F) - \(k/U_{\mathrm{ref}}^2\) (column
k_ratio; dimensionless) — turbulent kinetic energy over the square of the reference speed, where only normalised values are published (Case G)
Concentration and temperature
- \(C\) (column
C; ppm) — mean tracer-gas concentration as published (volume fraction \(\times 10^6\) of the released gas diluted in air) - \(c_{\mathrm{rms}}\) (column
c_rms; ppm) — root mean square of the concentration fluctuation - \(c_0\) (ppm) — reference concentration, defined for each case on its Data page: \(c_0 = \dfrac{c_{\mathrm{gas}}\thinspace q}{U_H H^2}\) (\(U_R H_R^2\) in the denominator in Case M)
- \(c^\ast\) (column
concentration_ratio; dimensionless) — dimensionless concentration \(c^\ast = C/c_0\); the value of a solve is \(c\thinspace U_H H^2/Q\) for its mean concentration \(c\) and its source strength \(Q\) - \(\mathrm{Sc_t}\) (dimensionless) — turbulent Schmidt number of a solve (0.3, 0.5 and 0.7)
- \(T\), \(T_{\mathrm{rms}}\) (columns
T,T_rms; °C, K) — mean air temperature as published (Cases J and L) and the root mean square of its fluctuation - \(\overline{u'c'}\), \(\overline{v'c'}\), \(\overline{w'c'}\) (columns
uc,vc,wc; ppm·m/s) — turbulent concentration fluxes - \(\overline{u'\theta'}\), \(\overline{v'\theta'}\), \(\overline{w'\theta'}\) (columns
uT,vT,wT; K·m/s) — turbulent heat fluxes (\(\theta'\): the temperature fluctuation) - \(\theta = \dfrac{T_f - T}{T_f - T_H}\) (dimensionless) — floor-referenced temperature the solves of Cases J and L are scored on: 0 at the floor, 1 in the approach air at \(H\); \(T_f\) is the floor temperature and \(T_H\) the temperature of the approach flow at \(H\)
Scores
- \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
- \(R\) — Pearson correlation coefficient of the pairs
- \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
- \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
- \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
- \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component
- \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\) — normalised mean square error
- \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\) — normalised absolute difference
- \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\) — geometric mean bias
- \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\) — geometric variance
- The ratio of \(\mathrm{FAC2}\) keeps its sign, so a model of the other sign than the measurement is a miss; \(\mathrm{FB}\), \(\mathrm{NMSE}\) and \(\mathrm{NAD}\) divide by the means, so they are undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive
- VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings (\(q\) the tracer release rate, \(D\) the block size of Case C, \(W\) the vertical velocity)
- The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
- The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.
Quantities scored
- \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
- \(k\) (turbulent kinetic energy) over \(U_H^2\)
- the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
- \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
- \(\theta\) (floor-referenced temperature)
Measurement locations
Explore the points
OpenFOAM 12
Status: scored
An Eddy3D production run, realizable k-ε at numerics level 3 and 3000 iterations, at 1:60, with the probe at the tunnel’s own coordinates. Case I was also solved with standard k-ε and with k-ω SST on the same mesh.
- Velocities are divided by \(U_H\), \(k\) by \(U_H^2\), with \(U_H\) = 1.7 m/s for the measurement and for the model.
- Concentration: the cleaned table’s \(C/c_0\) (\(c_0\) = 39.22 ppm) against the model’s \(c^\ast = c\thinspace U_H H^2/Q\), scored at the turbulent Schmidt numbers 0.3, 0.5 and 0.7 (the figures show 0.5). One detection limit, \(10^{-3}\) of the largest measured value.
Measured against modelled
FB is positive when the model is low.
- Flow. \(U\): R 1.00, FAC2 0.88. \(W\): R 0.97. \(k\): R 0.91, FAC2 0.92. \(V\) is scored at the 38 points off the \(y = 0\) plane; on the plane the workbook holds numerical zeros, which are left out. \(V\): FB −2.0 on a near-zero mean, R 0.99.
- Concentration. R is 0.92 to 0.95 and FAC2 stays below 0.5. FB +0.40 at 0.3, +0.08 at 0.5 and −0.10 at 0.7. Case H has a negative FB (model high) at all three.
Interpretation: Case I scores
- Velocity and \(k\) agree closely. \(V\)’s FB of −2.0 comes from a near-zero mean.
- The concentration is well correlated, with a wide scatter at low concentration (visible in panel d). The bias changes sign with the Schmidt number: the model is low at 0.3, nearly unbiased at 0.5 and high at 0.7. Case H over-predicts at all three, so the Schmidt number that suits one case does not suit the other.
Turbulence closures
| Closure | \(U\) R | \(U\) FAC2 | \(k\) R | \(k\) FAC2 | \(k\) FB | \(c^\ast\) (\(\mathrm{Sc_t}\) 0.5) R | \(c^\ast\) FAC2 | \(c^\ast\) FB |
|---|---|---|---|---|---|---|---|---|
| standard k-ε | +0.990 | 0.846 | +0.870 | 0.912 | −0.037 | +0.955 | 0.519 | −0.035 |
| realizable k-ε | +0.998 | 0.879 | +0.905 | 0.923 | +0.106 | +0.940 | 0.475 | +0.083 |
| k-ω SST | +0.992 | 0.879 | +0.880 | 0.396 | +0.495 | +0.466 | 0.403 | −0.153 |
- \(U\): R is 0.99 to 1.00 and FAC2 0.85 to 0.88 under all three.
- The two k-ε closures keep \(k\) within a factor of two at more than 90 % of the points (FAC2 0.912 and 0.923) and give a concentration with R of 0.94 to 0.96. Under k-ω SST, \(k\) FAC2 is 0.40 (FB +0.49, model low) and the concentration’s R is 0.47.
- Mean \(k/U_H^2\) of the approach profile: 0.0006 under k-ω SST, 0.0075 under both k-ε closures.
- Reattachment length \(L_r/H\): 2.23 (standard k-ε), 2.51 (realizable k-ε), 2.68 (k-ω SST).
Interpretation: Case I turbulence closures
- The mean velocity does not tell the closures apart. \(k\) and the concentration do.
- Under k-ω SST the turbulence is lower from the inlet on.
- This ranks three closures on one cube, one mesh and this release’s numerics.
Vertical profiles
Where along the height the model meets the measurement, station by station.
Field maps
The solved velocity on the measured planes, every point drawn on it as a dot coloured by its measured value on the same scale: a dot that stands out from the colour around it is a point the model misses.
OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor)
Reattachment length \(L_r/H\) = 2.51.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate | n |
|---|---|---|---|---|---|---|
| \(U\) | +0.998 | 0.997 | 0.879 | +0.020 | 0.813 | 91 |
| \(V\) | +0.985 | 0.970 | 0.895 | — | 1.000 | 38 |
| \(W\) | +0.972 | 0.945 | 0.659 | -0.049 | 0.967 | 91 |
| \(k\) | +0.905 | 0.818 | 0.923 | +0.106 | 0.582 | 91 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.3 | +0.922 | 0.850 | 0.360 | +0.398 | 0.106 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.5 | +0.940 | 0.884 | 0.475 | +0.083 | 0.284 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.7 | +0.946 | 0.896 | 0.477 | -0.100 | 0.291 | 539 |
- Per-point values:
/api/cases/I/pairs/openfoam12-realizableKE.csv - Reproduce: Eddy3D fixture TestAijCubeContainerE2E (realizable k-ε, 1:60) (numerics level 3, 3000 iterations); figures validation/plot_cases_hi_openfoam.py
OpenFOAM 12, standard k-epsilon (Eddy3D Outdoor)
Reattachment length \(L_r/H\) = 2.23.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate | n |
|---|---|---|---|---|---|---|
| \(U\) | +0.990 | 0.981 | 0.846 | -0.003 | 0.769 | 91 |
| \(V\) | +0.966 | 0.934 | 0.789 | — | 0.895 | 38 |
| \(W\) | +0.943 | 0.889 | 0.615 | +0.080 | 0.901 | 91 |
| \(k\) | +0.870 | 0.757 | 0.912 | -0.037 | 0.637 | 91 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.3 | +0.938 | 0.879 | 0.442 | +0.308 | 0.134 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.5 | +0.955 | 0.913 | 0.519 | -0.035 | 0.378 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.7 | +0.954 | 0.909 | 0.519 | -0.248 | 0.288 | 539 |
- Per-point values:
/api/cases/I/pairs/openfoam12-kEpsilon.csv - Reproduce: Eddy3D fixture TestAijCubeContainerE2E (standard k-ε, 1:60) (numerics level 3, 3000 iterations); figures validation/plot_cases_hi_openfoam.py
OpenFOAM 12, k-omega SST (Eddy3D Outdoor)
Reattachment length \(L_r/H\) = 2.68.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate | n |
|---|---|---|---|---|---|---|
| \(U\) | +0.992 | 0.983 | 0.879 | +0.091 | 0.802 | 91 |
| \(V\) | +0.968 | 0.937 | 0.789 | -2.000 | 0.921 | 38 |
| \(W\) | +0.922 | 0.850 | 0.615 | -0.214 | 0.868 | 91 |
| \(k\) | +0.880 | 0.775 | 0.396 | +0.495 | 0.264 | 91 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.3 | +0.514 | 0.265 | 0.432 | -0.047 | 0.236 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.5 | +0.466 | 0.217 | 0.403 | -0.153 | 0.202 | 539 |
| \(c^\ast\), \(\mathrm{Sc_t}\) 0.7 | +0.439 | 0.193 | 0.380 | -0.221 | 0.195 | 539 |
- Per-point values:
/api/cases/I/pairs/openfoam12-kOmegaSST.csv - Reproduce: Eddy3D fixture TestAijCubeContainerE2E (k-ω SST, 1:60) (numerics level 3, 3000 iterations); figures validation/plot_cases_hi_openfoam.py
Not run
- OpenLB, FluidX3D and the ML surrogate.
Notes on every run
-
Normalisation: velocities by \(U_H\), \(k\) by \(U_H^2\) (the benchmark's own convention; Case E by \(U\) at 15.9 m, as it is published); hit rate \(\delta_{\mathrm{rel}} = 0.25\), \(\delta_{\mathrm{abs}} = 0.05\) in those units
-
Scores recomputed from the pairs by validation/dispersion_metrics.py.
Leaderboard
| Rank | Run | Source | Hit rate, \(U\) | Hit rate, \(V\) | Hit rate, \(W\) | Hit rate, \(\lvert\mathbf{U}\rvert\) | Hit rate, \(k\) | Hit rate, \(c^\ast\) | Hit rate | FAC2 | R | \(R^2\) | Points scored |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | OpenFOAM 12 · \(\mathrm{Sc_t}\) 0.7 | Eddy3D | 0.813 | 1.000 | 0.967 | 0.789 | 0.582 | 0.291 | 0.741 | 0.797 | +0.967 | 0.936 | 89 % |
| 2 | OpenFOAM 12 · \(\mathrm{Sc_t}\) 0.5 | Eddy3D | 0.813 | 1.000 | 0.967 | 0.789 | 0.582 | 0.284 | 0.739 | 0.796 | +0.966 | 0.934 | 89 % |
| 3 | OpenFOAM 12 k-ε · \(\mathrm{Sc_t}\) 0.5 | Eddy3D | 0.769 | 0.895 | 0.901 | 0.789 | 0.637 | 0.378 | 0.728 | 0.763 | +0.952 | 0.909 | 89 % |
| 4 | OpenFOAM 12 k-ε · \(\mathrm{Sc_t}\) 0.7 | Eddy3D | 0.769 | 0.895 | 0.901 | 0.789 | 0.637 | 0.288 | 0.713 | 0.763 | +0.952 | 0.908 | 89 % |
| 5 | OpenFOAM 12 · \(\mathrm{Sc_t}\) 0.3 | Eddy3D | 0.813 | 1.000 | 0.967 | 0.789 | 0.582 | 0.106 | 0.710 | 0.777 | +0.963 | 0.929 | 89 % |
| 6 | OpenFOAM 12 k-ε · \(\mathrm{Sc_t}\) 0.3 | Eddy3D | 0.769 | 0.895 | 0.901 | 0.789 | 0.637 | 0.134 | 0.688 | 0.750 | +0.950 | 0.903 | 89 % |
| 7 | OpenFOAM 12 k-ω SST · \(\mathrm{Sc_t}\) 0.3 | Eddy3D | 0.802 | 0.921 | 0.868 | 0.816 | 0.264 | 0.236 | 0.651 | 0.685 | +0.878 | 0.799 | 89 % |
| 8 | OpenFOAM 12 k-ω SST · \(\mathrm{Sc_t}\) 0.5 | Eddy3D | 0.802 | 0.921 | 0.868 | 0.816 | 0.264 | 0.202 | 0.646 | 0.680 | +0.870 | 0.791 | 89 % |
| 9 | OpenFOAM 12 k-ω SST · \(\mathrm{Sc_t}\) 0.7 | Eddy3D | 0.802 | 0.921 | 0.868 | 0.816 | 0.264 | 0.195 | 0.644 | 0.677 | +0.865 | 0.787 | 89 % |
All runs scored by validation/score_submissions.py. Ranked by mean hit rate (VDI 3783) over the tasks; bold: best per column. Points scored: share of measured points with values. Not ranked: the ML surrogate (no point ids) and negative controls. Community rows are self-reported (Validate your own solver).