Case C: validation
Contents
Engine status: OpenFOAM 12 scored · OpenLB not comparable · FluidX3D pending · ML surrogate scored
Nomenclature: symbols, columns and scores
Geometry, reference scales and release
- \(H\), \(b\), \(D\) (m) — height of the building, cube, tower or tree belt; width of the building; edge of the cubes of Case C: the lengths the cases are drawn and normalised by
- \(U_H\) (m/s) — reference speed: the mean approach-flow speed at the height \(H\) (each Data page gives the value used)
- \(U_R\), \(H_R\), \(Z_R\) (m/s, m) — reference speed and height of a workbook: Case E, \(U_R\) at \(Z_R\) = 250 m full scale (1 m in the model); Case M, \(U_R\) and \(H_R\) at the reference point of the tunnel and of each field run
- \(U_{\mathrm{ref}}\), \(L_{\mathrm{ref}}\) (m/s, m) — reference speed and length of a case’s normalisation: \(U_H\) and \(H\) (\(U_R\) and \(H_R\) in Case M); for the velocity ratios the approach speed at 15.9 m (Case E) or at 500 m (Case F)
- \(U_b\) (m/s) — Case G: mean approach-flow speed at 9 m, the normalisation of its approach-flow table
- \(q\), \(c_{\mathrm{gas}}\) (m³/s, ppm) — tracer release of the experiment: volume flow of the released gas, and its tracer content (\(10^6\) ppm when pure); \(Q\) is the source strength given to a solve
- \(R_b\) (dimensionless) — bulk Richardson number of the heated-floor cases J and L (negative: unstable stratification)
- \(\mathrm{Re_H}\), \(\mathrm{Re_D}\) (dimensionless) — Reynolds numbers \(U_H H/\nu\) and \(U_H D/\nu\) (\(\nu\): kinematic viscosity of air)
- \(u_\ast\), \(z_0\), \(\kappa\) (m/s, mm, dimensionless) — friction velocity and roughness length of the log law fitted to the approach flow, and the von Kármán constant (0.41)
- \(\mathrm{WD}\) (column
wind_direction_deg; °) — wind direction (each frame says how it is counted); \(\mathrm{WD_R}\) is the mean direction of a field run at its reference point (Case M) - \(x\), \(y\), \(z\) (columns
x,y,z; m) — coordinates in the case’s frame (\(+x\), \(+y\) and \(+z\) are the positive directions; each Data page says which way \(+x\) points), \(z\) up from the floor of the tunnel or from the ground - \(\mathrm{d}x\) (m) — lattice spacing of the OpenLB and FluidX3D runs, in metres or as a fraction of \(H\) (\(H/64\))
- \(L_r\) (m) — reattachment length behind the building, read from a wake line at ground level; reported as \(L_r/H\)
Velocity and turbulence
- \(U\), \(V\), \(W\) (columns
U,V,W; m/s) — mean velocity along \(+x\), \(+y\), \(+z\); \(V\) is positive toward \(+y\) in every case - \(\lvert\mathbf{U}\rvert\) (m/s) — magnitude of the mean velocity vector, \(\sqrt{U^2+V^2+W^2}\)
- \(V_{\mathrm{s}}\) (column
Vs; m/s) — mean omnidirectional scalar speed of a thermistor (never a vector magnitude) - \(\overline{u'u'}\), \(\overline{v'v'}\), \(\overline{w'w'}\) (columns
uu,vv,ww; m²/s²) — velocity variances (overbar: time average, prime: fluctuation about it) - \(\overline{u'v'}\), \(\overline{u'w'}\), \(\overline{v'w'}\) (columns
uv,uw,vw; m²/s²) — Reynolds shear stresses - \(k = \tfrac{1}{2}\left(\overline{u'u'}+\overline{v'v'}+\overline{w'w'}\right)\) (column
k; m²/s²) — turbulent kinetic energy - \(U/U_{\mathrm{ref}}\) (column
velocity_ratio; dimensionless) — mean speed over the case’s reference speed (\(V_{\mathrm{s}}/U_{\mathrm{ref}}\) for a thermistor) - \(\sigma_{U/U_{\mathrm{ref}}}\) (column
velocity_ratio_sd; dimensionless) — standard deviation ofvelocity_ratioover the observations behind a field value (Case F) - \(k/U_{\mathrm{ref}}^2\) (column
k_ratio; dimensionless) — turbulent kinetic energy over the square of the reference speed, where only normalised values are published (Case G)
Concentration and temperature
- \(C\) (column
C; ppm) — mean tracer-gas concentration as published (volume fraction \(\times 10^6\) of the released gas diluted in air) - \(c_{\mathrm{rms}}\) (column
c_rms; ppm) — root mean square of the concentration fluctuation - \(c_0\) (ppm) — reference concentration, defined for each case on its Data page: \(c_0 = \dfrac{c_{\mathrm{gas}}\thinspace q}{U_H H^2}\) (\(U_R H_R^2\) in the denominator in Case M)
- \(c^\ast\) (column
concentration_ratio; dimensionless) — dimensionless concentration \(c^\ast = C/c_0\); the value of a solve is \(c\thinspace U_H H^2/Q\) for its mean concentration \(c\) and its source strength \(Q\) - \(\mathrm{Sc_t}\) (dimensionless) — turbulent Schmidt number of a solve (0.3, 0.5 and 0.7)
- \(T\), \(T_{\mathrm{rms}}\) (columns
T,T_rms; °C, K) — mean air temperature as published (Cases J and L) and the root mean square of its fluctuation - \(\overline{u'c'}\), \(\overline{v'c'}\), \(\overline{w'c'}\) (columns
uc,vc,wc; ppm·m/s) — turbulent concentration fluxes - \(\overline{u'\theta'}\), \(\overline{v'\theta'}\), \(\overline{w'\theta'}\) (columns
uT,vT,wT; K·m/s) — turbulent heat fluxes (\(\theta'\): the temperature fluctuation) - \(\theta = \dfrac{T_f - T}{T_f - T_H}\) (dimensionless) — floor-referenced temperature the solves of Cases J and L are scored on: 0 at the floor, 1 in the approach air at \(H\); \(T_f\) is the floor temperature and \(T_H\) the temperature of the approach flow at \(H\)
Scores
- \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
- \(R\) — Pearson correlation coefficient of the pairs
- \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
- \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
- \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
- \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component
- \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\) — normalised mean square error
- \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\) — normalised absolute difference
- \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\) — geometric mean bias
- \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\) — geometric variance
- The ratio of \(\mathrm{FAC2}\) keeps its sign, so a model of the other sign than the measurement is a miss; \(\mathrm{FB}\), \(\mathrm{NMSE}\) and \(\mathrm{NAD}\) divide by the means, so they are undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive
- VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings (\(q\) the tracer release rate, \(D\) the block size of Case C, \(W\) the vertical velocity)
- The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
- The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.
Quantities scored
- \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
- \(k\) (turbulent kinetic energy) over \(U_H^2\)
- the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
- \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
- \(\theta\) (floor-referenced temperature)
Measurement locations
Explore the points
OpenFOAM 12
Status: scored
All nine configurations (centre block 0D / 1D / 2D × WD 0°, 22.5°, 45°), solved at 1:100 on a cylindrical domain (the tunnel’s turntable: the array turns with WD, the domain does not), ~310,000 cells per centre-block height, 3000 iterations per direction, the 120 probes at exactly the tunnel’s coordinates.
Measured against modelled
Scores for all nine configurations (R, FAC2, FB, hit rate; |U| alone)
| centre block | WD | R | FAC2 | FB | hit rate | \(\lvert\mathbf{U}\rvert\) alone: FB / R |
|---|---|---|---|---|---|---|
| 0D | 0° | +0.824 | 0.908 | +0.266 | 0.567 | +0.466 / +0.804 |
| 0D | 22.5° | +0.857 | 0.983 | +0.106 | 0.750 | +0.295 / +0.846 |
| 0D | 45° | +0.831 | 0.942 | +0.121 | 0.783 | +0.307 / +0.801 |
| 1D | 0° | +0.801 | 0.858 | +0.341 | 0.433 | +0.551 / +0.741 |
| 1D | 22.5° | +0.896 | 0.975 | +0.192 | 0.675 | +0.394 / +0.882 |
| 1D | 45° | +0.872 | 0.942 | +0.198 | 0.733 | +0.353 / +0.873 |
| 2D | 0° | +0.857 | 0.850 | +0.251 | 0.492 | +0.443 / +0.848 |
| 2D | 22.5° | +0.906 | 0.983 | +0.091 | 0.875 | +0.240 / +0.894 |
| 2D | 45° | +0.909 | 0.958 | +0.103 | 0.883 | +0.243 / +0.912 |
- R 0.80–0.91 and FAC2 0.85–0.98 in every configuration.
- FB +0.25 … +0.34 at \(\mathrm{WD} = 0\) (hit rate 0.43–0.57); FB +0.09 … +0.20 at 22.5° and 45° (hit rate 0.68–0.88). FB is positive when the model is low.
- The tunnel’s speed is above the model’s \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) at 902 of the 1080 points (84 %); the model’s \(\lvert\mathbf{U}\rvert\) alone is lower still.
- Rotation sense mirrored (the paper figure’s y mirror), control: R −0.33 … +0.03 at 22.5° and 45°.
- Upwind and downwind swapped, control: R −0.41 … +0.14 in eight of nine configurations.
Interpretation: Case C pattern and deficit
- The pattern is right in every configuration.
- The model is slow, worst at \(\mathrm{WD} = 0\), and the map puts the deficit in the streets running along the wind.
- The scalar-vs-vector difference does not explain the deficit: the tunnel reads above even \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) at 902 of the 1080 points, and \(\lvert\mathbf{U}\rvert\) alone is lower still. Where the deficit comes from is not settled.
- Both frame controls above score far below the runs, so the frame is the right one.
Data notes: Approach-flow sampling
The approach flow at the probe height reads 0.83 of the tunnel’s; that reading is a sampling artefact. Upstream of the array the probe height lies below the centre of the first cell of the coarse background mesh, where the probes’ cell-to-point interpolation runs linearly to zero at the wall: the sampled \(\lvert\mathbf{U}\rvert\) is exactly proportional to z there (1.00 m/s at 1 m, 2.01 m/s at 2 m). At \(z = D\) the approach matches the tunnel (0.99). At the measurement points the sampling is sound: none of the 240 checked (2D at 0°, 0D at 45°) has its 2 m sample inside the first cell.
OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 0D
Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate |
|---|---|---|---|---|---|
| WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.824 | 0.679 | 0.908 | +0.266 | 0.567 |
| WD 0° \(\lvert\mathbf{U}\rvert\) | +0.804 | 0.647 | 0.658 | +0.466 | 0.308 |
| WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.857 | 0.735 | 0.983 | +0.106 | 0.750 |
| WD 22.5° \(\lvert\mathbf{U}\rvert\) | +0.846 | 0.715 | 0.833 | +0.295 | 0.433 |
| WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.831 | 0.690 | 0.942 | +0.121 | 0.783 |
| WD 45° \(\lvert\mathbf{U}\rvert\) | +0.801 | 0.642 | 0.800 | +0.307 | 0.533 |
Every score is over the same 120 points.
- Per-point values:
/api/cases/C/pairs/openfoam12-realizableKE.csv - Reproduce: Eddy3D fixture TestAijCaseC0DContainerE2E; figures validation/plot_case_c_openfoam.py
OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 1D
Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate |
|---|---|---|---|---|---|
| WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.801 | 0.642 | 0.858 | +0.341 | 0.433 |
| WD 0° \(\lvert\mathbf{U}\rvert\) | +0.741 | 0.549 | 0.550 | +0.551 | 0.292 |
| WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.896 | 0.804 | 0.975 | +0.192 | 0.675 |
| WD 22.5° \(\lvert\mathbf{U}\rvert\) | +0.882 | 0.777 | 0.725 | +0.394 | 0.367 |
| WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.872 | 0.760 | 0.942 | +0.198 | 0.733 |
| WD 45° \(\lvert\mathbf{U}\rvert\) | +0.873 | 0.761 | 0.850 | +0.353 | 0.433 |
Every score is over the same 120 points.
- Per-point values:
/api/cases/C/pairs/openfoam12-realizableKE.csv - Reproduce: Eddy3D fixture TestAijCaseC1DContainerE2E; figures validation/plot_case_c_openfoam.py
OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 2D
Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.
| Score | R | \(R^2\) | FAC2 | FB | Hit rate |
|---|---|---|---|---|---|
| WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.857 | 0.734 | 0.850 | +0.251 | 0.492 |
| WD 0° \(\lvert\mathbf{U}\rvert\) | +0.848 | 0.720 | 0.692 | +0.443 | 0.317 |
| WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.906 | 0.820 | 0.983 | +0.091 | 0.875 |
| WD 22.5° \(\lvert\mathbf{U}\rvert\) | +0.894 | 0.799 | 0.900 | +0.240 | 0.633 |
| WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | +0.909 | 0.826 | 0.958 | +0.103 | 0.883 |
| WD 45° \(\lvert\mathbf{U}\rvert\) | +0.912 | 0.832 | 0.900 | +0.243 | 0.608 |
Every score is over the same 120 points.
- Per-point values:
/api/cases/C/pairs/openfoam12-realizableKE.csv - Reproduce: Eddy3D fixture TestAijCaseC2DContainerE2E; figures validation/plot_case_c_openfoam.py
OpenLB
Status: not comparable
Not scored: the case measures only at pedestrian height, \(z = 0.1\thinspace D\) (2 m full scale), 1.25 cells above the ground at the \(\mathrm{d}x = 1.6\thinspace\mathrm{m}\) of the Case A and B runs. A point is scored only where every point at its height in an empty-domain control is within 5 % of the imposed log law. In an empty domain 32 m wide and 80 m tall with Case C’s inflow on Eddy3D’s openlb-wind:1.9.3-r6 (its ground row half a cell under the domain, so 2 m is 2 m above the wall), 80–320 m from the inlet (where the array’s points sit with the inlet 5 block heights upstream, as in the Case A and B runs), \(U\) at 2 m reads 0.48 (0.37–0.64) of the log law with the bounce-back ground, 0.85 (0.75–0.94) with OpenLB’s ground wall model and the rough law, and 0.93 (0.86–1.00) with the power law. The lowest height at which every point is within 5 % is 12.8 m (8 cells) with either wall law and 19.2 m (12 cells) with bounce-back. In a domain 256 m wide, 320 m tall and 1100 m long on 1.9.3-r6, averaged over two crossings of the 2 m layer, \(U\) at 2 m reads 0.88 (0.78–0.95) with the rough law, 0.96 (0.886–0.999) with the power law and 0.52 (0.05–0.77) with bounce-back. The power law holds every point within 5 % over the first half crossing (0.954–1.004), not over two: its mean falls as the averaging goes on. On the earlier 1.9.3-r4, whose bounce-back wall sat half a cell up and whose wall model’s first node a full cell up, the same domain read 1.04 (1.02–1.07) with the rough law over half a crossing.
The engine and the measurements are on the OpenLB and FluidX3D page.
FluidX3D
Status: pending
Not run yet. The case measures only at pedestrian height, \(z = 0.1\thinspace D\) (2 m full scale), 1.25 cells above the ground at \(\mathrm{d}x = 1.6\thinspace\mathrm{m}\). A point is scored only where every point at its height in an empty-domain control is within 5 % of the imposed log law. In an empty domain with Case C’s inflow at that lattice, 80–320 m from the inlet (where the array’s points sit), \(U\) at 2 m reads 1.00 (0.98–1.01) of the log law with FluidX3D’s ground wall function and a steady inflow, and 0.48 (0.38–0.60) with the bounce-back ground; in a domain 96 m wide instead of 32 m, 1.00 (0.99–1.01) and 0.44 (0.38–0.55). With the wall function every point is within 5 % in both domains, so the ground no longer keeps the case unscored. With synthetic inlet turbulence the 32 m domain reads 1.05 (0.99–1.10). The runs of Cases A and B predate the wall function.
The engine and the measurements are on the OpenLB and FluidX3D page.
ML surrogate
Status: scored
Eddy3D’s Wind Predictor (the public Yel-2.0 network) predicts \(\lvert\mathbf{U_{\mathrm{mean}}}\rvert/U_{\mathrm{ref}}\) at one height, 1.8 m, so the case is scaled ×90 to put its measurement plane there; the prediction is read at the tunnel’s own sensor coordinates. How the surrogate is run is on the ML surrogate page.
| Quantity | n | FB | FAC2 | R | hit rate | OpenFOAM, same points and quantity |
|---|---|---|---|---|---|---|
| \(V_{\mathrm{s}}/U_H\), 9 configurations | 1080 | +0.67 | 0.44 | +0.21 | 0.18 | \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\): FB +0.18, FAC2 0.93, R +0.87, hit rate 0.69 |
- FB is positive when the model is low. Reversing the wind changes the predicted field by at most \(1 \times 10^{-4}\) \(U_{\mathrm{ref}}\): the prediction does not depend on the wind direction.
Scores
| Score | R | \(R^2\) | FAC2 | FB | Hit rate | n |
|---|---|---|---|---|---|---|
| \(V_{\mathrm{s}}\), 9 configurations | +0.212 | 0.045 | 0.444 | +0.666 | 0.181 | 1080 |
| \(V_{\mathrm{s}}\), 2D WD 0, lattice cut at ±250 m | +0.075 | 0.006 | 0.875 | +0.192 | 0.400 | 120 |
| wind reversed (control) | -0.441 | 0.194 | 0.725 | +0.367 | 0.392 | 120 |
- Per-point values:
/api/cases/C/pairs/ml-surrogate.csv - Reproduce: Eddy3D fixture Outdoor.ML.Tests/TestAijSurrogateE2E; figure validation/plot_ml_surrogate.py
Notes on every run
- Normalisation: velocities by \(U_H\), \(k\) by \(U_H^2\) (the benchmark's own convention; Case E by \(U\) at 15.9 m, as it is published); hit rate \(\delta_{\mathrm{rel}} = 0.25\), \(\delta_{\mathrm{abs}} = 0.05\) in those units
Leaderboard
| Rank | Run | Source | Hit rate, \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) | Hit rate, \(\lvert\mathbf{U}\rvert\) | Hit rate | FAC2 | R | \(R^2\) |
|---|---|---|---|---|---|---|---|---|
| 1 | OpenFOAM simpleFoam (Eddy3D Outdoor case writer) (eddy3d-of12-nutcap-lifted) | community | 0.700 | 0.431 | 0.565 | 0.847 | +0.858 | 0.739 |
| 2 | OpenFOAM 12 | Eddy3D | 0.688 | 0.436 | 0.562 | 0.850 | +0.853 | 0.730 |
All runs scored by validation/score_submissions.py. Ranked by mean hit rate (VDI 3783) over the tasks; bold: best per column. Not ranked: the ML surrogate (no point ids) and negative controls. Community rows are self-reported (Validate your own solver).