Case C: validation

Contents
  1. Measurement locations
  2. Explore the points
  3. OpenFOAM 12
    1. Measured against modelled
    2. OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 0D
    3. OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 1D
    4. OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 2D
  4. OpenLB
  5. FluidX3D
  6. ML surrogate
    1. Scores
  7. Notes on every run
  8. Leaderboard

Engine status: OpenFOAM 12 scored · OpenLB not comparable · FluidX3D pending · ML surrogate scored

Nomenclature: symbols, columns and scores

Geometry, reference scales and release

  • \(H\), \(b\), \(D\) (m) — height of the building, cube, tower or tree belt; width of the building; edge of the cubes of Case C: the lengths the cases are drawn and normalised by
  • \(U_H\) (m/s) — reference speed: the mean approach-flow speed at the height \(H\) (each Data page gives the value used)
  • \(U_R\), \(H_R\), \(Z_R\) (m/s, m) — reference speed and height of a workbook: Case E, \(U_R\) at \(Z_R\) = 250 m full scale (1 m in the model); Case M, \(U_R\) and \(H_R\) at the reference point of the tunnel and of each field run
  • \(U_{\mathrm{ref}}\), \(L_{\mathrm{ref}}\) (m/s, m) — reference speed and length of a case’s normalisation: \(U_H\) and \(H\) (\(U_R\) and \(H_R\) in Case M); for the velocity ratios the approach speed at 15.9 m (Case E) or at 500 m (Case F)
  • \(U_b\) (m/s) — Case G: mean approach-flow speed at 9 m, the normalisation of its approach-flow table
  • \(q\), \(c_{\mathrm{gas}}\) (m³/s, ppm) — tracer release of the experiment: volume flow of the released gas, and its tracer content (\(10^6\) ppm when pure); \(Q\) is the source strength given to a solve
  • \(R_b\) (dimensionless) — bulk Richardson number of the heated-floor cases J and L (negative: unstable stratification)
  • \(\mathrm{Re_H}\), \(\mathrm{Re_D}\) (dimensionless) — Reynolds numbers \(U_H H/\nu\) and \(U_H D/\nu\) (\(\nu\): kinematic viscosity of air)
  • \(u_\ast\), \(z_0\), \(\kappa\) (m/s, mm, dimensionless) — friction velocity and roughness length of the log law fitted to the approach flow, and the von Kármán constant (0.41)
  • \(\mathrm{WD}\) (column wind_direction_deg; °) — wind direction (each frame says how it is counted); \(\mathrm{WD_R}\) is the mean direction of a field run at its reference point (Case M)
  • \(x\), \(y\), \(z\) (columns x, y, z; m) — coordinates in the case’s frame (\(+x\), \(+y\) and \(+z\) are the positive directions; each Data page says which way \(+x\) points), \(z\) up from the floor of the tunnel or from the ground
  • \(\mathrm{d}x\) (m) — lattice spacing of the OpenLB and FluidX3D runs, in metres or as a fraction of \(H\) (\(H/64\))
  • \(L_r\) (m) — reattachment length behind the building, read from a wake line at ground level; reported as \(L_r/H\)

Velocity and turbulence

  • \(U\), \(V\), \(W\) (columns U, V, W; m/s) — mean velocity along \(+x\), \(+y\), \(+z\); \(V\) is positive toward \(+y\) in every case
  • \(\lvert\mathbf{U}\rvert\) (m/s) — magnitude of the mean velocity vector, \(\sqrt{U^2+V^2+W^2}\)
  • \(V_{\mathrm{s}}\) (column Vs; m/s) — mean omnidirectional scalar speed of a thermistor (never a vector magnitude)
  • \(\overline{u'u'}\), \(\overline{v'v'}\), \(\overline{w'w'}\) (columns uu, vv, ww; m²/s²) — velocity variances (overbar: time average, prime: fluctuation about it)
  • \(\overline{u'v'}\), \(\overline{u'w'}\), \(\overline{v'w'}\) (columns uv, uw, vw; m²/s²) — Reynolds shear stresses
  • \(k = \tfrac{1}{2}\left(\overline{u'u'}+\overline{v'v'}+\overline{w'w'}\right)\) (column k; m²/s²) — turbulent kinetic energy
  • \(U/U_{\mathrm{ref}}\) (column velocity_ratio; dimensionless) — mean speed over the case’s reference speed (\(V_{\mathrm{s}}/U_{\mathrm{ref}}\) for a thermistor)
  • \(\sigma_{U/U_{\mathrm{ref}}}\) (column velocity_ratio_sd; dimensionless) — standard deviation of velocity_ratio over the observations behind a field value (Case F)
  • \(k/U_{\mathrm{ref}}^2\) (column k_ratio; dimensionless) — turbulent kinetic energy over the square of the reference speed, where only normalised values are published (Case G)

Concentration and temperature

  • \(C\) (column C; ppm) — mean tracer-gas concentration as published (volume fraction \(\times 10^6\) of the released gas diluted in air)
  • \(c_{\mathrm{rms}}\) (column c_rms; ppm) — root mean square of the concentration fluctuation
  • \(c_0\) (ppm) — reference concentration, defined for each case on its Data page: \(c_0 = \dfrac{c_{\mathrm{gas}}\thinspace q}{U_H H^2}\) (\(U_R H_R^2\) in the denominator in Case M)
  • \(c^\ast\) (column concentration_ratio; dimensionless) — dimensionless concentration \(c^\ast = C/c_0\); the value of a solve is \(c\thinspace U_H H^2/Q\) for its mean concentration \(c\) and its source strength \(Q\)
  • \(\mathrm{Sc_t}\) (dimensionless) — turbulent Schmidt number of a solve (0.3, 0.5 and 0.7)
  • \(T\), \(T_{\mathrm{rms}}\) (columns T, T_rms; °C, K) — mean air temperature as published (Cases J and L) and the root mean square of its fluctuation
  • \(\overline{u'c'}\), \(\overline{v'c'}\), \(\overline{w'c'}\) (columns uc, vc, wc; ppm·m/s) — turbulent concentration fluxes
  • \(\overline{u'\theta'}\), \(\overline{v'\theta'}\), \(\overline{w'\theta'}\) (columns uT, vT, wT; K·m/s) — turbulent heat fluxes (\(\theta'\): the temperature fluctuation)
  • \(\theta = \dfrac{T_f - T}{T_f - T_H}\) (dimensionless) — floor-referenced temperature the solves of Cases J and L are scored on: 0 at the floor, 1 in the approach air at \(H\); \(T_f\) is the floor temperature and \(T_H\) the temperature of the approach flow at \(H\)

Scores

  • \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
  • \(R\) — Pearson correlation coefficient of the pairs
  • \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
  • \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
  • \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
  • \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component
  • \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\) — normalised mean square error
  • \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\) — normalised absolute difference
  • \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\) — geometric mean bias
  • \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\) — geometric variance
  • The ratio of \(\mathrm{FAC2}\) keeps its sign, so a model of the other sign than the measurement is a miss; \(\mathrm{FB}\), \(\mathrm{NMSE}\) and \(\mathrm{NAD}\) divide by the means, so they are undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive
  • VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings (\(q\) the tracer release rate, \(D\) the block size of Case C, \(W\) the vertical velocity)
  • The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
  • The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.

Quantities scored

  • \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
  • \(k\) (turbulent kinetic energy) over \(U_H^2\)
  • the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
  • \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
  • \(\theta\) (floor-referenced temperature)

Measurement locations

Measurement points, Case C
Figure 1. Case C, measurement points: A street x < 0 beside the centre block; B street x > 0 beside the centre block; C street \(y \gt 0\) beside the centre block; D street \(y \lt 0\) beside the centre block; E crossings beyond the corners of the centre block. Positions in multiples of \(D\). Centre block absent (0D), D or 2D high; wind directions 0°, 22.5° and 45°.

Explore the points

model lowmodel high
Loading the points… Hover a point to see where it sits in 3D and how the model differs from the measurement; click to pin it. This needs JavaScript; the figures on this page show the same scores.

OpenFOAM 12

Status: scored

All nine configurations (centre block 0D / 1D / 2D × WD 0°, 22.5°, 45°), solved at 1:100 on a cylindrical domain (the tunnel’s turntable: the array turns with WD, the domain does not), ~310,000 cells per centre-block height, 3000 iterations per direction, the 120 probes at exactly the tunnel’s coordinates.

Measured against modelled

Case C, measured vs modelled
Figure 2. Case C, measured vs modelled. The dots take the colour of their street in Figure 1.
Scores for all nine configurations (R, FAC2, FB, hit rate; |U| alone)
centre block WD R FAC2 FB hit rate \(\lvert\mathbf{U}\rvert\) alone: FB / R
0D 0° +0.824 0.908 +0.266 0.567 +0.466 / +0.804
0D 22.5° +0.857 0.983 +0.106 0.750 +0.295 / +0.846
0D 45° +0.831 0.942 +0.121 0.783 +0.307 / +0.801
1D 0° +0.801 0.858 +0.341 0.433 +0.551 / +0.741
1D 22.5° +0.896 0.975 +0.192 0.675 +0.394 / +0.882
1D 45° +0.872 0.942 +0.198 0.733 +0.353 / +0.873
2D 0° +0.857 0.850 +0.251 0.492 +0.443 / +0.848
2D 22.5° +0.906 0.983 +0.091 0.875 +0.240 / +0.894
2D 45° +0.909 0.958 +0.103 0.883 +0.243 / +0.912
Case C, where the model is slow or fast
Figure 3. Case C, where the model is slow or fast.
  • R 0.80–0.91 and FAC2 0.85–0.98 in every configuration.
  • FB +0.25 … +0.34 at \(\mathrm{WD} = 0\) (hit rate 0.43–0.57); FB +0.09 … +0.20 at 22.5° and 45° (hit rate 0.68–0.88). FB is positive when the model is low.
  • The tunnel’s speed is above the model’s \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) at 902 of the 1080 points (84 %); the model’s \(\lvert\mathbf{U}\rvert\) alone is lower still.
  • Rotation sense mirrored (the paper figure’s y mirror), control: R −0.33 … +0.03 at 22.5° and 45°.
  • Upwind and downwind swapped, control: R −0.41 … +0.14 in eight of nine configurations.
Interpretation: Case C pattern and deficit
  • The pattern is right in every configuration.
  • The model is slow, worst at \(\mathrm{WD} = 0\), and the map puts the deficit in the streets running along the wind.
  • The scalar-vs-vector difference does not explain the deficit: the tunnel reads above even \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) at 902 of the 1080 points, and \(\lvert\mathbf{U}\rvert\) alone is lower still. Where the deficit comes from is not settled.
  • Both frame controls above score far below the runs, so the frame is the right one.
Data notes: Approach-flow sampling

The approach flow at the probe height reads 0.83 of the tunnel’s; that reading is a sampling artefact. Upstream of the array the probe height lies below the centre of the first cell of the coarse background mesh, where the probes’ cell-to-point interpolation runs linearly to zero at the wall: the sampled \(\lvert\mathbf{U}\rvert\) is exactly proportional to z there (1.00 m/s at 1 m, 2.01 m/s at 2 m). At \(z = D\) the approach matches the tunnel (0.99). At the measurement points the sampling is sound: none of the 240 checked (2D at 0°, 0D at 45°) has its 2 m sample inside the first cell.

OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 0D

Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.

Score R \(R^2\) FAC2 FB Hit rate
WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.824 0.679 0.908 +0.266 0.567
WD 0° \(\lvert\mathbf{U}\rvert\) +0.804 0.647 0.658 +0.466 0.308
WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.857 0.735 0.983 +0.106 0.750
WD 22.5° \(\lvert\mathbf{U}\rvert\) +0.846 0.715 0.833 +0.295 0.433
WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.831 0.690 0.942 +0.121 0.783
WD 45° \(\lvert\mathbf{U}\rvert\) +0.801 0.642 0.800 +0.307 0.533

Every score is over the same 120 points.

OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 1D

Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.

Score R \(R^2\) FAC2 FB Hit rate
WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.801 0.642 0.858 +0.341 0.433
WD 0° \(\lvert\mathbf{U}\rvert\) +0.741 0.549 0.550 +0.551 0.292
WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.896 0.804 0.975 +0.192 0.675
WD 22.5° \(\lvert\mathbf{U}\rvert\) +0.882 0.777 0.725 +0.394 0.367
WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.872 0.760 0.942 +0.198 0.733
WD 45° \(\lvert\mathbf{U}\rvert\) +0.873 0.761 0.850 +0.353 0.433

Every score is over the same 120 points.

OpenFOAM 12, realizable k-epsilon (Eddy3D Outdoor), configuration 2D

Quantity: \(V_{\mathrm{s}}\) (thermistor) scored against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) [scalar] and \(\lvert\mathbf{U}\rvert\) [vector]; both normalised by \(U_H\) = \(U(z = D)\) = 3.654 m/s.

Score R \(R^2\) FAC2 FB Hit rate
WD 0° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.857 0.734 0.850 +0.251 0.492
WD 0° \(\lvert\mathbf{U}\rvert\) +0.848 0.720 0.692 +0.443 0.317
WD 22.5° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.906 0.820 0.983 +0.091 0.875
WD 22.5° \(\lvert\mathbf{U}\rvert\) +0.894 0.799 0.900 +0.240 0.633
WD 45° \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) +0.909 0.826 0.958 +0.103 0.883
WD 45° \(\lvert\mathbf{U}\rvert\) +0.912 0.832 0.900 +0.243 0.608

Every score is over the same 120 points.

OpenLB

Status: not comparable

Not scored: the case measures only at pedestrian height, \(z = 0.1\thinspace D\) (2 m full scale), 1.25 cells above the ground at the \(\mathrm{d}x = 1.6\thinspace\mathrm{m}\) of the Case A and B runs. A point is scored only where every point at its height in an empty-domain control is within 5 % of the imposed log law. In an empty domain 32 m wide and 80 m tall with Case C’s inflow on Eddy3D’s openlb-wind:1.9.3-r6 (its ground row half a cell under the domain, so 2 m is 2 m above the wall), 80–320 m from the inlet (where the array’s points sit with the inlet 5 block heights upstream, as in the Case A and B runs), \(U\) at 2 m reads 0.48 (0.37–0.64) of the log law with the bounce-back ground, 0.85 (0.75–0.94) with OpenLB’s ground wall model and the rough law, and 0.93 (0.86–1.00) with the power law. The lowest height at which every point is within 5 % is 12.8 m (8 cells) with either wall law and 19.2 m (12 cells) with bounce-back. In a domain 256 m wide, 320 m tall and 1100 m long on 1.9.3-r6, averaged over two crossings of the 2 m layer, \(U\) at 2 m reads 0.88 (0.78–0.95) with the rough law, 0.96 (0.886–0.999) with the power law and 0.52 (0.05–0.77) with bounce-back. The power law holds every point within 5 % over the first half crossing (0.954–1.004), not over two: its mean falls as the averaging goes on. On the earlier 1.9.3-r4, whose bounce-back wall sat half a cell up and whose wall model’s first node a full cell up, the same domain read 1.04 (1.02–1.07) with the rough law over half a crossing.

The engine and the measurements are on the OpenLB and FluidX3D page.

FluidX3D

Status: pending

Not run yet. The case measures only at pedestrian height, \(z = 0.1\thinspace D\) (2 m full scale), 1.25 cells above the ground at \(\mathrm{d}x = 1.6\thinspace\mathrm{m}\). A point is scored only where every point at its height in an empty-domain control is within 5 % of the imposed log law. In an empty domain with Case C’s inflow at that lattice, 80–320 m from the inlet (where the array’s points sit), \(U\) at 2 m reads 1.00 (0.98–1.01) of the log law with FluidX3D’s ground wall function and a steady inflow, and 0.48 (0.38–0.60) with the bounce-back ground; in a domain 96 m wide instead of 32 m, 1.00 (0.99–1.01) and 0.44 (0.38–0.55). With the wall function every point is within 5 % in both domains, so the ground no longer keeps the case unscored. With synthetic inlet turbulence the 32 m domain reads 1.05 (0.99–1.10). The runs of Cases A and B predate the wall function.

The engine and the measurements are on the OpenLB and FluidX3D page.

ML surrogate

Status: scored

Eddy3D’s Wind Predictor (the public Yel-2.0 network) predicts \(\lvert\mathbf{U_{\mathrm{mean}}}\rvert/U_{\mathrm{ref}}\) at one height, 1.8 m, so the case is scaled ×90 to put its measurement plane there; the prediction is read at the tunnel’s own sensor coordinates. How the surrogate is run is on the ML surrogate page.

Measured vs predicted, ML surrogate, Case C
Figure 4. Case C, measured vs predicted pedestrian-height speed, ML surrogate, with the 1:1 line and the factor-of-two band.
Quantity n FB FAC2 R hit rate OpenFOAM, same points and quantity
\(V_{\mathrm{s}}/U_H\), 9 configurations 1080 +0.67 0.44 +0.21 0.18 \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\): FB +0.18, FAC2 0.93, R +0.87, hit rate 0.69
  • FB is positive when the model is low. Reversing the wind changes the predicted field by at most \(1 \times 10^{-4}\) \(U_{\mathrm{ref}}\): the prediction does not depend on the wind direction.

Scores

Score R \(R^2\) FAC2 FB Hit rate n
\(V_{\mathrm{s}}\), 9 configurations +0.212 0.045 0.444 +0.666 0.181 1080
\(V_{\mathrm{s}}\), 2D WD 0, lattice cut at ±250 m +0.075 0.006 0.875 +0.192 0.400 120
wind reversed (control) -0.441 0.194 0.725 +0.367 0.392 120

Notes on every run

  • Normalisation: velocities by \(U_H\), \(k\) by \(U_H^2\) (the benchmark's own convention; Case E by \(U\) at 15.9 m, as it is published); hit rate \(\delta_{\mathrm{rel}} = 0.25\), \(\delta_{\mathrm{abs}} = 0.05\) in those units

Leaderboard

Rank Run Source Hit rate, \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) Hit rate, \(\lvert\mathbf{U}\rvert\) Hit rate FAC2 R \(R^2\)
1 OpenFOAM simpleFoam (Eddy3D Outdoor case writer) (eddy3d-of12-nutcap-lifted) community 0.700 0.431 0.565 0.847 +0.858 0.739
2 OpenFOAM 12 Eddy3D 0.688 0.436 0.562 0.850 +0.853 0.730

All runs scored by validation/score_submissions.py. Ranked by mean hit rate (VDI 3783) over the tasks; bold: best per column. Not ranked: the ML surrogate (no point ids) and negative controls. Community rows are self-reported (Validate your own solver).