Validate your own solver
The benchmark’s points, the inlet, the scoring conventions and the metrics that score Eddy3D’s results are all served by the API, and one script in the repository applies them to your values. Ten steps, Case A end to end; the same walk scores every case. Your result can then be submitted and shown on the Community results page.
Contents
1. Get a checkout
git clone https://github.com/Eddy3D-Dev/AIJ-Data-Validation
Nothing to install beyond uv (uv sync reads pyproject.toml and uv.lock; NumPy is all the scorer needs). The scorer, validation/score_submissions.py, uses validation/dispersion_metrics.py, the transcription of Eddy3D’s metrics that scores every served result; its self-test re-scores every served run from its pairs and matches the headline scores to \(10^{-9}\). The API itself needs no checkout: every file below is a plain GET.
2. Pick a case
GET /api/cases.json: every case with its rows, points and which engines Eddy3D has scored. Pin a release (/api/3.2.0/...) when the numbers must stay reproducible.
3. Read the case
GET /api/cases/A.json. Read frame, units and known_issues before anything else: they differ per case. scoring says what the measured side is divided by (U_ref_m_s, Case A’s \(U_H = 4.491\) m/s) and how each instrument is scored; inflow is the inlet contract (step 5).
4. Fetch the points to probe
GET /api/cases/A/probes.csv: one row per measured row of the point table: the key columns (point_id, config, campaign, instrument, run, wind_direction_deg), x y z in the case frame and units, and scored_as, the families the row is scored in. Probe your field at exactly these coordinates; do not interpolate the reference. For Case A that is 186 points. Cases C, D, E and F have several configurations and wind directions in one file: keep the rows of the ones you solve, and each is scored as its own group.
5. Set the inlet
GET /api/cases/A/inflow_openfoam.txt holds 0/U, 0/k, 0/epsilon (the atmBoundaryLayerInlet* conditions of OpenFOAM 12), 0/nut with the nutkAtmRoughWallFunction ground patch, and system/probes, a probes function object with the case’s points, each after a ==== <path> line. The inlet contract page says what is the benchmark’s data (the measured profile, approach_flow.csv) and what is this repository’s recommendation (the log-law fit, \(k\) and \(\varepsilon\) from the ABL relations). Any inlet is allowed; a submission records which one it used.
6. Run your solver and probe
Solve the case in its own frame and units (Case A is model scale: \(H = 0.16\) m, the tunnel’s own speeds). Probe at the points of step 4. Write your values into new columns of your copy of probes.csv:
| column | what | units (units: normalised, the default) |
|---|---|---|
U, V, W | the mean velocity components in the case frame | over \(U_H\) |
speed | the speed \(\lvert\mathbf{U}\rvert\) (computed from the components when absent) | over \(U_H\) |
k | turbulent kinetic energy | over \(U_H^2\) |
cstar | the dimensionless concentration \(c^\ast = c\thinspace U_H H^2/Q\) (the case’s normalisation) | — |
theta or T | the floor-referenced temperature fraction, or the temperature in °C (Cases J and L) | — or °C |
Nomenclature: symbols, columns and scores
Geometry, reference scales and release
- \(H\), \(b\), \(D\) (m) — height of the building, cube, tower or tree belt; width of the building; edge of the cubes of Case C: the lengths the cases are drawn and normalised by
- \(U_H\) (m/s) — reference speed: the mean approach-flow speed at the height \(H\) (each Data page gives the value used)
- \(U_R\), \(H_R\), \(Z_R\) (m/s, m) — reference speed and height of a workbook: Case E, \(U_R\) at \(Z_R\) = 250 m full scale (1 m in the model); Case M, \(U_R\) and \(H_R\) at the reference point of the tunnel and of each field run
- \(U_{\mathrm{ref}}\), \(L_{\mathrm{ref}}\) (m/s, m) — reference speed and length of a case’s normalisation: \(U_H\) and \(H\) (\(U_R\) and \(H_R\) in Case M); for the velocity ratios the approach speed at 15.9 m (Case E) or at 500 m (Case F)
- \(U_b\) (m/s) — Case G: mean approach-flow speed at 9 m, the normalisation of its approach-flow table
- \(q\), \(c_{\mathrm{gas}}\) (m³/s, ppm) — tracer release of the experiment: volume flow of the released gas, and its tracer content (\(10^6\) ppm when pure); \(Q\) is the source strength given to a solve
- \(R_b\) (dimensionless) — bulk Richardson number of the heated-floor cases J and L (negative: unstable stratification)
- \(\mathrm{Re_H}\), \(\mathrm{Re_D}\) (dimensionless) — Reynolds numbers \(U_H H/\nu\) and \(U_H D/\nu\) (\(\nu\): kinematic viscosity of air)
- \(u_\ast\), \(z_0\), \(\kappa\) (m/s, mm, dimensionless) — friction velocity and roughness length of the log law fitted to the approach flow, and the von Kármán constant (0.41)
- \(\mathrm{WD}\) (column
wind_direction_deg; °) — wind direction (each frame says how it is counted); \(\mathrm{WD_R}\) is the mean direction of a field run at its reference point (Case M) - \(x\), \(y\), \(z\) (columns
x,y,z; m) — coordinates in the case’s frame (\(+x\), \(+y\) and \(+z\) are the positive directions; each Data page says which way \(+x\) points), \(z\) up from the floor of the tunnel or from the ground - \(\mathrm{d}x\) (m) — lattice spacing of the OpenLB and FluidX3D runs, in metres or as a fraction of \(H\) (\(H/64\))
- \(L_r\) (m) — reattachment length behind the building, read from a wake line at ground level; reported as \(L_r/H\)
Velocity and turbulence
- \(U\), \(V\), \(W\) (columns
U,V,W; m/s) — mean velocity along \(+x\), \(+y\), \(+z\); \(V\) is positive toward \(+y\) in every case - \(\lvert\mathbf{U}\rvert\) (m/s) — magnitude of the mean velocity vector, \(\sqrt{U^2+V^2+W^2}\)
- \(V_{\mathrm{s}}\) (column
Vs; m/s) — mean omnidirectional scalar speed of a thermistor (never a vector magnitude) - \(\overline{u'u'}\), \(\overline{v'v'}\), \(\overline{w'w'}\) (columns
uu,vv,ww; m²/s²) — velocity variances (overbar: time average, prime: fluctuation about it) - \(\overline{u'v'}\), \(\overline{u'w'}\), \(\overline{v'w'}\) (columns
uv,uw,vw; m²/s²) — Reynolds shear stresses - \(k = \tfrac{1}{2}\left(\overline{u'u'}+\overline{v'v'}+\overline{w'w'}\right)\) (column
k; m²/s²) — turbulent kinetic energy - \(U/U_{\mathrm{ref}}\) (column
velocity_ratio; dimensionless) — mean speed over the case’s reference speed (\(V_{\mathrm{s}}/U_{\mathrm{ref}}\) for a thermistor) - \(\sigma_{U/U_{\mathrm{ref}}}\) (column
velocity_ratio_sd; dimensionless) — standard deviation ofvelocity_ratioover the observations behind a field value (Case F) - \(k/U_{\mathrm{ref}}^2\) (column
k_ratio; dimensionless) — turbulent kinetic energy over the square of the reference speed, where only normalised values are published (Case G)
Concentration and temperature
- \(C\) (column
C; ppm) — mean tracer-gas concentration as published (volume fraction \(\times 10^6\) of the released gas diluted in air) - \(c_{\mathrm{rms}}\) (column
c_rms; ppm) — root mean square of the concentration fluctuation - \(c_0\) (ppm) — reference concentration, defined for each case on its Data page: \(c_0 = \dfrac{c_{\mathrm{gas}}\thinspace q}{U_H H^2}\) (\(U_R H_R^2\) in the denominator in Case M)
- \(c^\ast\) (column
concentration_ratio; dimensionless) — dimensionless concentration \(c^\ast = C/c_0\); the value of a solve is \(c\thinspace U_H H^2/Q\) for its mean concentration \(c\) and its source strength \(Q\) - \(\mathrm{Sc_t}\) (dimensionless) — turbulent Schmidt number of a solve (0.3, 0.5 and 0.7)
- \(T\), \(T_{\mathrm{rms}}\) (columns
T,T_rms; °C, K) — mean air temperature as published (Cases J and L) and the root mean square of its fluctuation - \(\overline{u'c'}\), \(\overline{v'c'}\), \(\overline{w'c'}\) (columns
uc,vc,wc; ppm·m/s) — turbulent concentration fluxes - \(\overline{u'\theta'}\), \(\overline{v'\theta'}\), \(\overline{w'\theta'}\) (columns
uT,vT,wT; K·m/s) — turbulent heat fluxes (\(\theta'\): the temperature fluctuation) - \(\theta = \dfrac{T_f - T}{T_f - T_H}\) (dimensionless) — floor-referenced temperature the solves of Cases J and L are scored on: 0 at the floor, 1 in the approach air at \(H\); \(T_f\) is the floor temperature and \(T_H\) the temperature of the approach flow at \(H\)
Scores
- \(n\) — number of pairs of a measured value \(o_i\) and a modelled value \(p_i\) at one point
- \(R\) — Pearson correlation coefficient of the pairs
- \(R^2\) — square of \(R\): share of the measured variance a linear fit to the model explains
- \(\mathrm{FAC2}\) — fraction of pairs with \(0.5 \le p_i/o_i \le 2\)
- \(\mathrm{FB}\) — fractional bias \(2(\overline{o}-\overline{p})/(\overline{o}+\overline{p})\), positive when the model is low
- \(\mathrm{HR}\) (hit rate) — fraction of pairs with \(\lvert p_i-o_i\rvert/\lvert o_i\rvert \le \delta_{\mathrm{rel}}\) or \(\lvert p_i-o_i\rvert \le \delta_{\mathrm{abs}}\), where \(\delta_{\mathrm{rel}} = 0.25\) and \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\) for a velocity component
- \(\mathrm{NMSE} = \dfrac{\overline{(o-p)^2}}{\overline{o}\thinspace \overline{p}}\) — normalised mean square error
- \(\mathrm{NAD} = \dfrac{\overline{\lvert o-p\rvert}}{\overline{o}+\overline{p}}\) — normalised absolute difference
- \(\mathrm{MG} = \exp\overline{(\ln o-\ln p)}\) — geometric mean bias
- \(\mathrm{VG} = \exp\overline{(\ln o-\ln p)^2}\) — geometric variance
- The ratio of \(\mathrm{FAC2}\) keeps its sign, so a model of the other sign than the measurement is a miss; \(\mathrm{FB}\), \(\mathrm{NMSE}\) and \(\mathrm{NAD}\) divide by the means, so they are undefined (shown as —) for a signed component unless the measured and the modelled mean are both positive
- VDI 3783-9 calls the hit rate \(q\) and the tolerances \(D\) and \(W\); on this site those letters keep their other meanings (\(q\) the tracer release rate, \(D\) the block size of Case C, \(W\) the vertical velocity)
- The detection limit of a concentration is one value per case, \(10^{-3}\) of the largest measured value. It sets \(\delta_{\mathrm{abs}}\) of the hit rate and the both-below rule of \(\mathrm{FAC2}\), and floors the values in \(\mathrm{MG}\) and \(\mathrm{VG}\).
- The scores \(\mathrm{FB}\), \(\mathrm{FAC2}\), \(\mathrm{NMSE}\), \(\mathrm{NAD}\), \(\mathrm{MG}\) and \(\mathrm{VG}\) follow Chang & Hanna (2004); the hit rate follows VDI 3783-9.
Quantities scored
- \(U\), \(V\) and \(W\) (mean velocity components) over \(U_H\)
- \(k\) (turbulent kinetic energy) over \(U_H^2\)
- the speed \(\lvert\mathbf{U}\rvert\) and \(\sqrt{\lvert\mathbf{U}\rvert^2+2k}\), against a thermistor’s \(V_{\mathrm{s}}\)
- \(c^\ast = C/c_0\) (dimensionless concentration) at turbulent Schmidt numbers \(\mathrm{Sc_t}\) of 0.3, 0.5 and 0.7
- \(\theta\) (floor-referenced temperature)
With units: si in submission.yaml you give m/s and m²/s², and the script divides by the case’s U_ref_m_s. Cases E, F and G publish ratios and have no reference speed: give ratios over your own approach flow, as their scoring.U_ref_meaning says.
7. Score
Put the file in a submission folder (step 10 shows the layout; submission.yaml is ten lines about your solver) and run, from the checkout:
uv run python validation/score_submissions.py submissions/my-solver
It checks the folder, prints one line per score, and with --out results/ writes <case>.json (the same shape as the served results: every metric, the headline map, the coverage of each score). Case A scores \(U\), \(V\), \(W\) as signed velocities (FAC2’s ratio keeps the sign; hit rate \(\delta_{\mathrm{rel}} = 0.25\), \(\delta_{\mathrm{abs}} = 0.05\thinspace U_H\)), and \(k\) by the plain definitions, exactly as the method page defines them. If you give the served pairs of Eddy3D’s Case A run as your file, you get the served scores back: that is the example submission.
8. The thermistor bracket
Cases C, D, E and F (and G’s sonic anemometer) measured a mean scalar speed, which a RANS mean velocity cannot reproduce. Give U, V, W (or speed) and k, and the script scores the measurement twice: against \(\lvert\mathbf{U}\rvert\) (vs.vector) and against \(\sqrt{\lvert\mathbf{U}\rvert^2 + 2k}\) (vs.scalar), the two ends of the bracket the thermistor reading has to fall in. Quote both.
9. Partial coverage
You may score a subset: one configuration, one wind direction, a few quantities. Each score then carries its coverage, the points scored over the points the benchmark measured in that group, and the community page shows it as n/N, so a partial submission cannot pass for a full one. A row that matches no published point is counted and reported, not silently dropped.
10. Submit
Make the folder submissions/<your-slug>/ in a fork of the repository:
submissions/my-solver/
submission.yaml solver, version, url, author, licence, contact, inlet, mesh, turbulence_model (or ml_description), compute
cases/A/probes.csv the probe keys plus your value columns (step 6)
cases/C/probes.csv more cases, each in its own folder
Run step 7 once more, then open a pull request that adds the folder. CI scores it again in the pinned build (the result is a file under validation/results/submissions/), the API serves it as /api/submissions/<slug>.json, and the Community results page gets a row, labelled self-reported, scored by this repository’s CI. Eddy3D’s own results stay on the Validation page: a submission is scored by the same code but is not one of them. See CONTRIBUTING.md for the rules of a pull request; the example submissions/example-openfoam12/ is the layout to copy.
What is not scored
Controls (sign flips, mirrored frames, ceilings), subsets (points above two cells, far from the source) and the ML surrogate’s runs are Eddy3D’s own analyses; the script scores the plain quantities at the published points. Case M’s field campaign and Case F’s field observations are groups like any other: give values for them and they are scored.