Skip to content

Parity evidence

The claim — "RetroAchievementsSharp produces 100 % identical output to the original" — is an engineering claim with receipts. This page lists the evidence and how to reproduce it.

Current status

Metric Value
Fast suite (RetroAchievementsSharp.Tests, in the solution) 415/415 green (Debug and Release)
Slow suite (RetroAchievementsSharp.Slow.Tests, run manually) 172/172 green
Tier-2 parity cases 90/90 byte-identical vs the rcheevos 12.4.0-built oracle
RVZ-vs-ISO cases 6/6 — real GameCube/Wii RVZ images hashed live via RVZSharp equal the DolphinTool-converted ISO hashes
Real-ROM parity cases 61/61 byte-identical vs the pinned 1.8.3 binary (user libraries, incl. 3DS with user keys)
Tier-1 ported vectors all upstream test/rhash vectors green
CLI output byte-identical (stdout + stderr + exit codes), including verbose mode and error paths
Platforms net8.0;net9.0;net10.0 (415/415 + 172/172 green on each); CLI publishes win-x64/arm64 + linux-x64/arm64

The tiers

  1. Tier 1 — ported vectors: every upstream test from test/rhash/ ported with expected MD5s verbatim (deterministic, offline).
  2. Tier 2 — parity harness: both executables run with identical arguments over a generated corpus; stdout/stderr compared raw byte for byte (\r\n included); exit codes must match. See the parity harness.
  3. RVZ validation: real GameCube/Wii RVZ images are converted to ISO with DolphinTool and hashed; the same RV files are hashed live through RVZSharp (RvzFilereader, no conversion) and the hashes must be identical — 6/6 (3 GameCube + 3 Wii, e.g. 007 - Agent Under Fire (USA): 11e819767dd570389fde339a301ab23e). Skips with a note when the libraries, References\DolphinTool.exe, or the CLI are absent.
  4. Tier 3 — published hashes: spot checks against the official game-identification docs. RetroAchievements does not publish a database dump on the docs site, so the spot check uses a snapshot generated by the RetroAchievements.DataFetcher tool (API_GetGameList per console → a RetroAchievements.json with ID/Title/ConsoleID/Hashes): TestPublishedHashMatch hashes the first 50 files of each of 15 cartridge libraries with identify <console> <file> --db <snapshot> and requires at least one official-hash match per library — 15/15 libraries green (e.g. Master System 19/50, Genesis 11/50, 2600 14/50 on the 2026-08-15 snapshot). The snapshot's coverage is partial by construction (games with sets only), so a miss is not a failure; a library with zero matches is.

What the harness caught (evidence the harness works)

The Tier-2 suite is not decorative — it found real port bugs in Part I:

  1. Arg-count guard off-by-one — RetroAchievementsSharp.Cli.exe 4 crashed with an unhandled exception instead of printing usage (fixed).
  2. Wildcard path construction diverged from FindFirstFile/util::directory semantics (fixed, incl. the dir/*.bin quirk reproduction).
  3. Usage banner had an extra blank line vs. the C (fixed).

And in Part II, the audit found the port had inherited the 12.2.1 merge_callbacks bug (fixed with 12.4.0).

The real-ROM sweep (Part III) caught a fourth, memory-model port bug: zip entries >= 2 GiB crashed the CLI — FileUtil.LoadZippedFile materializes the entry in a byte[], which the CLR caps below 2 GiB, while the C mallocs the full entry. Fixed with a disk-backed fallback (FileUtil.LoadZippedFileToTemp + GenerateHashes), verified against 2 GiB 3DS dumps (7th Dragon III Code - VFD, both Ace Combat releases).

Oracle matrix

Oracle Source Used for
References\rcheevos-12.4.0\bin64\RAHasher.exe rcheevos 12.4.0 + RAHasher CLI stack + libchdr v0.3.0 current default (Part II)
References\RAHasher-1.8.3\bin64\RAHasher.exe pinned 1.8.3 sources (12.2.1) Part I evidence
References\RAHasher.exe legacy 1.8.3 binary (numeric ids only) fallback

All are GPL-3.0-built, local-only, git-ignored. Building them: oracles.

Real-ROM parity (user libraries)

TestRealRomParity extends Tier 2 to real dumps: for each configured library it takes the first 50 files (ordinal-sorted; recursive for MAME software-list layouts like pc98_cd, where each game lives in its own subfolder) and runs both executables with the numeric console id and the same absolute paths. stdout, stderr, and exit code must be byte-identical — the same rule as the synthetic corpus.

Console ids Libraries (path stems) Formats exercised
1, 10, 11, 15, 33, 68 Sega Genesis, 32X, Master System, Game Gear, SG-1000, PICO zip (single/multi-entry)
2, 3, 4, 5, 6, 7, 18, 28, 30, 45, 50, 51, 53, 59, 78, 81 Nintendo 64, SNES, GB/GBC/GBA, NES, DS/DSi, Virtual Boy, C64, Intellivision, 5200/7800, WonderSwan, ZX Spectrum, FDS zip / cartridge
8, 49, 58, 76 PC Engine, PC-FX, FM Towns, PC Engine CD ISO / CD
9, 39, 40, 43, 56, 77, 42 Sega CD, Saturn, Dreamcast, 3DO, Neo Geo CD, Jaguar CD, CD-i predominantly CHD + cue/bin
12, 21, 41 PlayStation, PS2, PSP (3 sub-libraries) ISO/PBP + CD
13, 14, 17, 23, 25, 27, 35, 36, 37, 44, 52, 70 Lynx, Neo Geo Pocket (+Color), Jaguar, Odyssey 2, 2600, Arcade (MAME romsets), Amiga, Atari ST, Amstrad CPC, ColecoVision, X68000, Zeebo zip / disk images
22, 82 Xbox, PlayStation 3 whole-file; PS3 is unsupported in both binaries — the case pins the identical rejection
62 Nintendo 3DS (+ CDN, DSiWare) zip; needs the user-supplied aes_keys.txt/seeddb.bin (repo root, git-ignored); CDN/DSiWare multi-entry zips are rejected identically by both binaries; the main library uses 25 files (multi-GiB entries make 50 too slow)

Notes:

  • The oracle is the pinned 1.8.3 binary (References\RAHasher-1.8.3\RAHasher.exe) — the Part I reference — not the 12.4.0 default, so the results line up with the historical 1.8.3 evidence.
  • Cases skip with a note when the host is not Windows, the oracle is missing, or a library path is absent; the suite stays green on machines without the ROM libraries.
  • The CD libraries (Saturn, Dreamcast, Neo Geo CD, 3DO, CD-i) are predominantly CHD — closing the "real CHDs not available locally" gap from the historical record below.
  • The 3DS cases skip when aes_keys.txt/seeddb.bin are missing from the repo root (they are user-supplied and git-ignored).

Reproduce

# fast suite (in the solution; Debug + Release)
dotnet test RetroAchievementsSharp.sln -c Debug
dotnet test RetroAchievementsSharp.sln -c Release

# slow suite — parity/real-world, run manually (not in the sln)
dotnet test RetroAchievementsSharp.Slow.Tests -c Release

# parity corpus only (slow suite)
dotnet test RetroAchievementsSharp.Slow.Tests --filter FullyQualifiedName~TestParity

# force a specific oracle
RASHARP_ORACLE=C:\path\to\RAHasher.exe dotnet test RetroAchievementsSharp.Slow.Tests --filter FullyQualifiedName~TestParity

Any mismatch in the parity suite is a port bug — the project never "accepts" a difference.

Historical record (Part I)

  • Phase 3: real-file parity vs. the C oracle — PSX cue db433fb038cde4fb15c144e8c7dea6e3, 3DO bin 257d1d19365a864266b236214dbea29c.
  • Phase 4: real PS2 ISO parity — 01a517e4ad72c6c2654d1b839be7579d.
  • Phase 5: CHD parity — PSX CHD db433fb038cde4fb15c144e8c7dea6e3, PSP CHD a7070bf07f5c1a0afb2b2d202d7e3893 (byte-for-byte vs RAHasher 1.8.3).
  • Phase 6: 3DS synthetic-fixture parity 10/10 (byte-identical verbose).
  • Phase 8: 82/82 corpus cases vs the source-built 1.8.3 oracle.

Known gaps (documented, not hidden)

  • 3DS real retail CIAs — user-supplied aes_keys.txt/seeddb.bin were provided and the real retail 3DS / CDN / DSiWare libraries are now part of TestRealRomParity (byte-identical vs the oracle).
  • Tier 3 spot checks — done (2026-08-15): TestPublishedHashMatch vs the user-generated RetroAchievements.json snapshot; needs real ROM libraries and the snapshot, both skipped-with-note when absent.
  • DOS (id 26) and Pokemon Mini (id 24) libraries — not yet configured in TestRealRomParity (paths not provided); the consoles themselves are covered by the synthetic corpus + upstream vectors.
  • 3DS CDN/DSiWare downloads — multi-entry zips (00000000 + 00000001 + tmd.0) cannot be hashed by either binary (both reject identically); the CDN entries must be extracted to .cia/.cxi first.
  • RVZ/WIA (GameCube/Wii) are hashed live through RVZSharp (RvzFilereader, GPL-2.0-or-later) and validated against the DolphinTool-converted ISO (6/6) — a net-new capability beyond rcheevos, implemented and validated in-project. WUX/WUD (Wii U) have no implementation in either engine; Wii U (id 20) has no hasher at all; both binaries reject it with Unsupported console for file hash: 20. See Known quirks.