You have an application that lives off data you do not own. A price feed, an external API whose schema occasionally changes without warning, a third-party catalogue. The supplier releases a new version whenever they feel like it - your release cycle does not interest them. And you need to get that data into production with no downtime and no risk of deploying something broken.
This is exactly the problem I solve in exile2exile, an open-source tool for Path of Exile 2. A game patch means new data: the passive skill tree, items, gems, thousands of icons. Everything is extracted directly from the official patch server and the application builds on it. The game gets patched twice a week without blinking, so "I will deploy it by hand when I am at the computer" is out of the question. PoE2 is just a concrete example here - the pattern I will describe fits anything that "releases patches" outside your control.
Why cron and overwriting files is not enough
The naive solution suggests itself: cron downloads new data once an hour and overwrites the old data on disk. But that asks for three disasters at once:
- Half-switched state. You overwrite thousands of files while requests keep coming in. The user gets the tree from the new version and the icons from the old one. With large data the overwrite takes minutes, not milliseconds.
- Broken data goes straight to production. The supplier changed the structure, renamed a field, half the icons are missing? You find out from the error logs, once the users have broken it for you.
- No way back. You have just overwritten the old data. A rollback means downloading and processing the old version again - if the supplier even still offers it.
A pipeline that solves this has five parts: detection, isolated extraction, a contract in CI, an atomic swap and a default failure mode of "no change".
Detection: a watcher, not a webhook
GGG (the PoE2 publisher) sends no webhooks, so what remains is polling: the poe2:watch-patch command asks the patch server for the current version every 5 minutes. A new version triggers two things: a notification (Discord + webhooks for subscribers) and an extraction job.
With a price feed or an API it will look the same - when the supplier cannot push, poll and compare a version or a checksum. What matters is that detection only triggers the pipeline, it never changes live data itself.
Isolated extraction: the new version next to it, not over it
Extraction writes the new version to releases/<version> - next to the live data, not over it. Production runs on the old version the whole time and does not know about the extraction in progress.
The extraction itself from the GGPK game format lives in poe2-toolkit, a separate layer of MIT npm packages. The toolkit is pure code, no data - it can pull the tree, items and icons from the patch server, but it knows nothing about how the application uses them. This separation pays off: the extractor can be tested and versioned independently and the application only deals with "I have a directory with data in a known structure".
After extraction the application packs the release into a tarball, computes a sha256 and stages it for download.
A contract, not trust
This is the core of the whole pattern. Deploying data just because it was downloaded and unpacked successfully is trust. I want a contract: a set of tests that verifies the data has exactly the shape the application expects.
After staging a release, the application dispatches the GitHub Actions workflow data-contract.yml and passes it the version and sha256. CI then:
- downloads the tarball from the application and verifies the checksum - it is certain it is testing exactly the bytes the watcher staged, even if another patch came out in the meantime,
- unpacks it into a checkout and runs the Contract suite (
tests/Contract) - real Pest tests over real data: the tree structure, all referenced icons exist on disk, the Path of Building import passes end-to-end, seeded plans load, - on green, calls the application's activation endpoint.
A key detail: CI never touches the supplier's source data. It extracts nothing from the GGPK - it validates exactly the artifact the server staged and that will go to production. Build once, promote. The release that goes live is, byte for byte, the one that passed the tests.
The checksum has one more practical role in the pipeline: it is part of the cache key under which CI stores the downloaded tarball. I use that when running manually - when I dispatch the workflow with the same version but a deliberately different checksum, the cache is missed, the tarball is downloaded again and the Contract tests run fresh from scratch. A simple way to force a new test run without waiting for the next patch.
And because a contract has two sides, the same suite also runs on every push to main - this time against the version production is currently using. A code change that breaks the contract with the data therefore does not pass green, just like data that breaks the contract with the code.
GGG patch server Production GitHub Actions
| | |
|<-- poll (every 5 min)| |
|--- new version ----->| |
| |-- extraction to releases/<v> |
| | (live data untouched) |
| |-- tarball + sha256 |
| |-- dispatch (version, sha256) |
| |<------ downloads tarball ----|
| | verifies checksum |
| | Contract suite |
| | |
| green: |<--- POST /api/data/activate -|
| |-- atomic symlink swap |
| red: | no swap, the last
| | verified version keeps running
Atomic swap: one rename
The application reads all game data through a single symlink: storage/game-data/current -> releases/<version>. Activating a new version is one atomic rename of the symlink.
Why that matters: rename is an atomic operation on POSIX filesystems. A request that arrived a nanosecond before the swap reads its whole response from the old version; a request a nanosecond after reads it all from the new one. There is no moment when anyone would see a mix of both - the half-switched state from the naive solution simply cannot occur. No maintenance mode, no restart, no downtime.
The POST /api/data/activate endpoint is protected by a token and idempotent - a repeated call with the same version breaks nothing.
Failure = no change
The most valuable property of the whole pipeline is its default failure mode. Red tests? No swap. The tarball could not be downloaded? No swap. Server unavailable, checksum does not match, workflow crashed halfway? No swap. Every failure path ends the same way: production keeps running on the last verified version.
That is the exact opposite of the cron solution, where a failure halfway through the overwrite means broken production. Here nobody has to actively look after the deployment - a bad version does not get into production until someone (typically a new patch with corrected data) replaces it with a green one.
The watcher also guards against stuck validations: when a staged version waits for a long time, it re-dispatches the workflow, but at most once every 6 hours. Without a limit, unavailable CI would mean a dispatch every 5 minutes forever.
Rollback without re-extraction
Old releases stay on disk (older ones are pruned over time, the last few are kept). A rollback is therefore the same single call to POST /api/data/activate with an older version - one symlink rename, done in milliseconds. No downloading, no re-extraction, no dependence on whether the supplier still offers the old version.
When the supplier releases a faulty patch and fixes it only a day later, you switch back and calmly wait out the day.
Where else this fits
The pattern stands on four legs and none of them is specific to game data:
- versioned releases side by side, live data is never overwritten,
- a contract in tests that validates the exact artifact meant for deployment,
- an atomic switch via a symlink (or a pointer in the DB, an alias in S3, a traffic switch),
- failure leaves the last verified version.
It fits wherever an external source can break you independently of your release cycle: price and exchange-rate feeds, supplier product catalogues, ML models and their weights, geodata, code lists from government registers, third-party APIs with a schema that "does not change" (until it changes). The point is always the same: do not trust data you did not produce. Give it a contract and have it sign it in CI before it touches production.
The whole implementation is open source: github.com/rajtik76/exile2exile (pipeline, Contract suite, watcher) and github.com/rajtik76/poe2-toolkit (data extraction).