Rajtík's blog

← Back to home

Business 10 min read

Synchronous VIES in billify: 3 iterations for an invoicing API

When an invoicing API returns 201 Created with a PDF URL in a single HTTP cycle, an 80-second retry budget is unusable from the start. Three iterations of getting the synchronous path to VIES under control.

In the previous article about the VIES audit trail I ended with a teaser: for captchaapi.eu I can afford a serial retry budget of 3 × 20 s plus a sleep between attempts. The verification there runs either in a nightly cron before subscription renewal, or in the checkout wizard, where the user has a "Save and come back" fallback.

Billify.cz has a different mode. A client issues an invoice synchronously through the REST API: POST /v1/invoices returns 201 Created with the invoice number, a supplier snapshot and a PDF URL. Between the request and the response I must fit in VIES (if it is a reverse-charge supply under § 9(1) of the Czech VAT Act), the ČNB rate, generating the invoice number and saving it. The client on the other side typically runs inside its own payment gateway webhook and has its own timeout - for an 80-second serial retry I simply have no room there.

In this article I will show how I solved it in three iterations.

Version 0: just call VIES

The trivial version of synchronous verification looks like this:

public function issueInvoice(Customer $customer, ...): Invoice
{
    $result = $this->vies->verify($customer->vatId);

    if ($result->verdict === Verdict::Invalid) {
        throw new ReverseChargeIneligible('VAT ID invalid');
    }
    if ($result->verdict === Verdict::Unavailable) {
        throw new ServiceUnavailable('VIES down, try later');
    }

    // ... issue the invoice
}

It works - until one of three scenarios occurs:

1. Tail latency. VIES typically answers in 1-3 seconds, but occasionally it hangs for 25+ s. The client side has long since failed on a timeout and I never even got to the 201.

2. A burst for the same customer. A client issues 10 invoices in a row to one recipient (for example monthly batching). I call VIES 10×, and 10× I pay the price of a race condition with an outage in the EU.

3. Unavailable as a permanent 422. When VIES goes down briefly, I return the client a 422 "try again". The client has no way to handle that - its cron is ending in the meantime, its webhook will not repeat. For me that means a lost invoice and manual work.

Each of the three scenarios has its own iteration.

Iteration 1: a parallel race instead of a serial retry

The first iteration addresses tail latency. When I cannot afford serial calls (1 + 1 + 1 = 3× 30 s), I have to go parallel (2× 30 s = still 30 s, but two independent chances to get an answer).

The concept: I launch two concurrent requests to VIES and take the first definitive verdict (Valid or Invalid), skipping transient failures (timeout, 5xx). When both fail, I return Unavailable, not Invalid. Otherwise I would falsely flip the supply regime to a negative.

Laravel can do this through Http::pool:

$responses = Http::pool(fn (Pool $pool) => array_map(
    fn () => $pool->timeout(30)->acceptJson()->get($viesUrl),
    range(1, 2),
));

foreach ($responses as $response) {
    $verdict = $this->parse($response);

    if ($verdict->isDefinitive()) {
        return $verdict;   // Valid / Invalid - we are done right away
    }
}

return Verdict::unavailable();   // both attempts failed transiently

A trap I fell into: array_fill(0, 2, $pool->get(...)) evaluates $pool->get(...) once and fills the array with two references to the same promise. The pool then sends one request, not two. array_map with a closure forces its own call for every slot.

With VIES I see "one member-state node 503, the other OK" much more often than "the whole service down for five minutes". Pool(2) therefore significantly lowers the Unavailable rate at the cost of double the calls. For a synchronous invoicing API it is a good choice - VIES costs nothing, the client's time does.

What this iteration does not solve: 10 calls for 10 invoices to the same customer is still 10× pool(2) = 20 requests. And an Unavailable from both directions still ends up as a 422 for the client.

Iteration 2: cache until the end of the day

The second iteration solves the burst. The key observation: all invoices issued on the same day have the same DUZP - that is, the same moment at which I need the VAT ID verified. A verification from nine in the morning therefore also serves an invoice issued at five in the afternoon of the same day.

A cache strategy follows from that:

$key = "vies:{$supplierVat}:{$customerVat}";

if ($cached = Cache::get($key)) {
    return $cached;
}

$result = $this->verify(...);   // pool(2) from iteration 1

if ($result->isDefinitive()) {
    Cache::put($key, $result, now()->endOfDay());
}

return $result;

Three details that matter:

The key is the (supplier, customer) pair, not just the customer. The same customer can fall into a different regime towards different suppliers (VAT payer status ending / being renewed, invoicing through linked companies). If I cached only by customer, I could let through a verdict from a different legal context.

The TTL is endOfDay(), not 24 hours. A classic TTL could let a Sunday Valid verdict into Monday's invoices - and Monday's DUZP already wants a verification that is close to Monday. Midnight is a business boundary, not an arbitrary window.

I cache only Valid and Invalid. Unavailable never gets into the cache. Otherwise a second attempt five minutes later would return the cached Unavailable instead of going live and possibly succeeding. The cache would actually make the result worse.

What this iteration does not solve: still, when even pool(2) fails, the client gets a 422. And for a new customer I do not have in the cache yet, the first call is fully exposed to VIES.

Iteration 3: return Pending instead of 422

The third iteration changes the mental model. The defensive reaction to Unavailable is "return 422 and tell the client: try later". But operationally that is wrong. VIES can be down for hours and the client has no way to deal with it.

Instead: Unavailable lets the invoice through with a Pending status. The invoice is issued, gets a number, a 201 with a PDF URL is returned. In the database it has an attached audit row with what VIES answered (and what it did not answer). Unavailable is not a final state - a scheduled command settles it in the background.

Completion by a scheduled command that runs every 15 minutes:

// schedule
$schedule->command('billify:retry-pending-vies')
    ->everyFifteenMinutes()
    ->withoutOverlapping();

// command (simplified)
Invoice::query()
    ->where('vat_mode', VatMode::ReverseCharge)
    ->whereDate('issued_at', today())   // today only
    ->whereRelation('latestVies', 'verdict', Verdict::Unavailable)
    ->chunkById(200, fn ($invoices) =>
        $invoices->each(fn ($inv) => RetryViesJob::dispatch($inv->id))
    );

The retry job is one-off (tries = 1, 60 s timeout, a unique lock on the invoice ID so two runs of the command do not trample each other). It writes another audit row, the invoice's latestVies is updated and the compliance state is recomputed from the last row.

The cutoff is implicit: the query filters whereDate('issued_at', today()). As soon as a new day begins, the invoice drops out of the query, no further retry is dispatched, and the compliance state accessor reads the last Unavailable as Exhausted. No explicit state machine, no cancel logic - the day simply ends.

The client learns the result through a webhook. The synchronous 201 carried the status Pending; at that moment the client only knows that the invoice exists and that the verification will finish in the background. When the retry job reaches a definitive verdict, it fires an event:

// inside the retry job
if ($result->isDefinitive()) {
    InvoiceViesResolved::dispatch($invoice->id, $result->verdict);
}

A queued listener on the other end finds the client's registered webhook for the invoice.vies.resolved event (if it has one) and sends it an HMAC-signed POST with the new state. The client can then locally overwrite its invoice cache, send the customer an email with the definitive invoice or anything else it normally handled around the Verified state with synchronous issuing.

An important detail: the webhook fires only on an async change (Pending → Verified / Invalid / Exhausted). A synchronous Verified from the first path, where the client got it right in the 201 response, is not announced by a webhook - that would be duplicate information.

The six states that follow

When I sketch out the possible paths through the three iterations, I get a six-member compliance lifecycle:

State When it occurs
NotApplicable The invoice is not reverse-charge, VIES is not dealt with
Verified VIES returned Valid at issuing or on retry
Pending VIES returned Unavailable, waiting for the next retry tick
Invalid VIES returned Invalid (rejected via the API with 422 at input)
Acknowledged UI path: the user clicked "Issue anyway" over an Invalid verdict
Exhausted Pending VIES returned no answer by midnight, the retry window closed

A client through the API sees the same state as a user in the UI. The only difference is Acknowledged, a state that only the UI button can issue. The REST API deliberately does not have that path, otherwise the contract would not be deterministic.

What it showed

1. A synchronous write against a flaky service is not a place for a serial retry. When the client on the other end has its own timeout, you cannot build a series against it. Either a parallel race, or you move the retry to the background (Pending + cron). A serial retry makes sense where you can afford it (cron, queue worker, background job) - not in a synchronous HTTP cycle.

2. Pending is a promise, not an answer. A false Invalid on tax data would in the end be paid for by me as a penalty, so "I do not know, I keep trying" is a legitimate state in the API contract, not an error. The price, though, is a closed loop on both sides: inward (the cron + retry job that settles the state in the DB) and outward (the webhook through which I actively tell the client the result). Without that second side I would force the client to poll, and that is worse UX than if I had just returned a 422.

3. endOfDay() as a TTL where the day carries business weight. A classic TTL (24 h, 1 h) is arbitrary and shifted. endOfDay() corresponds to how the DUZP works - the day is the atomic unit here. A TTL tied to the business day will not let you serve a Sunday Valid as the answer to a Monday request.

4. An implicit cutoff through the query, not through a state machine. "Midnight flips invoices to Exhausted" I initially wanted to solve with an explicit transition at night. A more elegant solution came out of it though: the query sees only invoices issued today, the accessor derives the state from what the last validation says. No cron for the transition, no flag to synchronize - the day passed, the query saw it, the accessor saw it, done.

Next time I will follow up with an article about ČNB rates - how to handle the "rate for DUZP" when ČNB occasionally does not publish a fixing and the synchronous invoicing flow needs some answer right away.