How to Change an Event's Data Without Breaking Other Services

Your order service publishes an OrderPlaced event. Reporting, email and warehouse services all read it. Adding a delivery preference seems like a small change until an older worker rejects the new field, or a replay discovers that last year's events never contained it.

Events travel beyond the deployment that created them. They can wait in queues, remain in archives and be read by services owned by other teams. Changing their data therefore requires a plan for old readers, new readers and retained history, not just a successful build of the producer.

Introduction

An event is a record of something that happened. Its data format is a contract between the service that writes it and the services that interpret it. That contract includes field names and types, but also meaning: whether an amount is in pounds or pence, what a status represents and whether a missing value is allowed.

We will evolve a small OrderPlaced event step by step. The examples use JSON because it is easy to read, then explain how schema tools and formats such as Avro and Protocol Buffers help with particular parts of the problem.

The goal is to let services change independently without guessing what old data means. We will preserve useful compatibility, recognise changes that deserve a new event version and test the combinations that exist during real deployments and replays.

Start with a Small Existing Contract

Suppose the current event looks like this:

{
  "eventId": "evt_842_placed",
  "eventType": "OrderPlaced",
  "schemaVersion": 1,
  "orderId": "order_842",
  "totalMinor": 4500,
  "currency": "GBP"
}

The documented meaning is that order 842 was accepted with a total of 4,500 minor units in GBP. The reporting service records the amount, while notifications use the order identifier to create a confirmation.

The event identifier belongs to this logical fact. A retry delivering the same fact keeps that identity so consumers can recognise it. Schema version describes the format and meaning used to encode the fact; it is not a counter that increases whenever the order changes.

A separate order version may be useful if consumers maintain the order's latest state. Keep those concepts distinct. Event identity answers which fact this is, schema version answers how to read it and entity version can answer where it belongs in one order's history.

Write this contract down before changing it. A field that looks obvious to its author may have several plausible interpretations to another service. Compatibility testing cannot protect a meaning that nobody has specified.

Think about Readers before Writers

The producer creates events, and consumers read them. During a rolling deployment, old and new producer instances may run together, and consumer services may update on different days.

There are four combinations to consider: old reader with old data, new reader with old data, old reader with new data and new reader with new data. The first and last usually receive attention. The middle two cause many production surprises.

A new reader needs to handle messages already waiting in the queue. An old reader may see a new message before its own deployment finishes. Retained history can extend the first requirement for months or years after all running producers have changed.

Do not assume a successful producer deployment proves consumer readiness. The producer may serialize valid JSON while a warehouse consumer rejects unknown fields or expects a value that is no longer present.

List the known consumer purposes, owners and supported schema versions. Include scheduled exports and replay tools, not just continuously running services. A monthly report can hide an incompatible reader until long after a migration appeared complete.

Add an Optional Field Carefully

We want to add a delivery note. An additive event might contain deliveryInstructions when the customer supplied them:

{
  "eventId": "evt_843_placed",
  "eventType": "OrderPlaced",
  "schemaVersion": 1,
  "orderId": "order_843",
  "totalMinor": 4500,
  "currency": "GBP",
  "deliveryInstructions": "Leave with reception"
}

This can remain compatible if old readers ignore fields they do not understand and new readers accept events without the added field. Those are behaviours to verify in the actual serializers and validators, not universal properties of JSON.

A strict schema with additional properties forbidden may reject the new field. A constructor requiring deliveryInstructions may reject old events. Generated types and custom validation can make an apparently harmless addition incompatible.

Define absence as “no delivery instruction supplied”, not an arbitrary instruction invented by the new service. A default should express an existing business meaning. It should not quietly turn missing historical data into a claim about what the customer requested.

An optional field also needs limits. Free-form text can contain sensitive data, unusual characters and large content. Document its maximum size and which consumers are allowed to retain it. Adding data broadens the contract's operational and privacy responsibilities.

Separate Missing, Null and Empty

These three JSON forms need not mean the same thing:

{}

{ "deliveryInstructions": null }

{ "deliveryInstructions": "" }

Missing can mean an older producer did not support the field. Null can mean explicitly unknown or not applicable. An empty string can mean a supplied but blank value. The contract may choose to treat some forms equivalently, but it should do so deliberately.

The distinction becomes important for updates. A missing field in a patch-like message might mean leave the previous value unchanged, while null might mean clear it. Treating both as a default string could preserve something the user intended to remove or erase something they never changed.

For facts such as OrderPlaced, prefer a clear snapshot meaning over an accidental mixture of patch semantics. State which fields were known at the time and which are optional. Consumers should not need to infer whether absence represents a deletion instruction.

Avoid using zero as a universal substitute for unknown numeric data. A zero amount, zero quantity and unknown amount are different business states. A deserializer's convenient default can become an incorrect report if the application never checks that distinction.

Renaming a Field Is Usually Two Changes

Renaming totalMinor to amountMinor looks tidy in the producer's code. To an old reader, however, totalMinor disappeared. To a new reader replaying old data, amountMinor never existed.

A compatible transition can first teach readers to accept both names under an explicit rule. If both appear, they must agree; a mismatch should be rejected or quarantined rather than resolved by whichever property the serializer visits last.

The producer can then include the new name while maintaining the old one for the agreed overlap, if the format and compatibility rules permit that approach. Once all relevant readers are ready, a later version can retire the old field.

Another option is a new schema version with a small adapter that maps old events into the new internal representation. That avoids carrying duplicate names indefinitely and makes the transition visible.

Do not rewrite every archived event merely to match the new code's preferred property name. A read-time adapter can preserve historical bytes while presenting a consistent model to current handlers. It also makes it easier to compare a received event with the original record during an investigation.

The important question is whether readers can still interpret the fact. A producer-side rename is not an isolated refactor once other services depend on the serialized field.

Changing Units Can Break a Perfectly Valid Message

Suppose totalMinor remains an integer, but the producer starts placing whole pounds in it. An old report reads 45 where it previously expected 4500 and divides by one hundred. The JSON parses successfully, yet the reported revenue is wrong.

Schema validation may catch a changed type, but it usually cannot infer that the unit changed. The same problem appears when a timestamp switches from seconds to milliseconds or a weight changes from grams to kilograms.

Prefer explicit names and documented units. A new meaning deserves a new field or versioned contract, even if its underlying storage type is unchanged. Do not use a comment in the producer's code as the only announcement of a cross-service semantic change.

Currency also affects interpretation. A minor-unit amount must be interpreted with its currency rules; not every currency has two decimal places. An old consumer that assumes GBP cannot safely process newly introduced currencies simply because a currency string was always present.

Test business results, not just parsing. An example event should produce the expected displayed amount, accounting value or shipment weight in the consumer. Such tests catch changes that remain structurally valid while violating the original agreement.

Treat New Status Values as Contract Changes

An event may contain status with values accepted, cancelled and completed. Adding pending_review can break a reader that uses an exhaustive switch or, worse, falls through to a default that treats every unknown value as accepted.

Define unknown-value handling before extending the set. A reporting consumer might retain an unknown status visibly for later interpretation. A fulfilment consumer should not ship an order merely because it does not recognise the status.

There is no single correct fallback for every field. Unknown display categories can sometimes be shown as Other. Unknown payment or permission states often need to stop a sensitive action until the contract is understood.

If the new value changes which business actions are permitted, update relevant consumers before producers emit it. Structural forward compatibility is not enough when an old reader lacks the rules needed to act safely.

Preserve the original value in diagnostic records where appropriate. Mapping it irreversibly to a generic enum value can make later repair impossible. Keep logs bounded and avoid exposing sensitive payloads while retaining enough evidence to identify the unsupported case.

Understand Compatibility Direction

Backward compatibility generally means a new reader can read data written with an older schema. That supports replay and consumers upgrading before producers. Forward compatibility generally means an old reader can read data written with a newer schema.

Full compatibility combines both directions for the versions being checked. The terms can feel counterintuitive, so write the reader-and-writer combinations explicitly in deployment plans rather than relying on a label alone.

Confluent's schema evolution documentation describes these compatibility modes and transitive checking. A transitive check considers a broader history of registered schemas, rather than only the immediately preceding version.

That difference matters when a consumer jumps from version 1 to version 5 or rebuilds from old retained events. Passing a check against version 4 alone does not automatically prove version 1 is still readable.

Compatibility is also format-specific. A change accepted under one format's resolution rules may not be safe under another. Record the serializer, schema format and compatibility policy with the event contract so that tools and people use the same definition.

Use a Schema Registry for Structural Guardrails

A schema registry stores named schema versions and can reject changes that violate a configured compatibility policy. Producers and consumers can identify which schema was used and obtain the information needed to decode the event.

This is useful automation. It prevents a hurried deployment from quietly replacing a required integer with an unrelated object when that violates the selected rules. It also gives teams a discoverable history of the contract.

The registry does not decide whether the new meaning is commercially correct. It cannot generally prove that a shipping date still refers to the customer's local calendar or that a field named authorised actually means payment authorisation succeeded.

Choose ownership and review rules for schemas. A registered subject should represent a coherent contract, and changes should be traceable to a reason and supported reader behaviour. Allowing every service to redefine shared events independently defeats much of the benefit.

Plan registry outages too. Clients may cache known schemas, but introducing a brand-new schema can require a control-plane operation. Test the chosen client behaviour rather than assuming existing event processing must stop whenever the registry is briefly unavailable.

Learn the Rules of the Serialization Format

JSON is readable, but compatibility depends heavily on application conventions and validators. Avro and Protocol Buffers have more explicit rules for resolving differences between writer and reader schemas.

The Avro specification defines schema resolution, including how a reader can use a field default when the writer's schema lacks that field. A default is part of resolution semantics; it does not make arbitrary values or transformations safe.

Protocol Buffers identifies binary fields using numeric tags. Its proto3 language guide explains why removed field numbers should not be reused. Reassigning an old number to a different meaning can make historical bytes appear to describe the new field.

Binary and JSON mappings of the same format can have different compatibility properties. A change that works for the binary wire format may interact differently with field names or unknown fields in JSON. Test the representation actually crossing each boundary.

Generated code also evolves. Include the relevant compiler and runtime libraries in compatibility testing when their behaviour changes. A schema file passing validation does not prove every deployed language client handles it as intended.

Keep Event Envelopes Stable

The envelope contains the information used to route, identify and interpret an event, such as eventId, eventType, schemaVersion and source. Business data can sit inside a separate data object if that helps keep responsibilities clear.

Changing envelope fields can break processing before the business handler runs. A routing rule may use eventType, while an inbox uses eventId for duplicate detection. Renaming either deserves the same care as changing the payload.

Document identifier types and ranges. Large numeric identifiers can lose precision in some language representations. Strings are often a practical cross-language choice for opaque identifiers, but a switch from number to string still needs a migration for existing readers.

Timestamps should include an unambiguous format and meaning. occurredAt might describe when the business fact happened, while publishedAt describes when it entered the stream. Those can differ during delayed delivery or imports.

Avoid using a schema-version field as an escape hatch for undocumented changes. A version number is useful only if readers can obtain the associated contract and select a supported interpretation. Unknown versions should produce an explicit outcome, not a hopeful attempt to treat everything as the latest schema.

Adapt Old Events at the Read Boundary

An adapter, sometimes called an upcaster, converts an older event representation into the internal shape expected by current code. The stored event remains unchanged, and the transformation is applied when it is read.

For example, version 1 may contain totalMinor while version 2 uses amountMinor. A deterministic adapter can rename the field and preserve its value and currency. Deterministic means the same old event produces the same adapted result each time.

Read original event and schema version
Decode using that version's rules
Apply supported deterministic transformations
Validate the current internal model
Pass it to the business handler

Do not fill historical fields by casually querying today's database. If a product's category changed since the event, using its current category can rewrite historical reporting during replay. That may be an intentional restatement, but it is not a neutral format conversion.

If the old event never recorded information now required, admit the limitation. Use an explicit unknown state, a documented historical assumption or a separate enrichment process. An adapter cannot recover facts that were never stored.

Keep transformation chains small and tested against retained examples. A long sequence of partly documented migrations becomes another application with its own failure modes. Periodically evaluate whether the supported internal representation still makes old data understandable.

Roll Out Readers before New Data

For a compatible addition, deploy consumers that accept both old and new events first. Verify their readiness using representative fixtures and deployment inventory. Then allow producers to emit the added data.

During the transition, old producer instances may still send the previous shape. New consumers therefore need their backward-reading behaviour even after the producer release begins. Rollback can also reintroduce an older writer.

Do not retire old handling as soon as the last old producer stops. Messages can remain in retry queues, dead-letter storage and archives. The retirement date depends on the supported replay horizon and every place older data can return from.

For an incompatible change, create a new event version and an explicit migration plan. Some consumers can move quickly; others may temporarily continue reading the old contract through a translation path. Keep ownership and a completion condition for that temporary support.

Observe consumer errors by event type and version during rollout. A schema mismatch can appear as rising lag rather than a failed producer request, because publication succeeded and the failure happened elsewhere.

Be Careful When Publishing Two Versions

Publishing both OrderPlacedV1 and OrderPlacedV2 can help migrate consumers, but it can also cause the same business action twice if a service subscribes to both and treats them as unrelated facts.

Keep a stable logical event identity across representations when they describe the same occurrence. Transport message identifiers may differ, but the consumer's effect identity needs to recognise the relationship if both versions can reach it.

Define which version each consumer should use and how duplicate representations are handled. A reporting rebuild may deliberately compare both versions in an isolated environment, while a notification service should not send two confirmations during migration.

Dual publication also creates a durability question. If the original business change and the two delivery intents are not recorded reliably, one version may be published while the other is missing. Use the application's established durable publication mechanism and monitor delivery of both contracts.

Set an end date and exit criteria. Permanent dual publication increases storage, monitoring and support work, and future changes become harder when every event has several live interpretations.

Test Compatibility with Real Reader Code

Create a fixture set containing actual supported event shapes: an old event missing the new field, a new event with it, explicit nulls where allowed, boundary numeric values and every meaningful status.

Run old supported reader versions against new fixtures when forward compatibility is required. Run the new reader against retained historical fixtures for backward compatibility. Use the actual serializers, validators and business mapping code.

Assert the resulting action or state, not only that deserialization returned an object. The amount should remain correct, a cancelled order should remain cancelled and an unknown permission state should not grant access.

Include cross-language readers where the system uses them. Differences in numeric precision, date parsing, case handling and missing-field defaults can expose incompatibility that a single-language test suite misses.

Test deployment rollback and replay as named scenarios. A release can be compatible with current live traffic while failing on an archived version that a recovery tool still promises to support. The fixture set should reflect the real retention contract.

Keep Public Events Separate from Database Rows

A tempting shortcut is to publish the database row directly whenever it changes. That reduces mapping code initially, but it makes internal schema changes part of every consumer's contract. Adding a private support field or splitting one table into two can unexpectedly change the event stream.

Prefer a deliberate event model containing the facts that consumers are meant to depend on. The producer can change its storage layout while continuing to publish the same business representation. This small translation boundary gives the owning service room to evolve.

For example, the order database might replace one total column with several pricing components. Consumers that only need the accepted order total should not have to understand that internal migration. The producer can still publish amount, currency and the established meaning while its own schema changes in stages.

This does not mean hiding a genuine business change. If discounts are no longer included in the published amount, the event's meaning changed and needs explicit treatment. A mapping layer protects implementation independence; it should not disguise incompatible semantics behind an unchanged field name.

Review event payloads for accidental data exposure as well. Serializing an entire entity can begin distributing a newly added personal field to every subscriber without anyone deliberately requesting it. An explicit event contract makes that expansion visible during review and keeps irrelevant storage details out of downstream services.

Handle Unsupported Events without Guessing

When a consumer sees an unknown required version or invalid payload, preserve enough context to diagnose it and follow a documented failure policy. Repeatedly retrying the same unsupported bytes will not make the reader understand them.

A quarantine record can contain event identity, schema identity, consumer version and a bounded failure reason, with payload access restricted appropriately. Operators can then distinguish a genuine producer error from a consumer that missed its deployment.

Whether processing can continue past the event depends on the consumer. A best-effort dashboard may tolerate temporarily excluding one record with a visible error. An ordered account projection may need to stop that entity's processing until the missing meaning is resolved.

Do not acknowledge and discard an unsupported event merely to make queue lag look healthy. If the service accepted responsibility for the fact, it needs a durable route to resolution or an explicit contract permitting omission.

Repair the producer or consumer first, then redeliver under the same logical identity. If the original fact itself was wrong, use the domain's correction process. Editing bytes silently in a retry queue can destroy the evidence needed to explain what other consumers already observed.

Walk Through the Delivery-Instruction Release

The team proposes deliveryInstructions as optional text, with absence meaning no instruction was supplied. Reporting does not need the field. Notifications may show it, and the warehouse needs to include it in the picking information.

First, the team verifies that reporting ignores unknown properties and adds tests proving its amount calculations remain unchanged. Notifications and warehouse readers are updated to accept both absence and valid text, with a size limit and a defined display policy.

Next, the producer starts including the field when supplied. A rolling deployment creates both shapes for a while, which all supported readers can handle. Monitoring confirms that no schema-related lag or quarantine entries appear.

A month later, an old OrderPlaced event is replayed into a rebuilt report. It lacks deliveryInstructions, but the reporting result is unaffected. A separate historical warehouse reconstruction records that the instruction was not captured rather than inventing one from today's customer profile.

Now suppose the business wants deliveryInstructions to become mandatory for a new specialist service. That is a separate requirement. The new workflow can require it for newly created specialist orders while historical orders remain explicitly incomplete under their original contract.

This sequence preserves independent deployment without pretending every change is automatically compatible. The team tested what old readers accept, what new readers can reconstruct and what the business means by missing data.

Summary

Changing event data is a contract migration across readers, writers and retained history. Additive fields can be safe when real readers tolerate them and new readers handle absence correctly. Renames, units, required fields and new status values need deliberate compatibility decisions.

Use schemas and registries to catch structural mistakes, then test business meaning with actual consumer code. Keep event identity stable, adapt old data without inventing historical facts and make rollout and retirement depend on the full replay horizon.

A dependable change leaves every supported consumer with a clear interpretation. That is more valuable than merely producing valid JSON: the services continue to agree on what happened, even when their deployments and their data belong to different versions.