There Is No Now

You send a friend a message: “Book the cheap one.” It is a perfectly clear instruction. The two of you have been looking at the same flight, at the same price, with the same departure time. Except your friend has refreshed the page. The cheap one now leaves on Tuesday, requires an overnight connection, and lands at an airport neither of you has discussed. Your message arrives. Your friend books it. Everyone did something reasonable, and now you have a problem.

The message was delivered correctly. It may even have arrived in less than a second. What failed was the assumption that you and your friend meant the same thing by “the cheap one.” Your instruction referred to a version of the world your friend had already moved beyond. A faster connection might reduce the chance of this happening. It would not make the instruction unambiguous.

Software makes this mistake with impressive efficiency. Imagine an automated deployment system, the machinery that puts new code into a running service. An agent writes version 17 of a change. A human opens it and starts reading. While that happens, the agent produces version 18. A little later, the human clicks Approve. The deployment worker sees the newest version and a fresh approval, so it releases version 18. Every service reports success. The person who clicked the button has never seen the code now running in production.

You can make the clocks perfectly accurate and still ship the wrong version. The approval really did happen after version 18 was created. That fact tells you very little about what the person approved. Somewhere between the interface, the queue, and the worker, a relationship disappeared. The system kept when things happened and lost what depended on what.

This is the kind of problem that makes me enjoy old computer science papers. You arrive looking for a better counter and leave suspicious of an ordinary word. In this case, the word is “after.” We use it to mean several different things, then put a date beside an event and assume the ambiguity has gone away. It has acquired six decimal places.

In A Network That Can Wait, I wrote about networks where different parts of a route exist at different times. A satellite can collect a message now and find somewhere to deliver it later. That makes delay visible enough that the application has to deal with it. A fast network makes the same issue easier to overlook. Information still takes time to travel, and while it travels, other participants keep doing things.

We normally experience the present as something available without asking. Look around a room and you can see who is there, whether a chair is occupied, and whether someone has picked up your coffee. Those observations are imperfect too, but the delays are usually small enough for the task. It is easy to carry that intuition into software. A screen says “available,” so the thing is available. A document says “approved,” so the version in front of us must be the approved one. The interface looks like a window onto the world.

In a network, that window is assembled from messages. Some are recent, some are delayed, and some have not arrived. There is no single, instantly accessible present shared by all the participants. Physical time still exists, and clocks remain useful. A distributed system can even offer very strong guarantees about its shared state. But those guarantees have to be built. The word “current” does not supply them.

What could this event have known?

Leslie Lamport’s Time, Clocks, and the Ordering of Events in a Distributed System, published in 1978, begins with relationships we can establish without synchronized clocks. Within a process, its steps occur in sequence. Sending a message precedes receiving that message. These relationships also form chains: if one event precedes a second and the second precedes a third, the first precedes the third. The resulting relation is called happened-before. Events with no such relationship in either direction are called concurrent.

That last word is easy to misread. Concurrent events need not happen at the same physical instant. Two isolated machines could act an hour apart without either receiving information from the other. Their actions can be concurrent in this model even if a hypothetical observer with reliable clocks could place them neatly on a timeline. The relation describes possible influence through the execution we have modeled.

Think of two people opening sealed instructions in different houses. One opens an envelope at breakfast; the other opens theirs after lunch. If neither has received a message from the other, the later action has no new information about the earlier one. The wall clock can establish that breakfast came first. It cannot turn lunch into a response to breakfast. In a distributed program, that distinction determines which actions can reasonably be interpreted as reactions to other actions.

This is also why “causal” needs a little care here. The relation tracks paths along which influence could have travelled in the execution. It does not establish a complete explanation of someone’s intentions. Reading a warning before pressing a button does not prove that the warning caused the choice, or that the person even understood it. We are building a useful account of information flow, with limits we can state.

Lamport’s own account of how the paper began is wonderfully concrete. He was reading work by Paul Johnson and Robert Thomas about maintaining duplicate databases. His understanding of special relativity helped him see the distinction between a causal partial order and a total order imposed through timestamps. What started as a correction to an algorithm became an argument about implementing distributed state machines. He later remarked that readers often remembered the clocks and missed the state machines.

The mathematical terms sound more forbidding than the idea. A total order puts everything in one line. A partial order leaves some pairs undecided. When you cook, chopping an onion must come before frying that onion. Washing a plate might happen before or after either step without changing the recipe. You can write a complete schedule for the kitchen, but some of its ordering will be your choice rather than a requirement of the meal. A distributed execution has the same distinction between dependencies that matter and orderings we are free to choose.

The document that prompted him, RFC 677, The Maintenance of Duplicate Databases, appeared in January 1975. Its concern was how cooperating processes could maintain copies of a database while the network delayed updates or failed. This was an engineering problem about copies disagreeing, years before our current vocabulary of cloud regions and globally distributed services. The hardware has changed considerably. A copy that has not received an update remains a copy that has not received an update.

I like that route into the subject. Put the same information on two computers, let both do useful work, and try to explain what happens when their updates cross. The awkward questions arrive on their own. Lamport recognized a structure beneath those questions that applied far beyond the particular database in front of him. Relativity helped him see it, but the computer science does not require a story about servers travelling near the speed of light. It requires taking seriously the fact that one participant cannot react to information it has not received.

Return to the deployment. There are two branches after version 17 is sent for review. Along one branch, the author makes version 18. Along the other, the reviewer reads version 17 and approves it. Neither branch has to know what happened on the other. Eventually the deployment worker receives results from both. At that point, the worker knows that a newer artifact exists and an approval exists. It still has no reason to conclude that the approval covers that artifact.

An activity feed tends to flatten the two branches into rows: new version, approval, deployment. That is a convincing story because the rows have the shape of an explanation. We instinctively read the approval as a response to the new version above it. But that connection was supplied by us. The software only sorted some records. A chronology can be accurate and the story we read into it can still be wrong.

The immediate repair is pleasantly ordinary. An approval should identify the exact version it covers. Engineers can use an immutable version ID, or a content hash, a fingerprint calculated from the file’s contents. The worker checks that the version about to be released is the one the person approved. For the flight, the equivalent is “Book flight 412 on Monday at 9:10 for this price.” We have made the object of the instruction explicit, so a change elsewhere cannot silently change its meaning.

That still leaves a human question: did the reviewer actually see and approve that version? A notification reaching a machine does not mean it appeared on the review screen. Someone may have been reading an older tab. The event we record has to match the promise the product makes. “A message arrived at the server” and “the customer accepted these terms” are very different claims, even when both are represented by one row in a database.

Once you start looking for those references, a different picture of an application appears. A reply belongs to a message. A test result belongs to an artifact and an environment. An authorization belongs to an action, a scope, and a particular authority. A model’s answer belongs to the context supplied to that invocation. The date on each record answers one useful question, but none of these relationships can be reconstructed safely from dates alone.

A clock that does not tell the time

There is a compact way to carry some ordering information through messages. A Lamport clock is an integer maintained by a process. Increment it for each event, attach its value to outgoing messages, and on receipt set the local value to one more than the maximum of the local counter and the incoming timestamp. If event a precedes event b in the happened-before relation, its clock value is smaller. Smaller values alone do not establish that relationship in reverse.

For a small example, suppose the author sends a message stamped 7. The worker’s counter is only 2 when that message arrives. Recording the receive event as 3 would put it before the send in our numerical order, so the worker advances to 8. Its next event gets 9. We have arranged the labels so that following the message never takes us backward. No measurement of seconds was involved.

The counter is acting like a rule for numbering chapters in a story. If a chapter depends on a letter sent in an earlier chapter, we must not number it before the letter. Receiving a message stamped 7 means the receiver has just encountered a piece of history its local numbering must accommodate. Jumping to 8 preserves that relationship. The skipped numbers are harmless because we are numbering events, not measuring elapsed seconds.

Now imagine another machine doing unrelated work. It has already counted fifty events, so its next record gets 51. That number is larger than 8. It does not establish that this machine has heard from the author, seen the deployment, or done anything later in physical time. It may simply have been busy. Reading a logical timestamp as a date would be like reading a line number as a measure of how long a program took to write.

A few ordinary questions now turn out to require different evidence:

A single field called timestamp is a tempting place to hide all four questions. It keeps the schema small. The cost shows up when different parts of the application quietly expect different answers from it. I would rather have a slightly more annoying schema than an incident review in which everybody is correct about what their service did and the combined result is still wrong.

A larger logical clock value therefore cannot prove that one event has the other in its history. This explanation of logical clocks makes that limitation explicit. To put every event in one line, we can compare clock values and use a stable process identifier to break ties. The resulting total order respects the causal constraints. The order between concurrent events is a convention we introduced.

Suppose two requests carry the same number, one from computer A and one from computer B. A rule that puts A first gives everyone an unambiguous answer. That can be useful without discovering anything profound about physical time. A bakery ticket machine does something similar: it establishes an order people can follow. It does not measure who became hungry first. The danger is forgetting what the rule actually promises.

Who gets the last seats?

Imagine a worker has received requests numbered 8 and 12. It can sort them immediately. What it cannot conclude from those numbers is that request 7 is not still on its way. If the worker executes 8, then 12, and receives 7 afterward, inserting another row at the top of a log does not undo the actions already taken. A perfect comparator cannot tell you whether the input is complete.

The human version is closing an auction while bids are still arriving by post. Sorting the envelopes on your desk is easy. Knowing whether another valid envelope is stuck in transit is the hard part. If the rules permit a deadline based on arrival, you can use that deadline. If they promise to include every bid sent before a certain moment, you need a way to determine which bids qualify and when it is safe to stop waiting. The promise changes the problem.

This is where a neat timestamp exercise becomes a real protocol. Before applying an operation irreversibly, the system needs some basis for deciding that it is safe to proceed. That basis might involve a leader, acknowledgments, an agreed log, or assumptions about participants and message delivery. The right choice depends on failures you intend to survive. Sorting events you already have is a smaller problem than agreeing which events may be committed.

Imagine two booking offices selling seats on the same eight-seat boat trip. Each has a copy of the booking list. Two requests arrive: six seats for group A and four for group B. Each office accepts a request only if enough seats remain. If one processes A first, A succeeds and B fails. If the other processes B first, B succeeds and A fails. Both offices followed the same rule and saw the same requests. They disagree because they processed them in different orders.

Start them with the same list, give them the same sequence of requests, and require the same rule to produce the same answer every time. They will reach the same result. That is the idea behind a replicated state machine: several computers maintain copies of one evolving state by applying the same commands in the same order. The Raft paper describes a consensus protocol for maintaining such an agreed log while handling failures and changes of leader. Timestamps by themselves do not give us that agreement.

For the booking offices, agreement makes the allocation reproducible. Whether it is fair is another question. Should a smaller group get priority because it is easier to accommodate? Should a returning customer go first? Should an office hold seats while waiting for a payment? The software can make computers follow the same policy, but it cannot make the policy fair simply by executing it consistently. Agreement gives a decision a common place in the system’s history.

The same example works for a pool of eight GPUs, a warehouse with eight items left, or eight appointment slots in a clinic. Whenever one action changes what another action is allowed to do, order becomes part of the answer. That is why the idea reaches so much further than clock synchronization. A reliable shared service needs a way to decide which changes count and how they fit together, even when the machines responsible for it are having a bad day.

This is one reason I find the subject more interesting than a list of failure modes. It forces you to separate facts from decisions. An artifact’s parent version is a fact your system can record. Choosing which of two competing jobs gets capacity is a decision. An agreed sequence lets that decision become shared state. These are different pieces of the design, and they become much easier to discuss when a single word like latest is no longer standing in for all of them.

There is a price for providing that shared decision. If an office loses contact with everyone who could tell it whether the last seats are already taken, it needs a policy for what to do next. It might stop confirming bookings, accept a request provisionally, or sell only seats previously set aside for it. Each choice gives customers a different experience. The product cannot get an unconditional promise of fresh shared knowledge merely by displaying a confident green button.

When history branches

Sometimes preserving the branching history is more valuable than immediately picking one branch. The original Dynamo paper describes using vector clocks to distinguish versions descended from earlier versions from divergent versions requiring reconciliation. An application can then receive competing versions instead of having one silently erased. That is a different contract from promising one immediately authoritative value, and it gives the application a chance to decide what the divergence means.

A shared shopping list makes this easier to picture. You add coffee on your phone while someone else, offline, adds bread on theirs. Choosing whichever entire list has the later timestamp could discard a perfectly valid addition. In another case, one person deliberately replaces coffee with tea after seeing the coffee entry. Those are different histories. If we keep track of what each change was based on, the application has a chance to preserve independent additions while also recognizing intentional replacements.

Here is a deliberately small example using our author and reviewer. Give each event a pair of counters, with the author’s component first and the reviewer’s second. Start both processes at [0, 0]. Each increments its own component for an event. A received message merges the componentwise maxima before the receiver increments its own component. For this illustration, keep the participants fixed and account for every relevant event and message; creating and sending version 18 is one combined event.

You can read the pair as two bookmarks. The first says how far this history has progressed through the author’s events. The second says how far it has progressed through the reviewer’s. When a message arrives, it can move either bookmark forward by bringing news from the other side. Looking at both positions tells us something a single increasing number had hidden: two histories may each contain information missing from the other.

The important pair is [2, 0] and [1, 2]. The author has advanced to their second event but has no events from the reviewer in this history. The reviewer has reached their own second event but has only seen the author’s first. Each knows something the other does not. Neither pair is at least as far along in both positions, so the two histories branch. The larger author counter is evidence of a newer version; the larger reviewer counter is evidence of more review activity. Neither is evidence of a review of the newer version.

Compare [2, 0] with [2, 4] instead. The second history reaches at least as far in both positions and farther in one. The author’s creation of version 18 is now in the later approval’s causal past. These little pairs are called vector timestamps. Under the fixed-participant model and bookkeeping rules used here, comparing them lets us distinguish ancestry from concurrency.

The vector still does not tell us what the reviewer thought. It records the history represented by these events. We should continue to bind approval to the artifact identity. The useful gain is that the system can distinguish a history that incorporates the new version from one that branched before it. With scalar values, a larger number could hide either situation.

I would not respond to every application problem by putting a vector clock on every object. A fixed pair in a worked example is cheap. A system with changing participants, long histories, and many objects has to account for the size and lifetime of its metadata. Sometimes an explicit parent ID is enough. Sometimes a database transaction gives you the guarantee you need. The point of understanding the machinery is to choose what information the application must preserve.

The dashboard that counts two tools

Another place our expectation of a shared present causes trouble is a dashboard. Imagine a warehouse with exactly one special tool, moved between two rooms. A camera photographs room A before the tool leaves; another photographs room B after it arrives. Put those images side by side and the dashboard appears to show two tools. Use a picture of A after departure and B before arrival, and it shows none. Every photograph can be genuine while the combined picture gives the wrong count.

The computer version might track a token, a small piece of state representing the right to perform a job. Suppose that token moves between workers while a monitoring service asks each worker what it holds. Combining responses from different moments can produce the same apparent duplication or disappearance. If an automated repair now deletes a supposed duplicate, the misleading display can cause a real failure. The system has acted on a picture that never represented a coherent whole.

Chandy and Lamport’s 1985 paper on distributed snapshots addresses this problem. A snapshot records process states and messages in transit. Its consistency condition prevents recording a message as received without also recording its send. Their marker algorithm works under communication assumptions that include reliable channels preserving message order. It constructs a coherent cut through the execution while the computation continues.

For the warehouse, a coherent inventory might count the tool in A, in B, or being carried between them. That third possibility is easy to forget if our form has only two boxes. For the software, the token may be in a message travelling between workers. The channels are part of the system’s state. This connects directly to the network that can wait: information in transit can be durable, important state, even while no endpoint has a complete view of its journey.

“Cut” has a simple meaning here. Imagine stopping your reading of each worker’s history at some chosen point, then combining everything read so far. Those stopping points do not have to share the same clock reading. They do have to fit together. Including a response while excluding the request that produced it would make the combined account impossible. Including the request with the response still on its way can make perfect sense.

A fleet dashboard can still be useful without providing a consistent distributed snapshot. Most screens do not need that guarantee. But the interface should be honest about what its numbers mean. A collection of observations taken at different times is good for many operational questions. Treating it as an exact global inventory is an additional claim, and the implementation has to earn it.

Clocks are still useful

At this point it would be easy to conclude that physical clocks should be avoided. That would discard a tool we need. A five second timeout, a certificate expiry, and a schedule tied to sunrise concern physical time. Logical counters cannot tell you whether a patient has been waiting for ten minutes. They also cannot explain how long a GPU spent idle. We need clocks; we need to understand the guarantees attached to their readings.

The Spanner paper offers an illuminating example. Its TrueTime interface reports an interval of physical time with uncertainty, rather than pretending to know one exact instant. Spanner combines that information with replication and transaction protocols, including a commit wait that ensures a chosen timestamp is in the past before the transaction becomes visible under the described protocol. Together, these mechanisms support external consistency: the transaction order respects the real-time precedence of transactions that do not overlap.

In ordinary language, if your booking is confirmed before I even start trying to book, the system’s transaction history must put your completed booking before mine. That sounds like the most natural expectation in the world. Making it hold across separate computers requires work. If our attempts overlap, the system has room to choose an order; it still has to make that choice consistent with the rest of its guarantees.

For an invented numerical example, suppose one event’s time is bounded by [100, 104] milliseconds and another’s by [110, 114], both against the same reference. Those intervals establish a physical order. If the bounds were [100, 112] and [108, 120], they would overlap, and the bounds alone would not settle which came first. More decimal places in the displayed midpoint would not change that. This is what I want an API to tell me: enough about its uncertainty to decide what conclusions I can actually draw.

I find that much more satisfying than an API that returns an impressive number and leaves its uncertainty out of the documentation. Precision in presentation and certainty about the world are separate things. A system that exposes what it knows, including the limits, gives the rest of the application something solid to reason from. Sometimes waiting is the correct action because waiting allows an uncertainty interval to move safely behind us.

An intelligent system can still be out of the loop

I work on agent systems, so I keep returning to the approval example. Several agents can be competent and still act on incompatible histories. A planning agent revises a task while a tool worker executes the previous version. A human revokes permission while another process is offline. A summarizer describes a completed action before the corresponding result has reached the rest of the team. Making the models better at reasoning does not make the missing messages arrive sooner.

Consider a worker disconnected from the authority that issues permissions. It has a signed authorization permitting a particular action, but the human has just revoked it elsewhere. The worker’s signature check can establish who issued the authorization and whether it was altered. It cannot establish that no revocation has occurred since its last contact. That information is outside its current view.

Imagine telling a person to collect a parcel, then changing your mind after their phone loses service. Your cancellation is real, and so is their earlier instruction. They cannot make a decision using a message they have not received. Computers can make this feel less obvious because the interface updates immediately when you press Cancel. What updated immediately may be your local view. The person or process carrying out the instruction can still be operating from the previous one.

The product therefore needs a concrete rule. Perhaps sensitive actions require a fresh online check. Perhaps authorizations expire after a bounded interval, and the design accepts a corresponding exposure window. Perhaps the offline worker may prepare a result but cannot commit the external action. These choices affect availability and user expectations. A timestamp on the revocation record is not enough to cause the disconnected worker to know it.

There is also a race between checking and acting. A worker can ask whether it still has permission, receive “yes,” and then face a revocation before it performs the action. For a strict guarantee, the design must specify where authorization and execution become one enforceable decision, or what interval of validity the permission grants. Simply adding another check moves the question; it does not automatically settle it. This is why precise product promises are part of the engineering, even before the first network call is written.

This also changes how I would design an agent’s memory. A useful record should preserve the version of the task, the inputs available to that invocation, the approval it relied on, and the result returned by a tool. Then the question “why did it do that?” has something concrete behind it. An ordered transcript can be pleasant to read while still concealing the fact that the action was based on a branch the reader no longer sees.

Take a generated answer that uses an old price list. Its response may have been produced after a new list was published, but that does not tell us whether the invocation received the new list. If the result carries the document version it used, we can diagnose the problem directly. Otherwise we may spend an afternoon debating model quality when the system handed the model stale input. The dependency is the explanation we need.

In the article about Shannon, I was interested in the distinction between damage to a physical signal and failure to recover a message. Here, receiving the right bytes is only the beginning. The bytes may describe an older version, an approval for a different object, or an action on a concurrent branch. A message can survive its journey perfectly and still be unsafe to apply to the state in front of you.

That is why I would put version and dependency information close to the action itself. A button should make clear what object it approves. A worker should verify that reference at the point of execution. A result should identify the inputs from which it was produced. A retry should refer to the same intended operation so the receiving system has a chance to recognize it. These are fairly modest requirements individually. Together, they make a much more legible history than timestamps alone.

Many actions can proceed independently. Two people can edit unrelated documents without putting every keystroke into a global queue. A booking office can sell seats from an allocation it exclusively controls. An offline worker can prepare a draft that somebody will review later. The useful question is which relationships must be preserved for the promised behavior to hold. Our deployment needs approval for the exact version. Our boat needs a rule that prevents incompatible seat allocations. Coordination belongs where those promises depend on it.

What stays with me about Lamport’s paper is how a small change in the question opens up so much of the problem. Instead of treating a timestamp as an explanation, ask what information could have reached this event. Suddenly a reply, a booking, a database update, and an agent’s action become easier to reason about. Each belongs to a history. Some histories extend one another. Some branch. When we need them to meet, we need rules for what that meeting means.

Return to the deployment worker. It has received version 18 and an approval that names version 17. It can now do the boring, correct thing: keep the new version ready and wait for the approval it actually needs. Back in the flight conversation, sending the specific itinerary would have served the same purpose. We made the reference travel with the instruction. Nobody had to guess which version of the world the other person meant.

The interfaces we use are full of words like current, latest, approved, available, and cancelled. Each feels simple because our side of the interaction is simple. Somewhere behind the screen, messages are still travelling, copies are catching up, and decisions are being made from different histories. Good systems do the work that lets those words deserve our trust.

There is no shared present waiting inside the network to be read. We build a shared account from what its participants have seen, sent, received, and agreed to do. I think that is the beautiful part: a coherent experience can emerge from machines that never have the same view at the same instant. They need enough information to respect one another’s histories. So, quite often, do we.