High Performance Git

Section IV · Large-Repo Operations, Transport, and Scale

Chapter 15

Ref Negotiation and the Want/Have Conversation

Pencil sketch of a fishing boat returning to a harbor village, with crew on deck and rooftops including a church steeple across the water.

A clone and a fetch use the same plumbing. A clone is the trivial case where the client has nothing yet. A fetch is the same protocol running with state on both sides. Part of what makes fetch fast or slow lives in the structure of that conversation.

This chapter walks through the exchange in order: ref advertisement, wants, haves, ACKs, the chosen negotiation algorithm, and the pack that comes back. Then it covers the performance layers the conversation actually exercises, and how to read a packet trace to tell them apart.

A Clone Is a Fetch With Nothing On Your Side

When a clone runs against an empty local directory, the client says "I have no objects yet — give me everything reachable from these tips." When a fetch runs against an existing repository, the client says "I already have these objects — give me only what I'm missing for these tips." It uses the same wire format and the same server-side reachability work; only the ratio of wanted to already-have changes.

This, of course, is why a fetch can be very small while a clone usually cannot. A clone has nothing to subtract, so most of the reachable object set has to come down the wire. A fetch only has to transfer what is genuinely new, but only if the negotiation step succeeds at proving how much is already on the client.

The two limit cases:

Everything else (shallow clone, partial clone, mirror fetch, sparse working tree) sits between those extremes.

The Conversation Opens with Refs

Before any want/have exchange begins, the server tells the client which refs it has. In protocol v1, that meant dumping every advertised ref as the opening packet of the connection. In protocol v2, that step is its own command: ls-refs. Either way, ref advertisement is the first cost on the wire.

GIT_PROTOCOL=version=2 git ls-remote --heads --tags origin | wc -l

That line counts how many refs your server advertises to you. In a small repository, the answer is small enough that ref advertisement is unmeasurable. In a repository with branch-per-PR automation or aggressive tag policies, the answer can run into tens or hundreds of thousands, and the advertisement step alone can take seconds. Chapter 17 covers the ref backend side; the negotiation side is that every fetch begins with that listing arriving over the wire.

Protocol v2's separation has a practical effect. In v1, every connection paid for a full ref dump even if the client only wanted one branch. In v2, ls-refs accepts its own arguments, so a client that only cares about main can scope the listing:

GIT_PROTOCOL=version=2 git ls-remote --heads origin main

That is the same command shape protocol v2 uses internally during fetch. The advertisement is now scoped, and large-ref repositories no longer pay a full-dump cost for a narrow operation.

Wants: What the Client Is Asking For

Once names are resolved, the client picks a set of wants. A want is a commit object ID, not a ref name. By the time wants leave the client, names have been resolved into commit IDs; the server is asked "please make sure I can reach commit X," not "please update my idea of main." For a clone without filters, the wants list is every advertised ref's tip. For a fetch with a configured refspec, it is the tips of the configured refs. For a fetch with an explicit refspec on the command line, it is whatever you named.

You can see wants in a packet trace:

GIT_TRACE_PACKET=/tmp/fetch.packet git fetch origin
grep -E "want " /tmp/fetch.packet | head

Each want <40-hex-commit-id> line is one tip the client is asking the server to make reachable. A clone of a small repository might emit one or two wants. A fresh mirror clone of a busy monorepo can emit thousands.

Haves: Where Negotiation Actually Happens

After sending wants, the client walks its own commit graph and offers candidate object IDs as haves. The server checks each one against the reachable set the wants imply, and ACKs the ones it recognizes as common with what it would otherwise send. The conversation continues until the server has enough common ancestry to compute a minimal pack, or until the client runs out of haves to offer.

A simplified single round, with commit IDs abbreviated for readability:

client → server: want a1b2c3...   (the new tip you want)
client → server: want d4e5f6...   (another tip you want)
client → server: have 7a8b9c...   (try this commit)
client → server: have 0d1e2f...   (try this one)
client → server: have 3a4b5c...   (and this)
server → client: ACK 7a8b9c... common
client → server: have 8d9e0f...
client → server: have 1a2b3c...
server → client: ACK 1a2b3c... common
client → server: done
server → client: packfile (commits, trees, blobs needed
                          to satisfy wants given the commons)

The number of round trips depends on how lucky the client gets when picking haves, and on which negotiation algorithm it uses. Two extremes bound it:

Most real fetches sit between those, with two to five round trips before done. The latency of each round trip is set by your network, not by Git, which is why a slow link makes fetches feel disproportionately slow even when the eventual pack is tiny. A fetch over a 200ms-RTT link spending five rounds in negotiation has already paid a full second before any pack data moves.

A small-repository trace

Here is the want/have/ACK section of an actual fetch against a small repository, with non-essential lines removed and commit IDs abbreviated to 8 characters:

packet:        fetch> want a1b2c3d4
packet:        fetch> have 9d8c7b6a
packet:        fetch> done
packet:        fetch< ACK 9d8c7b6a common
packet:        fetch< packfile

One want, one have, one ACK. The server's history was shallow enough relative to the client's that the first have was recognized as common, so a single round was enough.

A deeper-repository trace

Same exchange against a repository with ~50,000 commits of history that the client is several months behind. Same protocol, same algorithm (consecutive):

packet:        fetch> want a1b2c3d4
packet:        fetch> have 9d8c7b6a
packet:        fetch> have 5e4f3a2b
packet:        fetch> have 1c2d3e4f
packet:        fetch> have 6a7b8c9d
packet:        fetch> have 2f3e4d5c
packet:        fetch> have 0b1a2c3d
packet:        fetch> have 4e5f6a7b
packet:        fetch> have 8c9d0e1f
packet:        fetch< ACK 8c9d0e1f common
packet:        fetch> have 5a6b7c8d
packet:        fetch> have 9e0f1a2b
packet:        fetch> have 3c4d5e6f
packet:        fetch< ACK 3c4d5e6f common
packet:        fetch> done
packet:        fetch< packfile

Three round trips. The client offered eight haves before the server recognized one, then three more haves before the second ACK pinned down a tighter common ancestor. On a 50ms link, that is roughly 150ms of negotiation latency before any pack work begins.

A large-ref-set trace

Same client, same network, but the remote has 30,000 advertised refs (a repository with one branch per open PR plus several years of automation tags). Just the opening section:

packet:        fetch< version 2
packet:        fetch< ls-refs=ok
packet:        fetch< fetch=ok
packet:        fetch< object-format=sha1
packet:        fetch> command=ls-refs
packet:        fetch> ref-prefix refs/heads/
packet:        fetch> ref-prefix refs/tags/
packet:        fetch< abc1230 refs/heads/branch-00001
packet:        fetch< abc1231 refs/heads/branch-00002
... thousands of lines elided ...
packet:        fetch< abcfffe refs/tags/v1.99.998
packet:        fetch< abcffff refs/tags/v1.99.999
packet:        fetch> command=fetch
packet:        fetch> want a1b2c3d4

The ls-refs response alone is many kilobytes of data and several hundred milliseconds of server work before negotiation proper starts. Protocol v2 makes scoped advertisement possible (a client that only needs main can ask for ref-prefix refs/heads/main), but a fetch with the default refspecs walks the whole heads-and-tags set.

Negotiation Algorithms Decide Which Haves to Try

Git ships three negotiation algorithms: consecutive, skipping, and noop.

consecutive walks the client's commit graph in chronological order, offering each commit as a have. It is thorough — if there is a common commit, it will eventually find it — but on deep history it takes many round trips before reaching far enough back to discover one.

skipping uses a skip-list pattern: offer a have, skip past several commits, offer the next batch, skip further. It reaches deeper into history in fewer round trips. The cost is that it can declare common ancestry slightly later than consecutive would, which means the server may send a few extra commits in the pack that the client already had. In practice the extra pack data is small, and the round-trip savings dominate on any non-trivial latency. Set it globally with:

git config --global fetch.negotiationAlgorithm skipping

noop sends no haves at all. The client tells the server "treat me as if I have nothing" and the server replies with a full pack reachable from the wants. Useful when negotiation cost itself is the bottleneck (heavily filtered partial clones where the haves walk is mostly overhead) or when a script wants deterministic fetch behavior. It is rarely the right default for developer machines, because it sends data the client already has.

Choose the algorithm as a function of network and history depth:

You can test what difference an algorithm makes for one fetch without changing global config:

GIT_TRACE2_PERF=/tmp/consecutive.perf \
  git -c fetch.negotiationAlgorithm=consecutive \
  fetch --negotiate-only --negotiation-tip=HEAD origin

GIT_TRACE2_PERF=/tmp/skipping.perf \
  git -c fetch.negotiationAlgorithm=skipping \
  fetch --negotiate-only --negotiation-tip=HEAD origin

--negotiate-only stops after negotiation completes — no pack is built or transferred — so the Trace2 timings isolate the negotiation cost.

Where the Time Actually Goes

A single fetch is six cost layers:

  1. ref advertisement — server sends names. Scales with ref count.
  2. want/have round trips — proportional to history depth and chosen algorithm.
  3. server-side reachability — server computes which objects to send given wants and ACKed haves. Bitmaps make this cheap; their absence makes it linear in reachable objects.
  4. pack generation — server builds the pack, reusing existing pack data where it can, computing deltas where it cannot.
  5. transfer — bytes on the wire.
  6. client-side unpack and ref update — client integrates the pack into its object store and updates refs.

Any of those can dominate, and each has its own remedy:

Shallow and Partial Clone

Shallow clone and partial clone are both negotiation modifiers, but they touch different parts of the exchange.

Shallow clone sets a depth boundary. The client tells the server "do not consider anything older than this depth as a have, even if I technically have it." Negotiation gets cheaper because the haves walk stops at the shallow line, and the pack gets smaller because the server stops walking history at the same boundary. The cost: shallow boundaries have to be re-established on every fetch, and operations that need history past the boundary (blame, deep log, certain merge bases) require deepening or unshallowing.

Partial clone does not change the want/have conversation directly. It applies an object filter to the resulting pack: send the commit and tree objects, but omit blob objects that match the filter (commonly --filter=blob:none).

That distinction shows up in packet traces. A shallow fetch's haves list is short because the shallow boundary clipped the walk. A partial fetch's haves list looks normal; the savings appear later, in pack size and in the absence of blob entries.

Narrowing the Frontier with --negotiation-tip

A clone of a busy repository tends to leave thousands of remote-tracking refs in refs/remotes/origin/. When the next fetch runs, the default haves walk starts from all of those, which makes the negotiation step proportional to the total ref count even when the user only cares about advancing one branch.

--negotiation-tip=<ref> narrows that walk:

git fetch --negotiation-tip=HEAD origin
git fetch --negotiation-tip=refs/heads/main origin main

The client now uses only the named tip as the haves frontier. On a fleet of CI runners that fetch one branch at a time against a heavily-branched origin, this can turn a multi-second negotiation step into a single-round exchange. It is useful for branch-per-task agents and merge-queue automation, where the surrounding repository has many branches the local worker does not care about.

--negotiate-only takes that further: run the negotiation step, print the ACKed commons, and stop without transferring any pack:

GIT_TRACE2_PERF=/tmp/neg.perf \
  git fetch --negotiate-only --negotiation-tip=HEAD origin

Server-Side: Bitmaps and Pack Reuse Decide Layers 3 and 4

The negotiation conversation as seen from the client looks the same regardless of what the server has to do internally. Layers 3 and 4 — server-side reachability and pack generation — are where the same fetch can take 200ms or 20 seconds depending on server state. When the server has a current reachability bitmap (chapter 8), it can answer "which objects are reachable from wants but not from ACKed haves" by intersecting two bitmaps. That is microseconds. Without a bitmap, the server walks the commit graph and tree objects to compute the set, which is linear in reachable objects.

Pack reuse is similar. A server with one large pack and a bitmap can copy pack data directly into the response. A server with many small packs and no MIDX has to delta-compress more aggressively at request time, which costs CPU and time.

Remember that client-side negotiation flags cannot fix server-side problems. Setting fetch.negotiationAlgorithm=skipping on every developer machine does nothing if the bottleneck is the server walking 10M objects without a bitmap.

Reading a Packet Trace End-to-End

Set GIT_TRACE_PACKET to a writable path before a fetch:

GIT_TRACE_PACKET=/tmp/fetch.packet git fetch origin main

The trace records every wire packet in both directions. The structure to look for, in order:

  1. capability exchange (protocol version, available commands)
  2. ls-refs request and response (protocol v2) or full ref advertisement (v1)
  3. want <oid> lines from the client
  4. have <oid> lines from the client
  5. ACK <oid> common lines from the server, possibly several
  6. done from the client
  7. packfile from the server, then pack bytes

To see just the want/have/ACK section:

grep -E "want |have |ACK |done" /tmp/fetch.packet

To see the ref advertisement size at a glance:

grep -c "refs/" /tmp/fetch.packet

To time the negotiation alone, pair the packet trace with a Trace2 region trace:

GIT_TRACE_PACKET=/tmp/fetch.packet \
GIT_TRACE2_PERF=/tmp/fetch.perf \
  git fetch origin main

Trace2 emits region markers around each major phase. The region_enter/region_leave pairs for negotiation separate that cost from the receive-and-unpack cost. If the negotiation region is short and the unpack region is long, the conversation was cheap and the pack was big.

A Decision Matrix for Slow Fetch

Map symptoms to the layer that is actually hot: