What the Kernel Sees That Your Proxy Doesn't

Quick answer. A proxy sees the traffic you deliberately routed through it, decoded, after you changed something to put it in the path. The kernel's TC layer sees every segment that actually crossed an interface, including loopback, with nothing rerouted and no application reconfigured. The proxy knows more about each request it handles. The TC layer knows about the requests your proxy never saw.

I'm Necco, and I run yeet, a JS runtime for writing eBPF programs that thousands of engineers use to see what their Linux boxes are actually doing. I don't run your production traffic for a living. What I do see is where engineers go looking when the numbers stop agreeing with each other, and the same argument comes up almost every time: the application says one thing, the wire says another, and nobody can prove which one is lying.

Two things go wrong often enough to be worth naming. A service is slow, the access log looks clean, and the latency is real. Or an endpoint returns 500s and the log line insists the handler completed fine. Both are the same problem wearing different clothes: the access log records what the application believes it served, and it is written by the process you are currently debugging, which makes it a witness with an interest in the outcome. It is not lying, exactly. It is answering a narrower question than the one you asked, because it can only describe the part of the request that reached it and the part it chose to record. What you actually want is what the box put on the wire, and there are six places to stand to see that, each showing a genuinely different slice.

My service is slow but the access log looks clean. Where else can I look?

The access log is written by the process under suspicion, which is what makes it a bad witness for this particular question. It records what the handler believes it did, after the request reached the handler. Everything that happens before that (the accept queue, TLS termination, a proxy hop, the network itself) is invisible to it, and that is exactly the territory where "slow but the logs look fine" lives.

There are six realistic places to look instead, and they are not interchangeable. Each one sees a different slice of the request's life, demands a different amount of change to your system, and answers a different question well. Picking the wrong one is how an afternoon disappears into a tool that was never going to show you the thing you were hunting.

Where you look	Tools	Sees pre-handler time	Needs a code change	Needs a path change	Scope
Kernel TC layer	httpwatch, or your own probe on yeet	Yes	No	No	One host, decoded HTTP
Kubernetes eBPF	Pixie, Coroot, Cilium Hubble	Yes	No	No	Cluster, with storage
Packet capture	tcpdump, Wireshark	Yes	No	No	One host, packets
Proxy or mesh	Envoy, nginx, Istio, mitmproxy	Yes, for routed traffic	No	Yes, a proxy hop	What you routed through it
Instrumentation	OpenTelemetry, Datadog, Prometheus	Partly	Yes	No	Fleet, with retention
Application log	your own	No	Usually	No	One process

The TC-layer route: httpwatch, and writing your own with yeet

No code change, no path change, no cluster install, and nothing added to the request path. One host, plaintext HTTP only, decoded continuously and ranked by endpoint. httpwatch is the ready-made version: one docker run and every METHOD host path crossing the box is ranked live by count, req/s or p95, with the decoded headers and body one click away. Because it attaches to interfaces that already exist rather than inserting itself between anything, starting it and stopping it are both non-events. Nothing reconnects, no configuration reloads, and no traffic is at risk if the container dies while you are reading it.

Underneath it is yeet, a JS runtime for writing eBPF programs, which matters when the ready-made tool is not quite the shape of your problem. The same capture written differently becomes grpcsnoop for HTTP/2 and gRPC, or redissnoop for Redis commands. If your question is "which endpoint is slow," httpwatch answers it as-is. If your question is "which of my Redis keys is being scanned," that is a different probe rather than a different vendor.

What you give up: retention, a query language, and any view wider than this box. This answers "what is this machine doing right now" and nothing about last Tuesday. When the question outgrows one host or one moment, the routes below are what it grows into.

The instrumentation route: OpenTelemetry, Datadog, Prometheus

This is the correct long-term answer and the wrong immediate one. Code-based instrumentation gives you the richest data, tied to your spans and your business context, with retention and a query language behind it. Nothing here replaces that.

The catch is the loop. Add a middleware, export a histogram, wire up a scrape target or an exporter, deploy, wait for data. If the service is one you did not write, cannot rebuild today, or inherited from a team that left, that loop is measured in days. And it still measures from inside the process, so a request that spent 400ms in the accept queue looks fast to a handler-side histogram.

OpenTelemetry's zero-code instrumentation narrows this gap considerably for supported runtimes, attaching to a process without source changes and emitting spans that carry real application context. It is worth reaching for before concluding you need anything packet-level, and for a service you own and can restart it is often the correct stopping point. The gap it does not close is the one that matters at the TC layer: it still measures from inside the process, so a request that spent time in the accept queue or died before the runtime saw it is not in the data.

The proxy route: Envoy, nginx, a service mesh

Put something in the path and it can report on everything it handles, with full request and response detail, including HTTPS, because it terminates TLS itself and therefore holds the plaintext. That is a genuine capability advantage over anything reading the wire, and it is why mitmproxy remains the answer when you truly need to see encrypted payloads.

The costs come in two shapes. The first is a scope problem: a proxy only sees traffic you routed through it, which makes it circular for the question "is something reaching this box that I don't know about." If the traffic you are hunting bypassed the proxy, the proxy cannot tell you it exists, and that is exactly the failure mode when a client is hitting an old address or a health checker is hammering an endpoint nobody remembers configuring. The second is a risk problem: the proxy is now in the request path, so a proxy that fails takes production traffic down with it, and a proxy that is merely slow adds a hop to the very latency measurement you were trying to make honest.

A mesh is the right answer when you need enforcement anyway. If mTLS between services, retries, circuit breaking or traffic shifting are on your roadmap, you are installing it regardless and the observability arrives as a bonus. Installing one purely to answer a debugging question is a large, permanent change to your architecture in exchange for a look at something that a passive capture would have shown you in a minute.

The packet route: tcpdump and Wireshark

No code change, no path change, and it sees everything on the wire rather than one protocol's worth. tcpdump is strictly more powerful than anything else on this list and it is the right tool for TCP retransmits, zero-window stalls, handshake failures, malformed frames, or any protocol that is not HTTP. What it costs is the loop: write a filter, capture, stop the capture, move the file somewhere with a GUI, open it, discover the filter was slightly wrong, repeat. For a question like "which endpoint is slow," that loop is doing a lot of work the question never required, and each iteration costs minutes during which the condition you are chasing may have stopped happening. Compared side by side below.

The Kubernetes eBPF route: Pixie, Coroot, Cilium Hubble

These are the closest neighbors and the most honest comparison to draw, because they run on the same kernel machinery. Pixie uses eBPF to capture telemetry automatically, including full-body requests, with no manual instrumentation, and stores what it collects locally on the node. If you are on Kubernetes and want visibility that spans services rather than machines, it does things a single-host capture never will: correlate a slow request across the three pods it touched, keep enough history to compare this morning to yesterday, and give a team one place to look instead of asking each engineer to SSH somewhere. Coroot works a similar seam, gathering metrics, logs, traces and profiles automatically with eBPF and capturing requests without code changes, then building a service map and SLO tracking on top. It is aimed at continuous cluster monitoring rather than a session at a terminal. Both are answering "how is the system behaving," which is a different question from "what is this box doing right now."

Cilium Hubble is the one worth reading carefully, because its position is more subtle than the other two. Flow-level visibility comes free with the CNI and is genuinely passive: it observes connections as Cilium's datapath already handles them. Layer 7 visibility is a different matter, and Cilium's own documentation is explicit that it requires enabling L7 proxy support. So the HTTP-level detail that looks equivalent to a TC-layer decode is arriving through a proxy path, with the latency and failure-mode implications that carries. If you already run Cilium, that may be a trade you are happy to make for the integration. It is not the same architectural posture, and conflating the two is the most common mistake in this comparison.

What unites all three is that they are platform decisions rather than tools. You install them, you run them continuously, you learn their query surfaces, and they become part of how the cluster is observed and budgeted. For a platform team standardizing observability across dozens of services, that is exactly the right trade and a single-host capture would be an odd thing to standardize on. For an engineer who has already narrowed a problem to one node and wants an answer before the incident call ends, installing a cluster-wide platform is the wrong shape of effort entirely.

Can I see HTTP requests without adding a sidecar or changing my app?

Yes, and it is the reason this vantage point exists. Nothing is routed through the capture, no port is pointed at it, no application is reconfigured, and nothing is redeployed. TC is the kernel's traffic control layer, and an eBPF program attached there runs as segments move through it, on both tcx/ingress and tcx/egress. The program observes bytes rather than intercepting them, which is the distinction that separates this from every proxy-shaped answer: a proxy has to receive a request and forward it, so it is by definition a participant, while a TC-layer program is a spectator the traffic never learns about.

For HTTP/1.x that spectator position turns out to be enough, because the request genuinely is text on the wire:

• A request opens with a request line. RFC 9112 defines it as method SP request-target SP HTTP-version, so GET /path HTTP/1.1, then headers, then a blank line. The first bytes of the TCP payload are that line.
• The endpoint is METHOD host path. Method and path from the request line, host from the Host: header, or the absolute-form target on a proxied or CONNECT request.
• A response opens with a status line. Pairing responses to requests on a flow is what produces on-the-wire latency.

No decryption, no reassembly of a capture file afterwards, no agent inside the process. The parsing is possible because the bytes on the wire are the request, and that same property is what makes detection cheap enough to run in kernel context at all. Matching a leading ASCII method token is a comparison against the first few bytes of a payload rather than a parse of the whole message, so traffic that is not HTTP gets dropped before it ever reaches userspace. The cost scales with matched events rather than with total traffic, which is why watching a busy interface does not turn into a firehose of copies, and why the overhead question ("will this slow down my box") has a different answer here than it does for a packet capture that copies everything and filters afterwards.

The method list is finite, which is what makes the match possible. RFC 9110 defines eight methods (GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE), and PATCH comes from RFC 5789 for nine in total. A request using some other extension method will not be recognized, which is rare in practice and worth knowing when a custom protocol borrows HTTP's shape.

How do I see traffic between two services on the same host?

Loopback crosses the TC layer like any other interface, and that single fact is the sharpest practical difference between this vantage point and the alternatives. Two services on one host talking over 127.0.0.1 are completely invisible to a network tap, which is watching a physical interface that traffic never reaches. They are half-visible to a sidecar, which sees the leg routed through it and has no idea the other leg exists. They are invisible to a switch-level capture for the same reason a tap misses them: the packets never leave the machine. At the TC layer, watching lo captures both sides of the conversation without instrumenting either service, without either process knowing, and without a single configuration change to the thing you are debugging.

This is the shape of a lot of real infrastructure: an app and its cache, a service and its local sidecar, a reverse proxy and the thing behind it. If your east-west traffic never leaves the box, watching lo is how you see it at all.

I turned it on and the dashboard is empty. What is it missing?

Four causes account for almost all of it. Work down them in this order, because the cheap checks are also the likeliest:

1. There was never any plaintext traffic on that interface. An empty dashboard and a broken one look identical, so generate some known traffic before concluding anything. On a quiet box this is the most common answer by a wide margin.
2. The traffic is HTTPS. There is nothing to parse at this layer, only ciphertext, and no flag changes that.
3. The traffic is HTTP/2 or h2c. An ASCII method match does not decode binary framing, so the table stays empty rather than reporting an error.
4. The probe never attached. That means a kernel older than 6.6, and it announces itself as tcx: -EINVAL in the container logs.

The limits behind those symptoms are worth understanding before you pick this vantage point at all, because two of them are permanent properties rather than configuration mistakes.

• HTTPS is invisible, permanently. TLS encrypts the payload before it reaches the wire, so at this layer there is no request line to parse, only ciphertext. This is a property of where the capture sits rather than a missing feature or a flag someone forgot to add. Reading TLS traffic requires a uprobe on SSL_write and SSL_read, which is a different mechanism and a different tool, and a proxy like mitmproxy can decrypt precisely because it terminates TLS in the path.
• HTTP/2 and HTTP/3 do not appear, also permanently. Detection matches a leading ASCII method token (GET, PUT, HEAD, POST, TRACE, PATCH, DELETE, OPTIONS, CONNECT) or a HTTP/ status line, as described above. Both newer versions are binary and compress their headers with HPACK, so cleartext h2c and prior-knowledge gRPC never match the pattern. If your internal services speak h2c, expect an empty table rather than an error message telling you why. For gRPC specifically, grpcsnoop handles the binary framing.
• Latency is on-the-wire, not server-internal. The measurement is the time between request and response segments as seen at this host's TC layer, so for a remote caller it includes network RTT. That is what the client experienced, which is usually the more honest number, but it is not what the handler spent. Responses pair to requests FIFO per flow, which is correct for ordered HTTP/1.x and approximate under pipelining. If it disagrees with your application's histogram, the gap is the interesting part.
• Counts are a close lower bound, not an exact tally. Under heavy load or on a slow link some segments are not captured, so treat the numbers as a floor. The aggregates are diffed from cumulative tallies rather than counted from the request stream, which is why the endpoint table stays trustworthy even when individual rows get dropped.
• It cannot intervene, by design. This is observability, not enforcement: it tells you what crossed the wire and does not stop, hold, modify or delay anything. If the answer to your question has to change behavior rather than just describe it, you need something in the path and this is the wrong category of tool.
• Unix domain sockets are outside its view. Traffic over a .sock path never crosses a network interface, so there are no segments at the TC layer to observe at all. Traffic on 127.0.0.1 is visible; traffic to /var/run/app.sock is not, which catches people fronting a service with nginx over a socket.

How do I get error rate and p95 for a route nobody instrumented?

Three of the four golden signals fall out of the capture with no client library, no exporter and no code change:

• Rate comes from the per-second request count per endpoint, which also makes a spike visible before it accumulates enough volume to move a total. That is the shape a retry storm has in its first thirty seconds.
• Errors come from the per-class status tally, so a route that starts returning 503s shows up as a change in the mix rather than as a number you have to compare against a remembered baseline.
• Latency comes from pairing each response to its request on the wire, giving p50, p95 and max per endpoint. Because it is measured at the interface, it is what the caller experienced rather than what the handler self-reported.

The fourth signal, saturation, is not there and cannot be. This measures traffic rather than the resources serving it, so CPU pressure, memory headroom, connection-pool exhaustion and queue depth are all invisible. If you need those numbers next to the request numbers, and for a real diagnosis you usually do, they stay with your existing metrics stack. What you get here is the request-side three, available immediately, for a route that has never been instrumented and might never be.

The reason this matters for a route nobody instrumented is that the usual path to those numbers runs through the application: add a middleware, export a histogram, wire it to a scrape endpoint, redeploy. Every step is a change to the thing you are trying to measure. Reading them off the wire skips all of it, which is why it works on a service you did not write, cannot rebuild, or inherited.

There is a real cost to reading them this way. These numbers have no retention. They describe what this box is doing now, not what it did last Tuesday, and there is no query language to ask historical questions with. For a route that has been quietly failing for a week, a metrics stack is the right tool. For a route that is failing right now on a box you are logged into, this is faster than instrumenting it.

Does this work inside containers and on Kubernetes nodes?

On a single host, yes. The probe attaches to the host's interfaces, so with host networking it observes traffic for every container on that box, including the loopback chatter between them that container-level tooling usually misses. A pod talking to a sidecar, a service calling a local cache, an init container phoning home during startup: all of it crosses an interface the probe is watching, and none of it requires touching the container images or the pod spec.

The important limit is scope. This is one instance per host with no aggregation layer, so it answers questions about a node, not about a cluster. There is no service map across your fleet, no cross-node correlation, and no single pane covering every pod. Those are real needs and they belong to Pixie, Coroot or an APM.

The practical read: it fits when you have narrowed a problem to a node and want to see what that node is actually doing. It does not fit as the fleet-wide observability layer, and a platform team standardizing on one tool for every service should not expect it to be that.

Two container-specific caveats matter more than they look:

• Cross-node traffic is only half-visible from one node. Pods on different machines talk over the network, so each end sees its own side of the conversation. Watching both nodes gets you both halves, but nothing correlates them for you the way a cluster-scoped tool would.
• Service-mesh traffic is usually mTLS, which at this layer is ciphertext like any other TLS. Ironically, the more thoroughly a platform team has secured east-west traffic, the less of it a passive HTTP decode can read.

How is this different from just running tcpdump?

Both read the same bytes off the same interfaces, and for a while the comparison looks like a tie. What differs is everything that happens after the bytes are read, and that difference decides which tool answers your question in two minutes and which one answers it in forty.

	tcpdump + Wireshark	TC-layer HTTP capture
Output	Packets, reassembled later by you	Decoded requests, aggregated continuously
Workflow	Capture to a file, open it afterwards	Live, already grouped by endpoint
Scope	Every protocol, full packet detail	Plaintext HTTP/1.x only
Reading it	A desktop, or a file copied to one	A browser over the network
Ranking	You write the filter	Sorted by count, rate or p95

The row that matters most is the last one. tcpdump answers "show me packets matching this filter," which means you have to already suspect something to write the filter. Continuous decoding answers "which endpoint is worst," which is the question you actually have before you suspect anything. Getting from the first to the second means capturing, transferring, opening and sorting, by hand, every time.

The reverse cuts just as hard. tcpdump sees TCP retransmits, zero-window stalls, malformed frames, handshake failures and every non-HTTP protocol. HTTP decoding at the TC layer throws all of that away. When the problem is below HTTP, the ranking is useless and the packets are everything.

When should I use a service mesh or a proxy instead?

Reach for a proxy or a mesh when the observation is not the goal:

• You need to change the traffic: retries, timeouts, traffic shifting, header injection.
• You need mTLS between services.
• You need HTTPS payloads and you accept terminating TLS to get them. This is mitmproxy's actual trade: it can decrypt because it is a proxy in the path.
• You need cluster-wide service maps, retention and fleet aggregation. That is Pixie, Coroot or an APM. A single-host capture answers "what is this box doing right now," not "what happened last Tuesday."

Reach for tcpdump and Wireshark when you need non-HTTP protocols or packet-level detail, compared side by side above. TC-layer HTTP capture deliberately throws that detail away in exchange for decoding continuously and not producing a capture file to open afterwards.

Put plainly: the TC layer is the cheapest way to see plaintext HTTP for a whole box, and it becomes the wrong tool the moment you need encryption, enforcement, or a view across more than one machine. Those are not edge cases, and a tool that pretended otherwise would be lying to you about which afternoon it was going to save.

How do I actually run this on a box right now?

The capture described here is what httpwatch does. It attaches at tcx/ingress and tcx/egress, ranks every METHOD host path by traffic, and serves a live dashboard you can click into for decoded headers and bodies.

docker run --rm -it \
  --cap-add SYS_ADMIN --cap-add NET_ADMIN --cap-add BPF --cap-add PERFMON \
  --security-opt apparmor=unconfined \
  --pid=host --network=host \
  -v /sys/kernel/btf/vmlinux:/sys/kernel/btf/vmlinux:ro \
  ghcr.io/yeet-src/httpwatch:latest      # → http://localhost:8080

It needs kernel 6.6+ with BTF and TCX, which is the default on current Fedora, Arch, Ubuntu and Debian 12+. On a quiet box, start some traffic before judging it: an empty dashboard and a broken one look identical.

The bottom line: pick the vantage point that matches the question

If you need to know what your application believes, read its log. If you need retention and a fleet view, instrument with OpenTelemetry and send it somewhere that stores it. If you need to change traffic or read HTTPS, put Envoy or mitmproxy in the path and accept the cost. If you are on Kubernetes and want this continuously across a cluster, Pixie or Coroot is the shape that fits. If the problem is below HTTP, it is a tcpdump problem.

If you need to know what actually crossed the wire on one box right now, including the loopback traffic nothing else sees, capture at the TC layer with httpwatch and accept that it is plaintext-only and cannot intervene.

The failure mode worth avoiding is reaching for a sidecar because it is the familiar answer, and then discovering it sees one leg of the conversation you were trying to debug.

Frequently asked questions

Does capturing at the TC layer slow down my traffic?

The eBPF programs observe segments as the kernel moves them. Traffic is copied rather than held, modified or redirected, so nothing is routed through the capture and no application is reconfigured. A service does not know it is being watched. This is a different posture from a proxy, which sits in the request path and can add latency or fail closed.

Can the TC layer see traffic between two containers on the same host?

Yes, if that traffic crosses an interface the probe is watching. Loopback crosses the TC layer like any other interface, so east-west traffic on a single box is captured without instrumenting either side. This is the traffic a sidecar proxy sees only half of and a network tap misses entirely.

Why can't the TC layer see HTTPS?

TLS encrypts the payload before it reaches the wire, so at this layer there is no request line to parse. That is a property of where the capture sits, not a flag that can be turned on. Reading it would require a uprobe on SSL_write and SSL_read, which is a different tool.

What kernel version do I need for TCX?

Kernel 6.6 or newer, with BTF. That is the default on current Fedora, Arch, Ubuntu and Debian 12 and later. If the kernel is older, a TCX attach fails with tcx: -EINVAL rather than silently capturing nothing.

Is TC-layer capture a replacement for a service mesh?

No. A mesh enforces: it does mTLS, retries, traffic shifting and policy. TC-layer capture is observability only. It tells you what crossed the wire and does not stop, hold or modify anything. The two answer different questions, and the mesh is the one you need if the answer must change behavior.

Why does the latency here disagree with my application's own metrics?

They measure different spans. On-the-wire latency is the time between request and response segments at this host's TC layer, so for a remote caller it includes network RTT. Your application's histogram measures what the handler spent inside the process. A gap between them is not an error in either; it is the network, the accept queue, and everything else between the wire and your code. That gap is often the most useful number of the two.

Can it see traffic if my service listens on a Unix socket?

No. A Unix domain socket never crosses a network interface, so there are no segments at the TC layer to observe. This catches people who front a service with nginx over a Unix socket rather than a TCP port on loopback. Traffic on 127.0.0.1 is visible; traffic over /var/run/app.sock is not.

Does the endpoint table stay accurate if requests get dropped?

Yes, and the two are tracked separately on purpose. The endpoint aggregates are diffed from cumulative tallies that never lose a response, while individual rows in the live request stream are subject to a per-frame budget. If a number in the table and a count in the stream disagree, trust the table.

What happens to the numbers when I change which interfaces I watch?

Every counter resets, because changing the watched interfaces restarts the probe. Capture settings are spawn-time arguments and there is no control channel into a running isolate, so a change means a restart and a restart means new counters. Held response bodies are invalidated at the same time.

Sources

• RFC 9112, "HTTP/1.1" (IETF, June 2022) — the request-line grammar method SP request-target SP HTTP-version that makes an HTTP/1.x request parseable from the first bytes of a TCP payload.
• RFC 9110, "HTTP Semantics" (IETF, June 2022) — the eight methods defined in section 9.3: GET, HEAD, POST, PUT, DELETE, CONNECT, OPTIONS, TRACE.
• RFC 5789, "PATCH Method for HTTP" (IETF, March 2010) — PATCH, the ninth method matched by the detection described here.
• RFC 9113, "HTTP/2" (IETF, June 2022) — the binary framing layer and HPACK header compression that make HTTP/2 and cleartext h2c invisible to an ASCII method match.
• RFC 8446, "TLS 1.3" (IETF, August 2018) — the record protocol that encrypts application payloads before they reach the wire, which is why HTTPS carries no readable request line at the TC layer.
• Linux kernel BPF documentation, program types — the tcx/ingress and tcx/egress attach points, corresponding to BPF_TCX_INGRESS and BPF_TCX_EGRESS.
• Pixie documentation, "What is Pixie" — eBPF-based automatic telemetry capture for Kubernetes applications without manual instrumentation, including full-body requests, with data stored locally on the node.
• Cilium documentation, "Layer 7 Protocol Visibility" — L7 visibility requires enabling L7 proxy support, which is why HTTP-level detail in Hubble comes from a proxy path rather than passive observation.
• Coroot documentation and its README — metrics, logs, traces and profiles gathered automatically via eBPF, request capture without code changes, service map and SLO tracking.
• OpenTelemetry documentation, "Instrumentation" — the distinction between zero-code and code-based instrumentation, and that code-based gives deeper telemetry from the application itself.
• httpwatch README (yeet-src) — the nine-method detection list, the on-the-wire latency and FIFO response-pairing behavior, the counts-as-lower-bound caveat, and the kernel 6.6+ with BTF requirement.

Related resources

• httpwatch on GitHub — the eBPF HTTP traffic inspector this post describes, and its full caveats section.
• grpcsnoop — HTTP/2, cleartext h2c and gRPC, which the ASCII method match here deliberately misses.
• yeet docs — the JS runtime for writing the eBPF programs described here.