
Fifteen years building engineering platforms, currently focused on advanced AI infrastructure at yeet. I love turning the deeply complex topics into something everyone can understand. I relate deeply with the core yeet philosophy that you can just build things.
Quick answer. To see HTTP traffic on a Linux host without touching the application, capture it in the kernel.
tcpdump -i any -A 'tcp port 80'gives you raw packets in seconds, and an eBPF tool like httpwatch decodes and ranks the same traffic by endpoint in onedocker run. Both see requests your access log never records, because a log can only describe what reached a handler, and anything that failed in the accept queue, at TLS termination, or at a proxy hop never becomes a line in it. A clean access log is not evidence that nothing went wrong.
I'm Necco, and I run yeet, a JS runtime for writing eBPF programs that thousands of engineers use to see what their Linux boxes are actually doing. I have never carried your pager and I do not operate your fleet. What I do watch is which tool people reach for first when they need to see traffic on a box, and then which one they are still running a week later. Those are frequently not the same tool, and the gap is almost never about features.
The gap is about what each option asks from you before it will show you anything. One wants a restart. One wants a proxy in the request path. One wants an agent on every host, forever. On a machine that is currently serving traffic, those are not equivalent requests, and picking without knowing which is which is how an afternoon disappears into a tool that was never going to answer the question. Six routes are worth knowing, and the choice between them is usually decided by constraints rather than capability.
An access log looks clean during an incident because it is written by the process under suspicion, and it only starts recording once a request reaches the handler. Everything before that point is outside its vision. A connection that was refused at the accept queue never becomes a log line. A TLS handshake that failed never becomes a log line. A request that a proxy rejected on its way through never reaches your process at all. From inside the application, none of those events happened, and the log is a faithful record of a world in which they did not.
There is a second and more uncomfortable version of this. The log is a record of what the application believes it did, written by the same code whose behavior is in question. If a handler thinks it returned a 200 and the response was truncated on the way out, the log says 200. If a request was retried three times by a client and each retry succeeded, the log shows three healthy requests and no sign of a problem, while the user experienced one slow interaction that failed twice. The log is not lying. It is answering a narrower question than the one you asked.
The practical consequence is that "the access log looks clean" should be read as a data point rather than a conclusion. It rules out one class of failure, which is the handler receiving a request and mishandling it. It says nothing about the larger class where the request never got that far, and that class is where deploys, config changes and capacity problems tend to live.
Seeing that second class means reading traffic somewhere the application is not, which is what the rest of this post is about: the six places you can stand on a Linux host, what each one costs you, and the commands to run.
Run ss -lntp to list every listening TCP socket on a Linux host along with the process holding it. This is the step people skip before setting up any capture, and skipping it is how an afternoon goes into monitoring a service that turns out not to be the one serving the traffic.
ss -lntp # every listening TCP socket, with the process holding it
ss -lntp | grep -v 127.0.0 # only the ones reachable from off the box
ss ships with iproute2 and is present on essentially every modern Linux system. The -l flag limits it to listening sockets, -n keeps ports numeric so nothing is helpfully renamed, -t restricts it to TCP, and -p names the process, which is the column you actually came for. Run it without sudo and the process column is blank for anything you do not own, which is the usual reason people think it is broken.
Two things in that output decide everything downstream. A socket bound to 127.0.0.1 is reachable only from the box itself, so traffic to it never crosses an external interface and any capture watching eth0 alone will miss it entirely. And a service you expected to find, missing from the list, usually means it is behind a Unix domain socket rather than a TCP port, which changes what can observe it at all.
lsof -i -P -n gives the same picture with more detail per connection, including established sessions rather than just listeners, at the cost of being slower and not installed by default everywhere. For a quick inventory, ss is the one to reach for.
Yes, and there are six ways to do it, only some of which leave your application untouched: an eBPF capture at the kernel's TC layer such as httpwatch, tcpdump, a Coroot node agent, Pixie or Cilium Hubble on Kubernetes, a proxy like Envoy or mitmproxy, and instrumentation such as OpenTelemetry. The first three require nothing from the application at all. The choice matters more than it looks, because each sees a different slice of the request's life and each asks for a different amount of change to a system you may not want to change during an incident.
| Tool | How it sees traffic | Sees requests your app never got | Needs a code change | Needs a path change | Scope |
|---|---|---|---|---|---|
| httpwatch, or your own probe on yeet | eBPF at the kernel's TC layer | Yes | No | No | One host, decoded HTTP |
tcpdump, Wireshark | Packet capture | Yes | No | No | One host, packets |
| Coroot | eBPF node agent | Yes | No | No | Host or cluster, with storage |
| Pixie, Cilium Hubble | eBPF in a Kubernetes cluster | Yes | No | Sometimes | Cluster, with storage |
| Envoy, nginx, Istio, mitmproxy | Proxy in the request path | Only what you routed through it | No | Yes | What you routed through it |
| OpenTelemetry, Datadog, Prometheus | Library or agent inside the process | Partly | Yes | No | Fleet, with retention |
The TC layer is Linux traffic control, the point in the kernel where packets pass on their way to and from a network interface. It is one of several places an eBPF program can attach, and it is the one that matters for HTTP, because segments cross it after the kernel has assembled them and before any application has seen them. When a tool is described as eBPF-based network monitoring, this is usually where it sits. Everything else in this section follows from that position.
The fastest of the six routes to go from nothing to a decoded request, and the only one that leaves no trace when you stop. No code change, no path change, and nothing added to the request path. One host, plaintext HTTP only, decoded continuously and ranked by endpoint.
httpwatch is the ready-made version. One docker run on the box, open a browser tab, and every METHOD host path crossing the host is ranked live by count, requests per second or p95 latency, with the decoded headers and body one click away.
docker run --rm -it \
--cap-add SYS_ADMIN --cap-add NET_ADMIN --cap-add BPF --cap-add PERFMON \
--security-opt apparmor=unconfined \
--pid=host --network=host \
-v /sys/kernel/btf/vmlinux:/sys/kernel/btf/vmlinux:ro \
ghcr.io/yeet-src/httpwatch:latest # then open http://localhost:8080
The capability list is doing real work and dropping one mostly fails quietly. NET_ADMIN attaches the TCX programs, BPF and PERFMON load the program and its maps, and SYS_ADMIN mounts the container-private bpffs. The AppArmor line matters more than it looks: Docker's default profile denies that bpffs mount even when CAP_SYS_ADMIN is present, and lifting the profile is the narrow fix where --privileged would be the broad one. --network=host is what makes the capture real rather than an inspection of an empty container network, and the BTF mount is a read-only, world-readable kernel file that lets the probe relocate to whatever kernel is running.
There is nothing to install on the host beyond the container itself, no build step, and no configuration file to write first. On a box already serving traffic you are reading real requests in well under a minute, which is a different category of effort from every other row in that table.
Two prerequisites worth knowing before you start, because both fail in confusing ways. The kernel needs to be 6.6 or newer with BTF, since that is where TCX landed, and an older kernel fails with tcx: -EINVAL in the container logs rather than an empty screen with an explanation. The host also needs to be logged in to yeet: the dashboard drives the ordinary device flow from a sign-in button, and YEET_AUTH_KEY skips the click on an unattended box. Neither is a surprise if you know about it and both look like a broken tool if you do not.
The property that makes it usable during an incident rather than only after one is that it attaches to interfaces that already exist instead of inserting itself between anything. Starting and stopping are both non-events for your traffic. Nothing reconnects, no configuration reloads, and no requests are at risk if the container dies while you are reading a response, because it was never in the path in the first place.
The honest boundary is that it is one instance per host with nothing behind it. No retention, no query language, no fleet view. It answers what this box is doing right now, and when the ready-made shape is wrong, yeet is the runtime underneath and the probe is a JavaScript file you can change.
Already installed, needs no permission to add software, and sees every protocol rather than just HTTP. That breadth is the reason to reach for it first when the disagreement might not be about HTTP at all: if the real story is retransmits or resets, an HTTP decoder shows a clean table and tells you nothing.
# plaintext HTTP on any interface, printed as ASCII
sudo tcpdump -i any -A -s 0 'tcp port 80'
# just the request lines, which is usually what you want
sudo tcpdump -i any -A -s 0 'tcp port 80' | grep -E '^(GET|POST|PUT|DELETE|PATCH) '
# loopback only, for the traffic between two services on this box
sudo tcpdump -i lo -A -s 0 'tcp port 8080'
-A prints payloads as ASCII, which is what makes an HTTP request readable rather than a hex dump, and -s 0 stops truncation so a long header block survives intact. The third command is the one worth remembering, because loopback is where the traffic a proxy never saw actually lives, and -i any does capture it while people often assume it does not.
What it costs is that you get packets and reassemble streams yourself, usually later, usually in Wireshark, usually after a file transfer. A response split across segments arrives as fragments you stitch together by eye. For a question you are trying to answer while people are waiting, that loop is slow, and it is the specific work an HTTP decoder does for you continuously.
Worth correcting an assumption that costs this tool consideration: Coroot does not require Kubernetes. It supports standalone Docker, Docker Swarm, containerd, CRI-O and systemd units as containers, with a minimum Linux kernel of 5.1, which is a lower floor than TCX requires.
What you accept is a node agent and a storage layer. That is the right trade when the answer is "we will be watching this continuously" and the wrong one when you need to look at one box for an hour.
If the host is a cluster node and you want this continuously across every node, these fit. Pixie requires Kubernetes v1.21 or later, runs on Linux nodes only and deploys as a DaemonSet, with the docs explicit that standalone hosts are not the target.
Cilium Hubble deserves a specific correction, because people generalize from its flow data. The Cilium documentation states that L7 visibility requires enabling L7 proxy support, and that traffic matching an L7 rule in a CiliumNetworkPolicy is redirected to a proxy so the details can be captured. The docs add that L7 policies restrict what traffic is allowed to flow, not just what is visible. That is a request-path change, and it is worth choosing deliberately rather than discovering it after enabling visibility.
The only route here that can change traffic rather than describe it, and the only one that sees HTTPS without a uprobe, because it terminates TLS itself. If what you need is mTLS, retries or traffic shifting, nothing observational substitutes.
mitmproxy is the honest version of that trade: it generates its own certificate authority and clients must install and trust it before interception works at all. A proxy also sees only what you routed through it, which is the same blind spot as the access log wearing different clothes.
The only route that gives retention and a fleet view, which is why it is the eventual answer for most teams and why nothing here replaces it. If the question is what the error rate has looked like for a month, this is the tool and host-side capture is not.
The catch during an incident is attachment. OpenTelemetry's zero-code instrumentation covers .NET, Go, Java, JavaScript, PHP and Python and genuinely does not require editing source, but the Java agent attaches through a -javaagent flag in the JVM's startup arguments. The process starts instrumented or it does not, so adding visibility to something already misbehaving means restarting it and destroying the state you were investigating.
All three fall out of watching the wire, with no client library, no exporter and no code change. Request rate is a count per second of matched requests. Error rate is the per-class status tally read off the response lines. Latency percentiles come from pairing each response to its request as both cross the interface, which gives p50, p95 and max per endpoint. That is three of the four golden signals for a route whose owner never added a metric.
The fourth, saturation, is not available this way, and the reason is worth stating: capture measures traffic rather than the resources serving it. Queue depth, connection-pool exhaustion and thread starvation are properties of the process, and a tool watching the wire has no view of them. Pair capture with whatever already reports resource metrics rather than expecting one tool to do both.
One caveat that changes how you read the numbers. Latency measured at the host is on-the-wire latency, so for a remote caller it includes network round-trip time and time spent waiting in the accept queue. Your application's histogram starts when the handler receives the request. Both are correct, they measure different spans, and the gap between them is often the most useful number available: a large gap puts the delay outside your code, and a small one puts it inside.
Yes. An eBPF probe attaches to network interfaces that already exist rather than to your process, so watching HTTP traffic on a running host requires no restart, no redeploy and no configuration change to the service. The mechanism is worth understanding, because everything else that is true about this approach follows from it.
eBPF programs attach at tcx/ingress and tcx/egress and observe segments as the kernel moves them. Nothing is routed through the capture, no port is pointed at it, no application is reconfigured, and traffic is copied rather than held, modified or redirected. The service does not know it is being watched, because from the service's perspective nothing about its environment changed. TCX landed in Linux 6.6 and gives BPF programs link semantics on the TC hooks, including auto-detach when the file descriptor closes, which is what makes stopping the capture as safe as starting it.
Two properties follow from being outside the request path. The capture cannot add latency to a request or fail closed, so a crash is a blank dashboard rather than an outage, which is the opposite of a sidecar where the observability layer is load-bearing for the traffic it observes. And removal is genuinely reversible: there is no agent to uninstall and no config to revert, because nothing was modified.
The limits follow from the same position. TLS payloads are ciphertext at this layer, so HTTPS is invisible and reading it would require a uprobe on SSL_write and SSL_read, which is a different tool. HTTP/2 and cleartext h2c are binary with compressed headers, so an ASCII method match never sees them and produces an empty table with no error, and for gRPC specifically grpcsnoop is the sibling that handles the framing.
An eBPF HTTP capture showing zero requests almost always has one of four causes, listed here in the order they actually occur. On a quiet host an empty view and a broken tool look identical, which is why generating a little traffic before judging the setup is worth the thirty seconds.
SSL_write and SSL_read, or terminating TLS at a proxy, which is what mitmproxy does at the cost of distributing a CA certificate to every client.grpcsnoop decodes the binary framing that an ASCII match will never see, and is the right tool rather than a workaround.tcx: -EINVAL. This is a kernel older than 6.6, where the TCX attach point does not exist. The error appears in docker logs httpwatch and nowhere in the interface, so a dashboard that looks merely empty is often a dashboard that never started. Check the logs before concluding there is no traffic to see.127.0.0.1 is visible and traffic over /var/run/app.sock is not, and the difference is invisible from the outside.Cause three is worth checking first if you are on an older distribution, because the kernel floor varies by tool more than most comparisons mention. Each project states its own minimum, and on a fleet with mixed kernel versions this decides your options before any feature does:
| Tool | Minimum kernel | Also needs |
|---|---|---|
| httpwatch | 6.6, where TCX landed | BTF |
| Coroot node agent | 5.1 | Ubuntu 20.10+, Debian 11+, or RHEL 8.2+ for CO-RE profiling |
| Pixie | 4.14 | Kubernetes v1.21 or later, Linux nodes only |
A kernel below the floor is not a degraded mode. The attach fails outright, which is the better of the two failure modes: tcx: -EINVAL in the logs tells you what happened, whereas a tool that attached and quietly captured nothing would have you debugging your traffic instead of your kernel.
Reach for an APM the moment the question spans more than one host or more than the present moment. Host-level capture runs one instance per box, with no aggregation layer, no cross-host query and no retention beyond what is held in memory, so "what has this route's error rate been for a month" and "which service in the chain added the latency" are both outside it by construction. Running it on forty machines produces forty dashboards, which is forty times the work and none of the benefit you wanted. For fleet-wide and continuous, the honest recommendations are Coroot on Docker hosts or a cluster, Pixie on Kubernetes v1.21 or later, or OpenTelemetry into whatever already stores your metrics.
Reach for a service mesh when the answer has to change behavior rather than describe it. Retries, timeouts, circuit breaking, mTLS and traffic shifting are enforcement, and no amount of observability substitutes for them. The cost is that the mesh sits in the request path, so it can fail closed and take the service with it, which is exactly the property passive capture does not have.
What a per-host tool fits is the case in between, and most fleets have it somewhere. One box disagrees with its own dashboards. One node is the outlier and you want its actual traffic rather than its aggregates. One service was inherited and nobody knows its routes. For those, a tool that starts in thirty seconds and leaves nothing behind is the right instrument, and provisioning fleet-wide coverage to answer a question about one machine is a project rather than an answer.
tcpdump gives you packets and leaves you to reassemble streams. That is the right altitude when the question is below HTTP: a retransmit storm, a handshake that never completes, an MTU problem, or a protocol nobody documented. An HTTP decoder is actively worse for those, because it discards exactly the detail you need.
The difference is where the work happens. tcpdump captures and you interpret; an HTTP decoder parses and aggregates as it goes, so what you look at is already METHOD host path with counts, rates and percentiles. For "which endpoint changed after the deploy", that aggregation is the entire answer, and doing it by hand from a capture file is an afternoon.
The honest sequence when a host and its logs disagree is often both, in order. Start with the decoder to see whether the HTTP picture matches the log, and drop to tcpdump when the answer turns out to be that the problem is not HTTP at all.
Yes on both, with one boundary worth knowing in advance. A probe on the host sees container traffic that crosses an interface it watches, including loopback between two containers on the same box, because loopback crosses the TC layer like any other interface. That is the east-west traffic a sidecar proxy sees only half of and a network tap misses entirely, and it needs no instrumentation inside either container.
On a Kubernetes node the same thing holds: run it on the node and you see that node's traffic, pod-to-pod calls included. What a per-host tool does not give you is a cluster view, since there is no aggregation across nodes and no controller collecting from each one. Running it on one node to interrogate that node is a normal use. Running it on every node as a monitoring layer is not what it is for, and Pixie or Coroot is the right shape for that.
On Docker Desktop the picture shifts, because the capture watches the Linux VM rather than macOS. That is useful for inspecting your other containers and useless for your laptop's own applications, which live outside the VM entirely.
Running eBPF traffic capture on a production host is safe in the way that matters, because the capture is passive: it copies segments rather than holding, modifying or redirecting them, so there is no component in the request path that can add latency, fail closed, or drop connections when it restarts. Compare that to a sidecar, where a bad deploy of the observability layer is an outage of the service it observes.
The risk is not the capture, it is the dashboard. It shows decoded request and response bodies, and with request bodies enabled those include credentials, because that is the feature. Treat the port as the security boundary: a tailnet, a VPN, or a reverse proxy with real authentication. Captured data stays in the host's memory and travels to exactly two places you choose, which are your browser and any Slack channel you point an alert at, so nothing is shipped to a vendor.
Two operational details worth knowing before leaving it running. Changing the watched interfaces restarts the probe and resets every counter, because capture settings are spawn-time arguments with no control channel into a running isolate. And the captured-body store is bounded by both bytes and exchange count, oldest evicted first, so a body that has aged out says so rather than showing an empty panel.
If you need history across a fleet, instrument with OpenTelemetry and send it somewhere that stores it, or buy an APM and let someone else run the storage, and accept the restart that attaching costs. If you need continuous coverage on Docker hosts, Coroot runs without a cluster at kernel 5.1 or newer. On Kubernetes v1.21 or later, Pixie is built for the cluster view. If you already run Cilium, Hubble gives you flow data almost free, and L7 detail through a proxy redirect you should choose deliberately.
If you need to change traffic rather than observe it, that is Envoy or a mesh, and no observability tool substitutes. If you need HTTPS decoded from outside the process, mitmproxy does it and you will be distributing a CA certificate to every client. If you do not yet know what protocol you are looking at, it is a tcpdump question and always was.
If you need decoded HTTP for one Linux box right now, with no restart and nothing left behind when you stop, capture at the kernel's TC layer with httpwatch and accept that it is plaintext HTTP/1.x only, one host, and forgets everything when it exits.
The failure mode worth avoiding is reasoning from the access log alone. It is the account of one participant, written by that participant, covering only the requests that reached it, and during an incident it is usually accurate and usually incomplete at the same time. The kernel has no stake in the outcome: it sees every segment that crossed an interface, including the ones that never became a log line anywhere. That is the whole reason to capture there, and it is available on any Linux box without changing a single thing about the service you are investigating.
Not at the network layer. TLS encrypts the payload before it reaches the wire, so a probe attached to a network interface sees ciphertext with no request line to parse. Reading HTTPS with eBPF means moving the capture point to a uprobe on SSL_write and SSL_read inside the process, which requires knowing which TLS library the application uses. A proxy that terminates TLS is the other option.
You need privileges to load the eBPF program, though not necessarily an interactive root shell. Loading and attaching requires capabilities such as CAP_BPF, CAP_NET_ADMIN and CAP_PERFMON rather than full root, which is why these tools ship as containers with a specific capability set instead of running privileged. With httpwatch the yeet daemon handles the privileged load, so the command you type is unprivileged.
Position in the request path. A sidecar proxy is an endpoint of the connection, so traffic is routed through it, it can modify or block requests, and it can fail closed and take the service with it. An eBPF probe attached to an interface copies segments as the kernel moves them, so it cannot alter traffic and cannot break it. The proxy sees more, including HTTPS, and costs more to run.
They measure different spans of the same request. Latency measured at the host runs from the request segments arriving to the response segments leaving, so it includes network round-trip time and any wait in the accept queue. The application histogram starts when the handler receives the request. Neither is wrong, and the gap between them locates the delay: large means outside your code, small means inside it.
It depends on the attach point, and the range is wider than people expect. TCX, the modern TC attach point, landed in Linux 6.6 and needs BTF. Coroot's node agent supports kernels from 5.1, and Pixie states 4.14 as its floor. If you run older hosts, the kernel version may decide your tool before any feature comparison does.
For one host, yes: a container that attaches to existing interfaces leaves nothing behind when it stops. For fleet-wide coverage the honest answer is no. Every tool that provides a fleet view puts something on every host, whether a DaemonSet, a node agent, or an instrumentation library inside each process. The choice is which kind of resident you want, not whether you have one.
Kernel-side capture, because the filtering happens in the kernel and uninteresting traffic never reaches userspace, so cost scales with matched traffic rather than total traffic. More importantly for production, nothing sits in the request path, so the capture cannot add latency or fail closed. httpwatch is one ready-made implementation, and the trade is plaintext HTTP only, one host, and no retention.
No, and treating it as a replacement is how teams lose an entire class of answer. Kernel-side capture has no retention, no query language and no history beyond what is held in memory, so it answers what a host is doing right now rather than what happened last Tuesday. It is what you reach for when the dashboards say something is wrong and you need the actual bytes.
Run sudo tcpdump -i any -A -s 0 'tcp port 80'. The -A flag prints payloads as ASCII so request lines are readable rather than a hex dump, and -s 0 stops truncation so long header blocks survive intact. Pipe it through grep for GET or POST to see only request lines. Use -i lo instead of -i any to capture traffic between two services on the same host.
The kernel is older than 6.6, which is where the TCX attach point was added. A program targeting tcx/ingress or tcx/egress cannot attach on an earlier kernel and fails immediately rather than degrading. Check the kernel with uname -r. The failure appears in the tool's logs and not in its interface, so the symptom is usually an empty dashboard rather than an error on screen.
Run ss -lntp, which lists every listening TCP socket with the process holding it. Without sudo the process column is blank for anything you do not own, which is the usual reason it appears broken. A socket bound to 127.0.0.1 is reachable only from the host itself, so traffic to it never crosses an external interface and any capture watching only eth0 will miss it.
tcx/ingress and tcx/egress; requires kernel 6.6+ with BTF; detects nine ASCII method tokens (GET, PUT, HEAD, POST, TRACE, PATCH, DELETE, OPTIONS, CONNECT) plus HTTP/ status lines, which is why HTTP/2 and h2c never appear; captures roughly 64KB per message and ~256KB for a 4xx or 5xx, metered in aggregate with truncation flagged; runs one instance per host with no aggregation layer and no retention beyond memory; needs SYS_ADMIN, NET_ADMIN, BPF and PERFMON plus apparmor=unconfined, but not --privileged.cls_bpf path that required a clsact qdisc plus a filter handle and priority; gives BPF programs link semantics including safe ownership, auto-detach when the file descriptor closes, and explicit ordering through BPF_F_BEFORE and BPF_F_AFTER; an attach against an older kernel fails with tcx: -EINVAL.-javaagent JVM startup flag or JAVA_TOOL_OPTIONS, which ties instrumentation to process start: a JVM begins instrumented or it does not, so adding visibility to a misbehaving service means restarting it and losing the state you were investigating.~/.mitmproxy; clients must install and trust that CA or "click through a TLS certificate warning on every domain", so HTTPS interception costs a trust-store change on every machine whose traffic you want to read, which is the real price of the only route here that decrypts.