DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Sat, 21 Mar 2026 01:54:03 UTC
Peer count
250
Peer count at end
3732
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_23369109191
Create rate
The average number of created records per agent.
Mean
mean 66.69/s
Max
Highest per-partition mean rate
max 304/s
Min
Lowest per-partition mean rate
min 0/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean 136.10178s (SD = 37.082884s)
Max
Highest per-partition mean latency
max mean 251.214205s
Min
Lowest per-partition mean latency
min mean 0.118268s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean 62.11/s
Max
Highest per-partition mean rate
max 391.5/s
Min
Lowest per-partition mean rate
min 0/s
Error count
The number of errors encountered during the scenario.
30
Host Metrics
User CPU usage
CPU usage by user space
29.78%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 2.03 KiB/s < mean 901.54 KiB/s (SD = 735.20 KiB/s) < p952.06 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 156 B/s < mean 167.69 KiB/s (SD = 411.39 KiB/s) < p95659.37 KiB/s
Total bytes received
Total bytes received on primary network interface
count
81.78 GiB
mean
48.41 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
15.09 GiB
mean
8.93 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 640.93 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (39.0% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.15% < mean 42.74% (SD = 24.89%) < p95 83.72%
System
12.97%
CPU percentiles
CPU usage percentiles
p50
46.17%
p95
83.72%
p99
93.82%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
170 hosts
mean time
0.11s
Memory used percentage
Percentage of memory used
p5 5.4% < mean 10.97% (SD = 8.32%) < p95 20.49%
Memory available percentage
Percentage of available memory
p5 79.51% < mean 89.03% (SD = 8.32%) < p95 94.6%
Max host memory used
Maximum memory usage percentage across all hosts
max 78.69%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 39.02%
Memory growth rate
Rate of memory growth over time
growth 640.93MB/s
Disk read throughput
Disk read throughput in MB/s
0.99MB/s
Disk write throughput
Disk write throughput in MB/s
1210.47MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/198hosts
Mount Point /efi-boot
0/19hosts
Mount Point /etc/hostname
0/27hosts
Mount Point /etc/hosts
0/27hosts
Mount Point /etc/resolv.conf
0/27hosts
Mount Point /nix/store
0/23hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.70
5 min
0.87
15 min
0.37
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 19.2532% (SD = 16.8621%) < p95 48.18%
60 second average
16.7442%
300 second average
8.6257%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 7.265% (SD = 9.7761%) < p95 30.35%
60 second average
6.2060%
300 second average
3.1911%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 5.2829% (SD = 8.4957%) < p95 24.54%
60 second average
4.4895%
300 second average
2.3351%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 42.01% (SD = 22.76%) < p95 79.88%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 153.68 KiB < mean 267.58 KiB (SD = 56.27 KiB) < p95345.44 KiB
Holochain process threads
Number of threads in Holochain process
p5 5 threads < mean 23.28 threads (SD = 12.77 threads) < p95 45 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 32 file descriptors < mean 70.3 file descriptors (SD = 20.08 file descriptors) < p95 95 file descriptors

Validation Receipts

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Sat, 21 Mar 2026 02:08:45 UTC
Peer count
250
Peer count at end
7470
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
validation_receipts_23369109191
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean 50.49459s (SD = 11.31725s)
Max
Highest per-partition mean latency
max mean 230.435061s
Min
Lowest per-partition mean latency
min mean 11.692848s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean 0.51/s
Max
Highest per-partition mean rate
max 1/s
Min
Lowest per-partition mean rate
min 0/s
Host Metrics
User CPU usage
CPU usage by user space
7.52%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 2.12 KiB/s < mean 408.87 KiB/s (SD = 532.95 KiB/s) < p951.55 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1021 B/s < mean 42.42 KiB/s (SD = 52.07 KiB/s) < p95120.09 KiB/s
Total bytes received
Total bytes received on primary network interface
count
31.56 GiB
mean
30.78 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.19 GiB
mean
3.11 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 90.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 474.02 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (39.0% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.55% < mean 10.5% (SD = 16.66%) < p95 52.47%
System
2.98%
CPU percentiles
CPU usage percentiles
p50
5.62%
p95
52.47%
p99
90.76%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
119 hosts
mean time
0.04s
Memory used percentage
Percentage of memory used
p5 5.47% < mean 10.93% (SD = 8.64%) < p95 25.79%
Memory available percentage
Percentage of available memory
p5 74.21% < mean 89.07% (SD = 8.64%) < p95 94.53%
Max host memory used
Maximum memory usage percentage across all hosts
max 77.71%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 39.04%
Memory growth rate
Rate of memory growth over time
growth 474.02MB/s
Disk read throughput
Disk read throughput in MB/s
0.29MB/s
Disk write throughput
Disk write throughput in MB/s
142.99MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/188hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/18hosts
Mount Point /etc/hostname
0/27hosts
Mount Point /etc/hosts
0/27hosts
Mount Point /etc/resolv.conf
0/27hosts
Mount Point /nix/store
0/21hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.31
5 min
0.37
15 min
0.44
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 1.6529% (SD = 3.543%) < p95 6.21%
60 second average
1.5470%
300 second average
2.5445%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 1.0201% (SD = 3.194%) < p95 5.92%
60 second average
1.0201%
300 second average
2.0189%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.9495% (SD = 3.0512%) < p95 5.6%
60 second average
0.9522%
300 second average
1.7985%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.22% < mean 7.86% (SD = 15.16%) < p95 33.63%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 207.44 KiB < mean 242.55 KiB (SD = 31.94 KiB) < p95273.26 KiB
Holochain process threads
Number of threads in Holochain process
p5 9 threads < mean 25.44 threads (SD = 13.4 threads) < p95 46 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 39 file descriptors < mean 74.09 file descriptors (SD = 20.93 file descriptors) < p95 99 file descriptors