DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Tue, 31 Mar 2026 19:25:39 UTC
Peer count
38
Peer count at end
1444
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_23815394770_1
Create rate
The average number of created records per agent.
Mean
mean
40.82
/s
Max
Highest per-partition mean rate
max
95.6
/s
Min
Lowest per-partition mean rate
min
8.71
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
590.150406
s
SD
331.873542
s
Max
Highest per-partition mean latency
max mean
1429.875946
s
Min
Lowest per-partition mean latency
min mean
33.419559
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
50.59
/s
Max
Highest per-partition mean rate
max
211.13
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The number of errors encountered during the scenario.
174
Host Metrics
User CPU usage
CPU usage by user space
30.34
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
14.65
KiB/s
<
mean
183.44
KiB/s
SD
457.41
KiB/s
<
p95
743.16
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
13.53
KiB/s
<
mean
156.67
KiB/s
SD
353.15
KiB/s
<
p95
761.76
KiB/s
Total bytes received
Total bytes received on primary network interface
count
11.69
GiB
mean
6.20
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.98
GiB
mean
5.30
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1028.37 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
5.34
%
<
mean
34.46
%
SD
25.69
%
<
p95
96.99
%
System
4.11
%
CPU percentiles
CPU usage percentiles
p50
28.23
%
p95
96.99
%
p99
98.24
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
17
hosts
mean time
0.21
s
Memory used percentage
Percentage of memory used
p5
6.93
%
<
mean
13.62
%
SD
11.08
%
<
p95
18.54
%
Memory available percentage
Percentage of available memory
p5
81.46
%
<
mean
86.38
%
SD
11.08
%
<
p95
93.07
%
Max host memory used
Maximum memory usage percentage across all hosts
max
77.32
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
1028.37
MiB/s
Disk read throughput
Disk read throughput in MiB/s
5.70
MiB/s
Disk write throughput
Disk write throughput in MiB/s
365.31
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/32
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/26
hosts
Mount Point /etc/hostname
0/5
hosts
Mount Point /etc/hosts
0/5
hosts
Mount Point /etc/resolv.conf
0/5
hosts
Mount Point /nix/store
0/32
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.54
5 min
3.01
15 min
1.98
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
4.7133
%
SD
10.7509
%
<
p95
28.39
%
60 second average
4.5398
%
300 second average
3.7295
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
22.7668
%
SD
19.2807
%
<
p95
57.05
%
60 second average
22.2122
%
300 second average
19.2665
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
18.6668
%
SD
17.1425
%
<
p95
53.1
%
60 second average
18.1949
%
300 second average
15.8032
%
Holochain process CPU usage
CPU usage by Holochain process
p5
2.43
%
<
mean
30.8
%
SD
25.9
%
<
p95
96.19
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
240.86
KiB
<
mean
410.35
KiB
SD
175.53
KiB
<
p95
871.85
KiB
Holochain process threads
Number of threads in Holochain process
p5
13
threads
<
mean
37.53
threads
SD
16.35
threads
<
p95
66
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
60
file descriptors
<
mean
88
file descriptors
SD
24.04
file descriptors
<
p95
131
file descriptors

Remote Signals

This scenario tests the throughput of remote_signals operations.

Started
Tue, 31 Mar 2026 19:56:18 UTC
Peer count
10
Peer count at end
50
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
remote_signals_23815394770_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.312311
s
SD
0.756232
s
p50
0.136333
s
<
p95
0.703108
s
<
p99
1.838915
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
map[count:27 mean_rate_per_second:1.63 measurement_duration:map[nanos:483295945 secs:45] p5_rate_per_second:0 p95_rate_per_second:5.73 peak_rate_per_second:15.38 rate_trend:map[trend:[1.94 2.47 0.96 0.98 1.84 2.34] window_duration:10s] std_rate_per_second:2.7]
Host Metrics
User CPU usage
CPU usage by user space
15.68
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
10.61
KiB/s
<
mean
241.80
KiB/s
SD
908.90
KiB/s
<
p95
679.07
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
10.59
KiB/s
<
mean
145.39
KiB/s
SD
274.67
KiB/s
<
p95
782.33
KiB/s
Total bytes received
Total bytes received on primary network interface
count
1.06
GiB
mean
1.13
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
650.29
MiB
mean
693.64
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1453.70 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.63
%
<
mean
19.66
%
SD
25.12
%
<
p95
72.39
%
System
3.98
%
CPU percentiles
CPU usage percentiles
p50
5.26
%
p95
72.39
%
p99
98.27
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
3
hosts
mean time
0.04
s
Memory used percentage
Percentage of memory used
p5
7.75
%
<
mean
13.34
%
SD
3.45
%
<
p95
18.48
%
Memory available percentage
Percentage of available memory
p5
81.52
%
<
mean
86.66
%
SD
3.45
%
<
p95
92.25
%
Max host memory used
Maximum memory usage percentage across all hosts
max
16.97
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
1453.70
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.12
MiB/s
Disk write throughput
Disk write throughput in MiB/s
51.62
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/4
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/3
hosts
Mount Point /etc/hostname
0/1
hosts
Mount Point /etc/hosts
0/1
hosts
Mount Point /etc/resolv.conf
0/1
hosts
Mount Point /nix/store
0/4
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.89
5 min
2.18
15 min
2.15
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
5.6168
%
SD
12.9098
%
<
p95
32.31
%
60 second average
5.5549
%
300 second average
5.7449
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.9772
%
SD
16.47
%
<
p95
50.34
%
60 second average
9.7360
%
300 second average
13.4969
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
7.8206
%
SD
15.039
%
<
p95
46.64
%
60 second average
8.6347
%
300 second average
12.1137
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.19
%
<
mean
4.31
%
SD
8.33
%
<
p95
24.94
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
87.63
KiB
<
mean
210.56
KiB
SD
77.86
KiB
<
p95
285.46
KiB
Holochain process threads
Number of threads in Holochain process
p5
12
threads
<
mean
20.6
threads
SD
6.93
threads
<
p95
28
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
38
file descriptors
<
mean
62.03
file descriptors
SD
16.93
file descriptors
<
p95
86
file descriptors