DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Tue, 14 Apr 2026 19:17:13 UTC
Peer count
250
Peer count at end
62500
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.4
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_24417553515_1
Create rate
The average number of created records per agent.
Mean
mean
47.53
/s
Max
Highest per-partition mean rate
max
172.17
/s
Min
Lowest per-partition mean rate
min
9.08
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
281.973714
s
SD
127.272518
s
Max
Highest per-partition mean latency
max mean
1776.713372
s
Min
Lowest per-partition mean latency
min mean
28.40256
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
98.57
/s
Max
Highest per-partition mean rate
max
979
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The number of errors encountered during the scenario.
599
Holochain Metrics
Hidden by default. Click to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.014007
s
SD
0.063073
s
p50
0.010084
s
<
p95
0.175018
s
<
p99
0.32242
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
9
over 3078.169465528s
mean rate
358.03
/s
std
10544.15
/s
p5
0
/s
p95
76.39
/s
peak
552921.35
/s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
672105468
over 3057.800719544s
mean rate
2.47990945059e+10
/s
std
8.07709602759e+11
/s
p5
7.302413458e+07
/s
p95
5.58689914086e+10
/s
peak
9.19393111111e+13
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
6182812931
over 3137.779638357s
mean rate
2.28926245478e+11
/s
std
2.20342836391e+12
/s
p5
1.2844178261e+08
/s
p95
5.45982338808e+10
/s
peak
7.47745851782e+13
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
497018568
over 3139.511651029s
mean rate
3.00105509021e+09
/s
std
8.3174847218e+10
/s
p5
8.66440809e+06
/s
p95
7.95048382889e+09
/s
peak
1.04035461988e+13
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.014937
s
SD
0.009985
s
p50
0.016787
s
<
p95
0.033831
s
<
p99
0.045617
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.110431
s
SD
0.74657
s
p50
0.181311
s
<
p95
1.481477
s
<
p99
3.453677
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.013209
s
SD
0.009232
s
p50
0.014572
s
<
p95
0.030726
s
<
p99
0.042034
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.109097
s
SD
0.74649
s
p50
0.179287
s
<
p95
1.480803
s
<
p99
3.452819
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001705
s
SD
0.000756
s
p50
0.001911
s
<
p95
0.003136
s
<
p99
0.003735
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00222
s
SD
0.002872
s
p50
0.001845
s
<
p95
0.007745
s
<
p99
0.012844
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.002214
s
SD
0.002799
s
p50
0.001865
s
<
p95
0.007678
s
<
p99
0.012552
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.014956
s
SD
0.072253
s
p50
0.014642
s
<
p95
0.197785
s
<
p99
0.377312
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.003396
s
SD
0.008054
s
p50
0.005694
s
<
p95
0.025905
s
<
p99
0.039768
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003297
s
SD
0.00169
s
p50
0.003688
s
<
p95
0.006653
s
<
p99
0.007901
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001046
s
SD
0.000737
s
p50
0.001086
s
<
p95
0.002468
s
<
p99
0.003073
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
85.52
s
<
mean
905.34
s
SD
532.37
s
<
p95
1738.52
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
17259
over 3139.511684301s
mean rate
2.72501666e+06
/s
std
8.920162469e+07
/s
p5
9426.58
/s
p95
6.93095754e+06
/s
peak
1.11864754098e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
91.390354
s
SD
105.531644
s
p50
52.66897
s
<
p95
260.503117
s
<
p99
482.529235
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
1.307641
s
SD
0.294555
s
p50
1.310127
s
<
p95
1.999654
s
<
p99
2.015348
s
App validation workflow duration
Time spent running the app validation workflow.
mean
31.06316
s
SD
34.873436
s
p50
24.473898
s
<
p95
104.330585
s
<
p99
120.446814
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003511
s
SD
0.002783
s
p50
0.003036
s
<
p95
0.008664
s
<
p99
0.016536
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.029449
s
SD
1.898221
s
p50
0.028308
s
<
p95
0.11546
s
<
p99
8.021266
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
1.476162
s
SD
3.382576
s
p50
0.054755
s
<
p95
9.305566
s
<
p99
14.808872
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
37.688513
s
SD
67.26018
s
p50
53.34198
s
<
p95
200.866207
s
<
p99
297.260085
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.317271
s
SD
28.123783
s
p50
0.35869
s
<
p95
34.1588
s
<
p99
171.258681
s
Authored DB connection use time
Time spent holding authored database connections.
mean
0.009184
s
SD
0.011744
s
p50
0.007794
s
<
p95
0.019803
s
<
p99
0.07223
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002535
s
SD
0.002619
s
p50
0.002051
s
<
p95
0.007202
s
<
p99
0.014973
s
Conductor DB connection use time
Time spent holding conductor database connections.
mean
0.000306
s
SD
0.000164
s
p50
0.000315
s
<
p95
0.000612
s
<
p99
0.000717
s
Cache DB connection use time
Time spent holding cache database connections.
mean
0.005613
s
SD
0.011513
s
p50
0.000234
s
<
p95
0.017952
s
<
p99
0.066663
s
WASM DB connection use time
Time spent holding WASM database connections.
mean
0.000659
s
SD
0.004714
s
p50
0.000214
s
<
p95
0.000494
s
<
p99
0.013781
s
Peer meta store DB connection use time
Time spent holding peer meta store database connections.
mean
0.000144
s
SD
7.7e-05
s
p50
0.000158
s
<
p95
0.000285
s
<
p99
0.000424
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006322
s
SD
0.087944
s
p50
0.003751
s
<
p95
0.250863
s
<
p99
0.398932
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000597
s
SD
0.002395
s
p50
0.001244
s
<
p95
0.005833
s
<
p99
0.010884
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.336765
s
SD
29.549542
s
p50
3.522697
s
<
p95
61.687786
s
<
p99
67.26288
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.089981
s
SD
27.388713
s
p50
1.736512
s
<
p95
60.012934
s
<
p99
60.093516
s
Host Metrics
User CPU usage
CPU usage by user space
45.19
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
24.34
KiB/s
<
mean
910.40
KiB/s
SD
884.40
KiB/s
<
p95
2.47
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
2.67
KiB/s
<
mean
161.64
KiB/s
SD
412.11
KiB/s
<
p95
409.04
KiB/s
Total bytes received
Total bytes received on primary network interface
count
309.20
GiB
mean
100.83
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
53.00
GiB
mean
17.28
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 914.90 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 66% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.93
%
<
mean
59.82
%
SD
27.45
%
<
p95
92.41
%
System
14.63
%
CPU percentiles
CPU usage percentiles
p50
68.40
%
p95
92.41
%
p99
97.00
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
154
hosts
mean time
0.31
s
Memory used percentage
Percentage of memory used
p5
6.49
%
<
mean
10.85
%
SD
5.69
%
<
p95
16.05
%
Memory available percentage
Percentage of available memory
p5
83.95
%
<
mean
89.15
%
SD
5.69
%
<
p95
93.51
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.13
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
3.44
%
Memory growth rate
Rate of memory growth over time
growth
914.90
MiB/s
Disk read throughput
Disk read throughput in MiB/s
7.82
MiB/s
Disk write throughput
Disk write throughput in MiB/s
456.06
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/156
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/8
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/14
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.64
5 min
2.25
15 min
1.52
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
65.59
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
28.8849
%
SD
22.6691
%
<
p95
68.45
%
60 second average
28.1018
%
300 second average
23.8915
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0.0054
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.0242
%
SD
10.8996
%
<
p95
31.68
%
60 second average
7.8548
%
300 second average
6.8642
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.3734
%
SD
9.856
%
<
p95
26
%
60 second average
5.2576
%
300 second average
4.6208
%
Holochain process CPU usage
CPU usage by Holochain process
p5
4.36
%
<
mean
57.61
%
SD
25.37
%
<
p95
88.73
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
229.69
KiB
<
mean
356.63
KiB
SD
102.57
KiB
<
p95
517.28
KiB
Holochain process threads
Number of threads in Holochain process
p5
13
threads
<
mean
29.99
threads
SD
12.68
threads
<
p95
52
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
58
file descriptors
<
mean
77.06
file descriptors
SD
15.35
file descriptors
<
p95
98
file descriptors