DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Fri, 17 Apr 2026 01:54:35 UTC
Peer count
250
Peer count at end
8454
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.4
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_24541634156_1
Create rate
The average number of created records per agent.
Mean
mean
41.13
/s
Max
Highest per-partition mean rate
max
417.13
/s
Min
Lowest per-partition mean rate
min
0
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
323.565306
s
SD
162.575689
s
Max
Highest per-partition mean latency
max mean
1509.730916
s
Min
Lowest per-partition mean latency
min mean
9.111064
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
33.02
/s
Max
Highest per-partition mean rate
max
225
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0003
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
1305302
Agents affected
206 / 206
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
6336
Max
Highest count in any single partition
18349
Min
Lowest count in any single partition
44
Received count
The total number of entries received by record_lag agents, by agent.
Total
15135
Agents affected
33 / 35
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
432
Max
Highest count in any single partition
2050
Min
Lowest count in any single partition
0
Error count
The number of errors encountered during the scenario.
765
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.020733
s
SD
0.058524
s
p50
0.011758
s
<
p95
0.171212
s
<
p99
0.276279
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
10
over 26174.253498988s
mean rate
94.95
/s
std
1165.85
/s
p5
0
/s
p95
215.78
/s
peak
64850.75
/s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
683282750
over 26529.590269262s
mean rate
1.48957109224e+10
/s
std
1.24353977265e+11
/s
p5
1.725763927e+07
/s
p95
5.10749039808e+10
/s
peak
1.03306858704e+13
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
750361135
over 1956.544451475s
mean rate
2.01578317029e+12
/s
std
5.8517829225e+13
/s
p5
2.2100912975e+08
/s
p95
2.20610641354e+11
/s
peak
2.46082166118e+15
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
77318000
over 26529.590269262s
mean rate
9.13536844481e+09
/s
std
8.31246329254e+11
/s
p5
2.72507423e+06
/s
p95
8.46382364175e+09
/s
peak
1.185735e+14
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.014018
s
SD
0.005689
s
p50
0.014793
s
<
p95
0.022441
s
<
p99
0.039367
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.150543
s
SD
0.528473
s
p50
0.138973
s
<
p95
0.958519
s
<
p99
2.323959
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012004
s
SD
0.005119
s
p50
0.012505
s
<
p95
0.019201
s
<
p99
0.036216
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.149177
s
SD
0.528351
s
p50
0.138307
s
<
p95
0.956401
s
<
p99
2.321839
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001803
s
SD
0.000505
s
p50
0.001941
s
<
p95
0.002605
s
<
p99
0.002895
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.001332
s
SD
0.001618
s
p50
0.001128
s
<
p95
0.003091
s
<
p99
0.010695
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.001301
s
SD
0.001602
s
p50
0.001121
s
<
p95
0.002927
s
<
p99
0.010715
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.02179
s
SD
0.07853
s
p50
0.013615
s
<
p95
0.207032
s
<
p99
0.364034
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.007212
s
SD
0.010811
s
p50
0.004159
s
<
p95
0.030001
s
<
p99
0.062711
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003413
s
SD
0.001086
s
p50
0.003673
s
<
p95
0.005238
s
<
p99
0.005873
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001003
s
SD
0.000457
s
p50
0.001028
s
<
p95
0.001607
s
<
p99
0.002602
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
86.41
s
<
mean
908.5
s
SD
528.21
s
<
p95
1735.52
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
67528
over 26529.590226658s
mean rate
5.29391312e+06
/s
std
3.7541298078e+08
/s
p5
3332.43
/s
p95
7.39818745e+06
/s
peak
5.33888888889e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
113.607513
s
SD
104.362635
s
p50
59.150491
s
<
p95
306.071371
s
<
p99
439.544253
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
1.350161
s
SD
0.305929
s
p50
1.296124
s
<
p95
2.000563
s
<
p99
2.017365
s
App validation workflow duration
Time spent running the app validation workflow.
mean
7.05885
s
SD
22.340212
s
p50
17.273263
s
<
p95
67.36518
s
<
p99
102.19424
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.004527
s
SD
0.003716
s
p50
0.003662
s
<
p95
0.010177
s
<
p99
0.021109
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.047377
s
SD
2.130953
s
p50
0.042448
s
<
p95
0.811024
s
<
p99
9.060988
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.429595
s
SD
2.490405
s
p50
0.028615
s
<
p95
6.14014
s
<
p99
13.106445
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.61047
s
SD
60.78643
s
p50
40.235925
s
<
p95
174.587584
s
<
p99
246.883041
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.304987
s
SD
28.42772
s
p50
0.281509
s
<
p95
36.607815
s
<
p99
156.486522
s
Authored DB connection use time
Time spent holding authored database connections.
mean
0.012573
s
SD
0.011024
s
p50
0.010955
s
<
p95
0.029309
s
<
p99
0.054583
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.005686
s
SD
0.006044
s
p50
0.004609
s
<
p95
0.015713
s
<
p99
0.033962
s
Conductor DB connection use time
Time spent holding conductor database connections.
mean
0.000425
s
SD
0.000406
s
p50
0.000388
s
<
p95
0.000932
s
<
p99
0.002035
s
Cache DB connection use time
Time spent holding cache database connections.
mean
0.007492
s
SD
0.01091
s
p50
0.000273
s
<
p95
0.021288
s
<
p99
0.059182
s
WASM DB connection use time
Time spent holding WASM database connections.
mean
0.000273
s
SD
0.000233
s
p50
0.00025
s
<
p95
0.00045
s
<
p99
0.001686
s
Peer meta store DB connection use time
Time spent holding peer meta store database connections.
mean
0.000198
s
SD
0.000143
s
p50
0.000195
s
<
p95
0.000337
s
<
p99
0.000805
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008329
s
SD
0.095205
s
p50
0.005897
s
<
p95
0.288146
s
<
p99
0.347901
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000564
s
SD
0.00146
s
p50
0.000812
s
<
p95
0.002454
s
<
p99
0.009169
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.373718
s
SD
29.658298
s
p50
2.710916
s
<
p95
60.037952
s
<
p99
63.647811
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.566844
s
SD
27.405622
s
p50
1.07803
s
<
p95
60.002532
s
<
p99
60.012114
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
30.69
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
60
B/s
<
mean
502.66
KiB/s
SD
576.14
KiB/s
<
p95
1.62
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
60.35
KiB/s
SD
121.94
KiB/s
<
p95
220.15
KiB/s
Total bytes received
Total bytes received on primary network interface
count
301.53
GiB
mean
11.64
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
35.75
GiB
mean
1.38
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 499.51 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 66% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.45
%
<
mean
42.95
%
SD
33.8
%
<
p95
87.65
%
System
12.26
%
CPU percentiles
CPU usage percentiles
p50
52.97
%
p95
87.65
%
p99
96.34
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
208
hosts
mean time
0.21
s
Memory used percentage
Percentage of memory used
p5
8.35
%
<
mean
10.39
%
SD
2.14
%
<
p95
13.89
%
Memory available percentage
Percentage of available memory
p5
86.11
%
<
mean
89.61
%
SD
2.14
%
<
p95
91.65
%
Max host memory used
Maximum memory usage percentage across all hosts
max
29.45
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
2.65
%
Memory growth rate
Rate of memory growth over time
growth
499.51
MiB/s
Disk read throughput
Disk read throughput in MiB/s
1.32
MiB/s
Disk write throughput
Disk write throughput in MiB/s
256.39
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/209
hosts
Mount Point /efi-boot
0/7
hosts
Mount Point /etc/hostname
0/7
hosts
Mount Point /etc/hosts
0/7
hosts
Mount Point /etc/resolv.conf
0/7
hosts
Mount Point /nix/store
0/10
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.89
5 min
1.63
15 min
1.17
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
65.93
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
23.2779
%
SD
24.4048
%
<
p95
67.95
%
60 second average
22.7474
%
300 second average
19.7473
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
7.3143
%
SD
9.7623
%
<
p95
26.15
%
60 second average
7.1733
%
300 second average
6.2631
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.0395
%
SD
8.2375
%
<
p95
20.43
%
60 second average
4.9325
%
300 second average
4.3227
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
40.79
%
SD
32.22
%
<
p95
83.45
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
231.46
KiB
<
mean
306.14
KiB
SD
71.61
KiB
<
p95
409.10
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
20.59
threads
SD
13.35
threads
<
p95
42
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
32
file descriptors
<
mean
62.48
file descriptors
SD
23.08
file descriptors
<
p95
91
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Fri, 17 Apr 2026 01:04:34 UTC
Peer count
250
Peer count at end
3487
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.4
Wind Tunnel version
0.6.0
Run ID
remote_call_rate_24541634156_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-5.728033
s
SD
36.378334
s
Max
Highest per-partition mean latency
max mean
14.871005
s
Min
Lowest per-partition mean latency
min mean
-94.367243
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.43384
s
SD
1.624547
s
Max
Highest per-partition mean latency
max mean
3.567248
s
Min
Lowest per-partition mean latency
min mean
0.053023
s
Error count
The total number of errors accumulated during the run.
3878
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 892.373351746s
mean rate
2.43
/s
std
11.36
/s
p5
0
/s
p95
1.01
/s
peak
60.78
/s
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
29674384
over 2290.493567089s
mean rate
1.53780434864e+09
/s
std
1.25065732277e+10
/s
p5
7.20663054e+06
/s
p95
4.78956815488e+09
/s
peak
6.51517851973e+11
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
5854128
over 2290.493567089s
mean rate
2.7745041559e+08
/s
std
2.642438606e+09
/s
p5
1.02904411e+06
/s
p95
1.06812209277e+09
/s
peak
1.63506706552e+11
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 28611.79478798s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.339625
s
SD
0.890217
s
p50
0.388642
s
<
p95
1.989051
s
<
p99
3.708531
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.003564
s
SD
0.000888
s
p50
0.003763
s
<
p95
0.004821
s
<
p99
0.005614
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
1.017312
s
SD
3.259073
s
p50
1.244938
s
<
p95
8.782712
s
<
p99
14.847604
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00223
s
SD
0.000593
s
p50
0.002327
s
<
p95
0.003194
s
<
p99
0.003595
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003863
s
SD
0.002417
s
p50
0.003665
s
<
p95
0.006489
s
<
p99
0.010964
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
1.014546
s
SD
3.258996
s
p50
1.242272
s
<
p95
8.779913
s
<
p99
14.844645
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000676
s
SD
0.00052
s
p50
0.000627
s
<
p95
0.001235
s
<
p99
0.001958
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
2.3e-05
s
SD
1e-05
s
p50
1.8e-05
s
<
p95
4.1e-05
s
<
p99
4.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000414
s
SD
0.000314
s
p50
0.000417
s
<
p95
0.000938
s
<
p99
0.001704
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000787
s
SD
0.00077
s
p50
0.000531
s
<
p95
0.002535
s
<
p99
0.004166
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
45.98
s
<
mean
450.25
s
SD
259.48
s
<
p95
855.77
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
31.970015
s
SD
21.77169
s
p50
21.09968
s
<
p95
68.384552
s
<
p99
100.185607
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
2.103292
s
SD
0.287379
s
p50
2.069828
s
<
p95
2.262024
s
<
p99
2.40528
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.400834
s
SD
0.794578
s
p50
0.393275
s
<
p95
2.238918
s
<
p99
3.824706
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003787
s
SD
0.00299
s
p50
0.003259
s
<
p95
0.009229
s
<
p99
0.015146
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.00666
s
SD
0.007708
s
p50
0.005056
s
<
p95
0.013268
s
<
p99
0.028864
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.005527
s
SD
0.003139
s
p50
0.005111
s
<
p95
0.01214
s
<
p99
0.016767
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
12.355226
s
SD
13.330274
s
p50
12.734136
s
<
p95
41.049914
s
<
p99
68.122327
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
9.878693
s
SD
15.358921
s
p50
4.413342
s
<
p95
41.881446
s
<
p99
71.445566
s
Authored DB connection use time
Time spent holding authored database connections.
mean
0.000249
s
SD
9.6e-05
s
p50
0.000271
s
<
p95
0.000418
s
<
p99
0.000687
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000353
s
SD
0.000212
s
p50
0.000221
s
<
p95
0.000731
s
<
p99
0.001083
s
Conductor DB connection use time
Time spent holding conductor database connections.
mean
0.000194
s
SD
7.2e-05
s
p50
0.0002
s
<
p95
0.000303
s
<
p99
0.000475
s
Cache DB connection use time
Time spent holding cache database connections.
mean
0.00016
s
SD
0.000153
s
p50
0.000162
s
<
p95
0.000234
s
<
p99
0.000352
s
WASM DB connection use time
Time spent holding WASM database connections.
mean
0.000254
s
SD
0.000224
s
p50
0.000235
s
<
p95
0.000386
s
<
p99
0.000673
s
Peer meta store DB connection use time
Time spent holding peer meta store database connections.
mean
0.000125
s
SD
3.8e-05
s
p50
0.000127
s
<
p95
0.000186
s
<
p99
0.000252
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.001163
s
SD
0.086645
s
p50
0.001543
s
<
p95
0.263739
s
<
p99
0.32021
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000365
s
SD
0.000173
s
p50
0.000419
s
<
p95
0.000643
s
<
p99
0.00104
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.843258
s
SD
29.670454
s
p50
0.938403
s
<
p95
60.003391
s
<
p99
62.993681
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.545922
s
SD
28.70122
s
p50
0.533968
s
<
p95
60.001617
s
<
p99
60.002117
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.714963
s
SD
29.518295
s
p50
1.27274
s
<
p95
60.001403
s
<
p99
60.001865
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
6.33
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
1.99
KiB/s
<
mean
567.33
KiB/s
SD
553.17
KiB/s
<
p95
1.61
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
136
B/s
<
mean
62.08
KiB/s
SD
66.84
KiB/s
<
p95
183.73
KiB/s
Total bytes received
Total bytes received on primary network interface
count
150.16
GiB
mean
5.37
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
16.17
GiB
mean
592.07
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 879.54 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.9
%
<
mean
10.03
%
SD
13.97
%
<
p95
38.15
%
System
3.70
%
CPU percentiles
CPU usage percentiles
p50
6.52
%
p95
38.15
%
p99
73.72
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
144
hosts
mean time
0.01
s
Memory used percentage
Percentage of memory used
p5
5.68
%
<
mean
9.09
%
SD
4.47
%
<
p95
12.63
%
Memory available percentage
Percentage of available memory
p5
87.37
%
<
mean
90.91
%
SD
4.47
%
<
p95
94.32
%
Max host memory used
Maximum memory usage percentage across all hosts
max
77.39
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
2.65
%
Memory growth rate
Rate of memory growth over time
growth
879.54
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.15
MiB/s
Disk write throughput
Disk write throughput in MiB/s
35.90
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/223
hosts
Mount Point /efi-boot
0/7
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/12
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.31
5 min
0.24
15 min
0.14
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
2.3227
%
SD
6.9477
%
<
p95
6.23
%
60 second average
2.2541
%
300 second average
1.7602
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3417
%
SD
1.7195
%
<
p95
1.5
%
60 second average
0.3277
%
300 second average
0.2510
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2795
%
SD
1.4924
%
<
p95
1.26
%
60 second average
0.2657
%
300 second average
0.1985
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
7.35
%
SD
10.45
%
<
p95
20.78
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
209.67
KiB
<
mean
234.73
KiB
SD
24.82
KiB
<
p95
260.63
KiB
Holochain process threads
Number of threads in Holochain process
p5
6
threads
<
mean
26.5
threads
SD
13.21
threads
<
p95
49
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
32
file descriptors
<
mean
72.5
file descriptors
SD
19.56
file descriptors
<
p95
96
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Fri, 17 Apr 2026 01:25:56 UTC
Peer count
500
Peer count at end
15928
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.4
Wind Tunnel version
0.6.0
Run ID
remote_signals_24541634156_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
1.583892
s
SD
12.791858
s
p50
0.27353
s
<
p95
1.629211
s
<
p99
27.896072
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
326
over 27329.99690561s
mean rate
2.1578487e+06
/s
std
2.7832174134e+08
/s
p5
0
/s
p95
71668.82
/s
peak
5.2e+10
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0011
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4
over 1269.251148893s
mean rate
539.68
/s
std
8194.78
/s
p5
0
/s
p95
1271.51
/s
peak
514668.04
/s
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 27348.985806023s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
22155591
over 27348.98569569s
mean rate
8.96846925303e+09
/s
std
1.46333552921e+11
/s
p5
3.3604393e+06
/s
p95
2.16745568399e+10
/s
peak
1.2243233669e+13
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
15510915
over 27348.985806023s
mean rate
5.9493533729e+08
/s
std
1.17548101615e+10
/s
p5
0
/s
p95
9.7495745276e+08
/s
peak
1.16663403614e+12
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.003975
s
SD
0.001053
s
p50
0.004048
s
<
p95
0.005447
s
<
p99
0.006866
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.004435
s
SD
0.001171
s
p50
0.004403
s
<
p95
0.006204
s
<
p99
0.00713
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003728
s
SD
0.002676
s
p50
0.00309
s
<
p95
0.008465
s
<
p99
0.014276
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002409
s
SD
0.000647
s
p50
0.002477
s
<
p95
0.003322
s
<
p99
0.003861
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002503
s
SD
0.000642
s
p50
0.002512
s
<
p95
0.003441
s
<
p99
0.003755
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000654
s
SD
0.000612
s
p50
0.000517
s
<
p95
0.001543
s
<
p99
0.003593
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
9.6e-05
s
SD
3.1e-05
s
p50
9.9e-05
s
<
p95
0.000146
s
<
p99
0.000169
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
0.000104
s
SD
4.6e-05
s
p50
0.0001
s
<
p95
0.000184
s
<
p99
0.000252
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.7e-05
s
SD
7e-06
s
p50
1.7e-05
s
<
p95
2.8e-05
s
<
p99
3.7e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000621
s
SD
0.000743
s
p50
0.000406
s
<
p95
0.00202
s
<
p99
0.003144
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1036
over 27348.985603969s
mean rate
473522.12
/s
std
8.68652429e+06
/s
p5
161.49
/s
p95
878110.03
/s
peak
9.8742138365e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1096
over 27348.985782865s
mean rate
509438.08
/s
std
1.091245303e+07
/s
p5
92.14
/s
p95
829055.37
/s
peak
9.3755420642e+08
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000762
s
SD
0.00083
s
p50
0.000497
s
<
p95
0.00209
s
<
p99
0.005049
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
45.26
s
<
mean
448.19
s
SD
259.91
s
<
p95
855.22
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
58
over 27349.865480582s
mean rate
3.07248506e+06
/s
std
7.283961086e+07
/s
p5
0
/s
p95
3.82262219e+06
/s
peak
6.77272727273e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
51.214687
s
SD
31.245158
s
p50
32.158245
s
<
p95
83.479455
s
<
p99
142.822297
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
2.03631
s
SD
0.387108
s
p50
2.013312
s
<
p95
2.078563
s
<
p99
2.133672
s
App validation workflow duration
Time spent running the app validation workflow.
mean
1.472183
s
SD
2.070147
s
p50
0.910561
s
<
p95
5.638436
s
<
p99
9.797128
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.005661
s
SD
0.006714
s
p50
0.003756
s
<
p95
0.018995
s
<
p99
0.032572
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.012409
s
SD
0.052524
s
p50
0.005689
s
<
p95
0.041203
s
<
p99
0.186305
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.007454
s
SD
0.005212
s
p50
0.006526
s
<
p95
0.014694
s
<
p99
0.024265
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
23.551174
s
SD
21.525963
s
p50
20.05321
s
<
p95
60.225538
s
<
p99
85.21712
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
9.717216
s
SD
19.385265
s
p50
0.724363
s
<
p95
48.287258
s
<
p99
98.158609
s
Authored DB connection use time
Time spent holding authored database connections.
mean
0.000278
s
SD
0.00014
s
p50
0.000292
s
<
p95
0.000539
s
<
p99
0.000718
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001001
s
SD
0.000833
s
p50
0.000283
s
<
p95
0.002477
s
<
p99
0.00374
s
Conductor DB connection use time
Time spent holding conductor database connections.
mean
0.000235
s
SD
0.000144
s
p50
0.000227
s
<
p95
0.000408
s
<
p99
0.000883
s
Cache DB connection use time
Time spent holding cache database connections.
mean
0.000179
s
SD
9.2e-05
s
p50
0.000165
s
<
p95
0.000303
s
<
p99
0.000402
s
WASM DB connection use time
Time spent holding WASM database connections.
mean
0.000242
s
SD
0.000235
s
p50
0.000212
s
<
p95
0.000442
s
<
p99
0.001462
s
Peer meta store DB connection use time
Time spent holding peer meta store database connections.
mean
0.000169
s
SD
6.7e-05
s
p50
0.000157
s
<
p95
0.00027
s
<
p99
0.000347
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.001973
s
SD
0.118906
s
p50
0.002307
s
<
p95
0.339984
s
<
p99
0.494733
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000762
s
SD
0.00058
s
p50
0.000714
s
<
p95
0.001397
s
<
p99
0.002728
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.212976
s
SD
29.858775
s
p50
1.385516
s
<
p95
60.005035
s
<
p99
68.334433
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.131348
s
SD
28.976497
s
p50
0.540021
s
<
p95
60.001823
s
<
p99
60.008206
s
P2P received remote signals
Count of remote signals received.
total
1038
over 27348.985684961s
mean rate
508472.57
/s
std
1.131498636e+07
/s
p5
91.82
/s
p95
875822.44
/s
peak
1.39655172414e+09
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
11.32
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
198
B/s
<
mean
642.94
KiB/s
SD
888.51
KiB/s
<
p95
2.68
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
4
B/s
<
mean
73.96
KiB/s
SD
99.09
KiB/s
<
p95
252.09
KiB/s
Total bytes received
Total bytes received on primary network interface
count
225.05
GiB
mean
8.44
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
25.71
GiB
mean
987.28
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.5%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 585.40 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 3% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.75
%
<
mean
18.44
%
SD
23.8
%
<
p95
71.89
%
System
7.12
%
CPU percentiles
CPU usage percentiles
p50
9.04
%
p95
71.89
%
p99
95.53
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
207
hosts
mean time
0.04
s
Memory used percentage
Percentage of memory used
p5
7.14
%
<
mean
11.03
%
SD
4.22
%
<
p95
14.35
%
Memory available percentage
Percentage of available memory
p5
85.65
%
<
mean
88.97
%
SD
4.22
%
<
p95
92.86
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.73
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
2.65
%
Memory growth rate
Rate of memory growth over time
growth
585.40
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.11
MiB/s
Disk write throughput
Disk write throughput in MiB/s
80.22
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/208
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/7
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/10
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.74
5 min
0.57
15 min
0.36
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
3.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
9.7778
%
SD
17.3102
%
<
p95
48
%
60 second average
9.3443
%
300 second average
7.1800
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8583
%
SD
3.1416
%
<
p95
5.46
%
60 second average
0.8294
%
300 second average
0.6169
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6119
%
SD
2.4859
%
<
p95
3.58
%
60 second average
0.5883
%
300 second average
0.4354
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
7.21
%
SD
12.76
%
<
p95
29.13
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
187.66
KiB
<
mean
222.60
KiB
SD
37.67
KiB
<
p95
271.30
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
19.58
threads
SD
16.58
threads
<
p95
46
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
56.97
file descriptors
SD
28.73
file descriptors
<
p95
94
file descriptors

Two-party countersigning

↑ Back to index

This scenario tests the performance of countersigning operations. There are two roles: initiate and participate. The participants commit an entry to advertise that they are willing to participate in sessions. They listen for sessions and participate in one at a time.

Started
Fri, 17 Apr 2026 07:53:37 UTC
Peer count
3
Peer count at end
3
Behaviours
  • initiate (1 agent)
  • participate (1 agent)
Holochain version
0.6.1-rc.4
Wind Tunnel version
0.6.0
Run ID
two_party_countersigning_24541634156_1
Session accepted -- timing
The duration of the session from acceptance to completion.
Mean
mean
1.571474
s
SD
2.612133
s
Max
Highest per-partition mean latency
max mean
1.692732
s
Min
Lowest per-partition mean latency
min mean
1.450215
s
Session accepted -- success rate
The number of accepted sessions that completed per second.
Mean
mean
6.5
/s
Max
Highest per-partition mean rate
max
7
/s
Min
Lowest per-partition mean rate
min
6
/s
Session accepted -- success ratio
Fraction of accepted sessions that completed successfully (0–1).
1
Session accepted -- failures
Accepted sessions that failed to complete.
Total
3
Agents affected
1 / 2
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2
Max
Highest count in any single partition
3
Min
Lowest count in any single partition
0
Session initiated -- timing
The duration of the session from initiation to completion.
Mean
mean
1.867794
s
SD
2.626774
s
Max
Highest per-partition mean latency
max mean
1.867794
s
Min
Lowest per-partition mean latency
min mean
1.867794
s
Session initiated -- success rate
The number of initiated sessions that completed per second.
Mean
mean
12
/s
Max
Highest per-partition mean rate
max
12
/s
Min
Lowest per-partition mean rate
min
12
/s
Session initiated -- success ratio
Fraction of initiated sessions that completed successfully (0–1).
1
Session initiated -- failures
Initiated sessions that failed to complete.
Total
4
Agents affected
1 / 1
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4
Max
Highest count in any single partition
4
Min
Lowest count in any single partition
4
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.001692
s
SD
0.000374
s
p50
0.001759
s
<
p95
0.002159
s
≤
p99
0.002159
s
WASM usage: 'countersigning::accept_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
1932705
over 260.126950097s
mean rate
2.01673866e+06
/s
std
1.41172695e+06
/s
p5
65226.39
/s
p95
4.17316674e+06
/s
peak
4.17393353e+06
/s
WASM usage: 'countersigning::call_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
1064006
over 270.839062226s
mean rate
567775.56
/s
std
755280.9
/s
p5
5687.18
/s
p95
2.23614333e+06
/s
peak
2.23794131e+06
/s
WASM usage: 'countersigning::commit_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
2524343
over 260.126950097s
mean rate
1.597241e+06
/s
std
1.9105857e+06
/s
p5
196543.53
/s
p95
5.72208989e+06
/s
peak
5.72669077e+06
/s
WASM usage: 'countersigning::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 270.839062226s
Not enough time data to show a trend.
WASM usage: 'countersigning::initiator_hello'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 270.003385544s
Not enough time data to show a trend.
WASM usage: 'countersigning::list_participants'
Metered usage count of the WASM ribosome for this zome function.
total
1334730
over 270.003385544s
mean rate
4943.37
/s
std
13981.95
/s
p5
0
/s
p95
53388.1
/s
peak
53388.63
/s
WASM usage: 'countersigning::participant_hello'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 270.126994119s
Not enough time data to show a trend.
WASM usage: 'countersigning::start_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
5662935
over 270.003385544s
mean rate
20973.51
/s
std
59213.26
/s
p5
0
/s
p95
226978.93
/s
peak
226981.17
/s
WASM usage: 'countersigning_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
222005
over 280.839406348s
mean rate
126004.2
/s
std
122368.73
/s
p5
9689.31
/s
p95
410626.4
/s
peak
410956.57
/s
Zome call duration: 'countersigning::accept_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.687375
s
SD
1.608176
s
p50
1.189269
s
≤
p95
1.189269
s
<
p99
11.939906
s
Zome call duration: 'countersigning::call_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.004191
s
SD
0.000166
s
p50
0.004138
s
<
p95
0.00446
s
<
p99
0.004577
s
Zome call duration: 'countersigning::commit_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.343561
s
SD
0.306426
s
p50
0.036773
s
<
p95
0.671483
s
<
p99
1.103256
s
Zome call duration: 'countersigning::initiator_hello'
Duration of this zome call as measured by Holochain internally.
mean
0.004347
s
SD
0
s
p50
0.004347
s
≤
p95
0.004347
s
≤
p99
0.004347
s
Zome call duration: 'countersigning::list_participants'
Duration of this zome call as measured by Holochain internally.
mean
0.025241
s
SD
0.003634
s
p50
0.024846
s
<
p95
0.0384
s
≤
p99
0.0384
s
Zome call duration: 'countersigning::participant_hello'
Duration of this zome call as measured by Holochain internally.
mean
0.006614
s
SD
5e-05
s
p50
0.006664
s
≤
p95
0.006664
s
≤
p99
0.006664
s
Zome call duration: 'countersigning::start_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.257559
s
SD
0.023587
s
p50
0.25848
s
<
p95
0.348765
s
≤
p99
0.348765
s
WASM call duration: 'countersigning::accept_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.686053
s
SD
1.60814
s
p50
1.188028
s
≤
p95
1.188028
s
<
p99
11.93815
s
WASM call duration: 'countersigning::call_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002686
s
SD
0.000103
s
p50
0.002652
s
<
p95
0.002869
s
<
p99
0.003002
s
WASM call duration: 'countersigning::commit_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.342135
s
SD
0.306294
s
p50
0.035426
s
<
p95
0.669917
s
<
p99
1.101541
s
WASM call duration: 'countersigning::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008318
s
SD
0.001641
s
p50
0.007645
s
<
p95
0.010579
s
≤
p99
0.010579
s
WASM call duration: 'countersigning::initiator_hello'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002635
s
SD
0
s
p50
0.002635
s
≤
p95
0.002635
s
≤
p99
0.002635
s
WASM call duration: 'countersigning::list_participants'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.023564
s
SD
0.003683
s
p50
0.023153
s
<
p95
0.036905
s
≤
p99
0.036905
s
WASM call duration: 'countersigning::participant_hello'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005516
s
SD
0.000104
s
p50
0.00562
s
≤
p95
0.00562
s
≤
p99
0.00562
s
WASM call duration: 'countersigning::start_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
3.230962
s
SD
2.190975
s
p50
2.555071
s
<
p95
6.037198
s
≤
p99
6.037198
s
WASM call duration: 'countersigning_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00219
s
SD
0.000207
s
p50
0.00216
s
<
p95
0.00236
s
≤
p99
0.00236
s
Host function call duration: '__hc__accept_countersigning_preflight_request_1'
Duration of this host function call invoked from within WASM.
mean
0.002456
s
SD
0.000196
s
p50
0.002389
s
<
p95
0.002598
s
<
p99
0.003366
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.5e-05
s
SD
7e-06
s
p50
4.6e-05
s
<
p95
6.2e-05
s
≤
p99
6.2e-05
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
1.979218
s
SD
2.04006
s
p50
1.179336
s
<
p95
6.024658
s
<
p99
11.927878
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000646
s
SD
6.9e-05
s
p50
0.000532
s
<
p95
0.000679
s
≤
p99
0.000679
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000489
s
SD
2.4e-05
s
p50
0.000513
s
≤
p95
0.000513
s
≤
p99
0.000513
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
0.000101
s
SD
1.7e-05
s
p50
0.000102
s
<
p95
0.000131
s
<
p99
0.000195
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.334891
s
SD
0.306203
s
p50
0.027822
s
<
p95
0.662582
s
<
p99
1.094239
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.017282
s
SD
0.003741
s
p50
0.01686
s
<
p95
0.030825
s
≤
p99
0.030825
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.3e-05
s
SD
2e-06
s
p50
1.3e-05
s
<
p95
2.1e-05
s
≤
p99
2.1e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00343
s
SD
0.000585
s
p50
0.003339
s
<
p95
0.003849
s
<
p99
0.003876
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
15
over 270.838924248s
mean rate
8.59
/s
std
10.56
/s
p5
0
/s
p95
32.43
/s
peak
32.46
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000448
s
SD
7.6e-05
s
p50
0.000435
s
<
p95
0.000566
s
<
p99
0.000695
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
16.66
s
<
mean
146.66
s
SD
83.67
s
<
p95
276.67
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
176
over 280.839447238s
mean rate
4.94
/s
std
17.11
/s
p5
0
/s
p95
16.24
/s
peak
138.36
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0.628998
s
SD
0.558352
s
p50
0.882516
s
<
p95
0.948123
s
<
p99
3.855325
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
1.668537
s
SD
0.1826
s
p50
1.721311
s
<
p95
1.737705
s
≤
p99
1.737705
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005244
s
SD
0.001306
s
p50
0.005391
s
<
p95
0.005973
s
<
p99
0.011366
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.02074
s
SD
0.011658
s
p50
0.010967
s
<
p95
0.038859
s
<
p99
0.045387
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.002009
s
SD
0.000835
s
p50
0.001975
s
<
p95
0.004772
s
<
p99
0.006732
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.004865
s
SD
0.000844
s
p50
0.004303
s
<
p95
0.005896
s
<
p99
0.006197
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.08232
s
SD
0.096839
s
p50
0.104585
s
<
p95
0.144675
s
<
p99
0.777439
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.519804
s
SD
0.63806
s
p50
0.203513
s
<
p95
1.757091
s
<
p99
2.503905
s
Witnessing workflow duration
Time spent running the witnessing workflow.
mean
0.002589
s
SD
0.001622
s
p50
0.003242
s
<
p95
0.003709
s
≤
p99
0.003709
s
Authored DB connection use time
Time spent holding authored database connections.
mean
0.000398
s
SD
6.6e-05
s
p50
0.000337
s
<
p95
0.00047
s
<
p99
0.000495
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000198
s
SD
4.8e-05
s
p50
0.000192
s
<
p95
0.000269
s
<
p99
0.00027
s
Conductor DB connection use time
Time spent holding conductor database connections.
mean
0.000236
s
SD
2.1e-05
s
p50
0.000234
s
<
p95
0.000265
s
<
p99
0.000386
s
Cache DB connection use time
Time spent holding cache database connections.
mean
0.000255
s
SD
9e-05
s
p50
0.000208
s
<
p95
0.000407
s
<
p99
0.000474
s
WASM DB connection use time
Time spent holding WASM database connections.
mean
0.000235
s
SD
1.9e-05
s
p50
0.000226
s
<
p95
0.000262
s
≤
p99
0.000262
s
Peer meta store DB connection use time
Time spent holding peer meta store database connections.
mean
0.000205
s
SD
3.7e-05
s
p50
0.000187
s
<
p95
0.000242
s
≤
p99
0.000242
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000624
s
SD
0.081192
s
p50
0.001338
s
<
p95
0.238275
s
≤
p99
0.238275
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000496
s
SD
0.000218
s
p50
0.000509
s
<
p95
0.000539
s
<
p99
0.002133
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.393942
s
SD
5.520993
s
p50
0.068273
s
<
p95
11.859383
s
≤
p99
11.859383
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.376143
s
SD
0
s
p50
0.376143
s
≤
p95
0.376143
s
≤
p99
0.376143
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.292207
s
SD
0.255032
s
p50
0.027207
s
<
p95
0.661976
s
≤
p99
0.661976
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.574126
s
SD
23.365521
s
p50
0.205206
s
<
p95
60.00209
s
≤
p99
60.00209
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
1.978562
s
SD
22.909605
s
p50
0.243493
s
<
p95
60.001175
s
≤
p99
60.001175
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
5.25
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
896
B/s
<
mean
113.44
KiB/s
SD
629.93
KiB/s
≤
p95
9.04
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
196
B/s
<
mean
2.25
KiB/s
SD
8.50
KiB/s
<
p95
5.63
KiB/s
Total bytes received
Total bytes received on primary network interface
count
98.59
MiB
mean
336.53
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.95
MiB
mean
6.66
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 84.97 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.69
%
<
mean
7.13
%
SD
14.46
%
<
p95
33.4
%
System
1.89
%
CPU percentiles
CPU usage percentiles
p50
3.41
%
p95
33.40
%
p99
87.33
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
2
hosts
mean time
0.03
s
Memory used percentage
Percentage of memory used
p5
6.77
%
<
mean
8.76
%
SD
0.59
%
<
p95
9
%
Memory available percentage
Percentage of available memory
p5
91
%
<
mean
91.24
%
SD
0.59
%
<
p95
93.23
%
Max host memory used
Maximum memory usage percentage across all hosts
max
8.88
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
84.97
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
0.00
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.12
5 min
0.07
15 min
0.02
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.5291
%
SD
0.9873
%
<
p95
3.07
%
60 second average
0.5328
%
300 second average
0.3261
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.1623
%
SD
0.5265
%
<
p95
1.44
%
60 second average
0.1539
%
300 second average
0.0900
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.1203
%
SD
0.3895
%
<
p95
0.9
%
60 second average
0.1127
%
300 second average
0.0641
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.45
%
<
mean
3.9
%
SD
14.13
%
<
p95
7.4
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
231.71
KiB
<
mean
235.36
KiB
SD
30.96
KiB
<
p95
245.60
KiB
Holochain process threads
Number of threads in Holochain process
p5
10
threads
<
mean
12.56
threads
SD
1.43
threads
<
p95
14
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
45
file descriptors
<
mean
55.57
file descriptors
SD
9.8
file descriptors
<
p95
76
file descriptors