App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Tue, 28 Jul 2026 16:42:49 UTC
Peer count
1
Behaviours
  • large (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
app_install_large_30377987643_1
First install time
The time from conductor startup to completion of the first app installation attempt.
0.078626083
s
Install app timing
The time taken to install an app.
mean
0.037604
s
SD
0.006065
s
p50
0.035712
s
<
p95
0.049691
s
<
p99
0.063882
s
Holochain MetricsClick to toggle visibility.
WASM usage: 'large::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001929191s
Not enough time data to show a trend.
WASM call duration: 'large::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002069
s
SD
0.000367
s
p50
0.00197
s
<
p95
0.002484
s
<
p99
0.004141
s
Post-commit duration
Time spent executing post-commit workflows.
mean
5.6e-05
s
SD
8.7e-05
s
p50
4e-05
s
<
p95
6.7e-05
s
<
p99
0.000646
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.53
s
<
mean
148.53
s
SD
83.67
s
<
p95
278.53
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.00176906s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.000942
s
SD
4e-06
s
p50
0.000943
s
p95
0.000943
s
p99
0.000943
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.002978
s
SD
8e-06
s
p50
0.002979
s
p95
0.002979
s
p99
0.002979
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.00108
s
SD
0
s
p50
0.00108
s
p95
0.00108
s
p99
0.00108
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.002768
s
SD
1.4e-05
s
p50
0.002766
s
p95
0.002766
s
<
p99
0.002841
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.002537
s
SD
2.6e-05
s
p50
0.00254
s
p95
0.00254
s
p99
0.00254
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.002346
s
SD
2e-06
s
p50
0.002346
s
p95
0.002346
s
p99
0.002346
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
2.1e-05
s
SD
0
s
p50
2.1e-05
s
p95
2.1e-05
s
p99
2.1e-05
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.0005
s
SD
8e-05
s
p50
0.000608
s
<
p95
0.000616
s
p99
0.000616
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000143
s
SD
1e-06
s
p50
0.000143
s
p95
0.000143
s
<
p99
0.000146
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
28.37
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
8.90
KiB/s
<
mean
829.66
KiB/s
SD
1.46
MiB/s
<
p95
5.18
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
7.82
KiB/s
<
mean
380.11
KiB/s
SD
657.47
KiB/s
<
p95
1.73
MiB/s
Total bytes received
Total bytes received on primary network interface
count
234.96
MiB
mean
829.66
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
107.65
MiB
mean
380.11
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1366.87 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (62.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
14.71
%
<
mean
36.69
%
SD
27.27
%
<
p95
97.89
%
System
8.32
%
CPU percentiles
CPU usage percentiles
p50
29.25
%
p95
97.89
%
p99
97.98
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1
hosts
mean time
0.14
s
Memory used percentage
Percentage of memory used
p5
57.55
%
<
mean
59.81
%
SD
1.52
%
<
p95
62.1
%
Memory available percentage
Percentage of available memory
p5
37.9
%
<
mean
40.19
%
SD
1.52
%
<
p95
42.45
%
Max host memory used
Maximum memory usage percentage across all hosts
max
59.81
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
62.62
%
Memory growth rate
Rate of memory growth over time
growth
1366.87
MiB/s
Disk read throughput
Disk read throughput in MiB/s
4.09
MiB/s
Disk write throughput
Disk write throughput in MiB/s
61.47
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1
hosts
Mount Point /etc/hosts
0/1
hosts
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
14.19
5 min
9.58
15 min
5.11
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.03
%
<
mean
5.926
%
SD
10.5198
%
<
p95
29.65
%
60 second average
5.7117
%
300 second average
3.1867
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.044
%
SD
0.1437
%
<
p95
0.28
%
60 second average
0.0377
%
300 second average
0.0120
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.044
%
SD
0.1437
%
<
p95
0.28
%
60 second average
0.0377
%
300 second average
0.0120
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0003
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0003
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
3.52
%
<
mean
6.85
%
SD
3.86
%
<
p95
15.03
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
23.89
KiB
<
mean
1.23
MiB
SD
731.21
KiB
<
p95
1.93
MiB
Holochain process threads
Number of threads in Holochain process
p5
11
threads
<
mean
142.8
threads
SD
144.63
threads
<
p95
467
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
205.29
file descriptors
SD
297.16
file descriptors
<
p95
879
file descriptors

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Tue, 28 Jul 2026 16:44:38 UTC
Peer count
1
Behaviours
  • minimal (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
app_install_minimal_30377987643_1
First install time
The time from conductor startup to completion of the first app installation attempt.
0.117885264
s
Install app timing
The time taken to install an app.
mean
0.025293
s
SD
0.00584
s
p50
0.022701
s
<
p95
0.039282
s
<
p99
0.044737
s
Holochain MetricsClick to toggle visibility.
WASM usage: 'callback::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 0s
Not enough time data to show a trend.
WASM usage: 'callback_integrity::genesis_self_check_2'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 0s
Not enough time data to show a trend.
WASM call duration: 'callback::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000714
s
SD
0.000303
s
p50
0.000646
s
<
p95
0.000774
s
<
p99
0.002244
s
Not enough time data to show a trend.
WASM call duration: 'callback_integrity::genesis_self_check_2'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00068
s
SD
0
s
p50
0.00068
s
p95
0.00068
s
p99
0.00068
s
Not enough time data to show a trend.
Post-commit duration
Time spent executing post-commit workflows.
mean
4.3e-05
s
SD
1.4e-05
s
p50
3.7e-05
s
<
p95
7.6e-05
s
<
p99
8.5e-05
s
Not enough time data to show a trend.
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
8.57
s
mean
8.57
s
SD
0
s
p95
8.57
s
Not enough time data to show a trend.
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 0s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Not enough time data to show a trend.
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Not enough time data to show a trend.
App validation workflow duration
Time spent running the app validation workflow.
mean
0.001118
s
SD
0
s
p50
0.001118
s
p95
0.001118
s
p99
0.001118
s
Not enough time data to show a trend.
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.002373
s
SD
0
s
p50
0.002373
s
p95
0.002373
s
p99
0.002373
s
Not enough time data to show a trend.
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000734
s
SD
0
s
p50
0.000734
s
p95
0.000734
s
p99
0.000734
s
Not enough time data to show a trend.
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.003311
s
SD
0
s
p50
0.003311
s
p95
0.003311
s
p99
0.003311
s
Not enough time data to show a trend.
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.002367
s
SD
0
s
p50
0.002367
s
p95
0.002367
s
p99
0.002367
s
Not enough time data to show a trend.
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.001612
s
SD
0
s
p50
0.001612
s
p95
0.001612
s
p99
0.001612
s
Not enough time data to show a trend.
DHT DB connection use time
Time spent holding DHT database connections.
mean
5.1e-05
s
SD
0
s
p50
5.1e-05
s
p95
5.1e-05
s
p99
5.1e-05
s
Not enough time data to show a trend.
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000714
s
SD
0.000134
s
p50
0.000907
s
p95
0.000907
s
p99
0.000907
s
Not enough time data to show a trend.
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000157
s
SD
0
s
p50
0.000157
s
p95
0.000157
s
p99
0.000157
s
Not enough time data to show a trend.
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
12.73
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
71.44
KiB/s
mean
71.44
KiB/s
SD
0
B/s
p95
71.44
KiB/s
Not enough time data to show a trend.
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
31.67
KiB/s
mean
31.67
KiB/s
SD
0
B/s
p95
31.67
KiB/s
Not enough time data to show a trend.
Total bytes received
Total bytes received on primary network interface
count
714.41
KiB
mean
71.44
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
316.65
KiB
mean
31.67
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2211.90 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
18.64
%
mean
18.64
%
SD
0
%
p95
18.64
%
Not enough time data to show a trend.
System
5.92
%
CPU percentiles
CPU usage percentiles
p50
18.64
%
p95
18.64
%
p99
18.64
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
21.43
%
<
mean
28.45
%
SD
7.02
%
<
p95
35.46
%
Not enough time data to show a trend.
Memory available percentage
Percentage of available memory
p5
64.54
%
<
mean
71.55
%
SD
7.02
%
<
p95
78.57
%
Not enough time data to show a trend.
Max host memory used
Maximum memory usage percentage across all hosts
max
28.45
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
2211.90
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
11.70
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.08
5 min
1.88
15 min
1.25
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.265
%
SD
0.265
%
<
p95
0.53
%
Not enough time data to show a trend.
60 second average
0.2900
%
300 second average
0.5950
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
17.88
%
mean
17.88
%
SD
0
%
p95
17.88
%
Not enough time data to show a trend.
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
272.98
KiB
<
mean
825.01
KiB
SD
552.02
KiB
<
p95
1.34
MiB
Not enough time data to show a trend.
Holochain process threads
Number of threads in Holochain process
p5
11
threads
<
mean
154
threads
SD
143
threads
<
p95
297
threads
Not enough time data to show a trend.
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
11
file descriptors
<
mean
315
file descriptors
SD
304
file descriptors
<
p95
619
file descriptors
Not enough time data to show a trend.

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Tue, 28 Jul 2026 16:46:37 UTC
Peer count
30
Peer count at end
27
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_30377987643_1
Create rate
The average number of created records per agent.
Mean
mean
22.3
/s
Max
Highest per-partition mean rate
max
42.42
/s
Min
Lowest per-partition mean rate
min
8.69
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
187.107337
s
SD
107.904931
s
Max
Highest per-partition mean latency
max mean
683.730132
s
Min
Lowest per-partition mean latency
min mean
3.686157
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
0.75
/s
Max
Highest per-partition mean rate
max
2
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0001
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
99964
Agents affected
24 / 24
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4165
Max
Highest count in any single partition
7884
Min
Lowest count in any single partition
1596
Received count
The total number of entries received by record_lag agents, by agent.
Total
32
Agents affected
4 / 4
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
8
Max
Highest count in any single partition
15
Min
Lowest count in any single partition
5
Error count
The number of errors encountered during the scenario.
40
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.002547
s
SD
0.001088
s
p50
0.001289
s
<
p95
0.003205
s
<
p99
0.003252
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 1707.096039542s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
503437830
over 38373.460086001s
mean rate
6.7011360506e+08
/s
std
3.21068268818e+09
/s
p5
163763.69
/s
p95
2.40734995095e+09
/s
peak
5.32309116455e+10
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
7159424846
over 1790.297992653s
mean rate
9.3145096536e+08
/s
std
3.7711746036e+08
/s
p5
1.865158116e+08
/s
p95
1.40697510871e+09
/s
peak
1.44506807183e+09
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
64100064
over 38373.460086001s
mean rate
9.425192314e+07
/s
std
3.8607385466e+08
/s
p5
20851.03
/s
p95
4.2588595756e+08
/s
peak
6.77760915038e+09
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.012443
s
SD
0.019267
s
p50
0.009968
s
<
p95
0.06018
s
<
p99
0.095924
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.068745
s
SD
0.098688
s
p50
0.07376
s
<
p95
0.233603
s
<
p99
0.23658
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011586
s
SD
0.018832
s
p50
0.009005
s
<
p95
0.056204
s
<
p99
0.094213
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.067954
s
SD
0.098624
s
p50
0.072995
s
<
p95
0.232706
s
<
p99
0.235691
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001766
s
SD
0.001581
s
p50
0.001792
s
<
p95
0.005097
s
<
p99
0.008009
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002115
s
SD
0.006044
s
p50
0.000898
s
<
p95
0.015636
s
<
p99
0.029513
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.001993
s
SD
0.005702
s
p50
0.000863
s
<
p95
0.014064
s
<
p99
0.027981
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.002574
s
SD
0.001064
s
p50
0.00138
s
<
p95
0.003174
s
<
p99
0.003219
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002588
s
SD
0.002851
s
p50
0.002923
s
<
p95
0.007215
s
<
p99
0.00733
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002911
s
SD
0.003045
s
p50
0.002833
s
<
p95
0.010213
s
<
p99
0.014465
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000312
s
SD
0.000766
s
p50
0.000155
s
<
p95
0.002853
s
<
p99
0.003513
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
88.58
s
<
mean
900.88
s
SD
518.92
s
<
p95
1707.96
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
89.014746
s
SD
96.129308
s
p50
0.008197
s
<
p95
288.686684
s
<
p99
621.773162
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.091657
s
SD
0.513577
s
p50
0.238849
s
<
p95
1.213873
s
<
p99
2.448184
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003424
s
SD
0.001557
s
p50
0.002834
s
<
p95
0.006593
s
<
p99
0.008404
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.315034
s
SD
0.534514
s
p50
0.396683
s
<
p95
1.503582
s
<
p99
1.617823
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.321641
s
SD
0.213883
s
p50
0.245083
s
<
p95
0.675167
s
<
p99
0.940305
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.796128
s
SD
52.811615
s
p50
25.628345
s
<
p95
134.872164
s
<
p99
215.384573
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
46.511755
s
SD
58.350635
s
p50
0.004099
s
<
p95
179.711315
s
<
p99
219.970397
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001914
s
SD
0.002429
s
p50
0.001732
s
<
p95
0.007796
s
<
p99
0.012815
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00548
s
SD
0.014199
s
p50
0.001772
s
<
p95
0.047489
s
<
p99
0.057263
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.001682
s
SD
0.005116
s
p50
0.000632
s
<
p95
0.013083
s
<
p99
0.026306
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.757785
s
SD
28.662469
s
p50
0.581769
s
<
p95
60.005643
s
<
p99
60.103833
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.186345
s
SD
2.316414
s
p50
0.729585
s
<
p95
5.161382
s
<
p99
9.35575
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
63.30
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
79
B/s
<
mean
644.85
KiB/s
SD
552.21
KiB/s
<
p95
1.27
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
58
B/s
<
mean
571.21
KiB/s
SD
442.27
KiB/s
<
p95
1.28
MiB/s
Total bytes received
Total bytes received on primary network interface
count
30.96
GiB
mean
17.51
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
27.42
GiB
mean
15.51
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 483.73 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (37.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 71% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
29.46
%
<
mean
76.11
%
SD
24.25
%
<
p95
97.69
%
System
12.81
%
CPU percentiles
CPU usage percentiles
p50
86.85
%
p95
97.69
%
p99
98.05
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
20
hosts
mean time
0.86
s
Memory used percentage
Percentage of memory used
p5
10.82
%
<
mean
18.16
%
SD
7.82
%
<
p95
39.85
%
Memory available percentage
Percentage of available memory
p5
60.15
%
<
mean
81.84
%
SD
7.82
%
<
p95
89.18
%
Max host memory used
Maximum memory usage percentage across all hosts
max
42.31
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
37.80
%
Memory growth rate
Rate of memory growth over time
growth
483.73
MiB/s
Disk read throughput
Disk read throughput in MiB/s
4.36
MiB/s
Disk write throughput
Disk write throughput in MiB/s
209.20
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/8
hosts
Mount Point /etc/hosts
0/8
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/8
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
6.46
5 min
5.70
15 min
3.97
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
71.43
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
38.2577
%
SD
25.3833
%
<
p95
73.83
%
60 second average
37.2116
%
300 second average
32.4327
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0002
%
SD
0.0143
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
13.0056
%
SD
19.5047
%
<
p95
56.72
%
60 second average
12.6421
%
300 second average
10.9540
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1763
%
SD
7.1794
%
<
p95
19.06
%
60 second average
4.0392
%
300 second average
3.4918
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
41.18
%
SD
36.05
%
<
p95
94.96
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
229.81
KiB
<
mean
399.13
KiB
SD
184.87
KiB
<
p95
628.87
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
37.46
threads
SD
22.6
threads
<
p95
67
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
78.23
file descriptors
SD
30.5
file descriptors
<
p95
111
file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started
Tue, 28 Jul 2026 16:24:20 UTC
Peer count
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
first_call_30377987643_1
Zome call timing
The time elapsed between start of install and calling init.
mean
0.006622
s
SD
0.000735
s
p50
0.006681
s
<
p95
0.007718
s
<
p99
0.007848
s
Holochain MetricsClick to toggle visibility.
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 0s
Not enough time data to show a trend.
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 0s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 0s
Not enough time data to show a trend.
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.002346
s
SD
0
s
p50
0.002346
s
p95
0.002346
s
p99
0.002346
s
Not enough time data to show a trend.
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001959
s
SD
0.00021
s
p50
0.001918
s
<
p95
0.002463
s
<
p99
0.002577
s
Not enough time data to show a trend.
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000906
s
SD
5.4e-05
s
p50
0.0009
s
<
p95
0.00099
s
<
p99
0.00111
s
Not enough time data to show a trend.
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000638
s
SD
0
s
p50
0.000638
s
p95
0.000638
s
p99
0.000638
s
Not enough time data to show a trend.
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000174
s
SD
3.3e-05
s
p50
0.000206
s
p95
0.000206
s
p99
0.000206
s
Not enough time data to show a trend.
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000968
s
SD
0
s
p50
0.000968
s
p95
0.000968
s
p99
0.000968
s
Not enough time data to show a trend.
Post-commit duration
Time spent executing post-commit workflows.
mean
3.7e-05
s
SD
1.1e-05
s
p50
3.5e-05
s
<
p95
6.3e-05
s
<
p99
7.6e-05
s
Not enough time data to show a trend.
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
8.56
s
mean
8.56
s
SD
0
s
p95
8.56
s
Not enough time data to show a trend.
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 0s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Not enough time data to show a trend.
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Not enough time data to show a trend.
App validation workflow duration
Time spent running the app validation workflow.
mean
0.001086
s
SD
0
s
p50
0.001086
s
p95
0.001086
s
p99
0.001086
s
Not enough time data to show a trend.
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.001942
s
SD
0
s
p50
0.001942
s
p95
0.001942
s
p99
0.001942
s
Not enough time data to show a trend.
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.00067
s
SD
0
s
p50
0.00067
s
p95
0.00067
s
p99
0.00067
s
Not enough time data to show a trend.
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.002439
s
SD
0
s
p50
0.002439
s
p95
0.002439
s
p99
0.002439
s
Not enough time data to show a trend.
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.001881
s
SD
0
s
p50
0.001881
s
p95
0.001881
s
p99
0.001881
s
Not enough time data to show a trend.
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.001366
s
SD
0
s
p50
0.001366
s
p95
0.001366
s
p99
0.001366
s
Not enough time data to show a trend.
DHT DB connection use time
Time spent holding DHT database connections.
mean
5.2e-05
s
SD
0
s
p50
5.2e-05
s
p95
5.2e-05
s
p99
5.2e-05
s
Not enough time data to show a trend.
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000632
s
SD
0.00013
s
p50
0.000834
s
p95
0.000834
s
p99
0.000834
s
Not enough time data to show a trend.
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000141
s
SD
0
s
p50
0.000141
s
p95
0.000141
s
p99
0.000141
s
Not enough time data to show a trend.
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
6.03
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
64.69
KiB/s
mean
64.69
KiB/s
SD
0
B/s
p95
64.69
KiB/s
Not enough time data to show a trend.
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
26.34
KiB/s
mean
26.34
KiB/s
SD
0
B/s
p95
26.34
KiB/s
Not enough time data to show a trend.
Total bytes received
Total bytes received on primary network interface
count
646.95
KiB
mean
64.69
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
263.38
KiB
mean
26.34
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
11.31
%
mean
11.31
%
SD
0
%
p95
11.31
%
Not enough time data to show a trend.
System
5.28
%
CPU percentiles
CPU usage percentiles
p50
11.31
%
p95
11.31
%
p99
11.31
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
10.1
%
<
mean
16.69
%
SD
6.59
%
<
p95
23.27
%
Not enough time data to show a trend.
Memory available percentage
Percentage of available memory
p5
76.73
%
<
mean
83.31
%
SD
6.59
%
<
p95
89.9
%
Not enough time data to show a trend.
Max host memory used
Maximum memory usage percentage across all hosts
max
16.69
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
-2076.70
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.82
MiB/s
Disk write throughput
Disk write throughput in MiB/s
15.72
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.44
5 min
0.12
15 min
0.03
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.28
%
<
mean
0.415
%
SD
0.135
%
<
p95
0.55
%
Not enough time data to show a trend.
60 second average
0.1150
%
300 second average
0.0200
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
mean
0
%
SD
0
%
p95
0
%
Not enough time data to show a trend.
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
880.51
KiB
mean
880.51
KiB
SD
0
B
p95
880.51
KiB
Not enough time data to show a trend.
Holochain process threads
Number of threads in Holochain process
p5
177
threads
mean
177
threads
SD
0
threads
p95
177
threads
Not enough time data to show a trend.
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
373
file descriptors
mean
373
file descriptors
SD
0
file descriptors
p95
373
file descriptors
Not enough time data to show a trend.

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Tue, 28 Jul 2026 19:53:36 UTC
Peer count
30
Peer count at end
27
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_30377987643_1
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
41.98
/s
Max
Highest per-partition mean rate
max
48.61
/s
Min
Lowest per-partition mean rate
min
35.34
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
718.099751
s
SD
7.91581
s
Max
Highest per-partition mean latency
max mean
1385.560373
s
Min
Lowest per-partition mean latency
min mean
50.639129
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0089
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
7724
Agents affected
2 / 2
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3862
Max
Highest count in any single partition
4410
Min
Lowest count in any single partition
3314
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
1715
Agents affected
2 / 25
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
69
Max
Highest count in any single partition
872
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
19
<
mean
23.16
SD
2.84
<
p95
24
zero-arc
p5
0
<
mean
3.81
SD
5.98
<
p95
17
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
313
Agents affected
23 / 25
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
13
Max
Highest count in any single partition
14
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
316
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.021146
s
SD
0.049252
s
p50
0.077474
s
<
p95
0.143723
s
<
p99
0.202439
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4
over 1730.016674636s
mean rate
488.05
/s
std
1912.85
/s
p5
0
/s
p95
3490.48
/s
peak
13751.77
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
305522616
over 890.01133311s
mean rate
5.11427609e+06
/s
std
2.84479889e+06
/s
p5
511038.13
/s
p95
9.11833166e+06
/s
peak
1.038459175e+07
/s
WASM usage: 'timed_and_validated::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
4140668484
over 37461.205574383s
mean rate
6.7411729701e+08
/s
std
1.11393985211e+09
/s
p5
2.33673962e+06
/s
p95
2.8840164402e+09
/s
peak
7.35342889162e+09
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
430640794
over 37461.277677009s
mean rate
1.5548690607e+08
/s
std
4.161941298e+08
/s
p5
4780.57
/s
p95
1.16444972194e+09
/s
peak
2.44542738127e+09
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 37461.205574383s
Not enough time data to show a trend.
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006684
s
SD
0.001259
s
p50
0.005777
s
<
p95
0.009002
s
<
p99
0.009126
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.010681
s
SD
47.241187
s
p50
0.004859
s
<
p95
0.170935
s
<
p99
292.514661
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005989
s
SD
0.001219
s
p50
0.005089
s
<
p95
0.008234
s
<
p99
0.008354
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.101635
s
SD
34.107236
s
p50
0.199997
s
<
p95
89.169583
s
<
p99
192.72103
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000962
s
SD
0.0007
s
p50
0.000849
s
<
p95
0.003168
s
<
p99
0.003263
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008046
s
SD
0.002583
s
p50
0.001068
s
<
p95
0.007306
s
<
p99
0.01557
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000399
s
SD
0.000134
s
p50
0.000301
s
<
p95
0.000635
s
<
p99
0.000646
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000375
s
SD
0.000138
s
p50
0.000267
s
<
p95
0.000617
s
<
p99
0.000631
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.023469
s
SD
0.054327
s
p50
0.130925
s
<
p95
0.25417
s
<
p99
0.277264
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.011513
s
SD
1.175614
s
p50
0.022646
s
<
p95
3.008227
s
<
p99
6.646988
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.016799
s
SD
0.009082
s
p50
0.013425
s
<
p95
0.031382
s
<
p99
0.034818
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001513
s
SD
0.001064
s
p50
0.001314
s
<
p95
0.004587
s
<
p99
0.004852
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000104
s
SD
0.000132
s
p50
9.6e-05
s
<
p95
0.000443
s
<
p99
0.000601
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.96
s
<
mean
447.1
s
SD
256.92
s
<
p95
846.98
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 37461.27761346s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
57.442282
s
SD
14.143106
s
p50
0
s
<
p95
28.732194
s
<
p99
77.522193
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
4.368289
s
SD
1.700938
s
p50
0.001428
s
<
p95
3.856106
s
<
p99
9.718502
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003595
s
SD
0.001491
s
p50
0.003258
s
<
p95
0.005671
s
<
p99
0.008301
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.03406
s
SD
0.008873
s
p50
0.001823
s
<
p95
0.023967
s
<
p99
0.05352
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.141545
s
SD
0.036803
s
p50
0.018616
s
<
p95
0.100318
s
<
p99
0.205584
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
12.112688
s
SD
6.528025
s
p50
0.003269
s
<
p95
13.061251
s
<
p99
39.90104
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
4.513486
s
SD
2.760443
s
p50
0.002451
s
<
p95
4.056949
s
<
p99
17.374265
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000865
s
SD
0.002028
s
p50
0.000101
s
<
p95
0.005708
s
<
p99
0.010562
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008427
s
SD
0.007649
s
p50
0.001736
s
<
p95
0.013902
s
<
p99
0.050096
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000244
s
SD
0.000173
s
p50
0.000327
s
<
p95
0.000817
s
<
p99
0.000952
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.445133
s
SD
29.651152
s
p50
0.347466
s
<
p95
60.001477
s
<
p99
60.001662
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
11.483565
s
SD
19.435803
s
p50
7.443787
s
<
p95
60.001185
s
<
p99
60.00154
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.917773
s
SD
18.139015
s
p50
2.636417
s
<
p95
60.001271
s
p99
60.001271
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.93
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
64
B/s
<
mean
64.55
KiB/s
SD
266.20
KiB/s
<
p95
195.71
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
49
B/s
<
mean
36.91
KiB/s
SD
199.14
KiB/s
<
p95
102.53
KiB/s
Total bytes received
Total bytes received on primary network interface
count
1.49
GiB
mean
1.69
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
871.68
MiB
mean
991.77
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.1%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 507.74 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 4% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.78
%
<
mean
13.21
%
SD
20.83
%
<
p95
57.98
%
System
5.28
%
CPU percentiles
CPU usage percentiles
p50
3.24
%
p95
57.98
%
p99
95.14
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
8
hosts
mean time
0.11
s
Memory used percentage
Percentage of memory used
p5
7.02
%
<
mean
15.9
%
SD
6.49
%
<
p95
36.72
%
Memory available percentage
Percentage of available memory
p5
63.28
%
<
mean
84.1
%
SD
6.49
%
<
p95
92.98
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.13
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.09
%
Memory growth rate
Rate of memory growth over time
growth
507.74
MiB/s
Disk read throughput
Disk read throughput in MiB/s
4.81
MiB/s
Disk write throughput
Disk write throughput in MiB/s
48.32
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/5
hosts
Mount Point /efi-boot
0/4
hosts
Mount Point /etc/hostname
0/7
hosts
Mount Point /etc/hosts
0/7
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/7
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.99
5 min
1.20
15 min
1.54
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
3.70
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
2.6087
%
SD
11.0488
%
<
p95
7.23
%
60 second average
2.7350
%
300 second average
3.9152
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.1749
%
SD
7.9566
%
<
p95
24.2
%
60 second average
3.3510
%
300 second average
3.8379
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.1034
%
SD
7.8299
%
<
p95
23.76
%
60 second average
3.1849
%
300 second average
3.2147
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.19
%
<
mean
8.6
%
SD
18.97
%
<
p95
51.66
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
360.70
KiB
<
mean
448.92
KiB
SD
112.96
KiB
<
p95
550.49
KiB
Holochain process threads
Number of threads in Holochain process
p5
11
threads
<
mean
26.94
threads
SD
10.54
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
60.1
file descriptors
SD
17.76
file descriptors
<
p95
81
file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started
Tue, 28 Jul 2026 16:26:53 UTC
Peer count
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
local_signals_30377987643_1
Send timing
The time taken to emit a batch of 10,000 signals.
mean
1.844297
s
SD
0.927731
s
p50
2.268437
s
<
p95
2.950218
s
p99
2.950218
s
Receive timing
The time taken to receive the completed batch, to the nearest 250ms.
mean
27.925699
s
SD
6.226068
s
p50
30.001176
s
<
p95
30.002
s
p99
30.002
s
Success ratio
The proportion of signals successfully received in a batch.
min
0.7328000068664551
<
mean
0.8335
SD
0.081
<
max
0.949999988079071
Holochain MetricsClick to toggle visibility.
WASM usage: 'signal::emit_10k_signals'
Metered usage count of the WASM ribosome for this zome function.
total
277609437
over 280.003045709s
mean rate
991450.75
/s
std
1.44054454e+06
/s
p5
0
/s
p95
3.08453102e+06
/s
peak
3.08453548e+06
/s
Zome call duration: 'signal::emit_10k_signals'
Duration of this zome call as measured by Holochain internally.
mean
0.050423
s
SD
0.000477
s
p50
0.050571
s
<
p95
0.051353
s
p99
0.051353
s
WASM call duration: 'signal::emit_10k_signals'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.049889
s
SD
0.000499
s
p50
0.049973
s
<
p95
0.050873
s
p99
0.050873
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
4e-06
s
SD
0
s
p50
4e-06
s
p95
4e-06
s
p99
4e-06
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
90000
over 280.003068399s
mean rate
321.43
/s
std
467.02
/s
p5
0
/s
p95
999.99
/s
peak
999.99
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
7.9e-05
s
SD
0
s
p50
7.9e-05
s
p95
7.9e-05
s
p99
7.9e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.64
s
<
mean
148.64
s
SD
83.67
s
<
p95
278.64
s
Dropped WebSocket signals
Count of signals dropped from the app WebSocket due to channel overload.
total
1393
over 280.002931108s
mean rate
4.97
/s
std
10.2
/s
p5
0
/s
p95
29.2
/s
peak
39.4
/s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.002949605s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.001354
s
SD
0
s
p50
0.001354
s
p95
0.001354
s
p99
0.001354
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003429
s
SD
0
s
p50
0.003429
s
p95
0.003429
s
p99
0.003429
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000757
s
SD
0
s
p50
0.000757
s
p95
0.000757
s
p99
0.000757
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.004664
s
SD
0.000552
s
p50
0.005116
s
<
p95
0.0054
s
p99
0.0054
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.003134
s
SD
0
s
p50
0.003134
s
p95
0.003134
s
p99
0.003134
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.002991
s
SD
0
s
p50
0.002991
s
p95
0.002991
s
p99
0.002991
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000159
s
SD
2.3e-05
s
p50
0.000161
s
<
p95
0.000171
s
<
p99
0.000172
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000819
s
SD
9.8e-05
s
p50
0.000953
s
p95
0.000953
s
p99
0.000953
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000151
s
SD
0
s
p50
0.000151
s
p95
0.000151
s
p99
0.000151
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
1.59
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
279
B/s
<
mean
441.32
KiB/s
SD
1.39
MiB/s
<
p95
4.70
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
146
B/s
<
mean
18.11
KiB/s
SD
28.47
KiB/s
<
p95
99.28
KiB/s
Total bytes received
Total bytes received on primary network interface
count
124.98
MiB
mean
441.32
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.13
MiB
mean
18.11
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1312.23 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.46
%
<
mean
4.12
%
SD
4.11
%
<
p95
10.87
%
System
2.53
%
CPU percentiles
CPU usage percentiles
p50
2.73
%
p95
10.87
%
p99
17.52
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
12.89
%
<
mean
18.76
%
SD
5.22
%
<
p95
27.85
%
Memory available percentage
Percentage of available memory
p5
72.15
%
<
mean
81.24
%
SD
5.22
%
<
p95
87.11
%
Max host memory used
Maximum memory usage percentage across all hosts
max
18.76
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
1312.23
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
3.06
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.45
5 min
0.70
15 min
0.28
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.03
%
<
mean
0.8357
%
SD
1.3895
%
<
p95
2.69
%
60 second average
0.5540
%
300 second average
0.1943
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.21
%
<
mean
0.67
%
SD
0.49
%
<
p95
1.45
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
233.79
KiB
<
mean
248.06
KiB
SD
10.30
KiB
<
p95
262.18
KiB
Holochain process threads
Number of threads in Holochain process
p5
32
threads
<
mean
38.97
threads
SD
5.38
threads
<
p95
46
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
61
file descriptors
<
mean
73.07
file descriptors
SD
9.31
file descriptors
<
p95
86
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Tue, 28 Jul 2026 20:09:17 UTC
Peer count
30
Peer count at end
26
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_30377987643_1
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
109085
Agents affected
19 / 19
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
5741
Max
Highest count in any single partition
10689
Min
Lowest count in any single partition
1615
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2563.25
Max
Highest chain head value observed for any single write agent.
4314
Write agents observed
4
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
0.726309
s
SD
1.352477
s
Max
Highest per-partition mean latency
max mean
3.996339
s
Min
Lowest per-partition mean latency
min mean
-0.11236
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
15.42
/s
Max
Highest per-partition mean rate
max
27.3
/s
Min
Lowest per-partition mean rate
min
1
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
0.20685
s
SD
0.076587
s
Max
Highest per-partition mean latency
max mean
0.231445
s
Min
Lowest per-partition mean latency
min mean
0.190094
s
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
0
<
mean
21.93
SD
6.48
<
p95
24
  • behaviour: zero_read
p5
0
<
mean
13.04
SD
6.48
<
p95
23
  • behaviour: zero_write
p5
11
<
mean
14.96
SD
2.97
<
p95
20
Final error count
The total number of errors accumulated across all agents over the run.
1
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.047846
s
SD
0.123441
s
p50
0.168542
s
<
p95
0.416864
s
p99
0.416864
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 719.452040979s
Not enough time data to show a trend.
WASM usage: 'agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 1893.08776039s
Not enough time data to show a trend.
WASM usage: 'agent_activity::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
174745946
over 1893.08776039s
mean rate
2.64326333932e+09
/s
std
1.94234269488e+10
/s
p5
160542.47
/s
p95
3.70044051749e+09
/s
peak
2.83908516706e+11
/s
WASM usage: 'agent_activity::get_agent_activity_full'
Metered usage count of the WASM ribosome for this zome function.
total
3748160559
over 886.767613489s
mean rate
4.7610613456e+08
/s
std
5.2070960848e+08
/s
p5
3.15134097e+06
/s
p95
1.36453162608e+09
/s
peak
1.4148275473e+09
/s
WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
4053157980
over 33081.187033128s
mean rate
1.32344267e+06
/s
std
1.9897304e+06
/s
p5
0
/s
p95
5.48699964e+06
/s
peak
7.36721834e+06
/s
WASM usage: 'agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 37463.592653196s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006633
s
SD
0.006853
s
p50
0.004705
s
<
p95
0.034952
s
p99
0.034952
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.004135
s
SD
0.004184
s
p50
0.004009
s
<
p95
0.017618
s
<
p99
0.019736
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.1742
s
SD
0.029056
s
p50
0.165652
s
<
p95
0.202863
s
<
p99
0.222785
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.00302
s
SD
0.162421
s
p50
0.276393
s
<
p95
0.421959
s
p99
0.421959
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005693
s
SD
0.006515
s
p50
0.004124
s
<
p95
0.032764
s
p99
0.032764
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003505
s
SD
0.003702
s
p50
0.003395
s
<
p95
0.015364
s
<
p99
0.016777
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.173063
s
SD
0.029099
s
p50
0.164674
s
<
p95
0.201891
s
<
p99
0.221848
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002485
s
SD
0.1624
s
p50
0.275253
s
<
p95
0.421102
s
p99
0.421102
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001026
s
SD
0.00074
s
p50
0.001384
s
<
p95
0.003509
s
<
p99
0.003864
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
3.2e-05
s
SD
1.9e-05
s
p50
2.4e-05
s
<
p95
7.4e-05
s
p99
7.4e-05
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000532
s
SD
0.001477
s
p50
0.000368
s
<
p95
0.004719
s
<
p99
0.007332
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.001168
s
SD
0.003285
s
p50
0.000377
s
<
p95
0.015217
s
p99
0.015217
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.170416
s
SD
0.029055
s
p50
0.162081
s
<
p95
0.19934
s
<
p99
0.217755
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000353
s
SD
0.162431
s
p50
0.27038
s
<
p95
0.416933
s
p99
0.416933
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
5e-06
s
SD
3e-06
s
p50
8e-06
s
<
p95
1.4e-05
s
p99
1.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001593
s
SD
0.001489
s
p50
0.00208
s
<
p95
0.006458
s
<
p99
0.008002
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000157
s
SD
0.000303
s
p50
0.000143
s
<
p95
0.000971
s
<
p99
0.001581
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.94
s
<
mean
448.75
s
SD
257.9
s
<
p95
848.42
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 37463.592547408s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
6.924819
s
SD
6.766669
s
p50
0
s
<
p95
0.025598
s
<
p99
39.816967
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.022322
s
SD
0.111774
s
p50
0.00178
s
<
p95
0.216636
s
<
p99
0.729426
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.006156
s
SD
0.007184
s
p50
0.00435
s
<
p95
0.020401
s
<
p99
0.037013
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.064449
s
SD
0.409963
s
p50
0.001909
s
<
p95
1.175163
s
<
p99
1.450324
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.08889
s
SD
0.068883
s
p50
0.055824
s
<
p95
0.214865
s
<
p99
0.289295
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
2.396434
s
SD
17.151539
s
p50
0.004432
s
<
p95
20.847993
s
<
p99
132.263246
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.023822
s
SD
2.603658
s
p50
0.003054
s
<
p95
0.113757
s
<
p99
15.878149
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002354
s
SD
0.001742
s
p50
0.001568
s
<
p95
0.004607
s
<
p99
0.00908
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.002315
s
SD
0.002853
s
p50
0.001218
s
<
p95
0.006136
s
<
p99
0.018336
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000496
s
SD
0.001238
s
p50
0.00032
s
<
p95
0.00417
s
<
p99
0.006891
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
8.914047
s
SD
29.100243
s
p50
0.955994
s
<
p95
60.017976
s
p99
60.017976
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.08182
s
SD
0.146839
s
p50
0.267216
s
<
p95
0.414208
s
p99
0.414208
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.169814
s
SD
0.029083
s
p50
0.161323
s
<
p95
0.198836
s
<
p99
0.217236
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.159522
s
SD
0.092229
s
p50
0.05604
s
<
p95
0.262435
s
p99
0.262435
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
42.31
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
66
B/s
<
mean
302.17
KiB/s
SD
388.17
KiB/s
<
p95
809.31
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
37
B/s
<
mean
276.58
KiB/s
SD
245.24
KiB/s
<
p95
709.86
KiB/s
Total bytes received
Total bytes received on primary network interface
count
6.97
GiB
mean
7.85
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
6.38
GiB
mean
7.18
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 303.31 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 41% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.39
%
<
mean
52
%
SD
36.44
%
<
p95
95.31
%
System
9.69
%
CPU percentiles
CPU usage percentiles
p50
58.60
%
p95
95.31
%
p99
98.25
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
11
hosts
mean time
0.88
s
Memory used percentage
Percentage of memory used
p5
6.32
%
<
mean
14.13
%
SD
6.04
%
<
p95
31.9
%
Memory available percentage
Percentage of available memory
p5
68.1
%
<
mean
85.87
%
SD
6.04
%
<
p95
93.68
%
Max host memory used
Maximum memory usage percentage across all hosts
max
35.56
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.09
%
Memory growth rate
Rate of memory growth over time
growth
303.31
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
102.67
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/5
hosts
Mount Point /efi-boot
0/4
hosts
Mount Point /etc/hostname
0/7
hosts
Mount Point /etc/hosts
0/7
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/7
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.45
5 min
2.77
15 min
2.11
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
40.74
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
21.262
%
SD
25.1927
%
<
p95
65.26
%
60 second average
20.1663
%
300 second average
15.6104
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.8693
%
SD
8.5224
%
<
p95
26.01
%
60 second average
3.8266
%
300 second average
3.6457
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.7885
%
SD
6.9456
%
<
p95
16.93
%
60 second average
2.8095
%
300 second average
2.8773
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.25
%
<
mean
44.82
%
SD
35.33
%
<
p95
93.73
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
222.56
KiB
<
mean
314.31
KiB
SD
90.50
KiB
<
p95
455.33
KiB
Holochain process threads
Number of threads in Holochain process
p5
13
threads
<
mean
32.79
threads
SD
11.06
threads
<
p95
52
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
46
file descriptors
<
mean
73.16
file descriptors
SD
19.82
file descriptors
<
p95
97
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Tue, 28 Jul 2026 20:25:03 UTC
Peer count
30
Peer count at end
25
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_30377987643_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1305.75
Max
Highest chain head value observed for any single write agent.
1677
Write agents observed
4
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
156.151889
s
SD
31.468369
s
Max
Highest per-partition mean latency
max mean
1004.204333
s
Min
Lowest per-partition mean latency
min mean
8.496517
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
2.06
/s
Max
Highest per-partition mean rate
max
2.39
/s
Min
Lowest per-partition mean rate
min
1.77
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.411362
s
SD
0.335126
s
Max
Highest per-partition mean latency
max mean
0.576954
s
Min
Lowest per-partition mean latency
min mean
0.292285
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
21265
Agents affected
8 / 8
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2658
Max
Highest count in any single partition
8320
Min
Lowest count in any single partition
227
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
24
mean
23.9
SD
0.99
<
p95
24
  • behaviour: zero_must_get_agent_activity
p5
0
<
mean
12.04
SD
5.51
<
p95
21
  • behaviour: zero_write
p5
12
<
mean
16.68
SD
2.9
<
p95
21
Final error count
The total number of all types of error accumulated over the run by all agents.
36097
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.045575
s
SD
0.158165
s
p50
0.120848
s
<
p95
0.503524
s
<
p99
0.510006
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
2
over 705.904808079s
mean rate
0.54
/s
std
0.72
/s
p5
0
/s
p95
2.09
/s
peak
2.09
/s
WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 894.386617149s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'
Metered usage count of the WASM ribosome for this zome function.
total
2645200522
over 894.386617149s
mean rate
2.4541366174e+08
/s
std
5.9414849009e+08
/s
p5
0
/s
p95
1.27488489787e+09
/s
peak
4.41873638524e+09
/s
WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
11364398
over 1885.945723088s
mean rate
6.010832575e+07
/s
std
5.333954422e+07
/s
p5
0
/s
p95
1.5249846646e+08
/s
peak
1.6501653501e+08
/s
WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Metered usage count of the WASM ribosome for this zome function.
total
754248295
over 1885.945723088s
mean rate
2.761356439e+07
/s
std
3.141134075e+07
/s
p5
382311.07
/s
p95
1.0545753468e+08
/s
peak
1.3349430697e+08
/s
WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
84056
over 1885.945723088s
mean rate
9779.44
/s
std
10013.93
/s
p5
0
/s
p95
24945.2
/s
peak
24952.06
/s
WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 37461.622128554s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 37461.622128554s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007444
s
SD
0.002875
s
p50
0.007756
s
<
p95
0.013116
s
p99
0.013116
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.059794
s
SD
0.068629
s
p50
0.041296
s
<
p95
0.177728
s
<
p99
0.418728
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.005655
s
SD
0.002441
s
p50
0.005554
s
<
p95
0.010576
s
<
p99
0.010642
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.172417
s
SD
0.074149
s
p50
0.15339
s
<
p95
0.25555
s
<
p99
0.278693
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.334542
s
SD
0.162267
s
p50
0.394769
s
<
p95
0.515793
s
p99
0.515793
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006202
s
SD
0.002419
s
p50
0.006336
s
<
p95
0.010672
s
p99
0.010672
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.058965
s
SD
0.068011
s
p50
0.04054
s
<
p95
0.174182
s
<
p99
0.415217
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004703
s
SD
0.002111
s
p50
0.004597
s
<
p95
0.00895
s
<
p99
0.009002
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.172901
s
SD
0.118967
s
p50
0.196656
s
<
p95
0.478096
s
<
p99
0.519055
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.333542
s
SD
0.162193
s
p50
0.393721
s
<
p95
0.514802
s
p99
0.514802
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.0011
s
SD
0.000693
s
p50
0.000983
s
<
p95
0.003137
s
<
p99
0.003197
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009426
s
SD
0.022683
s
p50
0.003144
s
<
p95
0.073325
s
<
p99
0.097404
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
2e-05
s
SD
2.5e-05
s
p50
2.1e-05
s
<
p95
0.000102
s
<
p99
0.000117
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000389
s
SD
0.000397
s
p50
0.000337
s
<
p95
0.000778
s
<
p99
0.002089
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000448
s
SD
0.000475
s
p50
0.000442
s
<
p95
0.001202
s
<
p99
0.003045
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.168112
s
SD
0.153306
s
p50
0.270694
s
<
p95
0.510086
s
p99
0.510086
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.030284
s
SD
0.081327
s
p50
0.024995
s
<
p95
0.223209
s
<
p99
0.28625
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.024863
s
SD
0.044424
s
p50
0.009764
s
<
p95
0.059104
s
<
p99
0.270138
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.4e-05
s
SD
4e-06
s
p50
1.3e-05
s
<
p95
2.1e-05
s
p99
2.1e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001769
s
SD
0.001266
s
p50
0.002074
s
<
p95
0.004986
s
<
p99
0.006735
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000133
s
SD
8.1e-05
s
p50
0.000113
s
<
p95
0.000217
s
<
p99
0.000501
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.94
s
<
mean
449.56
s
SD
258.44
s
<
p95
854.7
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
103.711443
s
SD
55.364111
s
p50
0
s
<
p95
162.1451
s
<
p99
266.838254
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.366366
s
SD
1.406912
s
p50
0.002157
s
<
p95
3.486117
s
<
p99
6.638478
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003707
s
SD
0.001139
s
p50
0.003594
s
<
p95
0.005148
s
<
p99
0.006222
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.201254
s
SD
0.262859
s
p50
0.001448
s
<
p95
0.515058
s
<
p99
1.479038
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.087055
s
SD
0.133381
s
p50
0.014009
s
<
p95
0.265582
s
<
p99
0.777991
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
1.79369
s
SD
13.195429
s
p50
0.005064
s
<
p95
34.290016
s
<
p99
62.202126
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
25.650741
s
SD
26.944189
s
p50
0.004231
s
<
p95
57.38558
s
<
p99
143.830135
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002255
s
SD
0.001841
s
p50
0.000241
s
<
p95
0.004364
s
<
p99
0.00966
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00605
s
SD
0.005871
s
p50
0.001716
s
<
p95
0.018372
s
<
p99
0.026643
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000351
s
SD
0.000282
s
p50
0.000325
s
<
p95
0.000687
s
<
p99
0.001676
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.517724
s
SD
29.537244
s
p50
0.152254
s
<
p95
60.001433
s
<
p99
60.00153
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.129574
s
SD
24.436729
s
p50
0.266925
s
<
p95
60.001953
s
<
p99
60.00207
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.058471
s
SD
24.174836
s
p50
0.173218
s
<
p95
60.001926
s
<
p99
60.003616
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.151366
s
SD
2.524333
s
p50
0.189416
s
<
p95
5.532939
s
<
p99
5.6202
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
16.65
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
65
B/s
<
mean
75.77
KiB/s
SD
123.59
KiB/s
<
p95
214.19
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
50
B/s
<
mean
71.62
KiB/s
SD
93.88
KiB/s
<
p95
233.58
KiB/s
Total bytes received
Total bytes received on primary network interface
count
1.75
GiB
mean
1.97
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.66
GiB
mean
1.86
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.5%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 309.86 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 4% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.01
%
<
mean
23.27
%
SD
25.92
%
<
p95
83.09
%
System
6.62
%
CPU percentiles
CPU usage percentiles
p50
10.18
%
p95
83.09
%
p99
93.52
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
6
hosts
mean time
0.26
s
Memory used percentage
Percentage of memory used
p5
7.52
%
<
mean
15.71
%
SD
6.09
%
<
p95
31.26
%
Memory available percentage
Percentage of available memory
p5
68.74
%
<
mean
84.29
%
SD
6.09
%
<
p95
92.48
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.34
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.09
%
Memory growth rate
Rate of memory growth over time
growth
309.86
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
57.52
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/5
hosts
Mount Point /efi-boot
0/4
hosts
Mount Point /etc/hostname
0/7
hosts
Mount Point /etc/hosts
0/7
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/7
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.51
5 min
1.94
15 min
2.16
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
3.70
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
3.3879
%
SD
8.0298
%
<
p95
16.01
%
60 second average
3.8895
%
300 second average
7.7608
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.3107
%
SD
9.9025
%
<
p95
29.89
%
60 second average
4.2651
%
300 second average
4.1974
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.5765
%
SD
5.864
%
<
p95
15.94
%
60 second average
2.6382
%
300 second average
2.7880
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.25
%
<
mean
16.24
%
SD
22.63
%
<
p95
68.34
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
336.23
KiB
<
mean
425.74
KiB
SD
85.08
KiB
<
p95
583.54
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
31.54
threads
SD
9.56
threads
<
p95
52
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
56
file descriptors
<
mean
70.16
file descriptors
SD
17.22
file descriptors
<
p95
95
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Tue, 28 Jul 2026 16:28:49 UTC
Peer count
10
Peer count at end
8
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_30377987643_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
0.056751
s
SD
0.079445
s
Max
Highest per-partition mean latency
max mean
0.079427
s
Min
Lowest per-partition mean latency
min mean
0.024867
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.112199
s
SD
0.105404
s
Max
Highest per-partition mean latency
max mean
0.151937
s
Min
Lowest per-partition mean latency
min mean
0.055819
s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
194903118
over 892.208409733s
mean rate
7.0190068847e+08
/s
std
1.25401109965e+09
/s
p5
553625.92
/s
p95
3.7437535826e+09
/s
peak
5.07246891685e+09
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
53722906
over 892.282092321s
mean rate
4.06732952e+06
/s
std
6.0578068e+06
/s
p5
7551.85
/s
p95
1.91157181e+07
/s
peak
2.398119658e+07
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 892.282092321s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.112083
s
SD
0.072543
s
p50
0.15284
s
<
p95
0.209132
s
<
p99
0.433463
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.002977
s
SD
0.001246
s
p50
0.00279
s
<
p95
0.004565
s
<
p99
0.004908
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.11115
s
SD
0.072459
s
p50
0.151171
s
<
p95
0.207719
s
<
p99
0.432688
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001973
s
SD
0.0008
s
p50
0.001897
s
<
p95
0.003059
s
<
p99
0.003076
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002249
s
SD
0.001177
s
p50
0.001839
s
<
p95
0.004575
s
p99
0.004575
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.109086
s
SD
0.072259
s
p50
0.148554
s
<
p95
0.204205
s
<
p99
0.43098
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000413
s
SD
0.000306
s
p50
0.000315
s
<
p95
0.00121
s
p99
0.00121
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
6e-06
s
p50
8e-06
s
<
p95
2.7e-05
s
<
p99
2.8e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000223
s
SD
0.000314
s
p50
0.000224
s
<
p95
0.001487
s
p99
0.001487
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000162
s
SD
0.000141
s
p50
0.000108
s
<
p95
0.000481
s
p99
0.000481
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.97
s
<
mean
446.49
s
SD
256.6
s
<
p95
846.98
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
56
over 892.282003282s
mean rate
14.12
/s
std
94.43
/s
p5
0
/s
p95
41.5
/s
peak
1965.35
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
1.976166
s
SD
1.599229
s
p50
0.878161
s
<
p95
4.616683
s
<
p99
5.951198
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.002672
s
SD
0.002065
s
p50
0.00222
s
<
p95
0.007665
s
<
p99
0.008461
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.005064
s
SD
0.00411
s
p50
0.002642
s
<
p95
0.012535
s
p99
0.012535
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.0716
s
SD
0.034159
s
p50
0.086318
s
<
p95
0.135329
s
<
p99
0.168985
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.006308
s
SD
0.004251
s
p50
0.006267
s
<
p95
0.013805
s
<
p99
0.017438
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.404069
s
SD
0.545493
s
p50
0.185387
s
<
p95
1.700105
s
<
p99
2.145434
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.147914
s
SD
0.084507
s
p50
0.153918
s
<
p95
0.305806
s
<
p99
0.423351
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000128
s
SD
7e-05
s
p50
0.000125
s
<
p95
0.000274
s
<
p99
0.000371
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00134
s
SD
0.00121
s
p50
0.001047
s
<
p95
0.004485
s
p99
0.004485
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000364
s
SD
0.000203
s
p50
0.000389
s
<
p95
0.000767
s
<
p99
0.000906
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.086742
s
SD
0.068615
s
p50
0.074647
s
<
p95
0.254621
s
<
p99
0.368306
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.136103
s
SD
0.052535
s
p50
0.172005
s
<
p95
0.189899
s
<
p99
0.217574
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.108618
s
SD
0.072215
s
p50
0.14792
s
<
p95
0.203462
s
<
p99
0.430573
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
9.47
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
11.44
KiB/s
<
mean
185.44
KiB/s
SD
884.05
KiB/s
<
p95
397.82
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
9.12
KiB/s
<
mean
19.67
KiB/s
SD
21.77
KiB/s
<
p95
60.29
KiB/s
Total bytes received
Total bytes received on primary network interface
count
1.42
GiB
mean
1.62
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
154.28
MiB
mean
175.54
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 4981.72 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (46.2% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.38
%
<
mean
14.89
%
SD
10.79
%
<
p95
24.33
%
System
5.42
%
CPU percentiles
CPU usage percentiles
p50
13.82
%
p95
24.33
%
p99
64.28
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
3
hosts
mean time
0.02
s
Memory used percentage
Percentage of memory used
p5
11.33
%
<
mean
22.59
%
SD
14.39
%
<
p95
52.74
%
Memory available percentage
Percentage of available memory
p5
47.26
%
<
mean
77.41
%
SD
14.39
%
<
p95
88.67
%
Max host memory used
Maximum memory usage percentage across all hosts
max
53.79
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
46.21
%
Memory growth rate
Rate of memory growth over time
growth
4981.72
MiB/s
Disk read throughput
Disk read throughput in MiB/s
6.04
MiB/s
Disk write throughput
Disk write throughput in MiB/s
32.78
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/3
hosts
Mount Point /etc/hosts
0/3
hosts
Mount Point /etc/nomad.d/client-meta.json
0/7
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.02
5 min
0.77
15 min
0.47
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
3.6363
%
SD
4.9607
%
<
p95
7.39
%
60 second average
3.4852
%
300 second average
2.6500
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.097
%
SD
0.3452
%
<
p95
0.61
%
60 second average
0.1026
%
300 second average
0.0757
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.0844
%
SD
0.3052
%
<
p95
0.48
%
60 second average
0.0884
%
300 second average
0.0632
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
6.19
%
SD
6.02
%
<
p95
13.9
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
201.09
KiB
<
mean
272.80
KiB
SD
148.39
KiB
<
p95
469.72
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
34.79
threads
SD
37.48
threads
<
p95
83
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
68.9
file descriptors
SD
64.17
file descriptors
<
p95
83
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Tue, 28 Jul 2026 16:30:44 UTC
Peer count
10
Peer count at end
8
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
remote_signals_30377987643_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.184208
s
SD
0.802562
s
p50
0.061994
s
<
p95
0.235653
s
<
p99
0.26805
s
Timeout rate
The proportion of remote signal round-trips that timed out.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 892.203982675s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
31518310
over 892.203982675s
mean rate
7.183619141e+07
/s
std
2.4533472101e+08
/s
p5
0
/s
p95
3.8022750084e+08
/s
peak
2.29310671637e+09
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
13383280
over 892.203982675s
mean rate
9.11470825e+06
/s
std
8.049952262e+07
/s
p5
0
/s
p95
7.01451186e+06
/s
peak
1.61166666667e+09
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.002856
s
SD
0.001812
s
p50
0.002519
s
<
p95
0.005203
s
<
p99
0.006532
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.002741
s
SD
0.001617
s
p50
0.002342
s
<
p95
0.004773
s
<
p99
0.004887
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002495
s
SD
0.001418
s
p50
0.002344
s
<
p95
0.004737
s
p99
0.004737
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001867
s
SD
0.001202
s
p50
0.001722
s
<
p95
0.003578
s
<
p99
0.00456
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001761
s
SD
0.001019
s
p50
0.001562
s
<
p95
0.003327
s
<
p99
0.003408
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000423
s
SD
0.000278
s
p50
0.000348
s
<
p95
0.001018
s
p99
0.001018
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
5.7e-05
s
SD
3.6e-05
s
p50
5.9e-05
s
<
p95
0.000103
s
<
p99
0.000137
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
5.9e-05
s
SD
4.3e-05
s
p50
5.4e-05
s
<
p95
0.000135
s
<
p99
0.00017
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
5e-06
s
SD
4e-06
s
p50
6e-06
s
<
p95
1e-05
s
<
p99
1.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000247
s
SD
0.000132
s
p50
0.000241
s
<
p95
0.000526
s
p99
0.000526
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1735
over 892.20399254s
mean rate
12012.18
/s
std
136725.81
/s
p5
0
/s
p95
15562.51
/s
peak
3.28587076e+06
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1734
over 892.203987695s
mean rate
11470.44
/s
std
107091.77
/s
p5
0
/s
p95
19055.6
/s
peak
2.24382947e+06
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000239
s
SD
0.000256
s
p50
0.000195
s
<
p95
0.000912
s
p99
0.000912
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.47
s
<
mean
450.07
s
SD
258.98
s
<
p95
854.82
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
60
over 892.310011872s
mean rate
34327.31
/s
std
563276.67
/s
p5
0
/s
p95
10.96
/s
peak
1.320638821e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0.938762
s
SD
1.153884
s
p50
0.592488
s
<
p95
4.109618
s
<
p99
5.048987
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.004312
s
SD
0.003993
s
p50
0.003927
s
<
p95
0.012127
s
<
p99
0.01431
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.006791
s
SD
0.004054
s
p50
0.004335
s
<
p95
0.013089
s
p99
0.013089
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.058598
s
SD
0.042999
s
p50
0.080723
s
<
p95
0.14024
s
<
p99
0.17826
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.007426
s
SD
0.005009
s
p50
0.007264
s
<
p95
0.016012
s
<
p99
0.027557
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.059806
s
SD
0.149531
s
p50
0.018687
s
<
p95
0.453484
s
<
p99
0.550707
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.196161
s
SD
0.100203
s
p50
0.247972
s
<
p95
0.341311
s
<
p99
0.468196
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00015
s
SD
0.000121
s
p50
0.000168
s
<
p95
0.000361
s
<
p99
0.00064
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.001557
s
SD
0.00212
s
p50
0.001506
s
<
p95
0.006155
s
<
p99
0.008912
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000524
s
SD
0.000392
s
p50
0.00046
s
<
p95
0.001077
s
<
p99
0.001659
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.06538
s
SD
0.041262
s
p50
0.068395
s
<
p95
0.137429
s
<
p99
0.171215
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.138271
s
SD
0.048961
s
p50
0.160233
s
<
p95
0.190584
s
<
p99
0.203001
s
P2P received remote signals
Count of remote signals received.
total
1736
over 892.20397049s
mean rate
12498.01
/s
std
69113.48
/s
p5
0
/s
p95
57655.16
/s
peak
1.1741683e+06
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
12.06
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
7.68
KiB/s
<
mean
166.51
KiB/s
SD
775.69
KiB/s
<
p95
416.17
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.47
KiB/s
<
mean
23.88
KiB/s
SD
23.87
KiB/s
<
p95
69.62
KiB/s
Total bytes received
Total bytes received on primary network interface
count
723.59
MiB
mean
832.53
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
103.77
MiB
mean
119.40
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.9%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 6038.30 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (48.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.84
%
<
mean
18.73
%
SD
14.03
%
<
p95
28.46
%
System
6.68
%
CPU percentiles
CPU usage percentiles
p50
18.06
%
p95
28.46
%
p99
97.88
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
4
hosts
mean time
0.03
s
Memory used percentage
Percentage of memory used
p5
17.23
%
<
mean
32.05
%
SD
14.13
%
<
p95
57.35
%
Memory available percentage
Percentage of available memory
p5
42.65
%
<
mean
67.95
%
SD
14.13
%
<
p95
82.77
%
Max host memory used
Maximum memory usage percentage across all hosts
max
54.99
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
48.56
%
Memory growth rate
Rate of memory growth over time
growth
6038.30
MiB/s
Disk read throughput
Disk read throughput in MiB/s
7.14
MiB/s
Disk write throughput
Disk write throughput in MiB/s
37.93
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/3
hosts
Mount Point /etc/hosts
0/3
hosts
Mount Point /etc/nomad.d/client-meta.json
0/3
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.12
5 min
1.50
15 min
0.91
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
4.6634
%
SD
7.4343
%
<
p95
9.35
%
60 second average
4.6143
%
300 second average
3.6719
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.0192
%
SD
0.1284
%
<
p95
0.04
%
60 second average
0.0263
%
300 second average
0.0279
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.0134
%
SD
0.0837
%
<
p95
0.03
%
60 second average
0.0181
%
300 second average
0.0181
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
4.2
%
SD
6.6
%
<
p95
21.39
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
198.67
KiB
<
mean
338.61
KiB
SD
312.69
KiB
<
p95
556.05
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
41.1
threads
SD
54.83
threads
<
p95
83
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
67.62
file descriptors
SD
98.02
file descriptors
<
p95
84
file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started
Tue, 28 Jul 2026 16:33:02 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
single_write_many_read_30377987643_1
read call timing
The time it takes to complete the zome call that retrieves the record.
mean
0.002712
s
SD
0.000216
s
p50
0.002679
s
<
p95
0.002972
s
<
p99
0.003623
s
read call rate
The number of read zome calls completed per second.
mean
3655.17
/s
Error count
The number of errors collected.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.000267
s
SD
1e-06
s
p50
0.000267
s
<
p95
0.000268
s
p99
0.000268
s
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001992879s
Not enough time data to show a trend.
WASM usage: 'crud::get_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
3404982343
over 280.001992879s
mean rate
1.216056469e+07
/s
std
80902.92
/s
p5
1.201663737e+07
/s
p95
1.228614793e+07
/s
peak
1.230277718e+07
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001992879s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001992879s
Not enough time data to show a trend.
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.002155
s
SD
0
s
p50
0.002155
s
p95
0.002155
s
p99
0.002155
s
Zome call duration: 'crud::get_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.001193
s
SD
7e-06
s
p50
0.001192
s
<
p95
0.001196
s
p99
0.001196
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00177
s
SD
0
s
p50
0.00177
s
p95
0.00177
s
p99
0.00177
s
WASM call duration: 'crud::get_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000929
s
SD
4e-06
s
p50
0.000928
s
<
p95
0.000931
s
p99
0.000931
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000919
s
SD
0
s
p50
0.000919
s
p95
0.000919
s
p99
0.000919
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000638
s
SD
0
s
p50
0.000638
s
p95
0.000638
s
p99
0.000638
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000175
s
SD
3.6e-05
s
p50
0.00021
s
p95
0.00021
s
p99
0.00021
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.000279
s
SD
1e-06
s
p50
0.000279
s
<
p95
0.00028
s
p99
0.00028
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000926
s
SD
0
s
p50
0.000926
s
p95
0.000926
s
p99
0.000926
s
Post-commit duration
Time spent executing post-commit workflows.
mean
3.9e-05
s
SD
0
s
p50
3.9e-05
s
p95
3.9e-05
s
p99
3.9e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.6
s
<
mean
148.6
s
SD
83.67
s
<
p95
278.6
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.001997352s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.000687
s
SD
0
s
p50
0.000687
s
p95
0.000687
s
p99
0.000687
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.001351
s
SD
0
s
p50
0.001351
s
p95
0.001351
s
p99
0.001351
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000942
s
SD
0
s
p50
0.000942
s
p95
0.000942
s
p99
0.000942
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.003523
s
SD
0.000651
s
p50
0.003668
s
<
p95
0.004766
s
p99
0.004766
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.002063
s
SD
0
s
p50
0.002063
s
p95
0.002063
s
p99
0.002063
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.000653
s
SD
0
s
p50
0.000653
s
p95
0.000653
s
p99
0.000653
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
4.5e-05
s
SD
0
s
p50
4.5e-05
s
p95
4.5e-05
s
p99
4.5e-05
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000592
s
SD
9e-05
s
p50
0.000724
s
p95
0.000724
s
p99
0.000724
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000205
s
SD
0
s
p50
0.000205
s
p95
0.000205
s
p99
0.000205
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
6.72
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
16.66
KiB/s
<
mean
18.73
KiB/s
SD
1.51
KiB/s
<
p95
21.39
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
14.22
KiB/s
<
mean
15.46
KiB/s
SD
1.33
KiB/s
<
p95
18.77
KiB/s
Total bytes received
Total bytes received on primary network interface
count
5.31
MiB
mean
18.73
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.38
MiB
mean
15.46
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 30.17 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
15.98
%
<
mean
16.13
%
SD
0.08
%
<
p95
16.24
%
System
9.41
%
CPU percentiles
CPU usage percentiles
p50
16.13
%
p95
16.24
%
p99
16.27
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
26.49
%
<
mean
27.37
%
SD
0.72
%
<
p95
28.68
%
Memory available percentage
Percentage of available memory
p5
71.32
%
<
mean
72.63
%
SD
0.72
%
<
p95
73.51
%
Max host memory used
Maximum memory usage percentage across all hosts
max
27.37
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
30.17
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
1.93
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.68
5 min
1.38
15 min
0.73
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.83
%
<
mean
1.0603
%
SD
0.1304
%
<
p95
1.24
%
60 second average
0.9613
%
300 second average
0.7147
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
13.84
%
<
mean
13.89
%
SD
0.05
%
<
p95
13.96
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
269.46
KiB
<
mean
274.41
KiB
SD
2.48
KiB
<
p95
276.64
KiB
Holochain process threads
Number of threads in Holochain process
p5
32
threads
<
mean
32.4
threads
SD
0.8
threads
<
p95
34
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
57
file descriptors
<
mean
58
file descriptors
SD
2
file descriptors
<
p95
62
file descriptors

Two-party countersigning

↑ Back to index

This scenario tests the performance of countersigning operations. There are two roles: initiate and participate. The participants commit an entry to advertise that they are willing to participate in sessions. They listen for sessions and participate in one at a time.

Started
Tue, 28 Jul 2026 18:05:42 UTC
Peer count
3
Peer count at end
3
Behaviours
  • initiate (1 agent)
  • participate (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
two_party_countersigning_30377987643_1
Session accepted -- timing
The duration of the session from acceptance to completion.
Mean
mean
0.730274
s
SD
0.036492
s
Max
Highest per-partition mean latency
max mean
0.736619
s
Min
Lowest per-partition mean latency
min mean
0.723928
s
Session accepted -- success rate
The number of accepted sessions that completed per second.
Mean
mean
6.44
/s
Max
Highest per-partition mean rate
max
7.63
/s
Min
Lowest per-partition mean rate
min
5.25
/s
Session accepted -- success ratio
Fraction of accepted sessions that completed successfully (0–1).
1
Session initiated -- timing
The duration of the session from initiation to completion.
Mean
mean
0.901612
s
SD
0.05328
s
Max
Highest per-partition mean latency
max mean
0.901612
s
Min
Lowest per-partition mean latency
min mean
0.901612
s
Session initiated -- success rate
The number of initiated sessions that completed per second.
Mean
mean
10.78
/s
Max
Highest per-partition mean rate
max
10.78
/s
Min
Lowest per-partition mean rate
min
10.78
/s
Session initiated -- success ratio
Fraction of initiated sessions that completed successfully (0–1).
1
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.15923
s
SD
0.101386
s
p50
0.218381
s
<
p95
0.254209
s
<
p99
0.257042
s
WASM usage: 'countersigning::accept_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
21114875
over 270.003084709s
mean rate
1.48950859e+06
/s
std
1.14021097e+06
/s
p5
111641.46
/s
p95
2.44225783e+06
/s
peak
2.46120557e+06
/s
WASM usage: 'countersigning::call_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
10355508
over 270.003084709s
mean rate
1.00876472e+06
/s
std
627412.22
/s
p5
120337.82
/s
p95
1.94240728e+06
/s
peak
1.98838289e+06
/s
WASM usage: 'countersigning::commit_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
49870205
over 270.003084709s
mean rate
4.88771457e+06
/s
std
5.55491589e+06
/s
p5
149939.69
/s
p95
1.302988269e+07
/s
peak
1.327255585e+07
/s
WASM usage: 'countersigning::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 285.281775561s
Not enough time data to show a trend.
WASM usage: 'countersigning::initiator_hello'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 260.010159051s
Not enough time data to show a trend.
WASM usage: 'countersigning::list_participants'
Metered usage count of the WASM ribosome for this zome function.
total
13799093
over 260.010159051s
mean rate
53071.52
/s
std
17778.56
/s
p5
37407.76
/s
p95
79873.15
/s
peak
79873.24
/s
WASM usage: 'countersigning::participant_hello'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.003062811s
Not enough time data to show a trend.
WASM usage: 'countersigning::start_two_party'
Metered usage count of the WASM ribosome for this zome function.
total
44009771
over 260.010159051s
mean rate
169261.83
/s
std
9981.96
/s
p5
157185.28
/s
p95
188521.6
/s
peak
188523.7
/s
WASM usage: 'countersigning_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
3013029
over 285.281775561s
mean rate
1.42053297e+06
/s
std
858093.94
/s
p5
7604.3
/s
p95
2.6064399e+06
/s
peak
2.67983402e+06
/s
Zome call duration: 'countersigning::accept_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.234735
s
SD
0.006797
s
p50
0.234768
s
<
p95
0.243633
s
<
p99
0.245335
s
Zome call duration: 'countersigning::call_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.003383
s
SD
0.00121
s
p50
0.002489
s
<
p95
0.004806
s
<
p99
0.004957
s
Zome call duration: 'countersigning::commit_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.232072
s
SD
0.013528
s
p50
0.229793
s
<
p95
0.257807
s
<
p99
0.259893
s
Zome call duration: 'countersigning::initiator_hello'
Duration of this zome call as measured by Holochain internally.
mean
0.003364
s
SD
0
s
p50
0.003364
s
p95
0.003364
s
p99
0.003364
s
Zome call duration: 'countersigning::list_participants'
Duration of this zome call as measured by Holochain internally.
mean
0.008929
s
SD
0.000291
s
p50
0.008841
s
<
p95
0.009256
s
<
p99
0.009809
s
Zome call duration: 'countersigning::participant_hello'
Duration of this zome call as measured by Holochain internally.
mean
0.003097
s
SD
0.000511
s
p50
0.003599
s
p95
0.003599
s
p99
0.003599
s
Zome call duration: 'countersigning::start_two_party'
Duration of this zome call as measured by Holochain internally.
mean
0.241379
s
SD
0.001589
s
p50
0.241301
s
<
p95
0.244662
s
<
p99
0.247402
s
WASM call duration: 'countersigning::accept_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.233898
s
SD
0.006932
s
p50
0.233852
s
<
p95
0.243201
s
<
p99
0.244906
s
WASM call duration: 'countersigning::call_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002452
s
SD
0.000966
s
p50
0.001673
s
<
p95
0.003674
s
<
p99
0.00377
s
WASM call duration: 'countersigning::commit_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.23112
s
SD
0.013753
s
p50
0.228871
s
<
p95
0.257417
s
<
p99
0.259499
s
WASM call duration: 'countersigning::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006379
s
SD
0.003643
s
p50
0.005467
s
<
p95
0.011426
s
p99
0.011426
s
WASM call duration: 'countersigning::initiator_hello'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002619
s
SD
0
s
p50
0.002619
s
p95
0.002619
s
p99
0.002619
s
WASM call duration: 'countersigning::list_participants'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007847
s
SD
0.000275
s
p50
0.007787
s
<
p95
0.008153
s
<
p99
0.008794
s
WASM call duration: 'countersigning::participant_hello'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002554
s
SD
0.000367
s
p50
0.002915
s
p95
0.002915
s
p99
0.002915
s
WASM call duration: 'countersigning::start_two_party'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.240295
s
SD
0.001576
s
p50
0.240242
s
<
p95
0.243603
s
<
p99
0.246116
s
WASM call duration: 'countersigning_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002235
s
SD
0.000791
s
p50
0.001653
s
<
p95
0.002703
s
<
p99
0.002936
s
Host function call duration: '__hc__accept_countersigning_preflight_request_1'
Duration of this host function call invoked from within WASM.
mean
0.003591
s
SD
0.001668
s
p50
0.002202
s
<
p95
0.005908
s
<
p99
0.006264
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
2.8e-05
s
SD
2e-05
s
p50
2.3e-05
s
<
p95
6.6e-05
s
p99
6.6e-05
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.225774
s
SD
0.006562
s
p50
0.226028
s
<
p95
0.238178
s
<
p99
0.242073
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000652
s
SD
0.000578
s
p50
0.000447
s
<
p95
0.001844
s
p99
0.001844
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000239
s
SD
5.7e-05
s
p50
0.000295
s
p95
0.000295
s
p99
0.000295
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
0.000171
s
SD
0.000116
s
p50
4.8e-05
s
<
p95
0.000358
s
<
p99
0.000374
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.224743
s
SD
0.015327
s
p50
0.22197
s
<
p95
0.255005
s
<
p99
0.257062
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001105
s
SD
4.9e-05
s
p50
0.001098
s
<
p95
0.001174
s
<
p99
0.001243
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
5e-06
s
SD
0
s
p50
5e-06
s
p95
5e-06
s
p99
5e-06
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003394
s
SD
0.001251
s
p50
0.002329
s
<
p95
0.004147
s
<
p99
0.004338
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
207
over 270.003081945s
mean rate
29.85
/s
std
34.29
/s
p5
1.06
/s
p95
82.86
/s
peak
84.9
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000251
s
SD
0.000119
s
p50
0.00018
s
<
p95
0.000363
s
<
p99
0.000402
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
16.83
s
<
mean
146.78
s
SD
83.68
s
<
p95
276.84
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1157
over 285.281710855s
mean rate
46.96
/s
std
33.52
/s
p5
2.47
/s
p95
85.15
/s
peak
91.18
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0.381794
s
SD
0.626154
s
p50
0.418373
s
<
p95
1.235234
s
<
p99
5.458269
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.001969
s
SD
0.002345
s
p50
0.001371
s
<
p95
0.005223
s
<
p99
0.013492
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.011066
s
SD
0.004822
s
p50
0.006772
s
<
p95
0.017128
s
<
p99
0.018219
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.04808
s
SD
0.031585
s
p50
0.041824
s
<
p95
0.103594
s
<
p99
0.175255
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.014425
s
SD
0.007966
s
p50
0.006251
s
<
p95
0.026851
s
<
p99
0.029703
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.011282
s
SD
0.073162
s
p50
0.011095
s
<
p95
0.088838
s
<
p99
0.642448
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.400816
s
SD
0.243498
s
p50
0.28632
s
<
p95
0.880623
s
<
p99
0.897049
s
Witnessing workflow duration
Time spent running the witnessing workflow.
mean
0.000522
s
SD
0.000367
s
p50
0.000403
s
<
p95
0.00088
s
<
p99
0.000897
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000505
s
SD
0.000283
s
p50
0.000321
s
<
p95
0.001011
s
<
p99
0.001116
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000992
s
SD
0.00135
s
p50
0.001696
s
<
p95
0.004314
s
p99
0.004314
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000382
s
SD
0.000185
s
p50
0.000342
s
<
p95
0.000668
s
<
p99
0.000679
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.208542
s
SD
0.005444
s
p50
0.209006
s
<
p95
0.214256
s
<
p99
0.224777
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.223626
s
SD
0.015738
s
p50
0.220448
s
<
p95
0.254797
s
<
p99
0.256852
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.215586
s
SD
0.003833
s
p50
0.21348
s
<
p95
0.218854
s
<
p99
0.236949
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.224918
s
SD
0.006749
s
p50
0.225221
s
<
p95
0.237988
s
<
p99
0.241884
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
35.70
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
0
B/s
<
mean
59.74
KiB/s
SD
46.13
KiB/s
<
p95
118.81
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
39.98
KiB/s
SD
33.55
KiB/s
<
p95
104.58
KiB/s
Total bytes received
Total bytes received on primary network interface
count
51.34
MiB
mean
175.23
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
34.35
MiB
mean
117.26
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 99.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 117.62 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 33% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
7.56
%
<
mean
47.17
%
SD
35.44
%
<
p95
96.63
%
System
11.47
%
CPU percentiles
CPU usage percentiles
p50
39.36
%
p95
96.63
%
p99
99.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1
hosts
mean time
1.00
s
Memory used percentage
Percentage of memory used
p5
15.61
%
<
mean
20.15
%
SD
3.08
%
<
p95
23.32
%
Memory available percentage
Percentage of available memory
p5
76.68
%
<
mean
79.85
%
SD
3.08
%
<
p95
84.39
%
Max host memory used
Maximum memory usage percentage across all hosts
max
22.69
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.02
%
Memory growth rate
Rate of memory growth over time
growth
117.62
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
15.91
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/2
hosts
Mount Point /etc/hosts
0/2
hosts
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.44
5 min
1.39
15 min
1.19
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
33.33
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
25.6073
%
SD
32.3488
%
<
p95
78.2
%
60 second average
22.8671
%
300 second average
12.5188
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2247
%
SD
0.3508
%
<
p95
0.93
%
60 second average
0.1875
%
300 second average
0.0990
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0025
%
300 second average
0.0044
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.64
%
<
mean
6.51
%
SD
8.31
%
<
p95
13.29
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
257.60
KiB
<
mean
281.07
KiB
SD
30.89
KiB
<
p95
314.99
KiB
Holochain process threads
Number of threads in Holochain process
p5
27
threads
<
mean
30.73
threads
SD
4.53
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
69.04
file descriptors
SD
8.6
file descriptors
<
p95
78
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Tue, 28 Jul 2026 16:48:20 UTC
Peer count
20
Peer count at end
19
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
validation_receipts_30377987643_1
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
6.940289
s
SD
2.588274
s
Max
Highest per-partition mean latency
max mean
8.057141
s
Min
Lowest per-partition mean latency
min mean
6.380221
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
1.32
/s
Max
Highest per-partition mean rate
max
1.42
/s
Min
Lowest per-partition mean rate
min
1.19
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
3829958
over 908.063030997s
mean rate
5.69606941e+06
/s
std
1.234067557e+07
/s
p5
0
/s
p95
3.956527576e+07
/s
peak
5.360507955e+07
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
14894318
over 908.063030997s
mean rate
1.013431941e+07
/s
std
2.528098732e+07
/s
p5
30615.48
/s
p95
8.063383676e+07
/s
peak
1.284024091e+08
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 908.063030997s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
471014
over 908.063030997s
mean rate
691105.19
/s
std
1.50911002e+06
/s
p5
0
/s
p95
4.86473882e+06
/s
peak
6.59243337e+06
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006533
s
SD
0.003406
s
p50
0.005373
s
<
p95
0.011876
s
<
p99
0.019723
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.022401
s
SD
0.016065
s
p50
0.015373
s
<
p95
0.041042
s
<
p99
0.10133
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00559
s
SD
0.0029
s
p50
0.004545
s
<
p95
0.010968
s
<
p99
0.016491
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.021324
s
SD
0.015614
s
p50
0.014362
s
<
p95
0.039241
s
<
p99
0.097985
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002815
s
SD
0.002036
s
p50
0.001732
s
<
p95
0.006515
s
<
p99
0.009359
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001864
s
SD
0.000899
s
p50
0.001543
s
<
p95
0.003737
s
<
p99
0.005074
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000615
s
SD
0.000727
s
p50
0.000472
s
<
p95
0.002519
s
<
p99
0.003471
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.019005
s
SD
0.014844
s
p50
0.012208
s
<
p95
0.035309
s
<
p99
0.092358
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00297
s
SD
0.001556
s
p50
0.002485
s
<
p95
0.005283
s
<
p99
0.008811
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000312
s
SD
0.000427
s
p50
0.000213
s
<
p95
0.000665
s
<
p99
0.00194
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.2
s
<
mean
449.94
s
SD
259.67
s
<
p95
856.21
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
5741
over 908.063012984s
mean rate
163.61
/s
std
212.55
/s
p5
4.54
/s
p95
571.63
/s
peak
1925.1
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
1.303953
s
SD
1.141657
s
p50
1.30568
s
<
p95
3.9364
s
<
p99
6.458409
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005216
s
SD
0.008194
s
p50
0.002345
s
<
p95
0.022707
s
<
p99
0.030472
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.005788
s
SD
0.002694
s
p50
0.005462
s
<
p95
0.013717
s
p99
0.013717
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.038062
s
SD
0.035312
s
p50
0.041984
s
<
p95
0.120272
s
<
p99
0.210518
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.048506
s
SD
0.033887
s
p50
0.029975
s
<
p95
0.078601
s
<
p99
0.212578
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.036625
s
SD
0.259824
s
p50
0.034444
s
<
p95
0.314631
s
<
p99
1.25172
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.737476
s
SD
9.299345
s
p50
2.275627
s
<
p95
27.465415
s
<
p99
42.410385
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000891
s
SD
0.00069
s
p50
0.000557
s
<
p95
0.001691
s
<
p99
0.00434
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008283
s
SD
0.009251
s
p50
0.001874
s
<
p95
0.030133
s
<
p99
0.033245
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000346
s
SD
0.000254
s
p50
0.000312
s
<
p95
0.00117
s
<
p99
0.00138
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.051945
s
SD
0.037493
s
p50
0.073552
s
<
p95
0.139301
s
<
p99
0.141051
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.142641
s
SD
0.040897
s
p50
0.134818
s
<
p95
0.199843
s
<
p99
0.21209
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
68.38
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
240.92
KiB/s
<
mean
674.45
KiB/s
SD
342.08
KiB/s
<
p95
1.22
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
99.76
KiB/s
<
mean
595.50
KiB/s
SD
400.51
KiB/s
<
p95
1.24
MiB/s
Total bytes received
Total bytes received on primary network interface
count
11.61
GiB
mean
12.78
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.25
GiB
mean
11.29
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 254.15 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (36.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 90% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
49.05
%
<
mean
83.52
%
SD
19.35
%
<
p95
97.75
%
System
15.14
%
CPU percentiles
CPU usage percentiles
p50
91.31
%
p95
97.75
%
p99
98.15
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
18
hosts
mean time
0.81
s
Memory used percentage
Percentage of memory used
p5
16.24
%
<
mean
20.92
%
SD
6.89
%
<
p95
40.75
%
Memory available percentage
Percentage of available memory
p5
59.25
%
<
mean
79.08
%
SD
6.89
%
<
p95
83.76
%
Max host memory used
Maximum memory usage percentage across all hosts
max
41.70
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
36.33
%
Memory growth rate
Rate of memory growth over time
growth
254.15
MiB/s
Disk read throughput
Disk read throughput in MiB/s
7.47
MiB/s
Disk write throughput
Disk write throughput in MiB/s
168.17
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/4
hosts
Mount Point /etc/hosts
0/4
hosts
Mount Point /etc/nomad.d/client-meta.json
0/11
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/4
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
7.80
5 min
6.21
15 min
3.65
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
90.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.41
%
<
mean
50.1306
%
SD
19.9039
%
<
p95
77.74
%
60 second average
49.4579
%
300 second average
39.1333
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0027
%
SD
0.0397
%
p95
0
%
60 second average
0.0016
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
16.479
%
SD
23.5769
%
<
p95
62.68
%
60 second average
15.7491
%
300 second average
11.6641
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.7153
%
SD
6.2891
%
<
p95
18.04
%
60 second average
3.5352
%
300 second average
2.6074
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.07
%
<
mean
5.46
%
SD
7.43
%
<
p95
11.28
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
113.23
KiB
<
mean
253.19
KiB
SD
77.67
KiB
<
p95
433.54
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
33.94
threads
SD
22.45
threads
<
p95
46
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
73.16
file descriptors
SD
28.05
file descriptors
<
p95
89
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Tue, 28 Jul 2026 17:17:10 UTC
Peer count
30
Peer count at end
27
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_30377987643_1
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2810
Max
Highest chain head value observed for any single write agent.
6725
Write agents observed
7
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.181493
s
SD
0.227618
s
Max
Highest per-partition mean latency
max mean
0.807588
s
Min
Lowest per-partition mean latency
min mean
0.014411
s
Error count
The number of errors accumulated.
8
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.008481
s
SD
0.124256
s
p50
0.000613
s
<
p95
0.158315
s
<
p99
0.440299
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
2
over 1806.270858969s
mean rate
0.4
/s
std
1.11
/s
p5
0
/s
p95
2.76
/s
peak
5.13
/s
WASM usage: 'agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 919.228656014s
Not enough time data to show a trend.
WASM usage: 'agent_activity::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
104961734
over 919.228656014s
mean rate
3.0337895774e+08
/s
std
5.9694556553e+08
/s
p5
3.66793325e+06
/s
p95
1.8587709283e+09
/s
peak
1.90670090358e+09
/s
WASM usage: 'agent_activity::get_agent_activity_full'
Metered usage count of the WASM ribosome for this zome function.
total
961270743
over 1895.333927882s
mean rate
3.20586021617e+09
/s
std
6.26876562837e+09
/s
p5
901862
/s
p95
1.9978707909e+10
/s
peak
2.98330088839e+10
/s
WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
5645873825
over 37462.097519352s
mean rate
8.9736486287e+08
/s
std
1.72338003953e+09
/s
p5
0
/s
p95
5.34818495838e+09
/s
peak
7.78072646434e+09
/s
WASM usage: 'agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
400894255
over 37462.097519352s
mean rate
2.5954538466e+08
/s
std
1.67392983057e+09
/s
p5
0
/s
p95
5.8313551532e+08
/s
peak
3.69154912172e+10
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.005129
s
SD
0.002409
s
p50
0.004192
s
<
p95
0.010902
s
p99
0.010902
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.005423
s
SD
0.006403
s
p50
0.006211
s
<
p95
0.021336
s
<
p99
0.022395
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.021592
s
SD
0.151934
s
p50
0.021979
s
<
p95
0.111858
s
<
p99
0.124282
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.003086
s
SD
0.103927
s
p50
0.00336
s
<
p95
0.445774
s
p99
0.445774
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004206
s
SD
0.0019
s
p50
0.003504
s
<
p95
0.008459
s
p99
0.008459
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004638
s
SD
0.005579
s
p50
0.005498
s
<
p95
0.017786
s
<
p99
0.018803
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.022711
s
SD
0.151614
s
p50
0.031791
s
<
p95
0.111413
s
<
p99
0.12357
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00253
s
SD
0.103797
s
p50
0.002726
s
<
p95
0.44473
s
p99
0.44473
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000956
s
SD
0.000922
s
p50
0.000994
s
<
p95
0.003899
s
<
p99
0.004691
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
2.2e-05
s
SD
1.3e-05
s
p50
1.8e-05
s
<
p95
5.6e-05
s
p99
5.6e-05
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000842
s
SD
0.001961
s
p50
0.000716
s
<
p95
0.005902
s
<
p99
0.006546
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000393
s
SD
0.000281
s
p50
0.000242
s
<
p95
0.001052
s
p99
0.001052
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.02149
s
SD
0.151609
s
p50
0.02973
s
<
p95
0.110074
s
<
p99
0.122548
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000244
s
SD
0.103361
s
p50
0.000201
s
<
p95
0.440366
s
p99
0.440366
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
7e-06
s
SD
7e-06
s
p50
6e-06
s
<
p95
2.5e-05
s
p99
2.5e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00147
s
SD
0.00161
s
p50
0.001602
s
<
p95
0.007537
s
<
p99
0.00817
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000223
s
SD
0.000353
s
p50
0.000114
s
<
p95
0.00094
s
<
p99
0.001847
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.93
s
<
mean
447.49
s
SD
257.34
s
<
p95
847.73
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 37462.097565422s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
121.696088
s
SD
110.623894
s
p50
0.882932
s
<
p95
292.151317
s
<
p99
665.752956
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.032963
s
SD
0.177313
s
p50
0.07153
s
<
p95
0.456654
s
<
p99
1.309303
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.004068
s
SD
0.003412
s
p50
0.003275
s
<
p95
0.005863
s
<
p99
0.020867
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.625438
s
SD
0.450088
s
p50
0.712634
s
<
p95
1.220524
s
<
p99
1.642181
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.125431
s
SD
0.11697
s
p50
0.026499
s
<
p95
0.239733
s
<
p99
0.322276
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.345094
s
SD
19.80695
s
p50
9.832381
s
<
p95
53.390842
s
<
p99
78.222331
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
19.153577
s
SD
38.945189
s
p50
0.058765
s
<
p95
82.392042
s
<
p99
197.480124
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002345
s
SD
0.001365
s
p50
0.002165
s
<
p95
0.004422
s
<
p99
0.005418
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003289
s
SD
0.008892
s
p50
0.001216
s
<
p95
0.031073
s
<
p99
0.039373
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000507
s
SD
0.0012
s
p50
0.000344
s
<
p95
0.00423
s
<
p99
0.005942
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
4.902733
s
SD
28.597214
s
p50
0.120801
s
<
p95
60.001544
s
<
p99
60.002061
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.08267
s
SD
0.169054
s
p50
0.155946
s
<
p95
0.437934
s
p99
0.437934
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.03516
s
SD
19.311182
s
p50
0.043562
s
<
p95
60.001209
s
<
p99
60.001436
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.126776
s
SD
1.196145
s
p50
0.144872
s
<
p95
5.010507
s
<
p99
5.22762
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
48.09
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
80
B/s
<
mean
359.49
KiB/s
SD
314.84
KiB/s
<
p95
1.06
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
50
B/s
<
mean
356.09
KiB/s
SD
377.20
KiB/s
<
p95
1.05
MiB/s
Total bytes received
Total bytes received on primary network interface
count
8.61
GiB
mean
9.68
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
8.52
GiB
mean
9.59
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 238.05 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (34.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 61% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
8.1
%
<
mean
58.6
%
SD
29.2
%
<
p95
97.29
%
System
10.51
%
CPU percentiles
CPU usage percentiles
p50
56.03
%
p95
97.29
%
p99
98.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
11
hosts
mean time
0.91
s
Memory used percentage
Percentage of memory used
p5
9.73
%
<
mean
15.25
%
SD
7.43
%
<
p95
36.12
%
Memory available percentage
Percentage of available memory
p5
63.88
%
<
mean
84.75
%
SD
7.43
%
<
p95
90.27
%
Max host memory used
Maximum memory usage percentage across all hosts
max
39.90
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
34.34
%
Memory growth rate
Rate of memory growth over time
growth
238.05
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.92
MiB/s
Disk write throughput
Disk write throughput in MiB/s
132.28
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/8
hosts
Mount Point /etc/hosts
0/8
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/8
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
4.22
5 min
4.72
15 min
5.05
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
60.71
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
20.416
%
SD
24.8528
%
<
p95
80.65
%
60 second average
20.8356
%
300 second average
24.8846
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
10.4268
%
SD
14.6652
%
<
p95
38.3
%
60 second average
10.6127
%
300 second average
11.2081
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
6.4813
%
SD
10.5629
%
<
p95
28.36
%
60 second average
6.5501
%
300 second average
6.0506
%
Holochain process CPU usage
CPU usage by Holochain process
p5
3.13
%
<
mean
51.65
%
SD
29.15
%
<
p95
95.67
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
227.27
KiB
<
mean
343.21
KiB
SD
136.24
KiB
<
p95
554.26
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
37.92
threads
SD
23.23
threads
<
p95
64
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
57
file descriptors
<
mean
79.5
file descriptors
SD
27.85
file descriptors
<
p95
105
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Tue, 28 Jul 2026 17:32:54 UTC
Peer count
30
Peer count at end
24
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_30377987643_1
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
50.27
/s
Max
Highest per-partition mean rate
max
145.98
/s
Min
Lowest per-partition mean rate
min
12.66
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.198725
s
SD
0.239465
s
Max
Highest per-partition mean latency
max mean
0.51658
s
Min
Lowest per-partition mean latency
min mean
0.044325
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
11
conductors
<
mean
15.51
conductors
SD
2.37
conductors
<
p95
19
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
714.20412
s
SD
414.368725
s
Max
Highest per-partition mean latency
max mean
872.214286
s
Min
Lowest per-partition mean latency
min mean
572.416667
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
42.870847
s
SD
25.739075
s
Max
Highest per-partition mean latency
max mean
51.559301
s
Min
Lowest per-partition mean latency
min mean
34.391038
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
37.002234
s
SD
27.078827
s
Max
Highest per-partition mean latency
max mean
52.732755
s
Min
Lowest per-partition mean latency
min mean
25.88053
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk6D8pdk_ROswIvW6fRmrsdViYDK0Zz45lhwmB2Ttzog7xUCx4
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk6Zr_D3S4zgLWhChMz82luOPyCMpGI81_b126xIaUZvaT2Jjl
p5
0
<
mean
0.4
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkAzNV9GsMdmrSISQsUiOASPgKavgrOU3XcsHF_ogM0xgWNChS
p5
0
<
mean
0.39
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkB8agCN58Rd2WTCSQMumfLiFgD_i8G3FYyiJwZcvSPQHOSWop
p5
0
<
mean
0.44
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIjzpvnPc-GYJzdHyj-5wjDkqe363QLoe-6NgAeKH6hN1OVsa
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQkcCAKaeoxQ_QMBDwJxCH58GuIkRzYx5MtPMN4_ep6N40DaU
p5
0
<
mean
0.34
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXm2jQz_ll4cWu9TUqzqPlyCjzKJOU9B9G8abDVAR_NNbZKAm
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbQW4jMjFAcdIScnyzkbDDfMvzKYLA442vdhOVRv2c3pl39ul
p5
0
<
mean
0.28
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbgwmTKAmNgBKg4FuTwHlhAVMYRiMpZ3x7nz17OPAiEZqN58-
p5
0
<
mean
0.27
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcL-SEIEEM1vHFY46wMN1dmD4LOA-wk65LdHj_Ossweg--gOt
p5
0
<
mean
0.39
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkc_aXgOBGxu7-KUnNVfPAHLh-L2UEIdpSbMvCadJmWNuTbYhO
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdyMjlEmuI3HU3s5sM4NxyYZGX9pIJXXsbqh3f-c_QXDqvkE2
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkeBxDlUT6Zdar0-M0zXGkWRJMNmzhcT6qpzKQCY7HRFTZ7v5R
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAki1baSL7ukz1vaB2jxBY-BrJ-qC1ssPaXurw-v51RgHTEp2r0
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiEe-LnAypEFyYUf8C5xMm86YCWfo0B37zRS1LMFi-HxwxGiq
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjhgYQwdwaMekjYyBBPR5uOCFqo75AIzhzEGVDwM_v5i1YyTs
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpTUD1iJLevL06Qw_gDIzaIVksDENGtzyJQRlFxzd77FCJ25I
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkruu3aoWzb_2HbLKHT-sb9IvoSjjRYDVlevdIUMTIsenM8TJW
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkuo-pQmeMJIaM-vanHFFFSQzYbmmU7VU4RNh0KRjvLD2veDoz
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkyiqtNXTMx2O8O67aNwHrj02zJCcxHjmy9OruFlk_eSwn_7ov
p5
0
<
mean
0.35
SD
0.48
<
p95
1
Error count
The number of errors accumulated.
312
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.083212
s
SD
6.102881
s
p50
0.262672
s
<
p95
0.999249
s
<
p99
1.689576
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
15
over 2655.15098877s
mean rate
3.01
/s
std
6.15
/s
p5
0
/s
p95
10.29
/s
peak
49.88
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.005561
s
SD
0.001353
s
p50
0.005624
s
<
p95
0.007059
s
p99
0.007059
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.004788
s
SD
0.00124
s
p50
0.005217
s
<
p95
0.007066
s
<
p99
0.007312
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.11043
s
SD
0.34514
s
p50
0.274594
s
<
p95
1.006663
s
<
p99
1.586876
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.003646
s
SD
0.289966
s
p50
0.004399
s
<
p95
0.858285
s
p99
0.858285
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004668
s
SD
0.00118
s
p50
0.004745
s
<
p95
0.006199
s
p99
0.006199
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004002
s
SD
0.001141
s
p50
0.004346
s
<
p95
0.006118
s
<
p99
0.006248
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.120541
s
SD
6.202207
s
p50
0.283109
s
<
p95
1.005689
s
<
p99
1.700241
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003019
s
SD
0.150055
s
p50
0.003789
s
<
p95
0.484715
s
p99
0.484715
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00135
s
SD
0.000386
s
p50
0.001482
s
<
p95
0.002046
s
<
p99
0.002289
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
2.3e-05
s
SD
5e-06
s
p50
2.2e-05
s
<
p95
3.2e-05
s
p99
3.2e-05
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000454
s
SD
0.000188
s
p50
0.000476
s
<
p95
0.000779
s
<
p99
0.000865
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000538
s
SD
0.000329
s
p50
0.000464
s
<
p95
0.001371
s
p99
0.001371
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.11789
s
SD
6.202375
s
p50
0.277927
s
<
p95
0.999287
s
<
p99
1.689698
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000377
s
SD
0.149948
s
p50
0.000307
s
<
p95
0.481211
s
p99
0.481211
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
7e-06
s
SD
4e-06
s
p50
1e-05
s
<
p95
1.7e-05
s
<
p99
2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002119
s
SD
0.000596
s
p50
0.002394
s
<
p95
0.003127
s
<
p99
0.003443
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000138
s
SD
5.6e-05
s
p50
0.000139
s
<
p95
0.000256
s
<
p99
0.000293
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
5.34
s
<
mean
322.53
s
SD
508.29
s
<
p95
1527.78
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
201.927017
s
SD
262.579413
s
p50
44.426907
s
<
p95
837.802797
s
<
p99
1279.756893
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.119585
s
SD
0.110541
s
p50
0.005419
s
<
p95
0.331305
s
<
p99
0.533208
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.007839
s
SD
0.006261
s
p50
0.005102
s
<
p95
0.018662
s
<
p99
0.033535
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.056905
s
SD
0.23264
s
p50
0.006219
s
<
p95
0.555272
s
<
p99
1.082282
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.20286
s
SD
0.250573
s
p50
0.082042
s
<
p95
0.611609
s
<
p99
1.301916
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
31.247516
s
SD
31.473434
s
p50
17.994429
s
<
p95
77.836766
s
<
p99
162.43355
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
43.425504
s
SD
47.018942
s
p50
0.069395
s
<
p95
116.079242
s
<
p99
178.206597
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.007942
s
SD
0.00331
s
p50
0.001202
s
<
p95
0.011169
s
<
p99
0.014229
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.015873
s
SD
0.058538
s
p50
0.001242
s
<
p95
0.025611
s
<
p99
0.091906
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000403
s
SD
0.000227
s
p50
0.000406
s
<
p95
0.00072
s
<
p99
0.00133
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
12.707794
s
SD
25.335173
s
p50
0.528601
s
<
p95
60.001599
s
<
p99
60.024366
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.237885
s
SD
0.187083
s
p50
0.478344
s
<
p95
0.662101
s
p99
0.662101
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.130919
s
SD
8.127875
s
p50
0.277365
s
<
p95
1.016539
s
<
p99
60.001131
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.327517
s
SD
15.846441
s
p50
0.540121
s
<
p95
49.044879
s
<
p99
60.000946
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
20.16
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
75
B/s
<
mean
126.16
KiB/s
SD
315.07
KiB/s
<
p95
523.07
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
50
B/s
<
mean
116.13
KiB/s
SD
193.85
KiB/s
<
p95
519.30
KiB/s
Total bytes received
Total bytes received on primary network interface
count
6.09
GiB
mean
3.33
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.60
GiB
mean
3.07
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 247.56 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (34.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 18% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.71
%
<
mean
25.79
%
SD
30.37
%
<
p95
96.69
%
System
5.63
%
CPU percentiles
CPU usage percentiles
p50
14.02
%
p95
96.69
%
p99
98.75
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
7
hosts
mean time
0.52
s
Memory used percentage
Percentage of memory used
p5
8.85
%
<
mean
14.03
%
SD
7.11
%
<
p95
33.39
%
Memory available percentage
Percentage of available memory
p5
66.61
%
<
mean
85.97
%
SD
7.11
%
<
p95
91.15
%
Max host memory used
Maximum memory usage percentage across all hosts
max
39.16
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
34.10
%
Memory growth rate
Rate of memory growth over time
growth
247.56
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.11
MiB/s
Disk write throughput
Disk write throughput in MiB/s
27.04
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/8
hosts
Mount Point /etc/hosts
0/8
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/8
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.99
5 min
2.27
15 min
3.06
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
17.86
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
11.5855
%
SD
25.4588
%
<
p95
80.73
%
60 second average
11.5519
%
300 second average
12.4124
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.4301
%
SD
11.1982
%
<
p95
34.04
%
60 second average
5.5733
%
300 second average
6.3241
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.6037
%
SD
5.8174
%
<
p95
16.67
%
60 second average
2.7464
%
300 second average
3.3289
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
20.48
%
SD
30.13
%
<
p95
96.21
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
101.08
KiB
<
mean
193.94
KiB
SD
100.99
KiB
<
p95
392.59
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
29.03
threads
SD
14.62
threads
<
p95
52
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
69.96
file descriptors
SD
29.12
file descriptors
<
p95
100
file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started
Tue, 28 Jul 2026 16:35:08 UTC
Peer count
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_query_30377987643_1
Write timing
The time it takes to complete the zome function call that writes the entry.
mean
0.021243
s
SD
0.011807
s
p50
0.019793
s
<
p95
0.041224
s
<
p99
0.054769
s
Write rate
The number of writes completed per second.
mean
122.18
/s
Query timing
The time it takes to complete the chain_query_count_len zome call that queries the source chain.
mean
0.053303
s
SD
0.033841
s
p50
0.047448
s
<
p95
0.112568
s
<
p99
0.137506
s
Query rate
The number of chain queries completed per second.
mean
122.18
/s
Error count
The number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'crud::chain_query_count_len'
Metered usage count of the WASM ribosome for this zome function.
total
167123650736
over 280.001611324s
mean rate
5.9686673532e+08
/s
std
4.956116298e+07
/s
p5
5.1239993792e+08
/s
p95
6.6426338955e+08
/s
peak
6.6602101017e+08
/s
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
122343892
over 280.001611324s
mean rate
436939.99
/s
std
242027.24
/s
p5
229080.25
/s
p95
923480.37
/s
peak
1.29931558e+06
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001611324s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
30096474
over 280.001611324s
mean rate
107486.8
/s
std
59555.48
/s
p5
56345.27
/s
p95
227582.21
/s
peak
319583.57
/s
Zome call duration: 'crud::chain_query_count_len'
Duration of this zome call as measured by Holochain internally.
mean
0.023871
s
SD
0.007516
s
p50
0.022052
s
<
p95
0.03169
s
<
p99
0.032357
s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.001975
s
SD
5.2e-05
s
p50
0.001953
s
<
p95
0.002022
s
<
p99
0.002029
s
WASM call duration: 'crud::chain_query_count_len'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.023418
s
SD
0.007462
s
p50
0.021607
s
<
p95
0.031187
s
<
p99
0.03185
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001636
s
SD
3.9e-05
s
p50
0.001618
s
<
p95
0.001672
s
<
p99
0.001678
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000767
s
SD
0
s
p50
0.000767
s
p95
0.000767
s
p99
0.000767
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000586
s
SD
1e-05
s
p50
0.000582
s
<
p95
0.000595
s
<
p99
0.000596
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00013
s
SD
9e-06
s
p50
0.000144
s
p95
0.000144
s
p99
0.000144
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.01043
s
SD
0.003296
s
p50
0.009702
s
<
p95
0.013707
s
<
p99
0.013974
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000826
s
SD
1.9e-05
s
p50
0.000817
s
<
p95
0.000846
s
<
p99
0.000848
s
Post-commit duration
Time spent executing post-commit workflows.
mean
2.6e-05
s
SD
1e-06
s
p50
2.6e-05
s
<
p95
2.8e-05
s
p99
2.8e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.57
s
<
mean
148.57
s
SD
83.67
s
<
p95
278.57
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.001608959s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.000641
s
SD
0
s
p50
0.000641
s
p95
0.000641
s
p99
0.000641
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.001907
s
SD
0
s
p50
0.001907
s
p95
0.001907
s
p99
0.001907
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000225
s
SD
3.7e-05
s
p50
0.000212
s
<
p95
0.000266
s
<
p99
0.000269
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.012338
s
SD
0.003889
s
p50
0.011566
s
<
p95
0.016093
s
<
p99
0.016399
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.002413
s
SD
0
s
p50
0.002413
s
p95
0.002413
s
p99
0.002413
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.000207
s
SD
0
s
p50
0.000207
s
p95
0.000207
s
p99
0.000207
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00158
s
SD
0.000495
s
p50
0.00148
s
<
p95
0.002059
s
<
p99
0.002099
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000337
s
SD
7.9e-05
s
p50
0.000478
s
p95
0.000478
s
p99
0.000478
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000114
s
SD
5e-06
s
p50
0.000111
s
<
p95
0.000118
s
<
p99
0.00012
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
11.37
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
18.04
KiB/s
<
mean
490.19
KiB/s
SD
1.09
MiB/s
<
p95
580.24
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
22.88
KiB/s
<
mean
60.75
KiB/s
SD
41.76
KiB/s
<
p95
168.32
KiB/s
Total bytes received
Total bytes received on primary network interface
count
138.82
MiB
mean
490.19
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
17.21
MiB
mean
60.75
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2812.94 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (45.5% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
7.1
%
<
mean
14.57
%
SD
9.08
%
<
p95
28.48
%
System
3.20
%
CPU percentiles
CPU usage percentiles
p50
13.16
%
p95
28.48
%
p99
50.31
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
52.25
%
<
mean
53.84
%
SD
1.41
%
<
p95
56
%
Memory available percentage
Percentage of available memory
p5
44
%
<
mean
46.16
%
SD
1.41
%
<
p95
47.75
%
Max host memory used
Maximum memory usage percentage across all hosts
max
53.84
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
45.45
%
Memory growth rate
Rate of memory growth over time
growth
2812.94
MiB/s
Disk read throughput
Disk read throughput in MiB/s
7.18
MiB/s
Disk write throughput
Disk write throughput in MiB/s
13.16
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1
hosts
Mount Point /etc/hosts
0/1
hosts
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.95
5 min
2.30
15 min
1.57
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.5507
%
SD
1.3536
%
<
p95
2.59
%
60 second average
0.3957
%
300 second average
0.1327
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
4.5
%
<
mean
4.95
%
SD
0.98
%
<
p95
5.54
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
428.96
KiB
<
mean
453.63
KiB
SD
15.43
KiB
<
p95
471.48
KiB
Holochain process threads
Number of threads in Holochain process
p5
81
threads
<
mean
82.4
threads
SD
1.05
threads
<
p95
84
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
62.2
file descriptors
SD
2.54
file descriptors
<
p95
66
file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started
Tue, 28 Jul 2026 16:37:20 UTC
Peer count
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_read_30377987643_1
Write timing
The time taken to write the entry.
mean
0.02858
s
SD
0.016164
s
p50
0.027229
s
<
p95
0.053799
s
<
p99
0.073524
s
Write rate
The number of writes completed per second.
mean
160.1
/s
Read timing
The time taken to read the entry.
mean
0.028774
s
SD
0.016696
s
p50
0.027642
s
<
p95
0.055289
s
<
p99
0.073874
s
Read rate
The number of reads completed per second.
mean
160.1
/s
Error count
The number of errors accumulated over the run.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.000295
s
SD
4e-05
s
p50
0.000291
s
<
p95
0.000325
s
<
p99
0.000326
s
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
158961154
over 280.001618662s
mean rate
567715.12
/s
std
279303.66
/s
p5
350778.95
/s
p95
1.19193259e+06
/s
peak
1.5677695e+06
/s
WASM usage: 'crud::get_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
148131720
over 280.001618662s
mean rate
529038.8
/s
std
260226.06
/s
p5
326956
/s
p95
1.10764453e+06
/s
peak
1.46129222e+06
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 280.001618662s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
19549282
over 280.001618662s
mean rate
69818.46
/s
std
34349.18
/s
p5
43139.32
/s
p95
146585.66
/s
peak
192806.65
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.002119
s
SD
6.6e-05
s
p50
0.002113
s
<
p95
0.002162
s
<
p99
0.002163
s
Zome call duration: 'crud::get_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.001399
s
SD
8.7e-05
s
p50
0.001392
s
<
p95
0.00146
s
<
p99
0.001461
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001764
s
SD
4.7e-05
s
p50
0.00176
s
<
p95
0.001795
s
<
p99
0.001796
s
WASM call duration: 'crud::get_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001045
s
SD
6.4e-05
s
p50
0.001038
s
<
p95
0.001091
s
<
p99
0.001092
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00087
s
SD
0
s
p50
0.00087
s
p95
0.00087
s
p99
0.00087
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000622
s
SD
1.1e-05
s
p50
0.000621
s
<
p95
0.000629
s
<
p99
0.00063
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000145
s
SD
1.8e-05
s
p50
0.000175
s
p95
0.000175
s
p99
0.000175
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.000317
s
SD
4.2e-05
s
p50
0.000312
s
<
p95
0.000348
s
<
p99
0.00035
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000895
s
SD
2.3e-05
s
p50
0.000893
s
<
p95
0.00091
s
<
p99
0.000911
s
Post-commit duration
Time spent executing post-commit workflows.
mean
3.2e-05
s
SD
2e-06
s
p50
3.2e-05
s
<
p95
3.3e-05
s
p99
3.3e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.56
s
<
mean
148.56
s
SD
83.67
s
<
p95
278.57
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.001604234s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.000841
s
SD
0
s
p50
0.000841
s
p95
0.000841
s
p99
0.000841
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.00279
s
SD
0
s
p50
0.00279
s
p95
0.00279
s
p99
0.00279
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000362
s
SD
7.9e-05
s
p50
0.000348
s
<
p95
0.000437
s
<
p99
0.000442
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.016355
s
SD
0.005244
s
p50
0.015218
s
<
p95
0.021437
s
<
p99
0.021838
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.002701
s
SD
0
s
p50
0.002701
s
p95
0.002701
s
p99
0.002701
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.002721
s
SD
0
s
p50
0.002721
s
p95
0.002721
s
p99
0.002721
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001689
s
SD
0.000539
s
p50
0.001582
s
<
p95
0.002199
s
<
p99
0.002238
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000388
s
SD
9e-05
s
p50
0.000536
s
p95
0.000536
s
p99
0.000536
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000126
s
SD
7e-06
s
p50
0.000126
s
<
p95
0.000131
s
p99
0.000131
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
15.88
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
222.23
KiB/s
<
mean
594.76
KiB/s
SD
1.10
MiB/s
p95
580.24
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
27.43
KiB/s
<
mean
50.25
KiB/s
SD
31.44
KiB/s
<
p95
126.25
KiB/s
Total bytes received
Total bytes received on primary network interface
count
168.44
MiB
mean
594.76
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
14.23
MiB
mean
50.25
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2650.01 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (45.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
13.16
%
<
mean
20.03
%
SD
8.69
%
<
p95
41.19
%
System
4.15
%
CPU percentiles
CPU usage percentiles
p50
16.80
%
p95
41.19
%
p99
50.32
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
53.71
%
<
mean
55.92
%
SD
1.31
%
<
p95
57.64
%
Memory available percentage
Percentage of available memory
p5
42.36
%
<
mean
44.08
%
SD
1.31
%
<
p95
46.29
%
Max host memory used
Maximum memory usage percentage across all hosts
max
55.92
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
45.65
%
Memory growth rate
Rate of memory growth over time
growth
2650.01
MiB/s
Disk read throughput
Disk read throughput in MiB/s
14.13
MiB/s
Disk write throughput
Disk write throughput in MiB/s
25.02
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1
hosts
Mount Point /etc/hosts
0/1
hosts
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
6.68
5 min
3.96
15 min
2.29
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.8613
%
SD
1.4818
%
<
p95
4.32
%
60 second average
0.7087
%
300 second average
0.3357
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0057
%
SD
0.0225
%
<
p95
0.04
%
60 second average
0.0027
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0057
%
SD
0.0225
%
<
p95
0.04
%
60 second average
0.0027
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.004
%
SD
0.0169
%
<
p95
0.03
%
60 second average
0.0013
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.004
%
SD
0.0169
%
<
p95
0.03
%
60 second average
0.0013
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
3.56
%
<
mean
4.98
%
SD
0.85
%
<
p95
5.6
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
395.85
KiB
<
mean
431.79
KiB
SD
80.18
KiB
<
p95
481.70
KiB
Holochain process threads
Number of threads in Holochain process
p5
81
threads
mean
80.29
threads
SD
12.13
threads
<
p95
84
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
60.65
file descriptors
SD
9.85
file descriptors
<
p95
66
file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started
Tue, 28 Jul 2026 16:39:10 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_validated_30377987643_1
Write timing
The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.
mean
0.054944
s
SD
0.026467
s
p50
0.054394
s
<
p95
0.095832
s
<
p99
0.101199
s
Write rate
The number of write zome function calls that complete per second.
mean
70.97
/s
Read timing
The time it takes to complete the zome function call that reads the data.
mean
0.078969
s
SD
0.034258
s
p50
0.077487
s
<
p95
0.133336
s
<
p99
0.139263
s
Read rate
The number of read zome function calls that complete per second.
mean
71
/s
Error count
The number of errors accumulated over the run.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.003669
s
SD
0.001118
s
p50
0.003445
s
<
p95
0.004751
s
<
p99
0.004903
s
WASM usage: 'validated::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
68743611
over 280.002149156s
mean rate
245511.04
/s
std
111902.64
/s
p5
151310.51
/s
p95
495198.17
/s
peak
618997.71
/s
WASM usage: 'validated::update_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
80959500
over 280.002149156s
mean rate
289138.88
/s
std
132165.26
/s
p5
174148.85
/s
p95
587245.93
/s
peak
728994.95
/s
WASM usage: 'validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
121631502
over 280.002149156s
mean rate
434394.93
/s
std
198349.28
/s
p5
263365.86
/s
p95
878320.52
/s
peak
1.09561901e+06
/s
WASM usage: 'validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
1207971213
over 280.002149156s
mean rate
4.31415024e+06
/s
std
1.96983273e+06
/s
p5
2.61235011e+06
/s
p95
8.71679826e+06
/s
peak
1.088142431e+07
/s
Zome call duration: 'validated::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.002103
s
SD
4e-05
s
p50
0.002099
s
<
p95
0.002133
s
<
p99
0.002135
s
Zome call duration: 'validated::update_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006095
s
SD
0.001192
s
p50
0.005812
s
<
p95
0.007179
s
<
p99
0.007268
s
WASM call duration: 'validated::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001802
s
SD
2.3e-05
s
p50
0.001801
s
<
p95
0.001819
s
<
p99
0.00182
s
WASM call duration: 'validated::update_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005796
s
SD
0.001173
s
p50
0.005514
s
<
p95
0.006867
s
<
p99
0.006955
s
WASM call duration: 'validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000587
s
SD
2e-06
s
p50
0.000587
s
<
p95
0.000589
s
p99
0.000589
s
WASM call duration: 'validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002215
s
SD
0.000206
s
p50
0.002161
s
<
p95
0.002411
s
<
p99
0.002427
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000169
s
SD
1e-06
s
p50
0.000169
s
<
p95
0.000171
s
p99
0.000171
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000158
s
SD
2e-06
s
p50
0.000158
s
<
p95
0.000161
s
p99
0.000161
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.003627
s
SD
0.001095
s
p50
0.003332
s
<
p95
0.004674
s
<
p99
0.004762
s
Host function call duration: '__hc__update_1'
Duration of this host function call invoked from within WASM.
mean
0.003965
s
SD
0.001143
s
p50
0.003687
s
<
p95
0.005015
s
<
p99
0.005101
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000858
s
SD
3.6e-05
s
p50
0.000906
s
<
p95
0.000941
s
<
p99
0.000944
s
Post-commit duration
Time spent executing post-commit workflows.
mean
3.4e-05
s
SD
1e-06
s
p50
3.4e-05
s
<
p95
3.5e-05
s
p99
3.5e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.58
s
<
mean
148.58
s
SD
83.67
s
<
p95
278.58
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.002152282s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.00126
s
SD
0
s
p50
0.00126
s
p95
0.00126
s
p99
0.00126
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.00294
s
SD
0
s
p50
0.00294
s
p95
0.00294
s
p99
0.00294
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000531
s
SD
5.4e-05
s
p50
0.000515
s
<
p95
0.000582
s
<
p99
0.000584
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.023859
s
SD
0.007845
s
p50
0.021958
s
<
p95
0.031159
s
<
p99
0.031765
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.003013
s
SD
0
s
p50
0.003013
s
p95
0.003013
s
p99
0.003013
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.002409
s
SD
0
s
p50
0.002409
s
p95
0.002409
s
p99
0.002409
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002287
s
SD
0.000785
s
p50
0.0021
s
<
p95
0.003016
s
<
p99
0.003075
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000723
s
SD
0.000122
s
p50
0.000943
s
p95
0.000943
s
p99
0.000943
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000136
s
SD
2e-06
s
p50
0.000135
s
<
p95
0.000138
s
p99
0.000138
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
15.97
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
11.33
KiB/s
<
mean
18.27
KiB/s
SD
2.67
KiB/s
<
p95
21.39
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
14.23
KiB/s
<
mean
16.26
KiB/s
SD
4.28
KiB/s
<
p95
18.68
KiB/s
Total bytes received
Total bytes received on primary network interface
count
5.17
MiB
mean
18.27
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.60
MiB
mean
16.26
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
21.09
%
<
mean
22.94
%
SD
0.84
%
<
p95
23.68
%
System
6.98
%
CPU percentiles
CPU usage percentiles
p50
23.16
%
p95
23.68
%
p99
23.89
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0
hosts
mean time
0.00
s
Memory used percentage
Percentage of memory used
p5
26.46
%
<
mean
30.31
%
SD
1.21
%
<
p95
31.32
%
Memory available percentage
Percentage of available memory
p5
68.68
%
<
mean
69.69
%
SD
1.21
%
<
p95
73.54
%
Max host memory used
Maximum memory usage percentage across all hosts
max
30.31
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.00
%
Memory growth rate
Rate of memory growth over time
growth
-153.06
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
1.98
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.88
5 min
1.59
15 min
1.04
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.2
%
<
mean
0.6883
%
SD
0.2749
%
<
p95
1
%
60 second average
0.6980
%
300 second average
0.7633
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
19.46
%
<
mean
21.42
%
SD
0.9
%
<
p95
22.2
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
24.16
KiB
<
mean
508.62
KiB
SD
126.18
KiB
<
p95
559.28
KiB
Holochain process threads
Number of threads in Holochain process
p5
12
threads
<
mean
33.91
threads
SD
5.84
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
60.53
file descriptors
SD
13.7
file descriptors
<
p95
68
file descriptors

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Tue, 28 Jul 2026 18:03:44 UTC
Peer count
30
Peer count at end
27
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_30377987643_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1156.33
Max
Highest chain head value observed for any single write agent.
1897
Write agents observed
3
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
119.400554
s
SD
38.093492
s
Max
Highest per-partition mean latency
max mean
1123.45875
s
Min
Lowest per-partition mean latency
min mean
6.21775
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.94
/s
Max
Highest per-partition mean rate
max
3.98
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.169603
s
SD
0.143018
s
Max
Highest per-partition mean latency
max mean
0.722651
s
Min
Lowest per-partition mean latency
min mean
0.014808
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
1054
Agents affected
9 / 11
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
96
Max
Highest count in any single partition
344
Min
Lowest count in any single partition
0
Final error count
The total number of all types of error accumulated over the run by all agents.
81079
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.012138
s
SD
0.057646
s
p50
0.001184
s
<
p95
0.039526
s
<
p99
0.336694
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 730.0069672s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.00429
s
SD
0.001918
s
p50
0.003467
s
<
p95
0.008821
s
p99
0.008821
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.056935
s
SD
0.015929
s
p50
0.049875
s
<
p95
0.07508
s
<
p99
0.082733
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.00652
s
SD
0.003084
s
p50
0.00497
s
<
p95
0.012142
s
<
p99
0.013766
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.007082
s
SD
0.003471
s
p50
0.005517
s
<
p95
0.014931
s
<
p99
0.015759
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007024
s
SD
0.098525
s
p50
0.005582
s
<
p95
0.339562
s
p99
0.339562
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003553
s
SD
0.001584
s
p50
0.002853
s
<
p95
0.007287
s
p99
0.007287
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.056147
s
SD
0.01589
s
p50
0.048993
s
<
p95
0.074345
s
<
p99
0.081987
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005546
s
SD
0.002591
s
p50
0.004024
s
<
p95
0.010841
s
<
p99
0.011022
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004208
s
SD
0.002336
s
p50
0.004044
s
<
p95
0.010069
s
<
p99
0.011583
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005945
s
SD
0.098596
s
p50
0.004709
s
<
p95
0.338919
s
p99
0.338919
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00111
s
SD
0.000493
s
p50
0.001093
s
<
p95
0.002333
s
<
p99
0.00271
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005826
s
SD
0.01056
s
p50
0.004307
s
<
p95
0.019644
s
<
p99
0.064222
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
2.3e-05
s
SD
2.6e-05
s
p50
1.8e-05
s
<
p95
0.000108
s
p99
0.000108
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000409
s
SD
0.000288
s
p50
0.000384
s
<
p95
0.001308
s
<
p99
0.001507
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000391
s
SD
0.000122
s
p50
0.000374
s
<
p95
0.00066
s
p99
0.00066
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000584
s
SD
0.073982
s
p50
0.000536
s
<
p95
0.305112
s
<
p99
0.336715
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.016519
s
SD
0.016188
s
p50
0.014662
s
<
p95
0.03797
s
<
p99
0.095458
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.021522
s
SD
0.010035
s
p50
0.015535
s
<
p95
0.032072
s
<
p99
0.037217
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1e-05
s
SD
5e-06
s
p50
8e-06
s
<
p95
2.3e-05
s
p99
2.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001791
s
SD
0.001114
s
p50
0.001839
s
<
p95
0.004779
s
<
p99
0.005759
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000174
s
SD
0.000139
s
p50
0.000116
s
<
p95
0.000516
s
<
p99
0.000768
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.95
s
<
mean
448.27
s
SD
257.69
s
<
p95
847.91
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
18.706605
s
SD
21.016701
s
p50
14.659824
s
<
p95
48.623534
s
<
p99
134.081856
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.339536
s
SD
1.076796
s
p50
0.585868
s
<
p95
2.645905
s
<
p99
6.144089
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.004489
s
SD
0.001835
s
p50
0.004208
s
<
p95
0.008955
s
<
p99
0.010226
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.078177
s
SD
0.178118
s
p50
0.080167
s
<
p95
0.462771
s
<
p99
1.044659
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.174525
s
SD
0.118893
s
p50
0.034968
s
<
p95
0.335706
s
<
p99
0.618003
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.301009
s
SD
2.115392
s
p50
0.39612
s
<
p95
6.077332
s
<
p99
9.672141
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
21.057828
s
SD
27.612493
s
p50
22.917389
s
<
p95
100.009074
s
<
p99
128.991136
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002455
s
SD
0.001887
s
p50
0.00164
s
<
p95
0.00476
s
<
p99
0.009709
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008004
s
SD
0.004841
s
p50
0.001784
s
<
p95
0.016829
s
<
p99
0.020851
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000345
s
SD
0.000251
s
p50
0.00032
s
<
p95
0.000863
s
<
p99
0.001265
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.213583
s
SD
29.827221
s
p50
0.175144
s
<
p95
60.001648
s
<
p99
60.002176
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.224593
s
SD
0.135496
s
p50
0.303984
s
<
p95
0.335931
s
p99
0.335931
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.20297
s
SD
24.848841
s
p50
0.237762
s
<
p95
60.001338
s
<
p99
60.00212
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
34.98
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
75
B/s
<
mean
85.26
KiB/s
SD
274.10
KiB/s
<
p95
125.85
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
37
B/s
<
mean
67.03
KiB/s
SD
65.30
KiB/s
<
p95
183.92
KiB/s
Total bytes received
Total bytes received on primary network interface
count
1.97
GiB
mean
2.19
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.55
GiB
mean
1.73
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.1%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 179.02 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.15
%
<
mean
44.34
%
SD
27.15
%
<
p95
90.86
%
System
9.37
%
CPU percentiles
CPU usage percentiles
p50
44.64
%
p95
90.86
%
p99
97.13
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
10
hosts
mean time
0.33
s
Memory used percentage
Percentage of memory used
p5
12.68
%
<
mean
16.52
%
SD
6.06
%
<
p95
33.27
%
Memory available percentage
Percentage of available memory
p5
66.73
%
<
mean
83.48
%
SD
6.06
%
<
p95
87.32
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.41
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.09
%
Memory growth rate
Rate of memory growth over time
growth
179.02
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.51
MiB/s
Disk write throughput
Disk write throughput in MiB/s
71.90
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/8
hosts
Mount Point /etc/hosts
0/8
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/8
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.50
5 min
2.33
15 min
2.39
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
21.43
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
10.7896
%
SD
15.412
%
<
p95
43.49
%
60 second average
10.7245
%
300 second average
11.2722
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0001
%
SD
0.0029
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0001
%
SD
0.0029
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.4348
%
SD
14.2844
%
<
p95
38.54
%
60 second average
8.1328
%
300 second average
7.1288
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.2866
%
SD
9.3328
%
<
p95
26.04
%
60 second average
5.1275
%
300 second average
4.3145
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.28
%
<
mean
36.55
%
SD
26.06
%
<
p95
82.61
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
326.82
KiB
<
mean
431.75
KiB
SD
73.95
KiB
<
p95
565.67
KiB
Holochain process threads
Number of threads in Holochain process
p5
16
threads
<
mean
35.63
threads
SD
11.59
threads
<
p95
52
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
43
file descriptors
<
mean
75.35
file descriptors
SD
19.58
file descriptors
<
p95
95
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Tue, 28 Jul 2026 19:22:20 UTC
Peer count
30
Peer count at end
26
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_30377987643_1
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
63.72
/s
Max
Highest per-partition mean rate
max
86.51
/s
Min
Lowest per-partition mean rate
min
47.34
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
353.07495
s
SD
127.103108
s
Max
Highest per-partition mean latency
max mean
510.03882
s
Min
Lowest per-partition mean latency
min mean
204.076373
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
120.51
/s
Max
Highest per-partition mean rate
max
403.5
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0289
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
53062
Agents affected
9 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
5896
Max
Highest count in any single partition
8073
Min
Lowest count in any single partition
4327
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
16844
Agents affected
11 / 11
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1531
Max
Highest count in any single partition
1939
Min
Lowest count in any single partition
730
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
15.27
SD
11.94
<
p95
25
zero-arc
p5
1
<
mean
10.13
SD
3.54
<
p95
15
Error count
The number of errors accumulated across all nodes.
101
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.005259
s
SD
0.049251
s
p50
0.056059
s
<
p95
0.157312
s
<
p99
0.197567
s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
315996285
over 1886.318897404s
mean rate
1.152856795e+08
/s
std
1.0139875456e+08
/s
p5
218350.63
/s
p95
2.9236071224e+08
/s
peak
3.1423040644e+08
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
51864535363
over 906.810283466s
mean rate
4.62345531102e+09
/s
std
6.45574215265e+09
/s
p5
5.40898101e+06
/s
p95
1.89925881428e+10
/s
peak
2.84672308848e+10
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 37462.606522995s
Not enough time data to show a trend.
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.004408
s
SD
0.001126
s
p50
0.005077
s
<
p95
0.006281
s
<
p99
0.006338
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
2.719259
s
SD
8.616443
s
p50
2.409307
s
<
p95
31.023606
s
<
p99
38.958236
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003895
s
SD
0.001028
s
p50
0.004482
s
<
p95
0.005691
s
<
p99
0.005744
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
2.718144
s
SD
8.616249
s
p50
2.407985
s
<
p95
31.021375
s
<
p99
38.955994
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000835
s
SD
0.000693
s
p50
0.001323
s
<
p95
0.003054
s
<
p99
0.003191
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000239
s
SD
6.6e-05
s
p50
0.000266
s
<
p95
0.000388
s
<
p99
0.000395
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000233
s
SD
7.2e-05
s
p50
0.000273
s
<
p95
0.000363
s
<
p99
0.000367
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.004936
s
SD
0.046016
s
p50
0.055491
s
<
p95
0.15013
s
<
p99
0.177773
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.229387
s
SD
0.286555
s
p50
0.252445
s
<
p95
0.856183
s
<
p99
1.758816
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001267
s
SD
0.001124
s
p50
0.001756
s
<
p95
0.004371
s
<
p99
0.004542
s
Post-commit duration
Time spent executing post-commit workflows.
mean
8e-05
s
SD
6.8e-05
s
p50
9.6e-05
s
<
p95
0.000242
s
<
p99
0.000337
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
46.96
s
<
mean
447.19
s
SD
257.03
s
<
p95
847.34
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 37462.606595878s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
5.836101
s
SD
4.301906
s
p50
0
s
<
p95
0.506131
s
<
p99
26.834826
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.07022
s
SD
0.0847
s
p50
0.001607
s
<
p95
0.166437
s
<
p99
0.467052
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.003868
s
SD
0.001938
s
p50
0.00372
s
<
p95
0.007448
s
<
p99
0.009701
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.060933
s
SD
0.399989
s
p50
0.001267
s
<
p95
1.10041
s
<
p99
1.274622
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.094272
s
SD
0.044747
s
p50
0.015928
s
<
p95
0.139215
s
<
p99
0.189626
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
9.220541
s
SD
9.244857
s
p50
0.004703
s
<
p95
23.356226
s
<
p99
39.045712
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.867338
s
SD
10.611833
s
p50
0.002461
s
<
p95
0.097
s
<
p99
67.219278
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000897
s
SD
0.00074
s
p50
0.000385
s
<
p95
0.001923
s
<
p99
0.003039
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004337
s
SD
0.009593
s
p50
0.001394
s
<
p95
0.029786
s
<
p99
0.047514
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000197
s
SD
0.000135
s
p50
0.000283
s
<
p95
0.000591
s
<
p99
0.0008
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.106009
s
SD
29.610092
s
p50
0.192661
s
<
p95
60.001427
s
<
p99
60.00186
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.095408
s
SD
16.967829
s
p50
0.128958
s
<
p95
60.000245
s
<
p99
60.001969
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.113222
s
SD
0.046401
s
p50
0.131089
s
<
p95
0.216435
s
p99
0.216435
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
24.63
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
88
B/s
<
mean
188.57
KiB/s
SD
303.41
KiB/s
<
p95
796.82
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
56
B/s
<
mean
186.68
KiB/s
SD
355.47
KiB/s
<
p95
646.31
KiB/s
Total bytes received
Total bytes received on primary network interface
count
4.79
GiB
mean
2.55
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.63
GiB
mean
2.47
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 750.57 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 25% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
0.93
%
<
mean
32.79
%
SD
29.48
%
<
p95
93.71
%
System
8.16
%
CPU percentiles
CPU usage percentiles
p50
28.65
%
p95
93.71
%
p99
96.59
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
5
hosts
mean time
0.57
s
Memory used percentage
Percentage of memory used
p5
5.79
%
<
mean
14.04
%
SD
5.87
%
<
p95
30.01
%
Memory available percentage
Percentage of available memory
p5
69.99
%
<
mean
85.96
%
SD
5.87
%
<
p95
94.21
%
Max host memory used
Maximum memory usage percentage across all hosts
max
34.85
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
0.09
%
Memory growth rate
Rate of memory growth over time
growth
750.57
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.08
MiB/s
Disk write throughput
Disk write throughput in MiB/s
79.53
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/6
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/7
hosts
Mount Point /etc/hosts
0/7
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/7
hosts
Mount Point /nix/store
0/6
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.30
5 min
2.31
15 min
2.23
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
25.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
10.1775
%
SD
16.0549
%
<
p95
46.4
%
60 second average
9.9988
%
300 second average
9.9383
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
mean
0
%
SD
0
%
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.9201
%
SD
10.0483
%
<
p95
27.16
%
60 second average
4.9914
%
300 second average
5.2026
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.5792
%
SD
7.6781
%
<
p95
20.59
%
60 second average
3.7046
%
300 second average
4.0861
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.22
%
<
mean
27.12
%
SD
26.99
%
<
p95
88.31
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
221.84
KiB
<
mean
291.20
KiB
SD
70.13
KiB
<
p95
450.11
KiB
Holochain process threads
Number of threads in Holochain process
p5
20
threads
<
mean
29.58
threads
SD
8.27
threads
<
p95
43
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
54
file descriptors
<
mean
67.21
file descriptors
SD
16.05
file descriptors
<
p95
81
file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started
Tue, 28 Jul 2026 16:40:59 UTC
Peer count
1
Behaviours
  • default (1 agent)
Holochain version
0.7.0-rc.1
Wind Tunnel version
0.7.0
Run ID
zome_call_single_value_30377987643_1
Call timing
The time taken for the call to complete.
mean
0.00203
s
SD
0.001644
s
p50
0.001663
s
<
p95
0.004041
s
<
p99
0.00998
s
Call rate
The number of zome calls completed per second.
mean
4885.79
/s
Error count
The number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'return_single_value::get_value'
Metered usage count of the WASM ribosome for this zome function.
total
543895916
over 280.001701227s
mean rate
1.94247369e+06
/s
std
670695.99
/s
p5
516538.46
/s
p95
2.64865627e+06
/s
peak
2.65985525e+06
/s
Zome call duration: 'return_single_value::get_value'
Duration of this zome call as measured by Holochain internally.
mean
0.000923
s
SD
3.9e-05
s
p50
0.000913
s
<
p95
0.000977
s
<
p99
0.001013
s
WASM call duration: 'return_single_value::get_value'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.000629
s
SD
2.1e-05
s
p50
0.000627
s
<
p95
0.00066
s
<
p99
0.000679
s
Post-commit duration
Time spent executing post-commit workflows.
mean
5.2e-05
s
SD
0
s
p50
5.2e-05
s
p95
5.2e-05
s
p99
5.2e-05
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
18.39
s
<
mean
148.39
s
SD
83.67
s
<
p95
278.39
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 280.001699234s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
p95
0
s
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.000894
s
SD
0
s
p50
0.000894
s
p95
0.000894
s
p99
0.000894
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0.001792
s
SD
0
s
p50
0.001792
s
p95
0.001792
s
p99
0.001792
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.000784
s
SD
0
s
p50
0.000784
s
p95
0.000784
s
p99
0.000784
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.001947
s
SD
0.00022
s
p50
0.001893
s
<
p95
0.002415
s
p99
0.002415
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.001878
s
SD
0
s
p50
0.001878
s
p95
0.001878
s
p99
0.001878
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.000383
s
SD
0
s
p50
0.000383
s
p95
0.000383
s
p99
0.000383
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
2.6e-05
s
SD
3e-06
s
p50
2.4e-05
s
<
p95
3.1e-05
s
p99
3.1e-05
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.000458
s
SD
6.8e-05
s
p50
0.00055
s
p95
0.00055
s
p99
0.00055
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000164
s
SD
0
s
p50
0.000164
s
p95
0.000164
s
p99
0.000164
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
26.47
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
8.90
KiB/s
<
mean
631.03
KiB/s
SD
1.44
MiB/s
<
p95
5.18
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
7.82
KiB/s
<
mean
43.10
KiB/s
SD
38.51
KiB/s
<
p95
155.97
KiB/s
Total bytes received
Total bytes received on primary network interface
count
178.71
MiB
mean
631.03
KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
12.21
MiB
mean
43.10
KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2194.17 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (55.4% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
8.19
%
<
mean
34.07
%
SD
28.84
%
<
p95
97.89
%
System
7.61
%
CPU percentiles
CPU usage percentiles
p50
18.49
%
p95
97.89
%
p99
97.98
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1
hosts
mean time
0.14
s
Memory used percentage
Percentage of memory used
p5
56.6
%
<
mean
58.62
%
SD
1.63
%
<
p95
61.71
%
Memory available percentage
Percentage of available memory
p5
38.29
%
<
mean
41.38
%
SD
1.63
%
<
p95
43.4
%
Max host memory used
Maximum memory usage percentage across all hosts
max
58.62
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
55.36
%
Memory growth rate
Rate of memory growth over time
growth
2194.17
MiB/s
Disk read throughput
Disk read throughput in MiB/s
9.63
MiB/s
Disk write throughput
Disk write throughput in MiB/s
62.70
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1
hosts
Mount Point /etc/hosts
0/1
hosts
Mount Point /etc/nomad.d/client-meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/1
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
12.84
5 min
7.48
15 min
3.95
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.08
%
<
mean
6.02
%
SD
10.496
%
<
p95
29.65
%
60 second average
4.8293
%
300 second average
1.9573
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0497
%
SD
0.1438
%
<
p95
0.28
%
60 second average
0.0403
%
300 second average
0.0120
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0497
%
SD
0.1438
%
<
p95
0.28
%
60 second average
0.0403
%
300 second average
0.0120
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.0043
%
SD
0.0169
%
<
p95
0.03
%
60 second average
0.0023
%
300 second average
0.0000
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.0043
%
SD
0.0169
%
<
p95
0.03
%
60 second average
0.0023
%
300 second average
0.0000
%
Holochain process CPU usage
CPU usage by Holochain process
p5
3.49
%
<
mean
4.37
%
SD
0.9
%
<
p95
5.53
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
24.60
KiB
<
mean
1.11
MiB
SD
747.22
KiB
<
p95
1.93
MiB
Holochain process threads
Number of threads in Holochain process
p5
11
threads
<
mean
142.82
threads
SD
146.52
threads
<
p95
467
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
200.48
file descriptors
SD
306.04
file descriptors
<
p95
879
file descriptors