DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Thu, 27 Aug 2026 14:54:13 BST
Duration
1800s
Peer count
150
Peer count at end
145
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_33072482790_1
Create rate
The average number of created records per agent.
Mean
mean
16.71
/s
Max
Highest per-partition mean rate
max
61.53
/s
Min
Lowest per-partition mean rate
min
5
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
267.902979
s
SD
191.873827
s
Max
Highest per-partition mean latency
max mean
1312.787787
s
Min
Lowest per-partition mean latency
min mean
3.54943
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
19.67
/s
Max
Highest per-partition mean rate
max
165
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0025
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
372728
Agents affected
120 / 120
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3106
Max
Highest count in any single partition
11571
Min
Lowest count in any single partition
919
Received count
The total number of entries received by record_lag agents, by agent.
Total
21106
Agents affected
21 / 23
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
918
Max
Highest count in any single partition
6855
Min
Lowest count in any single partition
0
Error count
The number of errors encountered during the scenario.
49
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.001122
s
SD
0.00157
s
p50
0.001753
s
<
p95
0.004803
s
<
p99
0.006681
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 1727.714093402s
mean rate
7.09
/s
std
15.35
/s
p5
0
/s
p95
37.64
/s
peak
84.56
/s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
118490243
over 1788.679483244s
mean rate
1.84474663046e+10
/s
std
1.91726341465e+11
/s
p5
5.127956622e+07
/s
p95
3.812883021e+10
/s
peak
8.15683168344e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
12164129715
over 1787.435799404s
mean rate
5.27724808255e+11
/s
std
1.9162924339e+12
/s
p5
3.302503271e+07
/s
p95
2.64787933647e+12
/s
peak
1.45621312249e+13
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
15108432
over 1788.679483244s
mean rate
6.5362979142e+09
/s
std
2.57314552576e+11
/s
p5
1.103764978e+07
/s
p95
5.93467838637e+09
/s
peak
2.5701660731e+13
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.024445
s
SD
0.036631
s
p50
0.026113
s
<
p95
0.119
s
<
p99
0.164334
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.104214
s
SD
0.148181
s
p50
0.07232
s
<
p95
0.448271
s
<
p99
0.644219
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.023083
s
SD
0.035462
s
p50
0.024364
s
<
p95
0.114331
s
<
p99
0.158159
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.103081
s
SD
0.148229
s
p50
0.071495
s
<
p95
0.447019
s
<
p99
0.643487
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001891
s
SD
0.001351
s
p50
0.002106
s
<
p95
0.005287
s
<
p99
0.006441
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.006214
s
SD
0.012497
s
p50
0.005799
s
<
p95
0.038631
s
<
p99
0.053204
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.006092
s
SD
0.012517
s
p50
0.0056
s
<
p95
0.038338
s
<
p99
0.055228
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001097
s
SD
0.001656
s
p50
0.002044
s
<
p95
0.005595
s
<
p99
0.006598
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.005184
s
SD
0.006494
s
p50
0.004565
s
<
p95
0.022323
s
<
p99
0.028152
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.004323
s
SD
0.0046
s
p50
0.004586
s
<
p95
0.015465
s
<
p99
0.020874
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001878
s
SD
0.003631
s
p50
0.001025
s
<
p95
0.010761
s
<
p99
0.016797
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
85.95
s
<
mean
874.46
s
SD
507.82
s
<
p95
1668.22
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
6176
over 1788.679586099s
mean rate
2.55338439e+06
/s
std
1.6117227825e+08
/s
p5
0
/s
p95
2.29315877e+06
/s
peak
1.79767932489e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
98.807494
s
SD
122.789736
s
p50
2.939767
s
<
p95
269.911173
s
<
p99
682.115251
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.361282
s
SD
1.123934
s
p50
0.292128
s
<
p95
2.652962
s
<
p99
6.434668
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.282084
s
SD
0.524938
s
p50
0.269162
s
<
p95
1.532411
s
<
p99
1.925538
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
2.261291
s
SD
3.535841
s
p50
1.58494
s
<
p95
10.313196
s
<
p99
13.082592
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
33.01278
s
SD
58.169037
s
p50
34.175973
s
<
p95
185.46398
s
<
p99
280.239003
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
19.69579
s
SD
44.066163
s
p50
0.016835
s
<
p95
97.363283
s
<
p99
228.581626
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003001
s
SD
0.003447
s
p50
0.004125
s
<
p95
0.011177
s
<
p99
0.013429
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.009062
s
SD
0.009839
s
p50
0.003896
s
<
p95
0.02988
s
<
p99
0.049734
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.005919
s
SD
0.013445
s
p50
0.002927
s
<
p95
0.036756
s
<
p99
0.054337
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.131714
s
SD
27.164584
s
p50
2.14797
s
<
p95
60.035202
s
<
p99
60.124292
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.72335
s
SD
25.313812
s
p50
0.812608
s
<
p95
60.023599
s
<
p99
60.082902
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
60.02
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
79.93
KiB/s
<
mean
567.93
KiB/s
SD
455.64
KiB/s
<
p95
1.33
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
10.82
KiB/s
<
mean
206.30
KiB/s
SD
343.48
KiB/s
<
p95
754.63
KiB/s
Total bytes received
Total bytes received on primary network interface
count
133.58
GiB
mean
76.42
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
48.52
GiB
mean
27.76
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 3.90 MiB/s

Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (25.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 92% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
40.99
%
<
mean
80.47
%
SD
18.18
%
<
p95
95.44
%
System
20.45
%
CPU percentiles
CPU usage percentiles
p50
87.78
%
p95
95.44
%
p99
97.60
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
137
hosts
mean time
1288.44
s
Memory used percentage
Percentage of memory used
p5
8.55
%
<
mean
14.32
%
SD
7.49
%
<
p95
31.14
%
Memory available percentage
Percentage of available memory
p5
68.86
%
<
mean
85.68
%
SD
7.49
%
<
p95
91.45
%
Max host memory used
Maximum memory usage percentage across all hosts
max
54.61
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
25.30
%
Memory growth rate
Rate of memory growth over time
growth
3.90
MiB/s
Disk read throughput
Disk read throughput in MiB/s
199.54
MiB/s
Disk write throughput
Disk write throughput in MiB/s
235.33
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/111
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
4.27
5 min
3.75
15 min
2.65
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
92.20
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
2.09
%
<
mean
45.0944
%
SD
22.0501
%
<
p95
72.43
%
60 second average
43.9103
%
300 second average
38.5290
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.01
%
SD
0.1316
%
≤
p95
0
%
60 second average
0.0109
%
300 second average
0.0137
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0048
%
SD
0.07
%
≤
p95
0
%
60 second average
0.0057
%
300 second average
0.0089
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
11.3778
%
SD
9.9266
%
<
p95
28.82
%
60 second average
11.1093
%
300 second average
9.7481
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.5461
%
SD
5.2572
%
<
p95
12.7
%
60 second average
3.4801
%
300 second average
3.1398
%
Holochain process CPU usage
CPU usage by Holochain process
p5
39.96
%
<
mean
75.11
%
SD
17.73
%
<
p95
92.03
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
239.07
KiB
<
mean
473.88
KiB
SD
183.99
KiB
<
p95
800.22
KiB
Holochain process threads
Number of threads in Holochain process
p5
31
threads
<
mean
36.01
threads
SD
11.33
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
77
file descriptors
<
mean
85.29
file descriptors
SD
15.49
file descriptors
<
p95
90
file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 27 Aug 2026 18:33:37 BST
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_33072482790_1
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
18.05
/s
Max
Highest per-partition mean rate
max
47.85
/s
Min
Lowest per-partition mean rate
min
7.27
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
370.177808
s
SD
41.528845
s
Max
Highest per-partition mean latency
max mean
1400.127132
s
Min
Lowest per-partition mean latency
min mean
0.306879
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
13.36
/s
Max
Highest per-partition mean rate
max
203
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0043
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
14648
Agents affected
9 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1628
Max
Highest count in any single partition
4392
Min
Lowest count in any single partition
623
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
8719
Agents affected
18 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
63
Max
Highest count in any single partition
2196
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
51
<
mean
93.76
SD
40.54
<
p95
146
zero-arc
p5
0
<
mean
17.32
SD
18.86
<
p95
54
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
1919
Agents affected
138 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
14
Max
Highest count in any single partition
15
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
1919
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.391211
s
SD
2.945719
s
p50
0.713298
s
<
p95
7.126983
s
<
p99
14.21954
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
7
over 799.539043838s
mean rate
1174.05
/s
std
7420.77
/s
p5
0
/s
p95
1087.72
/s
peak
83199.44
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
40690020
over 879.016680268s
mean rate
6.191381088e+07
/s
std
8.215016291e+07
/s
p5
3.47070017e+06
/s
p95
2.3883162047e+08
/s
peak
2.5736627036e+08
/s
WASM usage: 'timed_and_validated::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
486617141
over 889.177599017s
mean rate
9.14661730544e+09
/s
std
1.03247307222e+11
/s
p5
1.683084634e+07
/s
p95
2.75996425577e+10
/s
peak
6.02102999866e+12
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
7192630
over 889.177599017s
mean rate
1.30249862154e+09
/s
std
2.61688575091e+10
/s
p5
0
/s
p95
3.98178674351e+09
/s
peak
1.52369721116e+12
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.177599017s
Not enough time data to show a trend.
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.01241
s
SD
0.009031
s
p50
0.018955
s
<
p95
0.03693
s
<
p99
0.039458
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.043644
s
SD
86.638729
s
p50
0.020077
s
<
p95
146.293583
s
<
p99
488.635285
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011124
s
SD
0.008396
s
p50
0.016448
s
<
p95
0.034884
s
<
p99
0.037295
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.179122
s
SD
104.864801
s
p50
0.887783
s
<
p95
305.66253
s
<
p99
433.753123
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001134
s
SD
0.000775
s
p50
0.002118
s
<
p95
0.002943
s
<
p99
0.003368
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009355
s
SD
0.002724
s
p50
0.001827
s
<
p95
0.009056
s
<
p99
0.01233
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.001898
s
SD
0.00303
s
p50
0.002287
s
<
p95
0.011348
s
<
p99
0.012058
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.001733
s
SD
0.00272
s
p50
0.001859
s
<
p95
0.010097
s
<
p99
0.010834
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.675619
s
SD
3.344656
s
p50
1.274744
s
<
p95
8.026806
s
<
p99
15.620734
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.019377
s
SD
2.277685
s
p50
0.466421
s
<
p95
5.237187
s
<
p99
12.641869
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.012638
s
SD
0.00435
s
p50
0.008874
s
<
p95
0.01733
s
<
p99
0.02294
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001982
s
SD
0.001319
s
p50
0.003345
s
<
p95
0.004951
s
<
p99
0.005761
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000718
s
SD
0.000666
s
p50
0.00022
s
<
p95
0.001747
s
<
p99
0.002777
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.75
s
<
mean
427.53
s
SD
247.59
s
<
p95
815.33
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.177526209s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
90.871391
s
SD
16.791193
s
p50
0
s
<
p95
0.439189
s
<
p99
95.562124
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
1.616662
s
SD
0.932113
s
p50
0.00181
s
<
p95
0.367114
s
<
p99
3.289156
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.156102
s
SD
0.116497
s
p50
0.002032
s
<
p95
0.035756
s
<
p99
0.805064
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.236638
s
SD
0.081753
s
p50
0.019781
s
<
p95
0.136333
s
<
p99
0.500214
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
2.854077
s
SD
6.580895
s
p50
0.003176
s
<
p95
1.026804
s
<
p99
25.369108
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.675104
s
SD
1.369137
s
p50
0.001101
s
<
p95
0.005083
s
<
p99
7.902829
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002072
s
SD
0.00141
s
p50
0.000176
s
<
p95
0.003123
s
<
p99
0.00834
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008278
s
SD
0.009057
s
p50
0.003758
s
<
p95
0.023432
s
<
p99
0.041788
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.001576
s
SD
0.000924
s
p50
0.000697
s
<
p95
0.001939
s
<
p99
0.00388
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.936537
s
SD
29.851003
s
p50
0.725526
s
<
p95
60.001674
s
<
p99
60.009913
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.327418
s
SD
23.86601
s
p50
0.798488
s
<
p95
60.001403
s
<
p99
60.001834
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.350238
s
SD
0.686956
s
p50
0.065887
s
<
p95
2.419965
s
<
p99
2.914227
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.59
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
1.95
KiB/s
<
mean
119.82
KiB/s
SD
307.61
KiB/s
<
p95
413.77
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
394
B/s
<
mean
22.77
KiB/s
SD
107.11
KiB/s
<
p95
69.83
KiB/s
Total bytes received
Total bytes received on primary network interface
count
14.66
GiB
mean
16.87
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.79
GiB
mean
3.20
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (51.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 6% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.31
%
<
mean
12.26
%
SD
22.94
%
<
p95
86
%
System
4.67
%
CPU percentiles
CPU usage percentiles
p50
4.08
%
p95
86.00
%
p99
94.93
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
67
hosts
mean time
113.05
s
Memory used percentage
Percentage of memory used
p5
7.23
%
<
mean
13.76
%
SD
6.53
%
<
p95
27.94
%
Memory available percentage
Percentage of available memory
p5
72.06
%
<
mean
86.24
%
SD
6.53
%
<
p95
92.77
%
Max host memory used
Maximum memory usage percentage across all hosts
max
48.26
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
51.34
%
Memory growth rate
Rate of memory growth over time
growth
0.44
MiB/s
Disk read throughput
Disk read throughput in MiB/s
270.71
MiB/s
Disk write throughput
Disk write throughput in MiB/s
66.11
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/117
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.60
5 min
0.70
15 min
1.01
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
6.16
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
4.706
%
SD
16.835
%
<
p95
37.39
%
60 second average
4.6420
%
300 second average
5.7805
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0298
%
SD
0.3614
%
≤
p95
0
%
60 second average
0.0309
%
300 second average
0.0341
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0291
%
SD
0.3548
%
≤
p95
0
%
60 second average
0.0303
%
300 second average
0.0334
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.519
%
SD
6.1007
%
<
p95
10.85
%
60 second average
1.5206
%
300 second average
1.6612
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8805
%
SD
4.9216
%
<
p95
3.06
%
60 second average
0.8948
%
300 second average
0.9125
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.3
%
<
mean
8.57
%
SD
21.98
%
<
p95
81.04
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
307.98
KiB
<
mean
383.28
KiB
SD
75.76
KiB
<
p95
477.67
KiB
Holochain process threads
Number of threads in Holochain process
p5
21
threads
<
mean
28.6
threads
SD
7.64
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
54
file descriptors
<
mean
63.11
file descriptors
SD
10.77
file descriptors
<
p95
81
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Thu, 27 Aug 2026 18:50:40 BST
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_33072482790_1
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
462932
Agents affected
105 / 105
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4409
Max
Highest count in any single partition
13284
Min
Lowest count in any single partition
707
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2574.2
Max
Highest chain head value observed for any single write agent.
4800
Write agents observed
15
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
0.215262
s
SD
0.61471
s
Max
Highest per-partition mean latency
max mean
1.679754
s
Min
Lowest per-partition mean latency
min mean
-2.402478
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
9.76
/s
Max
Highest per-partition mean rate
max
78.96
/s
Min
Lowest per-partition mean rate
min
0
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
2.587684
s
SD
5.405065
s
Max
Highest per-partition mean latency
max mean
14.246841
s
Min
Lowest per-partition mean latency
min mean
0.068987
s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
162
Agents affected
43 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4
Max
Highest count in any single partition
9
Min
Lowest count in any single partition
1
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
32
<
mean
95.88
SD
37.08
<
p95
145
  • behaviour: zero_read
p5
14
<
mean
47.39
SD
19.69
<
p95
82
  • behaviour: zero_write
p5
4
<
mean
36.84
SD
17.33
<
p95
64
Final error count
The total number of errors accumulated across all agents over the run.
165
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.436472
s
SD
8.377777
s
p50
1.517082
s
<
p95
21.168697
s
<
p99
45.403863
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 818.726379416s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.00847
s
SD
0.003683
s
p50
0.008487
s
<
p95
0.015984
s
<
p99
0.019565
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007893
s
SD
0.014767
s
p50
0.005926
s
<
p95
0.049509
s
<
p99
0.061872
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.44116
s
SD
5.402407
s
p50
1.715153
s
<
p95
15.202498
s
<
p99
27.313971
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.302908
s
SD
10.353866
s
p50
0.92342
s
<
p95
23.266519
s
<
p99
45.410674
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006987
s
SD
0.003079
s
p50
0.007051
s
<
p95
0.012996
s
<
p99
0.016187
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006823
s
SD
0.013856
s
p50
0.004823
s
<
p95
0.045925
s
<
p99
0.057778
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.440918
s
SD
5.402462
s
p50
1.713644
s
<
p95
15.201458
s
<
p99
27.313026
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.350387
s
SD
10.35583
s
p50
1.126952
s
<
p95
23.265013
s
<
p99
45.40943
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001506
s
SD
0.00087
s
p50
0.0017
s
<
p95
0.003427
s
<
p99
0.004514
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.8e-05
s
SD
4.2e-05
s
p50
4.9e-05
s
<
p95
0.000166
s
<
p99
0.000195
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002372
s
SD
0.009614
s
p50
0.00061
s
<
p95
0.029871
s
<
p99
0.039003
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000707
s
SD
0.000472
s
p50
0.000646
s
<
p95
0.001353
s
<
p99
0.003379
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.438559
s
SD
5.40273
s
p50
1.710687
s
<
p95
15.199266
s
<
p99
27.31125
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.346463
s
SD
10.355581
s
p50
1.11454
s
<
p95
23.258462
s
<
p99
45.403939
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
8e-06
s
p50
1.6e-05
s
<
p95
2.8e-05
s
<
p99
4.9e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002828
s
SD
0.002631
s
p50
0.003261
s
<
p95
0.009663
s
<
p99
0.013848
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001006
s
SD
0.003055
s
p50
0.000442
s
<
p95
0.009263
s
<
p99
0.013881
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.69
s
<
mean
427.93
s
SD
247.57
s
<
p95
815.22
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
5
over 887.983073515s
mean rate
51099.36
/s
std
1.34011837e+06
/s
p5
0
/s
p95
98136.78
/s
peak
1.234387047e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
13.36916
s
SD
24.096879
s
p50
0
s
<
p95
34.97816
s
<
p99
81.349827
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.295002
s
SD
0.72144
s
p50
0.002178
s
<
p95
1.24951
s
<
p99
3.657117
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
1e-06
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.048702
s
SD
0.22252
s
p50
0.002513
s
<
p95
0.600787
s
<
p99
1.081085
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.197918
s
SD
0.135954
s
p50
0.150167
s
<
p95
0.379277
s
<
p99
0.506934
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
14.643331
s
SD
22.30543
s
p50
0.004007
s
<
p95
47.11424
s
<
p99
129.896627
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
3.13163
s
SD
14.223273
s
p50
0.001771
s
<
p95
2.441909
s
<
p99
38.476171
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002795
s
SD
0.00253
s
p50
0.001826
s
<
p95
0.007724
s
<
p99
0.010349
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00648
s
SD
0.004761
s
p50
0.002858
s
<
p95
0.015061
s
<
p99
0.024568
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002447
s
SD
0.008439
s
p50
0.00059
s
<
p95
0.025932
s
<
p99
0.038471
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
8.518584
s
SD
27.593174
s
p50
5.536775
s
<
p95
60.057525
s
<
p99
60.142577
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.969844
s
SD
10.361325
s
p50
0.757913
s
<
p95
23.253784
s
<
p99
45.386703
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.337847
s
SD
28.70479
s
p50
1.351135
s
<
p95
60.001566
s
<
p99
60.002477
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
2.077944
s
SD
15.621261
s
p50
0.885357
s
<
p95
60.001039
s
<
p99
60.027992
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
39.84
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
6.65
KiB/s
<
mean
521.17
KiB/s
SD
442.70
KiB/s
<
p95
1.31
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.35
KiB/s
<
mean
183.09
KiB/s
SD
286.42
KiB/s
<
p95
834.43
KiB/s
Total bytes received
Total bytes received on primary network interface
count
64.05
GiB
mean
73.69
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
22.50
GiB
mean
25.89
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (51.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 39% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.21
%
<
mean
54.08
%
SD
36.12
%
<
p95
93.64
%
System
14.23
%
CPU percentiles
CPU usage percentiles
p50
63.09
%
p95
93.64
%
p99
97.05
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
97
hosts
mean time
534.72
s
Memory used percentage
Percentage of memory used
p5
7.25
%
<
mean
12.78
%
SD
6.52
%
<
p95
27.7
%
Memory available percentage
Percentage of available memory
p5
72.3
%
<
mean
87.22
%
SD
6.52
%
<
p95
92.75
%
Max host memory used
Maximum memory usage percentage across all hosts
max
48.27
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
51.18
%
Memory growth rate
Rate of memory growth over time
growth
0.72
MiB/s
Disk read throughput
Disk read throughput in MiB/s
269.20
MiB/s
Disk write throughput
Disk write throughput in MiB/s
159.85
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/116
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/16
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.56
5 min
2.00
15 min
1.44
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
39.04
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
26.7266
%
SD
27.0699
%
<
p95
71.73
%
60 second average
25.4602
%
300 second average
19.8244
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.025
%
SD
0.3062
%
≤
p95
0
%
60 second average
0.0255
%
300 second average
0.0272
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0242
%
SD
0.2973
%
≤
p95
0
%
60 second average
0.0249
%
300 second average
0.0260
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.0884
%
SD
7.9799
%
<
p95
19.49
%
60 second average
4.8912
%
300 second average
3.9825
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.0074
%
SD
5.4922
%
<
p95
8.78
%
60 second average
1.9550
%
300 second average
1.6737
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.9
%
<
mean
50.55
%
SD
34.6
%
<
p95
89.36
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
203.76
KiB
<
mean
292.59
KiB
SD
98.84
KiB
<
p95
496.53
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
31.29
threads
SD
7.91
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
76.47
file descriptors
SD
12.88
file descriptors
<
p95
89
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Thu, 27 Aug 2026 19:07:56 BST
Duration
900s
Peer count
150
Peer count at end
139
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_33072482790_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
493.71
Max
Highest chain head value observed for any single write agent.
1248
Write agents observed
7
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
242.794406
s
SD
191.108845
s
Max
Highest per-partition mean latency
max mean
489.4497
s
Min
Lowest per-partition mean latency
min mean
26.911115
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.33
/s
Max
Highest per-partition mean rate
max
2.14
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.447119
s
SD
0.56912
s
Max
Highest per-partition mean latency
max mean
1.649244
s
Min
Lowest per-partition mean latency
min mean
0.032352
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
90468
Agents affected
44 / 44
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2056
Max
Highest count in any single partition
6491
Min
Lowest count in any single partition
360
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
38
<
mean
71.66
SD
33.26
<
p95
138
  • behaviour: zero_must_get_agent_activity
p5
17
<
mean
34.71
SD
16.62
<
p95
71
  • behaviour: zero_write
p5
5
<
mean
30.85
SD
18.6
<
p95
72
Final error count
The total number of all types of error accumulated over the run by all agents.
104872
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.088936
s
SD
1.811258
s
p50
0.103647
s
<
p95
1.016133
s
<
p99
2.880491
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 811.127040008s
mean rate
291.16
/s
std
1772.32
/s
p5
0
/s
p95
900.09
/s
peak
47740.67
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009545
s
SD
0.007136
s
p50
0.008346
s
<
p95
0.021755
s
<
p99
0.03413
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.165986
s
SD
0.206819
s
p50
0.079124
s
<
p95
0.658034
s
<
p99
0.969397
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007583
s
SD
0.00187
s
p50
0.007967
s
<
p95
0.010151
s
<
p99
0.011458
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.257473
s
SD
0.574653
s
p50
0.135911
s
<
p95
0.644537
s
<
p99
3.810314
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.352938
s
SD
0.59115
s
p50
0.090091
s
<
p95
1.548995
s
<
p99
2.885075
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008055
s
SD
0.006793
s
p50
0.007107
s
<
p95
0.019445
s
<
p99
0.030122
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.164336
s
SD
0.205842
s
p50
0.07752
s
<
p95
0.654583
s
<
p99
0.964714
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006275
s
SD
0.001549
s
p50
0.00663
s
<
p95
0.008315
s
<
p99
0.0096
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.632995
s
SD
3.494118
s
p50
0.412981
s
<
p95
1.199122
s
<
p99
24.927858
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.35165
s
SD
0.591203
s
p50
0.088342
s
<
p95
1.547144
s
<
p99
2.884339
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001953
s
SD
0.000951
s
p50
0.001927
s
<
p95
0.00393
s
<
p99
0.005462
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.024785
s
SD
0.060769
s
p50
0.006568
s
<
p95
0.145808
s
<
p99
0.2196
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
4.2e-05
s
SD
3.5e-05
s
p50
3.8e-05
s
<
p95
9.7e-05
s
<
p99
0.00018
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002124
s
SD
0.008373
s
p50
0.000614
s
<
p95
0.019887
s
<
p99
0.046043
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.003625
s
SD
0.008947
s
p50
0.000635
s
<
p95
0.019711
s
<
p99
0.052539
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.624821
s
SD
2.520556
s
p50
0.277554
s
<
p95
1.494018
s
<
p99
4.136436
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.064564
s
SD
0.246029
s
p50
0.05619
s
<
p95
0.523681
s
<
p99
1.016249
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.072797
s
SD
0.084414
s
p50
0.021167
s
<
p95
0.25171
s
<
p99
0.317379
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.7e-05
s
SD
9e-06
s
p50
1.7e-05
s
<
p95
2.9e-05
s
<
p99
6.1e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003824
s
SD
0.002371
s
p50
0.003242
s
<
p95
0.007857
s
<
p99
0.015273
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001154
s
SD
0.002498
s
p50
0.000429
s
<
p95
0.005506
s
<
p99
0.014979
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.24
s
<
mean
426.02
s
SD
249.11
s
<
p95
816.99
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
74
over 889.628227512s
mean rate
113341.41
/s
std
1.25227017e+06
/s
p5
0
/s
p95
194505.57
/s
peak
8.034714446e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
114.377205
s
SD
42.16933
s
p50
0
s
<
p95
62.219547
s
<
p99
246.385724
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.277457
s
SD
1.134081
s
p50
0.002538
s
<
p95
3.166712
s
<
p99
3.898106
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.536783
s
SD
0.37998
s
p50
0.002515
s
<
p95
1.230905
s
<
p99
1.434372
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.180608
s
SD
0.205647
s
p50
0.026402
s
<
p95
0.589321
s
<
p99
0.920565
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.135445
s
SD
10.780854
s
p50
0.004263
s
<
p95
24.032613
s
<
p99
45.46445
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
11.966978
s
SD
18.333678
s
p50
0.00184
s
<
p95
39.828237
s
<
p99
78.806343
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002117
s
SD
0.001783
s
p50
0.000215
s
<
p95
0.005447
s
<
p99
0.007393
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003736
s
SD
0.00518
s
p50
0.002902
s
<
p95
0.015835
s
<
p99
0.027762
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002419
s
SD
0.008165
s
p50
0.000547
s
<
p95
0.017498
s
<
p99
0.045267
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.594752
s
SD
29.745345
s
p50
0.325402
s
<
p95
60.017155
s
<
p99
60.046034
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.246464
s
SD
23.827546
s
p50
0.154493
s
<
p95
60.001901
s
<
p99
60.002113
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.177071
s
SD
29.209305
s
p50
0.211549
s
<
p95
60.002005
s
<
p99
60.002317
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.207268
s
SD
20.489122
s
p50
0.176392
s
<
p95
60.037306
s
<
p99
60.084942
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
21.93
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.72
KiB/s
<
mean
318.69
KiB/s
SD
364.61
KiB/s
<
p95
781.09
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.06
KiB/s
<
mean
84.79
KiB/s
SD
126.37
KiB/s
<
p95
290.40
KiB/s
Total bytes received
Total bytes received on primary network interface
count
37.67
GiB
mean
43.34
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.96
GiB
mean
11.46
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.6%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (51.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.21
%
<
mean
32.95
%
SD
33.93
%
<
p95
90.95
%
System
11.02
%
CPU percentiles
CPU usage percentiles
p50
13.21
%
p95
90.95
%
p99
94.60
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
107
hosts
mean time
241.05
s
Memory used percentage
Percentage of memory used
p5
7.38
%
<
mean
14.1
%
SD
6.7
%
<
p95
29.12
%
Memory available percentage
Percentage of available memory
p5
70.88
%
<
mean
85.9
%
SD
6.7
%
<
p95
92.62
%
Max host memory used
Maximum memory usage percentage across all hosts
max
48.51
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
51.19
%
Memory growth rate
Rate of memory growth over time
growth
0.50
MiB/s
Disk read throughput
Disk read throughput in MiB/s
192.33
MiB/s
Disk write throughput
Disk write throughput in MiB/s
123.18
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/114
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.55
5 min
1.63
15 min
1.64
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
28.67
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
16.462
%
SD
25.2677
%
<
p95
69.86
%
60 second average
15.6221
%
300 second average
16.2943
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0109
%
SD
0.1462
%
≤
p95
0
%
60 second average
0.0125
%
300 second average
0.0178
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0062
%
SD
0.0957
%
≤
p95
0
%
60 second average
0.0080
%
300 second average
0.0142
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.3777
%
SD
8.0706
%
<
p95
17.27
%
60 second average
4.2403
%
300 second average
4.2042
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.7757
%
SD
3.4328
%
<
p95
8.94
%
60 second average
1.7755
%
300 second average
1.8205
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.73
%
<
mean
29.38
%
SD
32.49
%
<
p95
84.66
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
245.68
KiB
<
mean
379.64
KiB
SD
88.43
KiB
<
p95
509.61
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
31.04
threads
SD
7.66
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
60
file descriptors
<
mean
73.95
file descriptors
SD
13.64
file descriptors
<
p95
87
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Thu, 27 Aug 2026 14:19:54 BST
Duration
900s
Peer count
150
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_33072482790_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-3.897556
s
SD
42.264439
s
Max
Highest per-partition mean latency
max mean
45.722806
s
Min
Lowest per-partition mean latency
min mean
-18.935181
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.309504
s
SD
0.513787
s
Max
Highest per-partition mean latency
max mean
14.213229
s
Min
Lowest per-partition mean latency
min mean
0.062552
s
Error count
The total number of errors accumulated during the run.
4897
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
23
over 690.01002976s
mean rate
0.03
/s
std
0.12
/s
p5
0
/s
p95
0.4
/s
peak
0.6
/s
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
41239991
over 889.577958519s
mean rate
3.61607379629e+09
/s
std
4.65418164369e+10
/s
p5
8.60882409e+06
/s
p95
5.01260982263e+09
/s
peak
2.45732624321e+12
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
6646063
over 889.577958519s
mean rate
8.7221561297e+08
/s
std
1.96908764373e+10
/s
p5
348610.29
/s
p95
1.37778558265e+09
/s
peak
1.29728173969e+12
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.577958519s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.142549
s
SD
0.519508
s
p50
0.136666
s
<
p95
0.292288
s
<
p99
0.618719
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.003824
s
SD
0.001098
s
p50
0.004036
s
<
p95
0.005198
s
<
p99
0.005424
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.415817
s
SD
1.929212
s
p50
0.389583
s
<
p95
0.807531
s
<
p99
2.43972
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002426
s
SD
0.000685
s
p50
0.002559
s
<
p95
0.003195
s
<
p99
0.003602
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004772
s
SD
0.005645
s
p50
0.004047
s
<
p95
0.00882
s
<
p99
0.041491
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.412908
s
SD
1.929251
s
p50
0.386384
s
<
p95
0.805291
s
<
p99
2.43652
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.001
s
SD
0.002571
s
p50
0.000645
s
<
p95
0.001286
s
<
p99
0.013749
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.3e-05
s
SD
6e-06
s
p50
1.2e-05
s
<
p95
2.4e-05
s
<
p99
2.7e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000348
s
SD
0.000942
s
p50
0.000364
s
<
p95
0.0011
s
<
p99
0.003973
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000533
s
SD
0.001017
s
p50
0.000212
s
<
p95
0.002273
s
<
p99
0.005491
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.46
s
<
mean
427.73
s
SD
248.9
s
<
p95
815.48
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1848
over 889.578048892s
mean rate
31496.41
/s
std
727991.24
/s
p5
7.59
/s
p95
71802.07
/s
peak
5.462040536e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
12.086697
s
SD
9.424413
s
p50
8.591237
s
<
p95
33.097842
s
<
p99
49.93782
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.010294
s
SD
0.018579
s
p50
0.010415
s
<
p95
0.045585
s
<
p99
0.078085
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.074807
s
SD
0.059626
s
p50
0.075461
s
<
p95
0.175626
s
<
p99
0.31231
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.01936
s
SD
0.012015
s
p50
0.018196
s
<
p95
0.045491
s
<
p99
0.06047
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.625961
s
SD
4.906498
s
p50
4.805943
s
<
p95
11.988081
s
<
p99
21.969687
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.06487
s
SD
2.551409
s
p50
0.580443
s
<
p95
7.133072
s
<
p99
12.923516
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000456
s
SD
0.000248
s
p50
0.000411
s
<
p95
0.000853
s
<
p99
0.001306
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00336
s
SD
0.001967
s
p50
0.002764
s
<
p95
0.006236
s
<
p99
0.008433
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000579
s
SD
0.000452
s
p50
0.00058
s
<
p95
0.000947
s
<
p99
0.001151
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.209676
s
SD
26.662125
s
p50
0.083617
s
<
p95
60.00315
s
<
p99
60.005459
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.108865
s
SD
3.992068
s
p50
0.063518
s
<
p95
0.216153
s
<
p99
0.272909
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.154546
s
SD
21.496397
s
p50
0.156031
s
<
p95
60.001161
s
<
p99
60.001846
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
5.14
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
9.96
KiB/s
<
mean
110.26
KiB/s
SD
304.57
KiB/s
<
p95
243.52
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.48
KiB/s
<
mean
16.55
KiB/s
SD
16.81
KiB/s
<
p95
42.57
KiB/s
Total bytes received
Total bytes received on primary network interface
count
12.57
GiB
mean
14.46
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.89
GiB
mean
2.17
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1.87 MiB/s

Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.96
%
<
mean
9.55
%
SD
10.88
%
<
p95
32.12
%
System
4.41
%
CPU percentiles
CPU usage percentiles
p50
6.25
%
p95
32.12
%
p99
62.80
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
67
hosts
mean time
11.15
s
Memory used percentage
Percentage of memory used
p5
5.84
%
<
mean
11.07
%
SD
6.42
%
<
p95
26.85
%
Memory available percentage
Percentage of available memory
p5
73.16
%
<
mean
88.93
%
SD
6.42
%
<
p95
94.16
%
Max host memory used
Maximum memory usage percentage across all hosts
max
43.41
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
1.69
%
Memory growth rate
Rate of memory growth over time
growth
1.87
MiB/s
Disk read throughput
Disk read throughput in MiB/s
3533.60
MiB/s
Disk write throughput
Disk write throughput in MiB/s
55.90
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/107
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/19
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.36
5 min
0.31
15 min
0.31
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
2.0081
%
SD
2.8659
%
<
p95
7.24
%
60 second average
1.9987
%
300 second average
1.6899
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.1003
%
SD
0.8737
%
≤
p95
0
%
60 second average
0.1027
%
300 second average
0.1072
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0985
%
SD
0.8627
%
≤
p95
0
%
60 second average
0.1010
%
300 second average
0.1056
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.9445
%
SD
4.911
%
<
p95
2.78
%
60 second average
1.1695
%
300 second average
1.3241
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8255
%
SD
4.5001
%
<
p95
2.53
%
60 second average
1.0418
%
300 second average
1.2048
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.63
%
<
mean
6.17
%
SD
9.13
%
<
p95
17.62
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
210.80
KiB
<
mean
236.19
KiB
SD
32.88
KiB
<
p95
276.94
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
33.27
threads
SD
11.32
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
80.39
file descriptors
SD
13.97
file descriptors
<
p95
88
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Thu, 27 Aug 2026 14:36:59 BST
Duration
900s
Peer count
300
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_signals_33072482790_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.356095
s
SD
2.209737
s
p50
0.20437
s
<
p95
0.406553
s
<
p99
4.494394
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
282
over 2614.41481666s
mean rate
559168.26
/s
std
3.975948769e+07
/s
p5
0
/s
p95
43395.12
/s
peak
5e+09
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0015
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 772.480058857s
Not enough time data to show a trend.
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 890.092421271s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
36562345
over 890.092421271s
mean rate
4.48552790257e+09
/s
std
5.29221158904e+10
/s
p5
4.23291608e+06
/s
p95
1.12421814155e+10
/s
peak
2.91431279621e+12
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
12322080
over 890.092421271s
mean rate
1.2671331195e+08
/s
std
3.57094444928e+09
/s
p5
0
/s
p95
1.449367554e+08
/s
peak
3.39287671233e+11
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.003988
s
SD
0.001354
s
p50
0.004349
s
<
p95
0.005605
s
<
p99
0.006639
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.003879
s
SD
0.001199
s
p50
0.004221
s
<
p95
0.005399
s
<
p99
0.005719
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005985
s
SD
0.014086
s
p50
0.003843
s
<
p95
0.013219
s
<
p99
0.067428
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002576
s
SD
0.000879
s
p50
0.002811
s
<
p95
0.003563
s
<
p99
0.004223
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002431
s
SD
0.000747
s
p50
0.002683
s
<
p95
0.003357
s
<
p99
0.003695
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000738
s
SD
0.00065
s
p50
0.000587
s
<
p95
0.001699
s
<
p99
0.003073
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.2e-05
s
SD
2.8e-05
s
p50
8.1e-05
s
<
p95
0.00011
s
<
p99
0.000128
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.2e-05
s
SD
3.4e-05
s
p50
7.9e-05
s
<
p95
0.000117
s
<
p99
0.000153
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
5e-06
s
p50
9e-06
s
<
p95
1.8e-05
s
<
p99
2.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001796
s
SD
0.012921
s
p50
0.000341
s
<
p95
0.002074
s
<
p99
0.062671
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1906
over 890.092306928s
mean rate
196492.44
/s
std
3.13423526e+06
/s
p5
141.4
/s
p95
530863.28
/s
peak
2.9213483146e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1990
over 890.092307003s
mean rate
153026.79
/s
std
1.08874268e+06
/s
p5
165.46
/s
p95
484372.5
/s
peak
6.319385141e+07
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000692
s
SD
0.002284
s
p50
0.000208
s
<
p95
0.002529
s
<
p99
0.009814
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
40.52
s
<
mean
425.44
s
SD
250.19
s
<
p95
815.64
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3576
over 890.092254388s
mean rate
1.55857137e+06
/s
std
4.792883947e+07
/s
p5
84.72
/s
p95
2.21714282e+06
/s
peak
3.64179104478e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
17.267742
s
SD
10.373476
s
p50
13.772029
s
<
p95
36.183587
s
<
p99
47.378762
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.020784
s
SD
0.030176
s
p50
0.019931
s
<
p95
0.075978
s
<
p99
0.150483
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.100397
s
SD
0.086348
s
p50
0.096377
s
<
p95
0.220641
s
<
p99
0.4286
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.025307
s
SD
0.016833
s
p50
0.024261
s
<
p95
0.05735
s
<
p99
0.083386
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
7.293757
s
SD
6.17682
s
p50
8.855811
s
<
p95
20.763395
s
<
p99
30.173854
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.256607
s
SD
7.559023
s
p50
1.313759
s
<
p95
8.5813
s
<
p99
38.484099
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000642
s
SD
0.000387
s
p50
0.000544
s
<
p95
0.001294
s
<
p99
0.001753
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003849
s
SD
0.00235
s
p50
0.002972
s
<
p95
0.007775
s
<
p99
0.009857
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000708
s
SD
0.000667
s
p50
0.000727
s
<
p95
0.001414
s
<
p99
0.003026
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.297168
s
SD
29.47788
s
p50
0.3416
s
<
p95
60.004086
s
<
p99
60.007382
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.160254
s
SD
18.625132
s
p50
0.128417
s
<
p95
60.001157
s
<
p99
60.001675
s
P2P received remote signals
Count of remote signals received.
total
1905
over 890.092268766s
mean rate
200996.27
/s
std
1.90338431e+06
/s
p5
141.32
/s
p95
677712.19
/s
peak
1.1986446421e+08
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
10.79
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
23.51
KiB/s
<
mean
198.36
KiB/s
SD
326.63
KiB/s
<
p95
588.21
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
18.29
KiB/s
<
mean
41.20
KiB/s
SD
38.81
KiB/s
<
p95
102.53
KiB/s
Total bytes received
Total bytes received on primary network interface
count
22.74
GiB
mean
26.16
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.61
GiB
mean
5.31
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.1%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 5.83 MiB/s

Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.95
%
<
mean
17.32
%
SD
15.81
%
<
p95
42.58
%
System
6.53
%
CPU percentiles
CPU usage percentiles
p50
12.95
%
p95
42.58
%
p99
95.08
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
102
hosts
mean time
32.98
s
Memory used percentage
Percentage of memory used
p5
7.53
%
<
mean
13.94
%
SD
6.83
%
<
p95
29.68
%
Memory available percentage
Percentage of available memory
p5
70.32
%
<
mean
86.06
%
SD
6.83
%
<
p95
92.47
%
Max host memory used
Maximum memory usage percentage across all hosts
max
48.39
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
1.69
%
Memory growth rate
Rate of memory growth over time
growth
5.83
MiB/s
Disk read throughput
Disk read throughput in MiB/s
3143.28
MiB/s
Disk write throughput
Disk write throughput in MiB/s
100.10
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/109
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.69
5 min
0.59
15 min
0.45
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
6.4416
%
SD
11.3784
%
<
p95
20.96
%
60 second average
6.2426
%
300 second average
5.2074
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0712
%
SD
0.591
%
≤
p95
0
%
60 second average
0.0713
%
300 second average
0.0763
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0684
%
SD
0.5726
%
≤
p95
0
%
60 second average
0.0686
%
300 second average
0.0737
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.3299
%
SD
5.2135
%
<
p95
4.21
%
60 second average
1.3728
%
300 second average
1.2875
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.0004
%
SD
4.3723
%
<
p95
3.13
%
60 second average
1.0409
%
300 second average
1.0127
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.9
%
<
mean
6.82
%
SD
7.75
%
<
p95
19.25
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
192.74
KiB
<
mean
222.99
KiB
SD
40.80
KiB
<
p95
268.01
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
33.19
threads
SD
11.39
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
74
file descriptors
<
mean
79.77
file descriptors
SD
15.83
file descriptors
<
p95
89
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Thu, 27 Aug 2026 15:26:27 BST
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
validation_receipts_33072482790_1
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
35.638993
s
SD
17.203713
s
Max
Highest per-partition mean latency
max mean
491.314495
s
Min
Lowest per-partition mean latency
min mean
16.354568
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
0.99
/s
Max
Highest per-partition mean rate
max
1.04
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4
over 766.810280024s
mean rate
6.86
/s
std
9.84
/s
p5
0
/s
p95
24.53
/s
peak
25.11
/s
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
679155
over 889.22719252s
mean rate
8.219727065e+07
/s
std
4.20254041655e+09
/s
p5
0
/s
p95
6.635997572e+07
/s
peak
3.39709609293e+11
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
11466581
over 889.22719252s
mean rate
4.9464967842e+08
/s
std
2.01121064313e+10
/s
p5
461371.89
/s
p95
5.789161745e+08
/s
peak
1.55182365364e+12
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.22719252s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
83638
over 889.22719252s
mean rate
9.9771401e+06
/s
std
5.1330230037e+08
/s
p5
0
/s
p95
8.05733819e+06
/s
peak
4.18352692714e+10
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006979
s
SD
0.00236
s
p50
0.00748
s
<
p95
0.01032
s
<
p99
0.011777
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.021402
s
SD
0.017948
s
p50
0.012381
s
<
p95
0.04831
s
<
p99
0.097825
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005737
s
SD
0.001937
s
p50
0.0062
s
<
p95
0.00842
s
<
p99
0.00957
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.019924
s
SD
0.017548
s
p50
0.010853
s
<
p95
0.046377
s
<
p99
0.095307
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004352
s
SD
0.005231
s
p50
0.003464
s
<
p95
0.011948
s
<
p99
0.022394
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001889
s
SD
0.000637
s
p50
0.002034
s
<
p95
0.002668
s
<
p99
0.003162
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000696
s
SD
0.002822
s
p50
0.000663
s
<
p95
0.001909
s
<
p99
0.009675
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.017663
s
SD
0.017286
s
p50
0.008368
s
<
p95
0.043517
s
<
p99
0.092383
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002945
s
SD
0.001029
s
p50
0.003187
s
<
p95
0.004305
s
<
p99
0.005502
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000663
s
SD
0.000651
s
p50
0.000499
s
<
p95
0.001897
s
<
p99
0.002776
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.67
s
<
mean
428.03
s
SD
247.57
s
<
p95
815.13
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
8389
over 889.227046686s
mean rate
762464.79
/s
std
4.031285205e+07
/s
p5
512.6
/s
p95
410652.4
/s
peak
3.10891089109e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
18.498995
s
SD
11.961986
s
p50
16.793288
s
<
p95
37.114276
s
<
p99
63.444186
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005361
s
SD
0.044962
s
p50
0.009023
s
<
p95
0.130603
s
<
p99
0.224979
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.042708
s
SD
0.116299
s
p50
0.074826
s
<
p95
0.373687
s
<
p99
0.519359
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.078334
s
SD
0.064386
s
p50
0.037189
s
<
p95
0.183209
s
<
p99
0.321506
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.361217
s
SD
7.824515
s
p50
1.013609
s
<
p95
20.651734
s
<
p99
31.244218
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
23.029733
s
SD
19.199098
s
p50
22.277579
s
<
p95
59.090966
s
<
p99
73.574932
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001338
s
SD
0.000874
s
p50
0.00086
s
<
p95
0.00304
s
<
p99
0.003807
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005164
s
SD
0.003392
s
p50
0.002997
s
<
p95
0.008049
s
<
p99
0.023487
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000645
s
SD
0.000268
s
p50
0.000599
s
<
p95
0.001124
s
<
p99
0.001532
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.435672
s
SD
29.512974
s
p50
0.246404
s
<
p95
60.003321
s
<
p99
60.005837
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.141402
s
SD
4.145911
s
p50
0.133
s
<
p95
0.247209
s
<
p99
0.305072
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
15.20
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
16.16
KiB/s
<
mean
141.16
KiB/s
SD
355.40
KiB/s
<
p95
399.12
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
7.28
KiB/s
<
mean
23.56
KiB/s
SD
24.65
KiB/s
<
p95
70.92
KiB/s
Total bytes received
Total bytes received on primary network interface
count
17.13
GiB
mean
19.70
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.86
GiB
mean
3.29
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (30.9% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 8% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.22
%
<
mean
22.3
%
SD
18.25
%
<
p95
62.88
%
System
7.10
%
CPU percentiles
CPU usage percentiles
p50
19.02
%
p95
62.88
%
p99
88.73
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
83
hosts
mean time
33.61
s
Memory used percentage
Percentage of memory used
p5
6.02
%
<
mean
11.75
%
SD
6.84
%
<
p95
25.93
%
Memory available percentage
Percentage of available memory
p5
74.07
%
<
mean
88.25
%
SD
6.84
%
<
p95
93.98
%
Max host memory used
Maximum memory usage percentage across all hosts
max
56.87
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
30.93
%
Memory growth rate
Rate of memory growth over time
growth
0.38
MiB/s
Disk read throughput
Disk read throughput in MiB/s
226.02
MiB/s
Disk write throughput
Disk write throughput in MiB/s
68.15
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/116
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.91
5 min
1.56
15 min
2.46
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
8.28
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
6.7479
%
SD
9.8823
%
<
p95
25.53
%
60 second average
7.0059
%
300 second average
14.4257
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0129
%
SD
0.1686
%
≤
p95
0
%
60 second average
0.0139
%
300 second average
0.0139
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0078
%
SD
0.1204
%
≤
p95
0
%
60 second average
0.0091
%
300 second average
0.0095
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.7177
%
SD
8.9953
%
<
p95
16.26
%
60 second average
3.7471
%
300 second average
5.1518
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.773
%
SD
7.3609
%
<
p95
10.33
%
60 second average
2.7453
%
300 second average
2.7797
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.4
%
<
mean
18.38
%
SD
16.23
%
<
p95
53.25
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
219.24
KiB
<
mean
263.63
KiB
SD
43.64
KiB
<
p95
309.64
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
32.15
threads
SD
11.41
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
79.33
file descriptors
SD
15.08
file descriptors
<
p95
87
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 27 Aug 2026 15:43:45 BST
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_33072482790_1
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1788.26
Max
Highest chain head value observed for any single write agent.
3930
Write agents observed
19
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.372821
s
SD
0.95077
s
Max
Highest per-partition mean latency
max mean
1.160463
s
Min
Lowest per-partition mean latency
min mean
0.005198
s
Error count
The number of errors accumulated.
345
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.060942
s
SD
3.521118
s
p50
0.072261
s
<
p95
7.542732
s
<
p99
14.641926
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 811.389241804s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.00707
s
SD
0.002568
s
p50
0.007011
s
<
p95
0.011977
s
<
p99
0.013869
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008656
s
SD
0.009151
s
p50
0.007667
s
<
p95
0.033008
s
<
p99
0.043524
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.132602
s
SD
1.31662
s
p50
0.143391
s
<
p95
2.566483
s
<
p99
4.376146
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006318
s
SD
3.379893
s
p50
0.008718
s
<
p95
11.117767
s
<
p99
14.248633
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005931
s
SD
0.0022
s
p50
0.005885
s
<
p95
0.010372
s
<
p99
0.011805
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007491
s
SD
0.008362
s
p50
0.006468
s
<
p95
0.029914
s
<
p99
0.03976
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.133402
s
SD
1.577186
s
p50
0.149819
s
<
p95
2.573585
s
<
p99
4.762477
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006065
s
SD
4.609932
s
p50
0.007354
s
<
p95
11.116413
s
<
p99
29.402632
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001547
s
SD
0.000736
s
p50
0.00169
s
<
p95
0.00296
s
<
p99
0.004299
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.5e-05
s
SD
4.7e-05
s
p50
4.3e-05
s
<
p95
0.000129
s
<
p99
0.000294
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002661
s
SD
0.005522
s
p50
0.001498
s
<
p95
0.017512
s
<
p99
0.024799
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000607
s
SD
0.000384
s
p50
0.000564
s
<
p95
0.001291
s
<
p99
0.002281
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.131463
s
SD
1.577004
s
p50
0.145759
s
<
p95
2.571182
s
<
p99
4.759851
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002126
s
SD
4.609868
s
p50
0.001038
s
<
p95
11.110373
s
<
p99
29.399947
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
8e-06
s
p50
1.1e-05
s
<
p95
3e-05
s
<
p99
4.6e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00255
s
SD
0.001677
s
p50
0.002796
s
<
p95
0.006272
s
<
p99
0.010202
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000975
s
SD
0.001213
s
p50
0.000257
s
<
p95
0.002831
s
<
p99
0.006835
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.06
s
<
mean
427.23
s
SD
248.48
s
<
p95
815.46
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
12259
over 889.08034743s
mean rate
319029.35
/s
std
1.65309067e+06
/s
p5
941.11
/s
p95
1.03855112e+06
/s
peak
5.497144555e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
80.53689
s
SD
64.305482
s
p50
20.599031
s
<
p95
152.561185
s
<
p99
353.923888
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.079613
s
SD
0.284192
s
p50
0.088218
s
<
p95
0.666137
s
<
p99
1.401811
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.126805
s
SD
0.426319
s
p50
0.220637
s
<
p95
1.256974
s
<
p99
1.454716
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.449839
s
SD
0.426932
s
p50
0.064297
s
<
p95
1.150157
s
<
p99
2.179335
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
9.34703
s
SD
20.594155
s
p50
14.870506
s
<
p95
63.791215
s
<
p99
90.683883
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
14.728896
s
SD
31.109524
s
p50
0.472421
s
<
p95
85.992158
s
<
p99
140.789725
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003434
s
SD
0.003234
s
p50
0.002429
s
<
p95
0.01005
s
<
p99
0.014486
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004678
s
SD
0.004377
s
p50
0.003154
s
<
p95
0.014207
s
<
p99
0.022651
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002778
s
SD
0.004548
s
p50
0.000847
s
<
p95
0.014257
s
<
p99
0.022906
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.231986
s
SD
29.233864
s
p50
0.931781
s
<
p95
60.013824
s
<
p99
60.040754
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
2.281026
s
SD
9.658633
s
p50
0.475882
s
<
p95
6.217802
s
<
p99
60.000983
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.115025
s
SD
28.872825
s
p50
0.148936
s
<
p95
60.003645
s
<
p99
60.015307
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.516076
s
SD
19.825007
s
p50
0.657041
s
<
p95
60.001251
s
<
p99
60.005554
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
47.88
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
7.48
KiB/s
<
mean
550.10
KiB/s
SD
459.71
KiB/s
<
p95
1.34
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.36
KiB/s
<
mean
169.18
KiB/s
SD
234.04
KiB/s
<
p95
640.51
KiB/s
Total bytes received
Total bytes received on primary network interface
count
65.19
GiB
mean
75.00
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
20.05
GiB
mean
23.07
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.4%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (31.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 57% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.8
%
<
mean
64.25
%
SD
31.24
%
<
p95
96.09
%
System
16.38
%
CPU percentiles
CPU usage percentiles
p50
77.35
%
p95
96.09
%
p99
98.40
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
115
hosts
mean time
515.30
s
Memory used percentage
Percentage of memory used
p5
6.45
%
<
mean
12.46
%
SD
7.25
%
<
p95
27.65
%
Memory available percentage
Percentage of available memory
p5
72.35
%
<
mean
87.54
%
SD
7.25
%
<
p95
93.55
%
Max host memory used
Maximum memory usage percentage across all hosts
max
57.40
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
31.23
%
Memory growth rate
Rate of memory growth over time
growth
0.52
MiB/s
Disk read throughput
Disk read throughput in MiB/s
124.86
MiB/s
Disk write throughput
Disk write throughput in MiB/s
221.05
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/114
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.50
5 min
2.70
15 min
2.28
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
57.34
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.12
%
<
mean
39.1094
%
SD
31.2678
%
<
p95
88.57
%
60 second average
36.6035
%
300 second average
27.5108
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0081
%
SD
0.1205
%
≤
p95
0
%
60 second average
0.0085
%
300 second average
0.0091
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0033
%
SD
0.0537
%
≤
p95
0
%
60 second average
0.0035
%
300 second average
0.0040
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.3385
%
SD
9.1417
%
<
p95
24.9
%
60 second average
7.9204
%
300 second average
6.5886
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.0836
%
SD
4.6779
%
<
p95
11.74
%
60 second average
2.9730
%
300 second average
2.8254
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.7
%
<
mean
60.64
%
SD
29.49
%
<
p95
92.68
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
215.47
KiB
<
mean
319.93
KiB
SD
97.15
KiB
<
p95
456.85
KiB
Holochain process threads
Number of threads in Holochain process
p5
26
threads
<
mean
32.99
threads
SD
11.45
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
69
file descriptors
<
mean
80.24
file descriptors
SD
15.38
file descriptors
<
p95
89
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 27 Aug 2026 16:00:54 BST
Duration
1800s
Peer count
150
Peer count at end
123
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_33072482790_1
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
264.92
/s
Max
Highest per-partition mean rate
max
640.25
/s
Min
Lowest per-partition mean rate
min
86.42
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.031475
s
SD
0.233233
s
Max
Highest per-partition mean latency
max mean
0.096522
s
Min
Lowest per-partition mean latency
min mean
0.009668
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
61
conductors
<
mean
77.06
conductors
SD
11.52
conductors
<
p95
91
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
971.983571
s
SD
581.653183
s
Max
Highest per-partition mean latency
max mean
1680.473684
s
Min
Lowest per-partition mean latency
min mean
586.488372
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
53.67647
s
SD
46.78068
s
Max
Highest per-partition mean latency
max mean
140.273007
s
Min
Lowest per-partition mean latency
min mean
25.030768
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
46.113498
s
SD
50.12743
s
Max
Highest per-partition mean latency
max mean
97.594728
s
Min
Lowest per-partition mean latency
min mean
8.548863
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk-ADfzK9oxpkkKNu521Dk-71y6wOmOIZL4PB17t-gYDQLFwFf
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0ROoliZHLd8TTv4-PH4H80AUlZFPH71aFB16HK66lB5jN6_e
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0_fjgVGlfVmneQyK1TUsth21xJsEsr1CWSFlyeVeCWNLd7XU
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0aUd_rhx_2IB4TWU63INFCVwQH418C9KT8DqMZ3Jv39sNua0
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2MOmD-c6Q-bhvYpQLDxNgOHZqOpqyph3E9x6OYcW-Os-KKnP
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3d0bVwXqW05z7WO5hRROpZ7jK1Dl-ZXdXPl2RPdMa3U4kCRJ
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk43vRZVEsXfjusmA9Xo83qRBx306JwX-Pq-VuujaoBdYKyBez
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4FlcTSMpKQSZKrTBwGXMBeL1CgDE20jtVGKekUyqjV45hBr0
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4P0Xq8uWoB631QMwXpiDvpBzzj8xUIMsScVIcrp3XOMJiRan
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4ZEPOwBAlE99HmqGp6EWlT5lKkQCgyCtrbpQWxM1OfYlFWkr
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4sanSmgyYfxup5abw29svUZt4D-ri94bB_j42Zz8XHuqpA-N
p5
0
<
mean
0.44
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk5Ryx5lgXxU4j3OzmovVFKKgaGSgDfs9E5XDcMBe6D1lekixD
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk663M3njXsYf8n94nlaWoPxvBvqYnCizYeAIYtykDWCJG6sSR
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk6EHiXpxHO7HI2wajCZmVRU3PfKXq5CeSWgo5Wh23huIpHoLe
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk7Q6zhHwRb9OcL1qaddsmk2_fD9WrOrWqY6afeyJf3GQSR0_3
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk89rN33Iuxpo4O5yKPPmoKTGGX-jrPGqm5b-qhJ04ayfpqHua
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk8Or_OHsbpVrPKjZdEVKVFWI79rfcWEb7sxCbDjgpsTw-Ga4L
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBFyjLmWIvh3bUE3BFUX8VbtWF-5iC4ErQlv4WeOk_I6vopbL
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBi1U8XAulpaJtJBjbk4T88OeiW9xAqTDltGn_5jolk3uW8Wz
p5
0
<
mean
0.42
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBwwivGgLpJAGHorV56L4XFO_ADmvL_PGKrkSm2yPOSKMktw8
p5
0
<
mean
0.18
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkC_S7Zt1AhBtyQFqj9mWdw_8BbfxLOmgX2uygqL7TOgcN2J69
p5
0
<
mean
0.46
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkDu2K2K_kmq-QdBH4IixqSlG3y_KuqNigGIM3y0HY9LtpauXk
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkE04YzHgDwqkaEFMj5E-KrOXdLpXPZVL6iiG1u0yB7FgFYXFy
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkEC0I3ArNnUILgj0hTuO0t7u9qp0kbwreOhm_iff8FkTMEHOL
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFIL8oaip05J1hJIWlsNb_vvALpIuYWk0b1BRGcjO1iJmmS-A
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkG4WiAkcgg5I46oun3eO4VZyuoaHzz5K7FxRHOnjRKK0UVX1H
p5
0
<
mean
0.48
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHUXvgDr8huBFMprzVUgMLQCa-gDSim1p-z2VxBsjx2ZDpdb6
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkI-At7hesGMRqwi8uwcQuWBNEPydrBT2lQsMwO1LRmPAZHO6B
p5
0
<
mean
0.4
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIv33zkaD-3DixejkrxwJnR8uMkmX6URPZp-dJcHMfMXHRzl6
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkJoJrcitTDQdFQVf0BxZ1620_Dp6MXmGmT_saWSEWv5U85k-D
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkK2nFj_flJVxUnIWrP2x0JeYj06_GBH9VRXiiLK0Loaf_4zuA
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkKimjmIDmIWzvOce8-LxWXZJVvWDcxCdN5xgQgjkSLrSyR06I
p5
0
<
mean
0.22
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkLk5FxPr_GPDZNR5pvG95b-Eacy4r8OieP3UE8c7ZKn6flupG
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMUhINXMQBiZOfjplIqlLLW7yz72JnuOagU1ck5VPnFkc0eJa
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkNDW2xZCdPpjMznLujkuN9hWghhzEe13e2vtzaXss817k1fjT
p5
0
<
mean
0.18
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkNpLh-5Gr9JIL03kjqEFEGpz-4L6JuMUMEXabiQJ9cbaNaHlX
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkObHQR6QkL2dhiIrdSgHiqJWzhFws4qu_WN1nCsfReFy3RTYa
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkOqCR3DKw2-55nhkWqsn2cyqRemKskvg2KBZvICcD_hNxeNp_
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkPSMkz4BMAbaLtdXOOWDOGjyK2NP7YaIkfdrgX-NjNWrBTPNF
p5
0
<
mean
0.27
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQB8sOX54oN8a_oWZtJGCUS8H7YwxztAAr7njUNF7siEtgQcr
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQxk8p0GzEJynLG45E5xju4hx1JbsuTxOZw3HwYMH9Yy4nb3M
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQyDuEfO9PFJAekEyylHEqdhlTdPT41QammKBkI8qQISCj1lk
p5
0
<
mean
0.28
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRJnM3v4cAFY6vQzKb0w2Ry9awR-l2ahD6b64h75fUpW7YRoA
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRM6GLONlwP9Gtmw5nMlh0emGpIyimDN13pp9pTdCe-ugVPzy
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRNF3Oc1mr9XyMSF0rpRL28XDXGEVospPRk-bTJAneWz_qIGA
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRU7QVOtjjgix981pMbG3JmBqfffaka0loPw-q4Is5FdLOlM-
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSInMQUJs-FI-TCyEPCd4wI4edKIMx28-WqkmT7JZTFX-vF5_
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSi6UUCjc1C5pYTWpjOseSxAzWjhMMc0yPSoUmjT3mhWlxlEN
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTCMr0PqHQBzu1_RDG1pchh33FQrhRNfEkvs9-5YcEhbFqlzv
p5
0
<
mean
0.43
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkUeZ_k_j9SK5Nd-SbCdRMlNifb0JQEVQ49DLd97Ee9ZP9r3fM
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkV6bemv-7PmX1OzfSlleuU8UksKp7oWfMSSk7Fj1yUkzw-tOe
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkV9rl3zhwGjTIp7iYIR0K6sihrsVUwVDNe8baQr7QfakHde7s
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVJPQSTLHFPfQylcA7t7qZ9kFyUEFIuXqkV9oX-J5KqjsRrIQ
p5
0
<
mean
0.54
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVOJUayBRSZOLT4jaVuFv63G7CW7U-z-XEl5w1t6cKei3iaw0
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVgh7DPEcX52RYbl26PZajy2xJFJkzICPqQ3x7ZLhtdUJw8S8
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkWAiMudwYpcbajUBCs_N3h9YuifL-0MM10FRw8nInHvF-OIf2
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXBuk1qhWWST0rvDKIhQlmbPACsoivJPv7uF8ot6e0o8-UVVy
p5
0
<
mean
0.45
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXx8A19OfROsn_1tUKU5duWs-3vHYLoElw6QHXEgZAqhRREHT
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkY2mG5i4q99EnnZxiD0Ux9hsjZolMdbriLWByOgH6k-jdLavY
p5
0
<
mean
0.19
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkY5mbJqQZNI0aHaNhC_Kioce5j56W-8pHQal_OyqcLPn1wGIK
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYXHu_oz9fdyu4EUfe2AKGQmJz12mDt-72hV3ZaHNL4tmpYYH
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkaLKCyfy8i69INEgMcHsA84y_KE0HdH6DGdgf7MvoesIY1Lta
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAka_umrzPhQqAygDxtfpD9gq7QVwoUWxfoBwYAMl3_-E8kWUuG
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkafwUrAq2fkXA8ibBSlZqCZtWl8pILMcLtbkt_AMk1a_VEogj
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkb5HI6owuXkG6ji_dCYwFfjajLXhSOM2sXyM-liwgzTtQx2zk
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbgH0KKIbNDtXiaNs7iw75qiQbXJ9A9Q1hePqpsN6Um_t9Mpm
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbkAQQbFXBuj-RrdBQDpCzIiuBy2DGYQ1xQpE4hmdXH2atFsb
p5
0
<
mean
0.22
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcAs9NyIf6xsQVPlKWq7okjAb4Rnxq2E0mfHXkzBL1WyAbdKj
p5
0
<
mean
0.5
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcRwf5Y4I9PH50Aqt84zTO1XsHBrMeK494ncBHc1SdkqqT4B0
p5
0
<
mean
0.24
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdatLJCsk2ltx_5FBL7X4xQTMa3IxktAnsPKRgG2_Nqv8rLwQ
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdnHKxPCzqHrCb-s0f_SfnhbzZbvWNU-Zh-Tlhn4wnpEr-enY
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkeUFMG1DxL8aOb9f0Wh8VI2tqRLLLEOC4e-m76O4iI5ZhHdFT
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAke_jv_j62YlUPTP_FtdQDpYdg3Q1JTxrM1pMv1aGdCy5riw9F
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkedDU-q5cjHPm6JmDJpx8yqMnG1VIc86AlBhPf7PWop2G4Y9J
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkg7Oypswy7l2ERf3JgloFGxgS0LzWMCWNWOUykvBmtUw8tpPh
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgQxQwp2YdAUEhGO4bCQnrKMqkVSxPyqCuQrUKkqXUipfzAad
p5
0
<
mean
0.22
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgb9LjqPkxu-3Rtb_yIJgW5_W_KM9BN_pnVpREvUzwz_pEVhn
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgfk2HHfeDgwbnQT8jAhLYIsAiTRPs19JoXjB26lNCjQiKRT3
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgfzH60rXgrXVwNSL-ft0NHDqoRru1Bg_JvjqfnsBnn7boXmg
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiQhEg2cXssPDPkS1fFwAGPKGahnq3Lu48PDuMDhKjFG4qvM0
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkimwNF27rJI6T3K-r_fhWzHTPMS8VTpmqfIngHpcVh24Lw6js
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjXxdwl7IuxE4wDQGZ8Ma7ArrfIP0DU37TTASqHovrYDlB65b
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjYh4gzVIeSkMKr4Z9OQWhW2wOrQ1YHO-U5sju6TQ25_mqjoE
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkk5Xv8M88roWulYveWZktk0gRplgSChHyM3SFOsLLyWSBPNAs
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkk8hLXWv4iyx9wCxTXx6IQwIvzV4SqHO9I0fQtWJkF7Ptcz2j
p5
0
<
mean
0.34
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkJHYaFPpLcJkQ0XrO_jNjOptnzu6mnai44VV5j0IsQr9qRcm
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkO_n2CfQWnFr8_ORiVoErNlWHUqz8YpNcS1xYuHvnWwpOSOl
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkPQibyJZ3bsbd5TbG8E3LPFtyoHEvCLKNw4S4zjYYlxoolho
p5
0
<
mean
0.29
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkm8jiMBbHjqgpJd80DxP7pDL59QYo_nWeVO8eiSYCPC2xMbe4
p5
0
<
mean
0.27
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmrPvxuAatKSgK7rHpS8_Z9Y0_6mmhoIivl3f9KahGBvk4INQ
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkn6Jj2d5WvKNRSpFbhd2th2erlJm6Q4JnHtrQ4gbz3d7yvon-
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAko-YIoSuyWBucU1zGA8AwsXBUtHEG5c6yobli5EphstVu9RDN
p5
0
<
mean
0.18
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpaK6x0GY8SbRV9egqN-i0NhmAp3ealBki_CMAMXGbRj921Zr
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpxjPgF08uWkDDAHEJYzxTfhH6OUuG9d734cuOnZy3c18-GTQ
p5
0
<
mean
0.39
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkq2EgeWu5gnDXkQcALq1A5FHrl0R9dYz2Fr8Nc2hF0Vt8qEk_
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrPFixB4Y8ReqzM5KkXhNqzdTxtomIZgTIR4s_S499OkCZNVm
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrgi3BCL_73uHBSdSiNQuOyFOxiqF01gvp1HD7vJcZFbrWA5G
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAktBxxo3Lk_lPaz2S3dIcATbDKy5yxNQe_tdMNIJUxYEEcjub7
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAktjpToTYzTxlJbkSa5zXQwtlFAnFgvkItyUqkPzoWw0VFB29m
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkun-WzkoJ3ySNVCVMEdgYsWbsvjbe3wjmRqIfIHcIoOjh0PhV
p5
0
<
mean
0.32
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxP4-1smuw2n91hf7l3Egecg_bQx9TYkM98X_5ehqdtXmj7vz
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxmd3u4ZJZ1H1cLkF9L6xcfDCbuYjYMmPWxaYeNITAJkM3MSC
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkyIa8ANuFxgzF7LHmvicz3t89HXlmHItrBXNc9PYzHtHAXDHU
p5
0
<
mean
0.21
SD
0.41
<
p95
1
Error count
The number of errors accumulated.
1053
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.038363
s
SD
25.818729
s
p50
0.068071
s
<
p95
43.126802
s
<
p99
131.822132
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
6
over 1688.845493974s
mean rate
64.49
/s
std
232.36
/s
p5
0
/s
p95
236.59
/s
peak
9022.99
/s
WASM usage: 'agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 1787.594101908s
Not enough time data to show a trend.
WASM usage: 'agent_activity::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
204290952
over 1787.594101908s
mean rate
3.5644683988e+08
/s
std
1.50662951198e+09
/s
p5
3.16237149e+06
/s
p95
1.50327592581e+09
/s
peak
5.45803511022e+10
/s
WASM usage: 'agent_activity::get_agent_activity_full'
Metered usage count of the WASM ribosome for this zome function.
total
1982194
over 1789.024802905s
mean rate
3.12350447338e+09
/s
std
6.80613402116e+10
/s
p5
1.61420894e+06
/s
p95
3.36888381872e+09
/s
peak
4.86390381142e+12
/s
WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
8234032
over 1484.193876637s
mean rate
6.3195473791e+08
/s
std
2.55569965245e+09
/s
p5
0
/s
p95
1.67630290679e+09
/s
peak
1.63050393712e+10
/s
WASM usage: 'agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
22945017
over 1789.348779195s
mean rate
4.738316317e+07
/s
std
1.8673101835e+08
/s
p5
0
/s
p95
1.9373805096e+08
/s
peak
6.68390232006e+09
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006866
s
SD
0.002419
s
p50
0.006928
s
<
p95
0.011487
s
<
p99
0.013129
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006307
s
SD
0.001687
s
p50
0.006503
s
<
p95
0.00924
s
<
p99
0.011729
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.019196
s
SD
4.640766
s
p50
0.061381
s
<
p95
1.421509
s
<
p99
22.184884
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.005999
s
SD
2.088849
s
p50
0.009692
s
<
p95
2.907698
s
<
p99
7.976676
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00562
s
SD
0.001938
s
p50
0.005622
s
<
p95
0.009176
s
<
p99
0.010999
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005064
s
SD
0.001307
s
p50
0.00528
s
<
p95
0.007274
s
<
p99
0.009199
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.042349
s
SD
26.490208
s
p50
0.073111
s
<
p95
43.262948
s
<
p99
133.347828
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006415
s
SD
4.27447
s
p50
0.008311
s
<
p95
4.251834
s
<
p99
22.012884
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001639
s
SD
0.000436
s
p50
0.001726
s
<
p95
0.002271
s
<
p99
0.002633
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.1e-05
s
SD
3.3e-05
s
p50
4.5e-05
s
<
p95
0.000103
s
<
p99
0.000189
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000608
s
SD
0.000228
s
p50
0.000627
s
<
p95
0.001092
s
<
p99
0.001273
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000578
s
SD
0.000275
s
p50
0.000555
s
<
p95
0.001003
s
<
p99
0.001789
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.040861
s
SD
26.490264
s
p50
0.070741
s
<
p95
43.260456
s
<
p99
133.346677
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.003527
s
SD
4.274682
s
p50
0.001407
s
<
p95
4.249223
s
<
p99
22.008317
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
7e-06
s
SD
7e-06
s
p50
1.1e-05
s
<
p95
2.4e-05
s
<
p99
3.7e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00263
s
SD
0.000921
s
p50
0.00273
s
<
p95
0.004275
s
<
p99
0.005633
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000391
s
SD
0.000539
s
p50
0.000425
s
<
p95
0.000818
s
<
p99
0.001538
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
4.32
s
<
mean
341.28
s
SD
496.74
s
<
p95
1505.2
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
2035
over 1789.948621604s
mean rate
2.07528726e+06
/s
std
1.5615452807e+08
/s
p5
0
/s
p95
1.3672508e+06
/s
peak
1.6679369251e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
230.500758
s
SD
184.297283
s
p50
104.890661
s
<
p95
546.871239
s
<
p99
850.415396
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.072086
s
SD
0.326609
s
p50
0.032337
s
<
p95
0.750262
s
<
p99
1.371825
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
7e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.075303
s
SD
0.394554
s
p50
0.037983
s
<
p95
0.46247
s
<
p99
1.384728
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.887002
s
SD
1.651864
s
p50
0.111914
s
<
p95
5.480031
s
<
p99
7.010629
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
13.763631
s
SD
59.130358
s
p50
41.040817
s
<
p95
183.698962
s
<
p99
274.057882
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
57.425069
s
SD
68.163409
s
p50
0.024198
s
<
p95
202.38644
s
<
p99
290.04795
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004924
s
SD
0.003187
s
p50
0.000976
s
<
p95
0.009787
s
<
p99
0.01279
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007454
s
SD
0.11676
s
p50
0.003445
s
<
p95
0.038844
s
<
p99
0.292971
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000546
s
SD
0.001047
s
p50
0.000741
s
<
p95
0.002414
s
<
p99
0.004437
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.039747
s
SD
28.427209
s
p50
0.451163
s
<
p95
60.003035
s
<
p99
60.004708
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.265798
s
SD
5.179755
s
p50
0.475333
s
<
p95
4.30241
s
<
p99
7.813456
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.148257
s
SD
17.013164
s
p50
0.194523
s
<
p95
60.00136
s
<
p99
60.001846
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.299502
s
SD
19.247675
s
p50
0.407241
s
<
p95
60.001282
s
<
p99
60.002932
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
26.18
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.36
KiB/s
<
mean
133.93
KiB/s
SD
282.83
KiB/s
<
p95
421.08
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.02
KiB/s
<
mean
35.71
KiB/s
SD
119.40
KiB/s
<
p95
132.25
KiB/s
Total bytes received
Total bytes received on primary network interface
count
32.88
GiB
mean
18.81
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
8.76
GiB
mean
5.01
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.8%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (31.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.73
%
<
mean
35.39
%
SD
31.17
%
<
p95
89.89
%
System
9.22
%
CPU percentiles
CPU usage percentiles
p50
29.53
%
p95
89.89
%
p99
96.82
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
95
hosts
mean time
361.21
s
Memory used percentage
Percentage of memory used
p5
6.44
%
<
mean
10.97
%
SD
6.96
%
<
p95
24.45
%
Memory available percentage
Percentage of available memory
p5
75.55
%
<
mean
89.03
%
SD
6.96
%
<
p95
93.56
%
Max host memory used
Maximum memory usage percentage across all hosts
max
54.75
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
31.71
%
Memory growth rate
Rate of memory growth over time
growth
0.18
MiB/s
Disk read throughput
Disk read throughput in MiB/s
152.81
MiB/s
Disk write throughput
Disk write throughput in MiB/s
88.06
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/116
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.36
5 min
1.56
15 min
1.89
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
21.23
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
12.3605
%
SD
17.3596
%
<
p95
50.12
%
60 second average
12.1473
%
300 second average
14.6604
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.025
%
SD
0.3743
%
≤
p95
0
%
60 second average
0.0254
%
300 second average
0.0233
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0032
%
SD
0.0934
%
≤
p95
0
%
60 second average
0.0069
%
300 second average
0.0067
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.9471
%
SD
7.0961
%
<
p95
12.05
%
60 second average
2.9702
%
300 second average
3.5648
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.7057
%
SD
3.8464
%
<
p95
7
%
60 second average
1.7208
%
300 second average
1.8730
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
31.89
%
SD
31.73
%
<
p95
86.01
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
65.52
KiB
<
mean
190.43
KiB
SD
109.03
KiB
<
p95
367.91
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
28.67
threads
SD
11.22
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
23
file descriptors
<
mean
75.73
file descriptors
SD
22.14
file descriptors
<
p95
89
file descriptors

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Thu, 27 Aug 2026 16:32:58 BST
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_33072482790_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
361.4
Max
Highest chain head value observed for any single write agent.
907
Write agents observed
5
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
168.365675
s
SD
54.857159
s
Max
Highest per-partition mean latency
max mean
1039.857467
s
Min
Lowest per-partition mean latency
min mean
6.802
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.72
/s
Max
Highest per-partition mean rate
max
7
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.138685
s
SD
0.098395
s
Max
Highest per-partition mean latency
max mean
0.566737
s
Min
Lowest per-partition mean latency
min mean
0.015023
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
42845
Agents affected
86 / 87
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
492
Max
Highest count in any single partition
795
Min
Lowest count in any single partition
0
Final error count
The total number of all types of error accumulated over the run by all agents.
133625
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.020659
s
SD
0.142864
s
p50
0.001787
s
<
p95
0.078121
s
<
p99
0.477525
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
7
over 760.607316547s
mean rate
18.48
/s
std
38.65
/s
p5
0
/s
p95
135.38
/s
peak
135.43
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009247
s
SD
0.00707
s
p50
0.007567
s
<
p95
0.01637
s
<
p99
0.047989
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.24642
s
SD
0.226378
s
p50
0.14123
s
<
p95
0.74927
s
<
p99
0.960075
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009916
s
SD
0.006217
s
p50
0.007997
s
<
p95
0.020375
s
<
p99
0.032536
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.007292
s
SD
0.043045
s
p50
0.00757
s
<
p95
0.01791
s
<
p99
0.422435
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.028599
s
SD
0.253895
s
p50
0.008438
s
<
p95
0.308012
s
<
p99
1.561799
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007667
s
SD
0.006728
s
p50
0.006265
s
<
p95
0.014104
s
<
p99
0.04607
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.244422
s
SD
0.225235
s
p50
0.139571
s
<
p95
0.745245
s
<
p99
0.954774
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00813
s
SD
0.005316
s
p50
0.006317
s
<
p95
0.016631
s
<
p99
0.028748
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005777
s
SD
0.037742
s
p50
0.006888
s
<
p95
0.013708
s
<
p99
0.365286
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.12977
s
SD
0.254882
s
p50
0.00709
s
<
p95
0.352985
s
<
p99
1.560953
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001528
s
SD
0.000966
s
p50
0.001704
s
<
p95
0.003634
s
<
p99
0.00499
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.014669
s
SD
0.0166
s
p50
0.010829
s
<
p95
0.049589
s
<
p99
0.084956
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
7.5e-05
s
SD
0.00011
s
p50
4.4e-05
s
<
p95
0.000207
s
<
p99
0.000945
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.003934
s
SD
0.006067
s
p50
0.000976
s
<
p95
0.014867
s
<
p99
0.033332
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.006119
s
SD
0.008906
s
p50
0.000724
s
<
p95
0.02623
s
<
p99
0.041407
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002506
s
SD
0.185274
s
p50
0.000944
s
<
p95
0.014715
s
<
p99
0.798057
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.021723
s
SD
0.027778
s
p50
0.015345
s
<
p95
0.080848
s
<
p99
0.138185
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.110158
s
SD
0.083953
s
p50
0.064403
s
<
p95
0.254412
s
<
p99
0.315398
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
2.3e-05
s
SD
1.2e-05
s
p50
1.1e-05
s
<
p95
2.1e-05
s
<
p99
5.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002917
s
SD
0.002366
s
p50
0.003071
s
<
p95
0.008041
s
<
p99
0.013431
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.00181
s
SD
0.002025
s
p50
0.000285
s
<
p95
0.004602
s
<
p99
0.011674
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.2
s
<
mean
426.46
s
SD
247.35
s
<
p95
815.09
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1269
over 888.934296519s
mean rate
1.67636154e+06
/s
std
1.787984197e+07
/s
p5
2500.34
/s
p95
2.00800377e+06
/s
peak
7.5910125142e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
96.261097
s
SD
66.283382
s
p50
25.468001
s
<
p95
189.563788
s
<
p99
296.292359
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.023232
s
SD
1.06411
s
p50
2.040221
s
<
p95
4.072469
s
<
p99
5.175255
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
3e-06
s
SD
4.1e-05
s
p50
0
s
≤
p95
0
s
<
p99
2e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.995055
s
SD
0.493603
s
p50
0.656944
s
<
p95
1.496784
s
<
p99
1.700504
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.608727
s
SD
0.626411
s
p50
0.094736
s
<
p95
1.079866
s
<
p99
3.127582
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
15.03085
s
SD
17.572589
s
p50
15.419468
s
<
p95
48.595513
s
<
p99
89.702497
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
19.013003
s
SD
30.963302
s
p50
1.707948
s
<
p95
89.869722
s
<
p99
141.469531
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004166
s
SD
0.003442
s
p50
0.003961
s
<
p95
0.011092
s
<
p99
0.014544
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005191
s
SD
0.004121
s
p50
0.00318
s
<
p95
0.012093
s
<
p99
0.02372
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.003028
s
SD
0.005807
s
p50
0.000917
s
<
p95
0.014368
s
<
p99
0.031936
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.043691
s
SD
29.184704
s
p50
0.426166
s
<
p95
60.003993
s
<
p99
60.011562
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
5.13957
s
SD
12.274878
s
p50
0.443078
s
<
p95
2.082768
s
<
p99
60.000998
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.114005
s
SD
0.223424
s
p50
0.159667
s
<
p95
0.583056
s
≤
p99
0.583056
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.830972
s
SD
17.298429
s
p50
0.992827
s
<
p95
60.00142
s
<
p99
60.008509
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
48.34
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
32.38
KiB/s
<
mean
381.03
KiB/s
SD
417.15
KiB/s
<
p95
1.28
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
337
B/s
<
mean
95.32
KiB/s
SD
140.37
KiB/s
<
p95
391.35
KiB/s
Total bytes received
Total bytes received on primary network interface
count
46.13
GiB
mean
53.07
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
11.54
GiB
mean
13.28
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (33.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 69% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
7.13
%
<
mean
69.06
%
SD
25.43
%
<
p95
93.61
%
System
20.73
%
CPU percentiles
CPU usage percentiles
p50
78.48
%
p95
93.61
%
p99
97.95
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
136
hosts
mean time
456.80
s
Memory used percentage
Percentage of memory used
p5
7.33
%
<
mean
13.82
%
SD
6.93
%
<
p95
29.18
%
Memory available percentage
Percentage of available memory
p5
70.82
%
<
mean
86.18
%
SD
6.93
%
<
p95
92.67
%
Max host memory used
Maximum memory usage percentage across all hosts
max
51.17
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
33.68
%
Memory growth rate
Rate of memory growth over time
growth
0.52
MiB/s
Disk read throughput
Disk read throughput in MiB/s
204.08
MiB/s
Disk write throughput
Disk write throughput in MiB/s
198.52
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/117
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.63
5 min
2.89
15 min
2.22
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
69.18
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.41
%
<
mean
42.6568
%
SD
30.7578
%
<
p95
86.92
%
60 second average
39.8514
%
300 second average
31.1084
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0332
%
SD
0.4406
%
≤
p95
0
%
60 second average
0.0339
%
300 second average
0.0333
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0066
%
SD
0.1052
%
≤
p95
0
%
60 second average
0.0085
%
300 second average
0.0104
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
10.2026
%
SD
9.0328
%
<
p95
25.46
%
60 second average
9.6603
%
300 second average
7.6125
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1261
%
SD
4.5354
%
<
p95
12.47
%
60 second average
4.0010
%
300 second average
3.4355
%
Holochain process CPU usage
CPU usage by Holochain process
p5
8.28
%
<
mean
65.66
%
SD
23.5
%
<
p95
89.84
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
114.38
KiB
<
mean
399.72
KiB
SD
133.38
KiB
<
p95
544.29
KiB
Holochain process threads
Number of threads in Holochain process
p5
12
threads
<
mean
34.99
threads
SD
12.43
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
40
file descriptors
<
mean
79.29
file descriptors
SD
19.73
file descriptors
<
p95
89
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Thu, 27 Aug 2026 17:58:59 BST
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_33072482790_1
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
49.83
/s
Max
Highest per-partition mean rate
max
90.4
/s
Min
Lowest per-partition mean rate
min
30.33
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
335.519259
s
SD
65.287248
s
Max
Highest per-partition mean latency
max mean
1264.580892
s
Min
Lowest per-partition mean latency
min mean
0.917829
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
62.06
/s
Max
Highest per-partition mean rate
max
1073.5
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0007
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
260870
Agents affected
58 / 58
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4498
Max
Highest count in any single partition
8076
Min
Lowest count in any single partition
2680
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
11459
Agents affected
56 / 59
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
194
Max
Highest count in any single partition
2378
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
35.92
SD
34.43
<
p95
103
zero-arc
p5
7
<
mean
24.52
SD
15.4
<
p95
56
Error count
The number of errors accumulated across all nodes.
638
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.236205
s
SD
1.840119
s
p50
0.41631
s
<
p95
4.189842
s
<
p99
9.514205
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
2
over 470.007180541s
mean rate
1.55
/s
std
2.43
/s
p5
0
/s
p95
7.62
/s
peak
7.64
/s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
340412452
over 879.321048015s
mean rate
4.45193518023e+09
/s
std
5.32070542039e+10
/s
p5
9.17862977e+06
/s
p95
1.18120171456e+10
/s
peak
1.46827036949e+12
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
187446016
over 879.678331534s
mean rate
2.78289426695e+10
/s
std
1.28182960648e+11
/s
p5
2.469336e+06
/s
p95
1.44376874006e+11
/s
peak
1.20390297997e+12
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
44665008
over 889.701439103s
mean rate
3.51510855504e+09
/s
std
2.7448965286e+10
/s
p5
0
/s
p95
1.22223799429e+10
/s
peak
1.17188133168e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008048
s
SD
0.004807
s
p50
0.008187
s
<
p95
0.019021
s
<
p99
0.022593
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
10.146098
s
SD
29.718892
s
p50
6.669023
s
<
p95
71.117015
s
<
p99
134.829389
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00708
s
SD
0.004267
s
p50
0.007203
s
<
p95
0.016801
s
<
p99
0.020131
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
10.144739
s
SD
29.718867
s
p50
6.667649
s
<
p95
71.116123
s
<
p99
134.828023
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001347
s
SD
0.000615
s
p50
0.0017
s
<
p95
0.002766
s
<
p99
0.003126
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000642
s
SD
0.000832
s
p50
0.000534
s
<
p95
0.00256
s
<
p99
0.003965
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.00057
s
SD
0.000621
s
p50
0.000481
s
<
p95
0.001956
s
<
p99
0.002845
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.234431
s
SD
2.182894
s
p50
0.419649
s
<
p95
4.373917
s
<
p99
9.971362
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.408492
s
SD
1.927876
s
p50
0.310998
s
<
p95
4.448227
s
<
p99
11.423958
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002124
s
SD
0.001036
s
p50
0.002556
s
<
p95
0.004375
s
<
p99
0.004868
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000422
s
SD
0.00226
s
p50
0.000268
s
<
p95
0.002001
s
<
p99
0.016519
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.64
s
<
mean
427.97
s
SD
247.79
s
<
p95
815.8
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.701336703s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
20.03936
s
SD
25.015534
s
p50
0
s
<
p95
13.9456
s
<
p99
77.208486
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.162155
s
SD
0.186911
s
p50
0.002037
s
<
p95
0.356972
s
<
p99
0.791867
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.050043
s
SD
0.419268
s
p50
0.002007
s
<
p95
1.2528
s
<
p99
1.385418
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.146236
s
SD
0.071324
s
p50
0.027858
s
<
p95
0.216932
s
<
p99
0.267098
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.859491
s
SD
9.481592
s
p50
0.003229
s
<
p95
19.437978
s
<
p99
47.829442
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.637347
s
SD
4.582966
s
p50
0.001408
s
<
p95
0.190022
s
<
p99
17.880959
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002391
s
SD
0.002041
s
p50
0.001102
s
<
p95
0.006044
s
<
p99
0.007619
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006485
s
SD
0.006702
s
p50
0.002982
s
<
p95
0.014279
s
<
p99
0.030403
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.00059
s
SD
0.006059
s
p50
0.000525
s
<
p95
0.002558
s
<
p99
0.042224
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.561582
s
SD
29.387037
s
p50
1.277047
s
<
p95
60.005871
s
<
p99
60.066908
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.249118
s
SD
13.052848
s
p50
0.172536
s
<
p95
60.00067
s
<
p99
60.001814
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.954779
s
SD
10.243354
s
p50
0.219141
s
<
p95
7.909258
s
<
p99
60.047254
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
33.99
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
6.48
KiB/s
<
mean
318.15
KiB/s
SD
356.34
KiB/s
<
p95
872.73
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
770
B/s
<
mean
113.81
KiB/s
SD
194.91
KiB/s
<
p95
516.44
KiB/s
Total bytes received
Total bytes received on primary network interface
count
38.72
GiB
mean
44.55
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.85
GiB
mean
15.94
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (51.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 47% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.95
%
<
mean
46.41
%
SD
36.28
%
<
p95
89.14
%
System
12.42
%
CPU percentiles
CPU usage percentiles
p50
59.66
%
p95
89.14
%
p99
95.04
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
94
hosts
mean time
433.92
s
Memory used percentage
Percentage of memory used
p5
6.84
%
<
mean
12.31
%
SD
6.39
%
<
p95
26.41
%
Memory available percentage
Percentage of available memory
p5
73.59
%
<
mean
87.69
%
SD
6.39
%
<
p95
93.16
%
Max host memory used
Maximum memory usage percentage across all hosts
max
47.18
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
51.77
%
Memory growth rate
Rate of memory growth over time
growth
0.94
MiB/s
Disk read throughput
Disk read throughput in MiB/s
288.53
MiB/s
Disk write throughput
Disk write throughput in MiB/s
140.89
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/117
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.92
5 min
1.97
15 min
2.12
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
47.26
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
18.5805
%
SD
19.6657
%
<
p95
50.93
%
60 second average
18.1602
%
300 second average
18.5931
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0435
%
SD
0.5586
%
≤
p95
0
%
60 second average
0.0419
%
300 second average
0.0344
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0407
%
SD
0.5239
%
≤
p95
0
%
60 second average
0.0384
%
300 second average
0.0283
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.8616
%
SD
7.7992
%
<
p95
14.32
%
60 second average
3.8124
%
300 second average
3.7617
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.2679
%
SD
6.316
%
<
p95
8.21
%
60 second average
2.2542
%
300 second average
2.2610
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.66
%
<
mean
41.6
%
SD
34.19
%
<
p95
84.56
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
204.08
KiB
<
mean
269.80
KiB
SD
66.96
KiB
<
p95
404.51
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
30.51
threads
SD
7.46
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
57
file descriptors
<
mean
73.77
file descriptors
SD
14.36
file descriptors
<
p95
89
file descriptors

Zero-Arc Create Data

↑ Back to index
A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 27 Aug 2026 17:07:22 BST
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data--high_zero_33072482790_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
51.02
/s
Max
Highest per-partition mean rate
max
99.12
/s
Min
Lowest per-partition mean rate
min
29.55
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
145.614429
s
SD
49.61946
s
Max
Highest per-partition mean latency
max mean
572.474128
s
Min
Lowest per-partition mean latency
min mean
12.294536
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
46.8
/s
Max
Highest per-partition mean rate
max
234
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0003
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
630719
Agents affected
137 / 137
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4604
Max
Highest count in any single partition
9066
Min
Lowest count in any single partition
2639
Received count
Total number of entries received by full-arc nodes, by agent.
Total
1527
Agents affected
2 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
170
Max
Highest count in any single partition
1352
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
3
<
mean
53.87
SD
46.76
<
p95
145
zero-arc
p5
4
<
mean
20.16
SD
16.15
<
p95
49
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.004769
s
SD
0.002739
s
p50
0.002239
s
<
p95
0.007938
s
<
p99
0.008833
s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007326
s
SD
0.003059
s
p50
0.00776
s
<
p95
0.01465
s
<
p99
0.015672
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.03426
s
SD
0.472107
s
p50
0.038734
s
<
p95
1.287369
s
<
p99
1.491814
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006406
s
SD
0.002636
s
p50
0.00675
s
<
p95
0.012772
s
<
p99
0.013627
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.033429
s
SD
0.471173
s
p50
0.03436
s
<
p95
1.28265
s
<
p99
1.488244
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001299
s
SD
0.000814
s
p50
0.001402
s
<
p95
0.002631
s
<
p99
0.005513
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000416
s
SD
0.000198
s
p50
0.000442
s
<
p95
0.000869
s
<
p99
0.000957
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000402
s
SD
0.000195
s
p50
0.000398
s
<
p95
0.00089
s
<
p99
0.000943
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.005529
s
SD
0.002314
s
p50
0.005004
s
<
p95
0.008127
s
<
p99
0.008627
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001649
s
SD
0.021383
s
p50
0.003213
s
<
p95
0.05656
s
<
p99
0.06251
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002041
s
SD
0.003058
s
p50
0.00209
s
<
p95
0.004377
s
<
p99
0.02082
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000375
s
SD
0.004117
s
p50
0.000468
s
<
p95
0.00131
s
<
p99
0.017566
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.51
s
<
mean
427.29
s
SD
247.5
s
<
p95
815.16
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.336487179s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
2.195191
s
SD
18.074101
s
p50
0
s
≤
p95
0
s
<
p99
38.229624
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.218099
s
SD
0.226272
s
p50
0.001739
s
<
p95
0.173152
s
<
p99
1.227504
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.009626
s
SD
0.280569
s
p50
0.001664
s
<
p95
1.037738
s
<
p99
1.400174
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.132401
s
SD
0.063294
s
p50
0.117052
s
<
p95
0.233949
s
<
p99
0.278386
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
6.869774
s
SD
8.975706
s
p50
0.003329
s
<
p95
5.880358
s
<
p99
45.776671
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.044807
s
SD
6.069935
s
p50
0.001484
s
<
p95
0.014947
s
<
p99
0.123539
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00701
s
SD
0.003959
s
p50
0.007326
s
<
p95
0.014626
s
<
p99
0.016303
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007737
s
SD
0.005737
s
p50
0.003714
s
<
p95
0.017147
s
<
p99
0.031479
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000411
s
SD
0.010176
s
p50
0.000346
s
<
p95
0.000848
s
<
p99
0.067022
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
10.34719
s
SD
27.100343
s
p50
16.564063
s
<
p95
60.133093
s
<
p99
60.160283
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.596492
s
SD
20.024504
s
p50
1.107407
s
<
p95
60.051048
s
<
p99
60.082074
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
50.01
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
13.99
KiB/s
<
mean
206.52
KiB/s
SD
352.04
KiB/s
<
p95
853.47
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.25
KiB/s
<
mean
56.08
KiB/s
SD
99.95
KiB/s
<
p95
191.34
KiB/s
Total bytes received
Total bytes received on primary network interface
count
25.34
GiB
mean
29.16
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
6.88
GiB
mean
7.92
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (53.4% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 48% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
18.1
%
<
mean
65.57
%
SD
21.32
%
<
p95
90.98
%
System
15.55
%
CPU percentiles
CPU usage percentiles
p50
68.25
%
p95
90.98
%
p99
95.17
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
112
hosts
mean time
363.79
s
Memory used percentage
Percentage of memory used
p5
6.64
%
<
mean
12.14
%
SD
6.56
%
<
p95
26.25
%
Memory available percentage
Percentage of available memory
p5
73.75
%
<
mean
87.86
%
SD
6.56
%
<
p95
93.36
%
Max host memory used
Maximum memory usage percentage across all hosts
max
50.06
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
53.42
%
Memory growth rate
Rate of memory growth over time
growth
0.45
MiB/s
Disk read throughput
Disk read throughput in MiB/s
231.53
MiB/s
Disk write throughput
Disk write throughput in MiB/s
179.05
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/117
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.32
5 min
2.28
15 min
2.44
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
47.95
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.79
%
<
mean
20.3444
%
SD
16.7006
%
<
p95
51.18
%
60 second average
19.9750
%
300 second average
20.4216
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0167
%
SD
0.2119
%
≤
p95
0
%
60 second average
0.0176
%
300 second average
0.0190
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0094
%
SD
0.1311
%
≤
p95
0
%
60 second average
0.0106
%
300 second average
0.0124
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.6812
%
SD
8.0673
%
<
p95
17.91
%
60 second average
4.6111
%
300 second average
4.5884
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.9938
%
SD
5.0359
%
<
p95
14.24
%
60 second average
2.9600
%
300 second average
2.8632
%
Holochain process CPU usage
CPU usage by Holochain process
p5
26.34
%
<
mean
62.31
%
SD
19.26
%
<
p95
86.41
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
212.70
KiB
<
mean
278.89
KiB
SD
65.15
KiB
<
p95
368.30
KiB
Holochain process threads
Number of threads in Holochain process
p5
27
threads
<
mean
32.37
threads
SD
6.79
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
68
file descriptors
<
mean
78.28
file descriptors
SD
11.5
file descriptors
<
p95
88
file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 27 Aug 2026 17:41:53 BST
Duration
900s
Peer count
150
Peer count at end
144
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data_validated--high_zero_33072482790_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
45.62
/s
Max
Highest per-partition mean rate
max
94.33
/s
Min
Lowest per-partition mean rate
min
24.67
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
145.519743
s
SD
72.277158
s
Max
Highest per-partition mean latency
max mean
347.660845
s
Min
Lowest per-partition mean latency
min mean
0.877136
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
130.69
/s
Max
Highest per-partition mean rate
max
459
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0021
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
552938
Agents affected
137 / 137
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4036
Max
Highest count in any single partition
8527
Min
Lowest count in any single partition
2033
Received count
Total number of entries received by full-arc nodes, by agent.
Total
10317
Agents affected
5 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1146
Max
Highest count in any single partition
3448
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
5
<
mean
56.26
SD
41.28
<
p95
144
zero-arc
p5
3
<
mean
20.98
SD
18.18
<
p95
52
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.002608
s
SD
0.004866
s
p50
0.008505
s
<
p95
0.017532
s
<
p99
0.020811
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
853
over 783.321599098s
mean rate
3665.27
/s
std
5714.09
/s
p5
30.35
/s
p95
18606.86
/s
peak
23246.73
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
278405400
over 870.980735926s
mean rate
4.71904106126e+09
/s
std
7.4641746151e+10
/s
p5
2.001668894e+07
/s
p95
1.08425594307e+10
/s
peak
5.53927120578e+12
/s
WASM usage: 'timed_and_validated::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
30562495092
over 885.997695213s
mean rate
4.25935285617e+10
/s
std
7.94237503835e+10
/s
p5
2.143872875e+07
/s
p95
2.59357841008e+11
/s
peak
2.89801866749e+11
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
152740170
over 888.632532697s
mean rate
2.53061953995e+09
/s
std
3.90686558569e+10
/s
p5
9.04523931e+06
/s
p95
6.27419196899e+09
/s
peak
3.0293058197e+12
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
1277960159
over 888.632532697s
mean rate
2.2103038021e+10
/s
std
3.30040105375e+11
/s
p5
1.2058121666e+08
/s
p95
5.5027839461e+10
/s
peak
2.53320732218e+13
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007727
s
SD
0.003419
s
p50
0.008168
s
<
p95
0.014833
s
<
p99
0.016519
s
Zome call duration: 'timed_and_validated::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.058305
s
SD
0.099992
s
p50
0.056147
s
<
p95
0.327416
s
<
p99
0.400993
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006764
s
SD
0.002969
s
p50
0.007116
s
<
p95
0.012905
s
<
p99
0.014386
s
WASM call duration: 'timed_and_validated::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.05742
s
SD
0.099829
s
p50
0.054669
s
<
p95
0.326581
s
<
p99
0.400142
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001364
s
SD
0.00071
s
p50
0.001386
s
<
p95
0.002452
s
<
p99
0.004895
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006059
s
SD
0.00372
s
p50
0.005457
s
<
p95
0.010476
s
<
p99
0.025716
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000423
s
SD
0.00021
s
p50
0.000448
s
<
p95
0.000825
s
<
p99
0.000938
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000406
s
SD
0.000208
s
p50
0.000402
s
<
p95
0.00083
s
<
p99
0.000966
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.00078
s
SD
0.002632
s
p50
0.000757
s
<
p95
0.007699
s
<
p99
0.008964
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002456
s
SD
0.003942
s
p50
0.002866
s
<
p95
0.011093
s
<
p99
0.013047
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.008122
s
SD
0.004677
s
p50
0.008909
s
<
p95
0.017766
s
<
p99
0.021024
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002281
s
SD
0.002075
s
p50
0.002174
s
<
p95
0.004327
s
<
p99
0.015615
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000472
s
SD
0.002209
s
p50
0.000582
s
<
p95
0.001377
s
<
p99
0.003297
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.48
s
<
mean
426.41
s
SD
247.22
s
<
p95
814.93
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.632512178s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
14.0487
s
SD
17.008518
s
p50
0
s
<
p95
0.303982
s
<
p99
117.545238
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
5.128035
s
SD
1.861769
s
p50
0.001809
s
<
p95
3.999413
s
<
p99
10.180011
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.009565
s
SD
0.257235
s
p50
0.00208
s
<
p95
0.92242
s
<
p99
1.224899
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.119474
s
SD
0.056063
s
p50
0.103223
s
<
p95
0.202495
s
<
p99
0.235532
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.124732
s
SD
7.059077
s
p50
0.003341
s
<
p95
6.249879
s
<
p99
46.554496
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.707353
s
SD
10.396397
s
p50
0.001112
s
<
p95
0.009309
s
<
p99
0.500227
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003544
s
SD
0.003173
s
p50
0.004575
s
<
p95
0.010768
s
<
p99
0.012406
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007062
s
SD
0.005421
s
p50
0.003708
s
<
p95
0.012844
s
<
p99
0.032809
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000335
s
SD
0.002602
s
p50
0.000346
s
<
p95
0.000725
s
<
p99
0.018185
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
15.998171
s
SD
32.873752
s
p50
11.395091
s
<
p95
75.511738
s
<
p99
122.498601
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.198651
s
SD
0.051697
s
p50
0.179623
s
<
p95
0.276348
s
<
p99
0.321851
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
49.58
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
21.26
KiB/s
<
mean
261.79
KiB/s
SD
419.37
KiB/s
<
p95
1.10
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
2.04
KiB/s
<
mean
79.47
KiB/s
SD
101.20
KiB/s
<
p95
272.63
KiB/s
Total bytes received
Total bytes received on primary network interface
count
32.00
GiB
mean
36.81
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.71
GiB
mean
11.18
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.7%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/15 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (52.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 37% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
17.46
%
<
mean
66.19
%
SD
21.2
%
<
p95
89.57
%
System
16.62
%
CPU percentiles
CPU usage percentiles
p50
74.15
%
p95
89.57
%
p99
96.69
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
111
hosts
mean time
303.64
s
Memory used percentage
Percentage of memory used
p5
7.44
%
<
mean
14.02
%
SD
6.86
%
<
p95
28.49
%
Memory available percentage
Percentage of available memory
p5
71.51
%
<
mean
85.98
%
SD
6.86
%
<
p95
92.56
%
Max host memory used
Maximum memory usage percentage across all hosts
max
48.18
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
52.17
%
Memory growth rate
Rate of memory growth over time
growth
0.69
MiB/s
Disk read throughput
Disk read throughput in MiB/s
232.61
MiB/s
Disk write throughput
Disk write throughput in MiB/s
173.98
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/117
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/16
hosts
Mount Point /etc/hostname
1/15
hosts
Mount Point /etc/hosts
1/15
hosts
Mount Point /etc/nomad.d/client-meta.json
1/15
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/15
hosts
Mount Point /nix/store
0/18
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.38
5 min
2.29
15 min
2.30
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
36.99
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1
%
<
mean
22.9942
%
SD
15.5551
%
<
p95
42.38
%
60 second average
22.1706
%
300 second average
21.3303
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0224
%
SD
0.2876
%
≤
p95
0
%
60 second average
0.0243
%
300 second average
0.0256
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.008
%
SD
0.1324
%
≤
p95
0
%
60 second average
0.0105
%
300 second average
0.0142
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.884
%
SD
6.8153
%
<
p95
15.42
%
60 second average
3.8302
%
300 second average
3.8409
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.2984
%
SD
4.1789
%
<
p95
11.21
%
60 second average
2.2950
%
300 second average
2.2880
%
Holochain process CPU usage
CPU usage by Holochain process
p5
25.88
%
<
mean
63.34
%
SD
18.93
%
<
p95
84.55
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
190.35
KiB
<
mean
393.30
KiB
SD
107.37
KiB
<
p95
546.28
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
33.9
threads
SD
6.86
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
67
file descriptors
<
mean
78.88
file descriptors
SD
11.96
file descriptors
<
p95
87
file descriptors