Conductor Startup

↑ Back to index

This scenario measures Holochain conductor startup time when many cells are installed. The agent starts an empty conductor, installs a configurable number of apps without enabling them (disabled cells), restarts the conductor, and then enables cells one by one — timing each enable call and periodically restarting the conductor to measure startup time as the share of enabled cells grows.

Started
Thu, 15 Oct 2026 03:12:58 UTC
Duration
900s
Peer count
150
Peer count at end
89
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
conductor_startup_38671000696_1
Initial conductor startup timing
Startup time with an empty conductor before any cells are installed.
mean
3.853375
s
SD
1.86405
s
p50
3.027348
s
<
p95
7.99619
s
<
p99
8.595945
s
Disabled-cell conductor startup timing
Startup time after all configured cells are installed but remain disabled.
mean
2.996996
s
SD
1.24422
s
p50
2.487992
s
<
p95
6.106334
s
<
p99
6.496684
s
Progressive conductor startup timing
Startup time sampled periodically as more installed cells are enabled.
mean
15.302574
s
SD
7.64522
s
p50
14.877647
s
<
p95
27.998637
s
<
p99
29.763973
s
Cell enable timing
Time to enable one previously disabled cell via the admin interface.
mean
0.29565
s
SD
0.229391
s
p50
0.23935
s
<
p95
0.641262
s
<
p99
1.195091
s
Installed cells
Number of cells installed on the conductor during setup.
500
cells
Maximum enabled cells
Highest percentage of installed cells enabled during the run.
100
%
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
7
over 631.427790373s
mean rate
1.93
/s
std
1.13
/s
p5
0.2
/s
p95
3.53
/s
peak
3.55
/s
WASM usage: 'callback_integrity::genesis_self_check_2'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 404.845215269s
Not enough time data to show a trend.
WASM call duration: 'callback_integrity::genesis_self_check_2'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002025
s
SD
0.001108
s
p50
0.002135
s
<
p95
0.004051
s
<
p99
0.005447
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
1.77
s
<
mean
159.66
s
SD
193.37
s
<
p95
594.15
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
56
over 851.648983623s
mean rate
147.68
/s
std
2210.36
/s
p5
0
/s
p95
78.2
/s
peak
52141.92
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
6.182153
s
SD
5.064849
s
p50
5.672263
s
<
p95
10.984203
s
<
p99
22.391848
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.009729
s
SD
0.0207
s
p50
0.001378
s
<
p95
0.009913
s
<
p99
0.095803
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
1.1e-05
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.089472
s
SD
0.061635
s
p50
0.001689
s
<
p95
0.170582
s
<
p99
0.224392
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.02215
s
SD
0.026158
s
p50
0.002039
s
<
p95
0.010495
s
<
p99
0.033038
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.91252
s
SD
1.134629
s
p50
0.002312
s
<
p95
2.510221
s
<
p99
4.012925
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.025223
s
SD
0.566925
s
p50
0.000775
s
<
p95
0.004125
s
<
p99
0.015502
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000783
s
SD
0.000641
s
p50
0.000272
s
<
p95
0.001037
s
<
p99
0.001244
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00346
s
SD
0.00554
s
p50
0.001729
s
<
p95
0.004967
s
<
p99
0.019372
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000603
s
SD
0.000964
s
p50
0.000682
s
<
p95
0.003096
s
<
p99
0.003911
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
4.647011
s
SD
20.357795
s
p50
0.060456
s
<
p95
60.003973
s
<
p99
60.00744
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
3.683676
s
SD
19.60413
s
p50
0.056194
s
<
p95
60.001155
s
≤
p99
60.001155
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
39.46
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
2.16
KiB/s
<
mean
383.39
KiB/s
SD
729.18
KiB/s
<
p95
1.97
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
163
B/s
<
mean
54.61
KiB/s
SD
50.99
KiB/s
<
p95
158.17
KiB/s
Total bytes received
Total bytes received on primary network interface
count
9.03
GiB
mean
10.39
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.27
GiB
mean
1.46
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1.10 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (11.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
8.58
%
<
mean
59.34
%
SD
27.67
%
<
p95
95.34
%
System
19.89
%
CPU percentiles
CPU usage percentiles
p50
69.37
%
p95
95.34
%
p99
96.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
44
hosts
mean time
295.40
s
Memory used percentage
Percentage of memory used
p5
5.99
%
<
mean
12.88
%
SD
5.35
%
<
p95
22.09
%
Memory available percentage
Percentage of available memory
p5
77.91
%
<
mean
87.12
%
SD
5.35
%
<
p95
94.01
%
Max host memory used
Maximum memory usage percentage across all hosts
max
32.86
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
11.17
%
Memory growth rate
Rate of memory growth over time
growth
1.10
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.05
MiB/s
Disk write throughput
Disk write throughput in MiB/s
106.93
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/72
hosts
Mount Point /etc/hostname
0/10
hosts
Mount Point /etc/hosts
0/10
hosts
Mount Point /etc/nomad.d/client-meta.json
0/12
hosts
Mount Point /etc/resolv.conf
0/10
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.46
5 min
1.49
15 min
0.87
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
9.57
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.99
%
<
mean
28.3235
%
SD
25.9829
%
<
p95
86.67
%
60 second average
22.8733
%
300 second average
14.1652
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.7546
%
SD
1.4308
%
<
p95
3.87
%
60 second average
0.6288
%
300 second average
0.3726
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2914
%
SD
0.7284
%
<
p95
1.39
%
60 second average
0.2765
%
300 second average
0.1820
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
45.84
%
SD
26.04
%
<
p95
89.34
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
24.36
KiB
<
mean
402.91
KiB
SD
354.86
KiB
<
p95
1.11
MiB
Holochain process threads
Number of threads in Holochain process
p5
6
threads
<
mean
709.9
threads
SD
739.06
threads
<
p95
1903
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
1609.3
file descriptors
SD
1618.13
file descriptors
<
p95
4096
file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Thu, 15 Oct 2026 03:24:35 UTC
Duration
1800s
Peer count
150
Peer count at end
140
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_38671000696_1
Create rate
The average number of created records per agent.
Mean
mean
17.66
/s
Max
Highest per-partition mean rate
max
59.55
/s
Min
Lowest per-partition mean rate
min
1.31
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
4.31612036384e+28
s
SD
1.12103577532e+30
s
Max
Highest per-partition mean latency
max mean
7.25239486191e+29
s
Min
Lowest per-partition mean latency
min mean
7.076543
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
66.17
/s
Max
Highest per-partition mean rate
max
524
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0015
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
393973
Agents affected
121 / 121
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3256
Max
Highest count in any single partition
10911
Min
Lowest count in any single partition
104
Received count
The total number of entries received by record_lag agents, by agent.
Total
13663
Agents affected
20 / 23
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
594
Max
Highest count in any single partition
4692
Min
Lowest count in any single partition
0
Error count
The number of errors encountered during the scenario.
55
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.001361
s
SD
0.014942
s
p50
0.002117
s
<
p95
0.00765
s
<
p99
0.095733
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 1723.592108515s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
206412470
over 1788.393154052s
mean rate
9.96644752444e+09
/s
std
2.09840785383e+11
/s
p5
3.746259819e+07
/s
p95
1.97338323159e+10
/s
peak
1.73513799138e+13
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
4770494965
over 1788.181894155s
mean rate
3.44524154486e+10
/s
std
4.68466848865e+10
/s
p5
1.0471564278e+08
/s
p95
1.09589460464e+11
/s
peak
9.35503536526e+11
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
642121218
over 1788.500936807s
mean rate
1.67301904702e+09
/s
std
3.03936296833e+10
/s
p5
6.63478222e+06
/s
p95
3.78007904776e+09
/s
peak
2.73271540118e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.0288
s
SD
0.17555
s
p50
0.030406
s
<
p95
0.097455
s
<
p99
1.16796
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.080929
s
SD
0.24418
s
p50
0.093803
s
<
p95
0.756903
s
<
p99
1.195107
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.027285
s
SD
0.16898
s
p50
0.028745
s
<
p95
0.094187
s
<
p99
1.125694
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.079682
s
SD
0.238306
s
p50
0.092648
s
<
p95
0.733484
s
<
p99
1.192393
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002051
s
SD
0.015616
s
p50
0.002178
s
<
p95
0.004606
s
<
p99
0.108788
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.007598
s
SD
0.05951
s
p50
0.007811
s
<
p95
0.031426
s
<
p99
0.402075
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.007394
s
SD
0.059359
s
p50
0.007315
s
<
p95
0.031126
s
<
p99
0.357119
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001337
s
SD
0.015001
s
p50
0.002501
s
<
p95
0.007613
s
<
p99
0.096305
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.004182
s
SD
0.018216
s
p50
0.005226
s
<
p95
0.046532
s
<
p99
0.104041
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.004817
s
SD
0.02414
s
p50
0.005015
s
<
p95
0.013814
s
<
p99
0.172842
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002286
s
SD
0.016446
s
p50
0.001709
s
<
p95
0.008478
s
<
p99
0.091418
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
82.41
s
<
mean
859.17
s
SD
510.32
s
<
p95
1668.11
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
147.861401
s
SD
138.01103
s
p50
2.972404
s
<
p95
349.225463
s
<
p99
725.909365
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.32484
s
SD
2.083686
s
p50
0.247223
s
<
p95
4.109506
s
<
p99
7.866883
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.288256
s
SD
0.574144
s
p50
0.217687
s
<
p95
1.420405
s
<
p99
2.937351
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
2.062278
s
SD
3.419256
s
p50
1.214833
s
<
p95
9.533381
s
<
p99
13.162384
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
23.897621
s
SD
63.923528
s
p50
28.393303
s
<
p95
178.374244
s
<
p99
266.978372
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
17.010296
s
SD
44.481495
s
p50
0.033601
s
<
p95
84.972499
s
<
p99
277.557568
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003786
s
SD
0.005303
s
p50
0.004567
s
<
p95
0.010622
s
<
p99
0.03487
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007819
s
SD
0.012075
s
p50
0.004094
s
<
p95
0.019377
s
<
p99
0.054834
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.006725
s
SD
0.053361
s
p50
0.004882
s
<
p95
0.030369
s
<
p99
0.298454
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
4.60946
s
SD
27.82266
s
p50
3.379006
s
<
p95
60.075771
s
<
p99
61.48005
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.556967
s
SD
24.643803
s
p50
1.200774
s
<
p95
60.017518
s
<
p99
60.097199
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
58.52
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
43.30
KiB/s
<
mean
561.31
KiB/s
SD
481.95
KiB/s
<
p95
1.54
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.77
KiB/s
<
mean
130.42
KiB/s
SD
219.16
KiB/s
<
p95
374.83
KiB/s
Total bytes received
Total bytes received on primary network interface
count
127.09
GiB
mean
72.70
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
29.52
GiB
mean
16.89
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (78.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 80% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
35.54
%
<
mean
79.42
%
SD
19.54
%
<
p95
96.05
%
System
20.90
%
CPU percentiles
CPU usage percentiles
p50
87.53
%
p95
96.05
%
p99
98.15
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
133
hosts
mean time
1287.71
s
Memory used percentage
Percentage of memory used
p5
7.61
%
<
mean
13.49
%
SD
8.38
%
<
p95
25.06
%
Memory available percentage
Percentage of available memory
p5
74.94
%
<
mean
86.51
%
SD
8.38
%
<
p95
92.39
%
Max host memory used
Maximum memory usage percentage across all hosts
max
89.56
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
78.14
%
Memory growth rate
Rate of memory growth over time
growth
0.37
MiB/s
Disk read throughput
Disk read throughput in MiB/s
3.77
MiB/s
Disk write throughput
Disk write throughput in MiB/s
167.09
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/111
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/18
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.96
5 min
3.53
15 min
2.60
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
79.72
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.93
%
<
mean
46.0137
%
SD
22.8546
%
<
p95
74.41
%
60 second average
44.7854
%
300 second average
39.4590
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0.0064
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0001
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0.0064
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0001
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
9.9037
%
SD
8.6691
%
<
p95
20.7
%
60 second average
9.7324
%
300 second average
8.7137
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.071
%
SD
4.9337
%
<
p95
8.79
%
60 second average
3.0665
%
300 second average
2.9600
%
Holochain process CPU usage
CPU usage by Holochain process
p5
35.49
%
<
mean
73.84
%
SD
19.07
%
<
p95
93.21
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
243.86
KiB
<
mean
440.26
KiB
SD
140.82
KiB
<
p95
655.31
KiB
Holochain process threads
Number of threads in Holochain process
p5
30
threads
<
mean
38.56
threads
SD
74.35
threads
<
p95
40
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
77
file descriptors
<
mean
92.01
file descriptors
SD
162.01
file descriptors
<
p95
93
file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 15 Oct 2026 07:08:03 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_38671000696_1
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
19.09
/s
Max
Highest per-partition mean rate
max
38.31
/s
Min
Lowest per-partition mean rate
min
2
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
107.77753
s
SD
5.908659
s
Max
Highest per-partition mean latency
max mean
1260.886071
s
Min
Lowest per-partition mean latency
min mean
4.857159
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0001
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
15831
Agents affected
9 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1759
Max
Highest count in any single partition
3598
Min
Lowest count in any single partition
90
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
233
Agents affected
12 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2
Max
Highest count in any single partition
89
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
47
<
mean
103.39
SD
34.47
<
p95
147
zero-arc
p5
0
<
mean
15.17
SD
13.89
<
p95
46
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
1930
Agents affected
139 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
14
Max
Highest count in any single partition
14
Min
Lowest count in any single partition
6
Error count
The number of errors accumulated across all nodes.
1931
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.544604
s
SD
4.359205
s
p50
1.008542
s
<
p95
13.427827
s
<
p99
17.863048
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
9
over 790.034966415s
mean rate
4806.34
/s
std
25225.39
/s
p5
0
/s
p95
15286.02
/s
peak
267843.55
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.014093
s
SD
0.159736
s
p50
0.01151
s
<
p95
0.482099
s
<
p99
0.813005
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.024798
s
SD
64.341556
s
p50
0.003618
s
<
p95
184.625837
s
<
p99
255.136192
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012805
s
SD
0.151619
s
p50
0.010369
s
<
p95
0.454328
s
<
p99
0.775403
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.191831
s
SD
108.826618
s
p50
0.540835
s
<
p95
316.222903
s
<
p99
434.068255
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001181
s
SD
0.003476
s
p50
0.001998
s
<
p95
0.003095
s
<
p99
0.020455
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011287
s
SD
0.011988
s
p50
0.001728
s
<
p95
0.009587
s
<
p99
0.047547
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002565
s
SD
0.044063
s
p50
0.001378
s
<
p95
0.138295
s
<
p99
0.216829
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.002404
s
SD
0.049298
s
p50
0.001228
s
<
p95
0.135932
s
<
p99
0.260962
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
2.545981
s
SD
4.462816
s
p50
2.696886
s
<
p95
14.393803
s
<
p99
19.184664
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.017884
s
SD
5.224149
s
p50
0.446997
s
<
p95
12.588366
s
<
p99
29.876628
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.012367
s
SD
0.009137
s
p50
0.011194
s
<
p95
0.027995
s
<
p99
0.055652
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002144
s
SD
0.008386
s
p50
0.003395
s
<
p95
0.00494
s
<
p99
0.049371
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000882
s
SD
0.002725
s
p50
0.000224
s
<
p95
0.00128
s
<
p99
0.003675
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.87
s
<
mean
433.08
s
SD
250.17
s
<
p95
824.48
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.681799287s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
103.535869
s
SD
18.494912
s
p50
0
s
<
p95
0.015136
s
<
p99
80.35051
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.288729
s
SD
0.818208
s
p50
0.001605
s
<
p95
1.462482
s
<
p99
3.811818
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.179709
s
SD
0.118405
s
p50
0.002011
s
<
p95
0.04998
s
<
p99
0.79689
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.245769
s
SD
0.143487
s
p50
0.020575
s
<
p95
0.154686
s
<
p99
0.687396
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
6.845342
s
SD
11.067058
s
p50
0.00274
s
<
p95
5.445742
s
<
p99
77.288343
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.335311
s
SD
5.344411
s
p50
0.001116
s
<
p95
0.013214
s
<
p99
0.320997
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002259
s
SD
0.002357
s
p50
0.000178
s
<
p95
0.00397
s
<
p99
0.009172
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007837
s
SD
0.132983
s
p50
0.004186
s
<
p95
0.04671
s
<
p99
0.097352
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002178
s
SD
0.010979
s
p50
0.000614
s
<
p95
0.001698
s
<
p99
0.011061
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
14.349108
s
SD
29.478733
s
p50
2.408415
s
<
p95
60.001735
s
<
p99
60.008611
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
4.957626
s
SD
18.614845
s
p50
1.255044
s
<
p95
60.001132
s
<
p99
60.001787
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.9539
s
SD
13.873435
s
p50
0.753828
s
<
p95
60.000917
s
≤
p99
60.000917
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.90
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
2.39
KiB/s
<
mean
143.54
KiB/s
SD
328.01
KiB/s
<
p95
633.70
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
404
B/s
<
mean
25.64
KiB/s
SD
121.98
KiB/s
<
p95
97.30
KiB/s
Total bytes received
Total bytes received on primary network interface
count
17.77
GiB
mean
20.44
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.17
GiB
mean
3.65
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.6%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (78.9% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 5% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.25
%
<
mean
12.53
%
SD
23.34
%
<
p95
85.23
%
System
4.63
%
CPU percentiles
CPU usage percentiles
p50
4.08
%
p95
85.23
%
p99
95.63
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
84
hosts
mean time
91.87
s
Memory used percentage
Percentage of memory used
p5
8.55
%
<
mean
13.44
%
SD
7.56
%
<
p95
23.94
%
Memory available percentage
Percentage of available memory
p5
76.06
%
<
mean
86.56
%
SD
7.56
%
<
p95
91.45
%
Max host memory used
Maximum memory usage percentage across all hosts
max
86.81
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
78.91
%
Memory growth rate
Rate of memory growth over time
growth
0.54
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.54
MiB/s
Disk write throughput
Disk write throughput in MiB/s
52.82
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.58
5 min
0.65
15 min
0.96
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
5.44
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
5.0713
%
SD
17.5876
%
<
p95
48.32
%
60 second average
4.9459
%
300 second average
6.0218
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0007
%
SD
0.0205
%
≤
p95
0
%
60 second average
0.0006
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0007
%
SD
0.0205
%
≤
p95
0
%
60 second average
0.0006
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.7105
%
SD
2.6136
%
<
p95
4.48
%
60 second average
0.7112
%
300 second average
0.8440
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2647
%
SD
1.0005
%
<
p95
1.62
%
60 second average
0.2679
%
300 second average
0.2844
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
8.03
%
SD
20.4
%
<
p95
66.44
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
315.55
KiB
<
mean
383.49
KiB
SD
67.20
KiB
<
p95
474.47
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
26.72
threads
SD
8.7
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
60.41
file descriptors
SD
16.94
file descriptors
<
p95
79
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Thu, 15 Oct 2026 07:25:25 UTC
Duration
900s
Peer count
150
Peer count at end
143
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_38671000696_1
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
472067
Agents affected
100 / 100
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4721
Max
Highest count in any single partition
11343
Min
Lowest count in any single partition
771
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1959.29
Max
Highest chain head value observed for any single write agent.
4634
Write agents observed
17
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
0.386207
s
SD
1.04437
s
Max
Highest per-partition mean latency
max mean
1.955341
s
Min
Lowest per-partition mean latency
min mean
-0.049486
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
7.5
/s
Max
Highest per-partition mean rate
max
51.69
/s
Min
Lowest per-partition mean rate
min
0
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
1.56738
s
SD
3.963069
s
Max
Highest per-partition mean latency
max mean
6.173355
s
Min
Lowest per-partition mean latency
min mean
0.006328
s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
145
Agents affected
41 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3
Max
Highest count in any single partition
10
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
8
<
mean
77.78
SD
34.74
<
p95
142
  • behaviour: zero_read
p5
3
<
mean
31.06
SD
16.08
<
p95
65
  • behaviour: zero_write
p5
4
<
mean
29.71
SD
16.3
<
p95
60
Final error count
The total number of errors accumulated across all agents over the run.
147
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.392951
s
SD
2.660275
s
p50
0.911283
s
<
p95
6.904983
s
<
p99
12.720892
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 807.676133564s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.008204
s
SD
0.005066
s
p50
0.007879
s
<
p95
0.022118
s
<
p99
0.030686
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008235
s
SD
0.015961
s
p50
0.006725
s
<
p95
0.051897
s
<
p99
0.07092
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.407525
s
SD
2.248486
s
p50
1.210167
s
<
p95
5.707736
s
<
p99
8.737768
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.067338
s
SD
3.011757
s
p50
0.606959
s
<
p95
10.90843
s
<
p99
12.726833
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006752
s
SD
0.004395
s
p50
0.006594
s
<
p95
0.013163
s
<
p99
0.028948
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00722
s
SD
0.015182
s
p50
0.005532
s
<
p95
0.048315
s
<
p99
0.067027
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.406412
s
SD
2.248429
s
p50
1.209126
s
<
p95
5.706249
s
<
p99
8.736822
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.066635
s
SD
3.011836
s
p50
0.606142
s
<
p95
10.907715
s
<
p99
12.725705
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00141
s
SD
0.000966
s
p50
0.001586
s
<
p95
0.003719
s
<
p99
0.005075
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
6.2e-05
s
SD
9e-05
s
p50
4.3e-05
s
<
p95
0.000174
s
<
p99
0.00072
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002849
s
SD
0.010627
s
p50
0.000859
s
<
p95
0.031368
s
<
p99
0.045463
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.00072
s
SD
0.000835
s
p50
0.000599
s
<
p95
0.00162
s
<
p99
0.007327
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.404397
s
SD
2.248106
s
p50
1.208279
s
<
p95
5.69099
s
<
p99
8.733389
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.063656
s
SD
3.011972
s
p50
0.601615
s
<
p95
10.903803
s
<
p99
12.720957
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
7e-06
s
SD
8e-06
s
p50
1.3e-05
s
<
p95
3.1e-05
s
<
p99
4.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002851
s
SD
0.003084
s
p50
0.002662
s
<
p95
0.00981
s
<
p99
0.017806
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.00113
s
SD
0.0032
s
p50
0.00048
s
<
p95
0.007811
s
<
p99
0.015877
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.22
s
<
mean
424.99
s
SD
251.87
s
<
p95
825.17
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
681
over 889.49826788s
mean rate
11658.5
/s
std
98719.06
/s
p5
0
/s
p95
25944.97
/s
peak
6.70589845e+06
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
5.01675
s
SD
21.034642
s
p50
0
s
<
p95
6.132089
s
<
p99
33.576405
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.194409
s
SD
0.753847
s
p50
0.002158
s
<
p95
1.420592
s
<
p99
3.377525
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.056284
s
SD
0.30869
s
p50
0.002032
s
<
p95
0.977241
s
<
p99
1.213771
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.188894
s
SD
0.14281
s
p50
0.130477
s
<
p95
0.364569
s
<
p99
0.591217
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
9.441953
s
SD
24.373876
s
p50
0.003539
s
<
p95
54.042283
s
<
p99
120.933909
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.872104
s
SD
14.269258
s
p50
0.001843
s
<
p95
1.872199
s
<
p99
54.74343
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003435
s
SD
0.002445
s
p50
0.00213
s
<
p95
0.007357
s
<
p99
0.009505
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006396
s
SD
0.003558
s
p50
0.00234
s
<
p95
0.011638
s
<
p99
0.016283
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002914
s
SD
0.009319
s
p50
0.000624
s
<
p95
0.02467
s
<
p99
0.045305
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
5.223294
s
SD
27.621751
s
p50
3.168277
s
<
p95
60.047756
s
<
p99
60.071131
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.214865
s
SD
2.116812
s
p50
0.526037
s
<
p95
6.321615
s
<
p99
10.900903
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.302502
s
SD
28.901662
s
p50
0.937634
s
<
p95
60.001496
s
<
p99
60.002551
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.719185
s
SD
14.459095
s
p50
0.44831
s
<
p95
60.050838
s
<
p99
60.180141
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
40.78
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.88
KiB/s
<
mean
581.01
KiB/s
SD
568.65
KiB/s
<
p95
1.68
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.33
KiB/s
<
mean
144.20
KiB/s
SD
199.26
KiB/s
<
p95
504.00
KiB/s
Total bytes received
Total bytes received on primary network interface
count
65.75
GiB
mean
75.65
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
16.28
GiB
mean
18.73
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.3%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (77.5% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 38% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.81
%
<
mean
54.89
%
SD
37.46
%
<
p95
93.81
%
System
14.11
%
CPU percentiles
CPU usage percentiles
p50
66.80
%
p95
93.81
%
p99
97.29
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
87
hosts
mean time
632.35
s
Memory used percentage
Percentage of memory used
p5
7.02
%
<
mean
12.61
%
SD
7.88
%
<
p95
23.53
%
Memory available percentage
Percentage of available memory
p5
76.47
%
<
mean
87.39
%
SD
7.88
%
<
p95
92.98
%
Max host memory used
Maximum memory usage percentage across all hosts
max
86.90
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
77.51
%
Memory growth rate
Rate of memory growth over time
growth
0.47
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.60
MiB/s
Disk write throughput
Disk write throughput in MiB/s
109.25
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/105
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/23
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.47
5 min
1.91
15 min
1.35
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
38.03
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
29.8954
%
SD
29.1237
%
<
p95
72.98
%
60 second average
28.3366
%
300 second average
21.8294
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1593
%
SD
5.8185
%
<
p95
14.95
%
60 second average
3.9670
%
300 second average
3.0661
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2088
%
SD
2.2343
%
<
p95
4.49
%
60 second average
1.1610
%
300 second average
0.9142
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
48.67
%
SD
36.63
%
<
p95
88.49
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
208.21
KiB
<
mean
303.63
KiB
SD
85.39
KiB
<
p95
464.01
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
29.69
threads
SD
9.18
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
0
file descriptors
<
mean
72.88
file descriptors
SD
21.31
file descriptors
<
p95
89
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Thu, 15 Oct 2026 07:42:38 UTC
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_38671000696_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
460.53
Max
Highest chain head value observed for any single write agent.
1633
Write agents observed
17
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
213.208517
s
SD
145.256511
s
Max
Highest per-partition mean latency
max mean
430.147818
s
Min
Lowest per-partition mean latency
min mean
27.552422
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.11
/s
Max
Highest per-partition mean rate
max
2.57
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
2.075792
s
SD
2.502416
s
Max
Highest per-partition mean latency
max mean
14.001081
s
Min
Lowest per-partition mean latency
min mean
0.160342
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
92265
Agents affected
44 / 44
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2097
Max
Highest count in any single partition
10783
Min
Lowest count in any single partition
45
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
39
<
mean
82.8
SD
38.76
<
p95
144
  • behaviour: zero_must_get_agent_activity
p5
20
<
mean
34.15
SD
14.16
<
p95
73
  • behaviour: zero_write
p5
16
<
mean
31.79
SD
12.05
<
p95
52
Final error count
The total number of all types of error accumulated over the run by all agents.
105165
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.098425
s
SD
1.484756
s
p50
0.270539
s
<
p95
2.253177
s
<
p99
11.036182
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 816.790622581s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007711
s
SD
0.005266
s
p50
0.006713
s
<
p95
0.017202
s
<
p99
0.024028
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.164362
s
SD
0.167753
s
p50
0.084271
s
<
p95
0.521125
s
<
p99
0.804911
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006138
s
SD
0.002153
s
p50
0.007502
s
<
p95
0.008796
s
<
p99
0.01092
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.327997
s
SD
0.770257
s
p50
0.366374
s
<
p95
1.998797
s
<
p99
4.379168
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.059201
s
SD
2.044918
s
p50
0.725407
s
<
p95
3.373618
s
<
p99
12.425487
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00629
s
SD
0.004445
s
p50
0.005436
s
<
p95
0.014929
s
<
p99
0.018846
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.162789
s
SD
0.167082
s
p50
0.082742
s
<
p95
0.51777
s
<
p99
0.801415
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005043
s
SD
0.001815
s
p50
0.006207
s
<
p95
0.007228
s
<
p99
0.009223
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.663655
s
SD
1.886442
s
p50
0.653356
s
<
p95
4.296291
s
<
p99
11.043366
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.0585
s
SD
2.044864
s
p50
0.724802
s
<
p95
3.372132
s
<
p99
12.424042
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001619
s
SD
0.000847
s
p50
0.001859
s
<
p95
0.003338
s
<
p99
0.004844
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.025456
s
SD
0.092278
s
p50
0.006484
s
<
p95
0.175527
s
<
p99
0.507225
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5e-05
s
SD
4e-05
s
p50
3.6e-05
s
<
p95
0.000129
s
<
p99
0.000211
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002076
s
SD
0.006273
s
p50
0.000617
s
<
p95
0.014677
s
<
p99
0.037027
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.003549
s
SD
0.006396
s
p50
0.000637
s
<
p95
0.013459
s
<
p99
0.035331
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.578983
s
SD
1.96717
s
p50
0.671972
s
<
p95
4.22195
s
<
p99
12.419384
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.077526
s
SD
0.385408
s
p50
0.064605
s
<
p95
0.796916
s
<
p99
2.253463
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.073139
s
SD
0.07406
s
p50
0.02437
s
<
p95
0.224545
s
<
p99
0.280545
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
6e-06
s
SD
9e-06
s
p50
1.6e-05
s
<
p95
2.5e-05
s
<
p99
5.5e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003107
s
SD
0.002142
s
p50
0.003066
s
<
p95
0.0065
s
<
p99
0.013844
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001059
s
SD
0.001525
s
p50
0.000596
s
<
p95
0.004525
s
<
p99
0.007981
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
45
s
<
mean
434.35
s
SD
250.79
s
<
p95
825.37
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.231485633s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
68.085767
s
SD
33.63139
s
p50
0
s
<
p95
34.647529
s
<
p99
211.10297
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.175699
s
SD
1.177139
s
p50
0.002649
s
<
p95
3.125176
s
<
p99
3.510174
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.678647
s
SD
0.429751
s
p50
0.002812
s
<
p95
1.271432
s
<
p99
1.430207
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.193135
s
SD
0.194748
s
p50
0.034661
s
<
p95
0.567197
s
<
p99
0.845044
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.892821
s
SD
9.017374
s
p50
0.004181
s
<
p95
22.34798
s
<
p99
33.06505
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
3.181538
s
SD
10.002592
s
p50
0.002027
s
<
p95
3.254924
s
<
p99
56.82422
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001967
s
SD
0.001505
s
p50
0.00021
s
<
p95
0.004358
s
<
p99
0.005646
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.002479
s
SD
0.005466
s
p50
0.002865
s
<
p95
0.017251
s
<
p99
0.027552
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002
s
SD
0.006067
s
p50
0.000546
s
<
p95
0.013967
s
<
p99
0.035874
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.160882
s
SD
29.034593
s
p50
0.620594
s
<
p95
60.005429
s
<
p99
60.033846
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.242943
s
SD
18.067166
s
p50
0.423904
s
<
p95
60.001725
s
<
p99
60.002132
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.193944
s
SD
27.793245
s
p50
0.287062
s
<
p95
60.001789
s
<
p99
60.002844
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.799804
s
SD
27.594243
s
p50
1.071265
s
<
p95
60.010064
s
<
p99
60.020249
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
25.91
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
11.87
KiB/s
<
mean
401.72
KiB/s
SD
387.30
KiB/s
<
p95
819.15
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.62
KiB/s
<
mean
94.40
KiB/s
SD
108.29
KiB/s
<
p95
324.48
KiB/s
Total bytes received
Total bytes received on primary network interface
count
48.50
GiB
mean
55.80
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
11.38
GiB
mean
13.10
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (79.9% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 31% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.6
%
<
mean
37.12
%
SD
34.97
%
<
p95
93.75
%
System
11.22
%
CPU percentiles
CPU usage percentiles
p50
16.96
%
p95
93.75
%
p99
97.90
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
102
hosts
mean time
327.24
s
Memory used percentage
Percentage of memory used
p5
7.96
%
<
mean
13.34
%
SD
7.56
%
<
p95
19.87
%
Memory available percentage
Percentage of available memory
p5
80.13
%
<
mean
86.66
%
SD
7.56
%
<
p95
92.04
%
Max host memory used
Maximum memory usage percentage across all hosts
max
88.55
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
79.89
%
Memory growth rate
Rate of memory growth over time
growth
0.36
MiB/s
Disk read throughput
Disk read throughput in MiB/s
2.04
MiB/s
Disk write throughput
Disk write throughput in MiB/s
168.81
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/116
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/11
hosts
Mount Point /etc/hosts
0/11
hosts
Mount Point /etc/nomad.d/client-meta.json
0/20
hosts
Mount Point /etc/resolv.conf
0/11
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.73
5 min
1.73
15 min
1.63
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
30.61
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
19.2013
%
SD
26.9789
%
<
p95
75.57
%
60 second average
18.3037
%
300 second average
18.6089
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0002
%
300 second average
0.0001
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0002
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.1246
%
SD
5.6684
%
<
p95
15.19
%
60 second average
3.0122
%
300 second average
2.9741
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.1618
%
SD
2.4401
%
<
p95
6.77
%
60 second average
1.1423
%
300 second average
1.1088
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.55
%
<
mean
31.15
%
SD
33.62
%
<
p95
88.78
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
275.04
KiB
<
mean
388.82
KiB
SD
83.90
KiB
<
p95
516.60
KiB
Holochain process threads
Number of threads in Holochain process
p5
9
threads
<
mean
31.02
threads
SD
8.89
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
23
file descriptors
<
mean
74.42
file descriptors
SD
18.63
file descriptors
<
p95
89
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Thu, 15 Oct 2026 02:38:06 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_38671000696_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-6.530127
s
SD
38.201986
s
Max
Highest per-partition mean latency
max mean
0.192059
s
Min
Lowest per-partition mean latency
min mean
-628.178059
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.178031
s
SD
0.405157
s
Max
Highest per-partition mean latency
max mean
0.949141
s
Min
Lowest per-partition mean latency
min mean
0.056311
s
Error count
The total number of errors accumulated during the run.
3732
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 713.066367954s
Not enough time data to show a trend.
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
142967750
over 888.482910173s
mean rate
3.82497859444e+09
/s
std
2.97147806517e+10
/s
p5
1.213890323e+07
/s
p95
1.04065306096e+10
/s
peak
1.0406689898e+12
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
21336351
over 888.482910173s
mean rate
5.8072161368e+08
/s
std
5.93202175473e+09
/s
p5
1.06909933e+06
/s
p95
1.36440924237e+09
/s
peak
2.24439525438e+11
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 888.482910173s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.160395
s
SD
0.732099
s
p50
0.165124
s
<
p95
0.392014
s
<
p99
1.129751
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.003859
s
SD
0.002534
s
p50
0.004039
s
<
p95
0.00596
s
<
p99
0.011942
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.177242
s
SD
0.731949
s
p50
0.182908
s
<
p95
0.418361
s
<
p99
1.139981
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002379
s
SD
0.001563
s
p50
0.002535
s
<
p95
0.003881
s
<
p99
0.007214
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005029
s
SD
0.00604
s
p50
0.003859
s
<
p95
0.010166
s
<
p99
0.025018
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.174605
s
SD
0.731939
s
p50
0.180538
s
<
p95
0.415394
s
<
p99
1.136638
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000649
s
SD
0.000414
s
p50
0.00053
s
<
p95
0.001251
s
<
p99
0.002302
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.4e-05
s
SD
1e-05
s
p50
1.1e-05
s
<
p95
2.4e-05
s
<
p99
3.2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000339
s
SD
0.003399
s
p50
0.000388
s
<
p95
0.00103
s
<
p99
0.01182
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000508
s
SD
0.000944
s
p50
0.000214
s
<
p95
0.001957
s
<
p99
0.004532
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.86
s
<
mean
430.08
s
SD
249.07
s
<
p95
817.25
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1930
over 888.482855967s
mean rate
11314.83
/s
std
109393.72
/s
p5
0
/s
p95
27798.75
/s
peak
5.02818215e+06
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
13.759568
s
SD
8.48111
s
p50
9.74067
s
<
p95
31.212675
s
<
p99
40.608605
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.012048
s
SD
0.021137
s
p50
0.011229
s
<
p95
0.046613
s
<
p99
0.125026
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.06852
s
SD
0.059737
s
p50
0.070016
s
<
p95
0.193405
s
<
p99
0.314745
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.02611
s
SD
0.059298
s
p50
0.017517
s
<
p95
0.04007
s
<
p99
0.181288
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
4.054054
s
SD
4.62328
s
p50
5.368409
s
<
p95
14.807812
s
<
p99
23.643381
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.646386
s
SD
2.886355
s
p50
1.438035
s
<
p95
7.510859
s
<
p99
14.794953
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000433
s
SD
0.000324
s
p50
0.000401
s
<
p95
0.000769
s
<
p99
0.001654
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003499
s
SD
0.002374
s
p50
0.002614
s
<
p95
0.00652
s
<
p99
0.008303
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000599
s
SD
0.000583
s
p50
0.000594
s
<
p95
0.000962
s
<
p99
0.003338
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.173737
s
SD
27.789388
s
p50
0.095087
s
<
p95
60.002962
s
<
p99
60.006137
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.183959
s
SD
4.289111
s
p50
0.168414
s
<
p95
0.479391
s
<
p99
1.408026
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.161703
s
SD
22.51991
s
p50
0.191723
s
<
p95
60.001468
s
<
p99
60.00199
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.44
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
14.02
KiB/s
<
mean
118.70
KiB/s
SD
315.80
KiB/s
<
p95
293.43
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
5.58
KiB/s
<
mean
16.05
KiB/s
SD
15.25
KiB/s
<
p95
43.15
KiB/s
Total bytes received
Total bytes received on primary network interface
count
14.49
GiB
mean
16.67
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.96
GiB
mean
2.26
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.3% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.46
%
<
mean
13.13
%
SD
12.24
%
<
p95
37.18
%
System
5.69
%
CPU percentiles
CPU usage percentiles
p50
9.94
%
p95
37.18
%
p99
72.25
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
56
hosts
mean time
11.59
s
Memory used percentage
Percentage of memory used
p5
6.72
%
<
mean
10.48
%
SD
7.57
%
<
p95
20.87
%
Memory available percentage
Percentage of available memory
p5
79.13
%
<
mean
89.52
%
SD
7.57
%
<
p95
93.28
%
Max host memory used
Maximum memory usage percentage across all hosts
max
86.62
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.34
%
Memory growth rate
Rate of memory growth over time
growth
0.23
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.82
MiB/s
Disk write throughput
Disk write throughput in MiB/s
31.75
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/119
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.45
5 min
0.36
15 min
0.21
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
3.7994
%
SD
6.5252
%
<
p95
10
%
60 second average
3.6725
%
300 second average
2.8624
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0014
%
SD
0.0581
%
≤
p95
0
%
60 second average
0.0007
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0014
%
SD
0.0581
%
≤
p95
0
%
60 second average
0.0006
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6443
%
SD
2.9318
%
<
p95
2.33
%
60 second average
0.8428
%
300 second average
0.8613
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.4502
%
SD
2.6293
%
<
p95
1.63
%
60 second average
0.6430
%
300 second average
0.7043
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.4
%
<
mean
9.81
%
SD
10.43
%
<
p95
23.03
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
211.31
KiB
<
mean
241.92
KiB
SD
31.93
KiB
<
p95
283.43
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
33.28
threads
SD
6.89
threads
<
p95
40
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
76
file descriptors
<
mean
81.5
file descriptors
SD
8.86
file descriptors
<
p95
89
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Thu, 15 Oct 2026 02:56:01 UTC
Duration
900s
Peer count
300
Peer count at end
290
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_signals_38671000696_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
2.587438
s
SD
13.181451
s
p50
0.156324
s
<
p95
9.551907
s
<
p99
67.737989
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
370
over 1364.353472624s
mean rate
821096.48
/s
std
7.164927523e+07
/s
p5
0
/s
p95
13889.91
/s
peak
7.5625e+09
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0016
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 710.824247706s
mean rate
0.64
/s
std
0.86
/s
p5
0
/s
p95
2.39
/s
peak
2.39
/s
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.17739798s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
24249168
over 889.17739798s
mean rate
7.14300229335e+09
/s
std
1.26122836968e+11
/s
p5
2.81715264e+06
/s
p95
1.11507824971e+10
/s
peak
8.23620224719e+12
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
12100074
over 889.17739798s
mean rate
1.0680195941e+08
/s
std
3.77583281364e+09
/s
p5
0
/s
p95
1.6637076141e+08
/s
peak
4.45970588235e+11
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.003552
s
SD
0.001799
s
p50
0.004174
s
<
p95
0.005892
s
<
p99
0.007514
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.003402
s
SD
0.001489
s
p50
0.003971
s
<
p95
0.005399
s
<
p99
0.006387
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005052
s
SD
0.008028
s
p50
0.003372
s
<
p95
0.010891
s
<
p99
0.054928
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002246
s
SD
0.001209
s
p50
0.002697
s
<
p95
0.003853
s
<
p99
0.005376
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002093
s
SD
0.000961
s
p50
0.002492
s
<
p95
0.003367
s
<
p99
0.00441
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000743
s
SD
0.000976
s
p50
0.000522
s
<
p95
0.001818
s
<
p99
0.003018
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
6.6e-05
s
SD
3e-05
s
p50
7.5e-05
s
<
p95
0.000104
s
<
p99
0.000127
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
6.1e-05
s
SD
5.6e-05
s
p50
6.9e-05
s
<
p95
0.000107
s
<
p99
0.000142
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
6e-06
s
p50
8e-06
s
<
p95
1.8e-05
s
<
p99
2.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001336
s
SD
0.006538
s
p50
0.000381
s
<
p95
0.001719
s
<
p99
0.04804
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1360
over 889.177179852s
mean rate
417450.81
/s
std
6.34630183e+06
/s
p5
98.68
/s
p95
645550.01
/s
peak
5.3741496599e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1432
over 889.177156655s
mean rate
718878.25
/s
std
2.660279936e+07
/s
p5
119.14
/s
p95
617101.11
/s
peak
2.83333333333e+09
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000844
s
SD
0.002004
s
p50
0.000212
s
<
p95
0.003471
s
<
p99
0.008842
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.67
s
<
mean
432.33
s
SD
249.77
s
<
p95
822.24
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3675
over 889.177434031s
mean rate
4.31472718e+06
/s
std
1.9056546124e+08
/s
p5
6.89
/s
p95
2.65245708e+06
/s
peak
1.31428571429e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
20.256713
s
SD
10.775189
s
p50
16.165628
s
<
p95
39.867706
s
<
p99
48.924085
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.019682
s
SD
0.044899
s
p50
0.019141
s
<
p95
0.126585
s
<
p99
0.230244
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.089917
s
SD
0.094612
s
p50
0.094324
s
<
p95
0.288998
s
<
p99
0.530725
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.029121
s
SD
0.023762
s
p50
0.023496
s
<
p95
0.067598
s
<
p99
0.125238
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.824896
s
SD
7.709043
s
p50
8.076811
s
<
p95
25.511087
s
<
p99
35.724901
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
6.688388
s
SD
17.875404
s
p50
4.904301
s
<
p95
48.685211
s
<
p99
85.371333
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000663
s
SD
0.000402
s
p50
0.00054
s
<
p95
0.001437
s
<
p99
0.001635
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003997
s
SD
0.003326
s
p50
0.002751
s
<
p95
0.009225
s
<
p99
0.011768
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000727
s
SD
0.00105
s
p50
0.000733
s
<
p95
0.002083
s
<
p99
0.004261
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.273995
s
SD
28.968283
s
p50
0.343848
s
<
p95
60.004071
s
<
p99
60.010947
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.582012
s
SD
24.94142
s
p50
0.575579
s
<
p95
60.001624
s
<
p99
60.002226
s
P2P received remote signals
Count of remote signals received.
total
1364
over 889.177147319s
mean rate
229360.25
/s
std
2.12128152e+06
/s
p5
106.07
/s
p95
639845.78
/s
peak
1.0133333333e+08
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
11.25
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
28.54
KiB/s
<
mean
258.88
KiB/s
SD
374.71
KiB/s
<
p95
949.75
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
14.85
KiB/s
<
mean
39.95
KiB/s
SD
37.76
KiB/s
<
p95
104.51
KiB/s
Total bytes received
Total bytes received on primary network interface
count
31.65
GiB
mean
36.41
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.88
GiB
mean
5.62
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.1%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (76.0% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 2% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.82
%
<
mean
17.71
%
SD
15.99
%
<
p95
43.19
%
System
6.46
%
CPU percentiles
CPU usage percentiles
p50
14.08
%
p95
43.19
%
p99
94.12
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
120
hosts
mean time
45.94
s
Memory used percentage
Percentage of memory used
p5
7.72
%
<
mean
13.26
%
SD
7.81
%
<
p95
24.4
%
Memory available percentage
Percentage of available memory
p5
75.6
%
<
mean
86.74
%
SD
7.81
%
<
p95
92.28
%
Max host memory used
Maximum memory usage percentage across all hosts
max
90.05
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
76.01
%
Memory growth rate
Rate of memory growth over time
growth
0.91
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.65
MiB/s
Disk write throughput
Disk write throughput in MiB/s
39.87
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.65
5 min
0.58
15 min
0.42
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
2.03
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
7.1763
%
SD
11.6755
%
<
p95
23.52
%
60 second average
7.0634
%
300 second average
6.2322
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0005
%
SD
0.0198
%
≤
p95
0
%
60 second average
0.0004
%
300 second average
0.0002
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0005
%
SD
0.0185
%
≤
p95
0
%
60 second average
0.0004
%
300 second average
0.0002
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.5438
%
SD
1.4208
%
<
p95
2.31
%
60 second average
0.5613
%
300 second average
0.5427
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3368
%
SD
1.1693
%
<
p95
1.47
%
60 second average
0.3497
%
300 second average
0.3539
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.26
%
<
mean
7.29
%
SD
7.71
%
<
p95
20.29
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
194.23
KiB
<
mean
231.09
KiB
SD
36.77
KiB
<
p95
276.09
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
32.95
threads
SD
6.99
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
81.46
file descriptors
SD
9.14
file descriptors
<
p95
90
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Thu, 15 Oct 2026 03:57:32 UTC
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
validation_receipts_38671000696_1
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
43.795338
s
SD
27.299955
s
Max
Highest per-partition mean latency
max mean
424.699496
s
Min
Lowest per-partition mean latency
min mean
13.061537
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
0.98
/s
Max
Highest per-partition mean rate
max
1.1
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
2
over 724.139766875s
mean rate
1.27
/s
std
2.96
/s
p5
0
/s
p95
7
/s
peak
13.25
/s
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
818256
over 888.993752136s
mean rate
8.594640038e+07
/s
std
3.43085759472e+09
/s
p5
0
/s
p95
6.383101612e+07
/s
peak
2.73747445255e+11
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
9845716
over 888.993752136s
mean rate
2.0814076367e+08
/s
std
2.77235413241e+09
/s
p5
809234.65
/s
p95
3.5984980436e+08
/s
peak
1.59683576642e+11
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 888.993752136s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
105696
over 888.993752136s
mean rate
1.095387706e+07
/s
std
4.3964052689e+08
/s
p5
0
/s
p95
8.35396132e+06
/s
peak
3.53605839416e+10
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006155
s
SD
0.003159
s
p50
0.006562
s
<
p95
0.010481
s
<
p99
0.015909
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.018636
s
SD
0.015799
s
p50
0.010579
s
<
p95
0.041401
s
<
p99
0.09332
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005043
s
SD
0.002725
s
p50
0.005359
s
<
p95
0.008561
s
<
p99
0.013892
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.01733
s
SD
0.015449
s
p50
0.009294
s
<
p95
0.039369
s
<
p99
0.090954
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003528
s
SD
0.003255
s
p50
0.002843
s
<
p95
0.008596
s
<
p99
0.021835
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001616
s
SD
0.000805
s
p50
0.001765
s
<
p95
0.002675
s
<
p99
0.003496
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000652
s
SD
0.001174
s
p50
0.00061
s
<
p95
0.002277
s
<
p99
0.00684
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.015396
s
SD
0.015091
s
p50
0.007386
s
<
p95
0.0367
s
<
p99
0.088095
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002544
s
SD
0.001469
s
p50
0.002699
s
<
p95
0.004211
s
<
p99
0.006067
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000684
s
SD
0.001002
s
p50
0.000528
s
<
p95
0.002054
s
<
p99
0.006067
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.73
s
<
mean
427.16
s
SD
249.51
s
<
p95
822.84
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
9598
over 888.993664717s
mean rate
509378.53
/s
std
9.1508513e+06
/s
p5
496.22
/s
p95
776192.85
/s
peak
5.8222932846e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
20.398686
s
SD
10.15403
s
p50
18.779386
s
<
p95
37.385386
s
<
p99
44.837302
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005158
s
SD
0.059841
s
p50
0.013204
s
<
p95
0.164025
s
<
p99
0.286047
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.037948
s
SD
0.132076
s
p50
0.082814
s
<
p95
0.396726
s
<
p99
0.655016
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.067277
s
SD
0.049431
s
p50
0.0366
s
<
p95
0.147219
s
<
p99
0.265866
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.531011
s
SD
8.322684
s
p50
3.278492
s
<
p95
24.817015
s
<
p99
34.57826
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
14.613083
s
SD
18.599701
s
p50
18.531481
s
<
p95
51.091632
s
<
p99
86.543221
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001056
s
SD
0.000649
s
p50
0.000735
s
<
p95
0.002342
s
<
p99
0.002823
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004494
s
SD
0.003714
s
p50
0.002731
s
<
p95
0.007796
s
<
p99
0.026994
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000548
s
SD
0.000404
s
p50
0.000574
s
<
p95
0.001281
s
<
p99
0.001869
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.511083
s
SD
29.682709
s
p50
0.358857
s
<
p95
60.003696
s
<
p99
60.007848
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.143892
s
SD
21.127525
s
p50
0.138963
s
<
p95
60.001411
s
<
p99
60.001735
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
15.74
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
19.00
KiB/s
<
mean
134.51
KiB/s
SD
264.38
KiB/s
<
p95
349.13
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
4.07
KiB/s
<
mean
22.55
KiB/s
SD
31.61
KiB/s
<
p95
68.03
KiB/s
Total bytes received
Total bytes received on primary network interface
count
16.30
GiB
mean
18.75
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.73
GiB
mean
3.14
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.1%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (17.5% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.19
%
<
mean
22.11
%
SD
20.27
%
<
p95
68.19
%
System
6.37
%
CPU percentiles
CPU usage percentiles
p50
15.96
%
p95
68.19
%
p99
94.07
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
76
hosts
mean time
52.89
s
Memory used percentage
Percentage of memory used
p5
6.08
%
<
mean
10.86
%
SD
4.27
%
<
p95
16.72
%
Memory available percentage
Percentage of available memory
p5
83.28
%
<
mean
89.14
%
SD
4.27
%
<
p95
93.92
%
Max host memory used
Maximum memory usage percentage across all hosts
max
35.45
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
17.54
%
Memory growth rate
Rate of memory growth over time
growth
0.50
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.15
MiB/s
Disk write throughput
Disk write throughput in MiB/s
46.86
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/10
hosts
Mount Point /etc/hosts
0/10
hosts
Mount Point /etc/nomad.d/client-meta.json
0/17
hosts
Mount Point /etc/resolv.conf
0/10
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.83
5 min
1.34
15 min
2.16
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
10.20
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
6.4469
%
SD
10.6961
%
<
p95
25
%
60 second average
6.6210
%
300 second average
13.4515
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.3951
%
SD
7.515
%
<
p95
5.72
%
60 second average
2.3774
%
300 second average
3.5376
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.9462
%
SD
7.118
%
<
p95
3.15
%
60 second average
1.8846
%
300 second average
1.9288
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.76
%
<
mean
14.7
%
SD
14.09
%
<
p95
43.78
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
231.48
KiB
<
mean
274.38
KiB
SD
40.04
KiB
<
p95
315.78
KiB
Holochain process threads
Number of threads in Holochain process
p5
1
threads
<
mean
30.16
threads
SD
9.15
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
5
file descriptors
<
mean
75.86
file descriptors
SD
19.72
file descriptors
<
p95
87
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 15 Oct 2026 04:16:30 UTC
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_38671000696_1
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2130.07
Max
Highest chain head value observed for any single write agent.
8113
Write agents observed
27
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.82006
s
SD
1.900231
s
Max
Highest per-partition mean latency
max mean
7.1813
s
Min
Lowest per-partition mean latency
min mean
0.017289
s
Error count
The number of errors accumulated.
353
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.081189
s
SD
2.030521
s
p50
0.033492
s
<
p95
4.3489
s
<
p99
11.822521
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
5
over 691.534138636s
mean rate
3.57
/s
std
5.77
/s
p5
0
/s
p95
16.84
/s
peak
17.18
/s
WASM usage: 'agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.054342814s
Not enough time data to show a trend.
WASM usage: 'agent_activity::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
43307460
over 889.054342814s
mean rate
5.0153854676e+08
/s
std
7.6106599627e+08
/s
p5
8.91029418e+06
/s
p95
2.33855263197e+09
/s
peak
3.92460867133e+09
/s
WASM usage: 'agent_activity::get_agent_activity_full'
Metered usage count of the WASM ribosome for this zome function.
total
57556749
over 890.269494805s
mean rate
4.10959251589e+10
/s
std
2.86852076582e+11
/s
p5
2.585871877e+07
/s
p95
7.73252796067e+10
/s
peak
4.26597253457e+12
/s
WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
391099165
over 890.435178731s
mean rate
2.22975972339e+10
/s
std
2.97293653402e+11
/s
p5
5634.94
/s
p95
3.26233371569e+10
/s
peak
1.29432430185e+13
/s
WASM usage: 'agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
28517772
over 890.435178731s
mean rate
1.29620267381e+09
/s
std
1.79377500885e+10
/s
p5
753700.3
/s
p95
2.14367395458e+09
/s
peak
9.43825500073e+11
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009237
s
SD
0.008464
s
p50
0.006161
s
<
p95
0.029488
s
<
p99
0.04351
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009293
s
SD
0.010778
s
p50
0.009995
s
<
p95
0.039807
s
<
p99
0.046989
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.18731
s
SD
2.29433
s
p50
0.436665
s
<
p95
5.881731
s
<
p99
12.363938
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006367
s
SD
1.672941
s
p50
0.005948
s
<
p95
0.820657
s
<
p99
9.237516
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007957
s
SD
0.007982
s
p50
0.005262
s
<
p95
0.027425
s
<
p99
0.041649
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008008
s
SD
0.010222
s
p50
0.008437
s
<
p95
0.037366
s
<
p99
0.044564
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.186621
s
SD
2.294148
s
p50
0.43511
s
<
p95
5.881059
s
<
p99
12.363303
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005282
s
SD
1.672899
s
p50
0.004905
s
<
p95
0.820208
s
<
p99
9.236224
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001554
s
SD
0.01191
s
p50
0.001589
s
<
p95
0.003087
s
<
p99
0.092848
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.1e-05
s
SD
5.6e-05
s
p50
3.7e-05
s
<
p95
0.000106
s
<
p99
0.000321
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.003203
s
SD
0.007106
s
p50
0.002623
s
<
p95
0.02475
s
<
p99
0.028647
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.002077
s
SD
0.003823
s
p50
0.000603
s
<
p95
0.01305
s
<
p99
0.015335
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.184883
s
SD
2.294015
s
p50
0.433694
s
<
p95
5.879658
s
<
p99
12.362566
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000794
s
SD
1.673321
s
p50
0.00051
s
<
p95
0.818069
s
<
p99
9.232531
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.3e-05
s
SD
0.000207
s
p50
1.1e-05
s
<
p95
2.5e-05
s
<
p99
0.000549
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002814
s
SD
0.018185
s
p50
0.002578
s
<
p95
0.008769
s
<
p99
0.143884
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001237
s
SD
0.033235
s
p50
0.000315
s
<
p95
0.006984
s
<
p99
0.052403
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.53
s
<
mean
425.13
s
SD
247.4
s
<
p95
815.01
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
2849
over 890.435140144s
mean rate
961591.93
/s
std
1.513549472e+07
/s
p5
2305.85
/s
p95
1.37954766e+06
/s
peak
7.1872787671e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
133.675737
s
SD
90.340286
s
p50
38.553228
s
<
p95
273.543881
s
<
p99
396.948678
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.087055
s
SD
0.480093
s
p50
0.167383
s
<
p95
0.761314
s
<
p99
2.6886
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.233129
s
SD
0.522952
s
p50
0.600072
s
<
p95
1.393193
s
<
p99
2.476958
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.6086
s
SD
0.778576
s
p50
0.100316
s
<
p95
2.117309
s
<
p99
3.512308
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.35944
s
SD
22.521849
s
p50
16.38381
s
<
p95
72.82946
s
<
p99
90.46229
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
18.024903
s
SD
29.329636
s
p50
3.488954
s
<
p95
73.041005
s
<
p99
146.523724
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004005
s
SD
0.003881
s
p50
0.003516
s
<
p95
0.00904
s
<
p99
0.024434
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004633
s
SD
0.006829
s
p50
0.00325
s
<
p95
0.01043
s
<
p99
0.026109
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.00212
s
SD
0.010259
s
p50
0.000861
s
<
p95
0.013102
s
<
p99
0.083029
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.625562
s
SD
29.253009
s
p50
0.919595
s
<
p95
60.006582
s
<
p99
60.021494
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.365395
s
SD
2.871994
s
p50
0.564416
s
<
p95
10.470789
s
≤
p99
10.470789
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.143453
s
SD
29.01658
s
p50
0.280416
s
<
p95
60.004186
s
<
p99
60.01215
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.318029
s
SD
22.384407
s
p50
0.329086
s
<
p95
60.001908
s
<
p99
60.00901
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
50.25
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
55.35
KiB/s
<
mean
484.82
KiB/s
SD
412.79
KiB/s
<
p95
1.15
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.80
KiB/s
<
mean
127.79
KiB/s
SD
204.29
KiB/s
<
p95
467.29
KiB/s
Total bytes received
Total bytes received on primary network interface
count
58.67
GiB
mean
66.75
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
15.46
GiB
mean
17.59
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.7%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (76.9% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 50% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
11.77
%
<
mean
68.19
%
SD
25.01
%
<
p95
94.98
%
System
17.94
%
CPU percentiles
CPU usage percentiles
p50
76.28
%
p95
94.98
%
p99
97.70
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
129
hosts
mean time
427.82
s
Memory used percentage
Percentage of memory used
p5
6.58
%
<
mean
12
%
SD
7.81
%
<
p95
24.13
%
Memory available percentage
Percentage of available memory
p5
75.87
%
<
mean
88
%
SD
7.81
%
<
p95
93.42
%
Max host memory used
Maximum memory usage percentage across all hosts
max
87.38
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
76.90
%
Memory growth rate
Rate of memory growth over time
growth
0.46
MiB/s
Disk read throughput
Disk read throughput in MiB/s
1.82
MiB/s
Disk write throughput
Disk write throughput in MiB/s
148.61
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/118
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/10
hosts
Mount Point /etc/hosts
0/10
hosts
Mount Point /etc/nomad.d/client-meta.json
0/17
hosts
Mount Point /etc/resolv.conf
0/10
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.29
5 min
2.47
15 min
2.00
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
49.66
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.2
%
<
mean
40.1042
%
SD
28.0509
%
<
p95
85.8
%
60 second average
37.3875
%
300 second average
27.3647
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.2616
%
SD
7.1269
%
<
p95
19.3
%
60 second average
7.8003
%
300 second average
5.9754
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.3727
%
SD
3.858
%
<
p95
10.37
%
60 second average
3.2268
%
300 second average
2.6922
%
Holochain process CPU usage
CPU usage by Holochain process
p5
2.9
%
<
mean
62.32
%
SD
25.09
%
<
p95
90.42
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
247.28
KiB
<
mean
340.33
KiB
SD
76.12
KiB
<
p95
459.04
KiB
Holochain process threads
Number of threads in Holochain process
p5
14
threads
<
mean
31.68
threads
SD
9.39
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
48
file descriptors
<
mean
78.14
file descriptors
SD
18.73
file descriptors
<
p95
90
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 15 Oct 2026 04:34:02 UTC
Duration
1800s
Peer count
150
Peer count at end
139
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_38671000696_1
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
71.15
/s
Max
Highest per-partition mean rate
max
320.37
/s
Min
Lowest per-partition mean rate
min
1
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.537704
s
SD
1.300945
s
Max
Highest per-partition mean latency
max mean
9.462042
s
Min
Lowest per-partition mean latency
min mean
0.019032
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
73
conductors
<
mean
83.04
conductors
SD
9.81
conductors
<
p95
94
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
793.824841
s
SD
547.103696
s
Max
Highest per-partition mean latency
max mean
1239.529412
s
Min
Lowest per-partition mean latency
min mean
463.95122
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
66.262131
s
SD
37.590748
s
Max
Highest per-partition mean latency
max mean
108.093631
s
Min
Lowest per-partition mean latency
min mean
27.36021
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
57.082515
s
SD
39.8074
s
Max
Highest per-partition mean latency
max mean
83.336141
s
Min
Lowest per-partition mean latency
min mean
15.775832
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk-qNfIe8t-bQeEUUo0icaeJ61E44EQp0Xh8EGZxri9e9HrrT_
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0CA6_EHupQS3AS6rxkDcNL29RVIPE_MbCHbC_KPe8A1aelYZ
p5
0
<
mean
0.43
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0OOGY-iFy3FShPt2_JRdORiZgpr-uL6df6yd1YN-r6WwgS5G
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0glMCn3u38uTWixw6wOuHlG9zVrf2oZzC3HeVUchFWna-im3
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk15Rdr60TUNFxvp0EkDo_YpmKb29vAdvc23LeTgTrT4wi86vP
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3onAq2VZsrG9LH_1uXeArqor4GMai_vwXJ7ZVrvrO0xQrTCc
p5
0
<
mean
0.32
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4X_AdzffhcmN9LF3uZYEUn0oGnCsOlDSQcSEQsgCrazP4m92
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk6N3dex6bhtY97OZsO6H-iWZpU704gF8BciXv9J5qSwg4VKcr
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk73Xcux_4O3eNyNoAjgL8l2607S6RtD2qJ7Me-yxA3xtMBolJ
p5
0
<
mean
0.43
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk79OCksmYAPKP-Ckahmu1IgSc4Gpu9NApPBDCIwQvjYgeky7q
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk8CxtfBgh30lUGKrcgn5Q1h4NtRwiKT6Dlo1jOYtGPQ4HX6Sj
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk9olHjZbE-sgMjOMYnt89J8L_1xNvh-QbwhqMBg9VNcYeqHZO
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkA_hIRx_cmrJtocjUhK0DnMNZ6EFcIGdqnxLzmVVrWCdN9AVB
p5
0
<
mean
0.28
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkC_bC79XGor5hB3iSMJZEjNlpFg7SphD8U7ZRLB1nM5HH3bHi
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkEt6QGgKJBCUbfeNiEUBI-7B9JJ9GZ5ajhbfjcsMxsCeBlDhH
p5
0
<
mean
0.31
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkGPd0v0ndXtLi25nKpnuOMI_dRD_55KkCQpk6j_gCoNiUslkf
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkH8jNOHqYJW_I9BQW3v_Wa-R6Czbunyy9pul28Nc_0cRcgbeo
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHO8bL7HpoBxbjViOGPu6HelvUHIi6csLQm5oT4LFdnxr8PCR
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHtMRrZEq77Kxjb8C9L-opg5HT4LrlKIUdlhNjdMI5G-A57cS
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIWffQNW9re6DEOzpwqHIYDo9XC1s1usvT4Srx3sWgZwsyIMr
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkKSlzkpK7GmM1guKyD_mX1UgMblCuPeOAP2NGVcKzXbfhZBXu
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkL6VXCJlspKn37Quktfn6kZx3x0ROd8SKNuW0SiQKZnmATg7N
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkLKqZyqoFH2wNWiq_9KTzyYtDJ8Ik9iC52Yr8dmaMZkudNqIl
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkLawBwt_koDCf6SxewLv_GV3qDa3JlbzcLUHOoG415gauTsr-
p5
0
<
mean
0.42
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMmbHgarkqlt7Hqu8zxAysvIP7zYFk6G-OXFrSGaTH6E3Dvob
p5
0
<
mean
0.58
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkNhzgZIdAUb08BeTyRR-DusONCl_85QpJQhwGTsTxTKu9zoUZ
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkO-NofNq2o2vbikhbnvPtkWT_QX6o7vvKj5QaQIaXHv6zw47-
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkOJxZIO06QY0fmYKgu0IB27wDIIQcqtkN5kFF6tMhILDPKIbz
p5
0
<
mean
0.4
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQ4iQqtXr5c7irVCrAQ_BfrXeovgU9tgEHADk5f2LK4x1ryHM
p5
0
<
mean
0.53
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQIx6SE1FHMiVDEwH5DKEiUzQkMvwN3goSkrCqHvjc44720B5
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQbsuYoAn7G-EZNsJRpalRIguZmu6QeLJVUsZqa_xs4k-K617
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRMFoNBB6-zO_HLS1p47GxTIT0IUM_JMJ35TkVm6nqehwqL59
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkS8AKums2_5lXolMsUPJJbXe0T22FjXTltOTL0gVzZD4r_WoH
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSO2t-6Fp6mMdQitzeV0m5UpxfJN77DiaWydUFw0qN6JYnq9_
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSQsX4HO6BJiH9e4ygArtD9cnFo-2n0V4CFy3BQmDB5x63vep
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkT2cP1LHiupq_VpGYQyYzBk0f9Yg76J7mNFvJQBHy65F-QWQ9
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkTDV0eplOXGl_GCd0yBJ_RdnSR1xyQstXN5WbRl74k792pHkQ
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTDm0WZqydhcvxK8ISSDsgLJEFDdWJZZ97DfCJLvUJenbUk_G
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTXKHwOP1biAzej53PWUM5p543T6RxNdijq-2SqMP0-rTzer1
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVf9wjvSx3nftqhMaSsky_HUZXdBPk8rLUGHchWdqbX212FBC
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVulAocvW58Zeu3J8mHpepcUw3merg-9yf0L0W07JCngb-MRA
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkWH0v4sw_s_-vkdn1THzWpt52uIGPPkBnMpxiahjghMJ01m9s
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkWyTUsooeS-lBHSaUVP-B-qgvFJ6FPmqItDLVLplQ-d5Y0-zB
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkX9ccAdKRO0o_Kl1KtUg06Q8mQ_x8kaLaNIsZLfvc7RU4N0HU
p5
0
<
mean
0.48
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXKo9EZCH4Yd-8b1i3eg6IUwmtBiwrfitAcVM6GQQWbd2TJDJ
p5
0
<
mean
0.52
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXUKk3kVxPcobnA4K59C9WI_8z8pfTgRbwI7lWCdP_5f04ia0
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXyFIqY-tbWBQLUNpe4wlA7WkzV7imM30emepcr_d8pE8UZ0u
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYR4bfIfZpHRCiSgzqUenVF0Ci4pq4tZh4RL_ZC8IJzLli6Xw
p5
0
<
mean
0.44
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYbeAREFxQioHiYffSZl707uYg0vtgb39wfp8UTofxXdldGac
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYrAwQReVwTkBRnBB_MEsLGe9QXyVqYb_RcZGFo8KsQhDgxxU
p5
0
<
mean
0.42
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkZap3meNZT7ii0dx2C54CT87Y0yhiPCP-YZEXJ_gE6PtPEr9S
p5
0
<
mean
0.47
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkZbO8IHxG1uve01jvmb00ao_xEVCek43oKbZ_dlRacSm4NBvR
p5
0
<
mean
0.55
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkZwDVh7L3JQywQITlShVn-4LMcTDikkySfNmDMR7Hy_DW0GlH
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_X13JiYeott23vNpw905ujeb55_U4VFK2-g272fIg7212Q4x
p5
0
<
mean
0.5
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_bucQMp2hoO3nF29mJZ22nko1vD5w4Iqa-5_LASSSg12aAxm
p5
0
<
mean
0.47
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_eidcE3h4AgZFYiyT8oJH8p5i1rqPLjBCj9RBfGDHgoC5lkF
p5
0
<
mean
0.39
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkaBj7-hMWkCebdBgkrP9_3rAMNhbXj3XGyqkelJzOoD05sGMm
p5
0
<
mean
0.48
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkaTssXMkWLFKK7eXN10X6BPjiQH55tNjH1FhzySKsJK9cdtvq
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkb3P6oJ8txTJkFxgB1Sh13W0-p1MseqAwXZBToZrh8kdnzJxT
p5
0
<
mean
0.4
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbEH_0yof260Dwe1QiX5XkDWaiwnSLW7gOO2OTsPHMfoKzLXF
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbdU83UeUiIpRacBnRigy5WmPWgLTq4sQC8GubH6maesTYVIa
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbq8OIquQeWAq-FqfXJ_BsJjAsoNp8NHXJjfLE9FYGdbQ-0bt
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkc5usCx7zBDEj6HPFymVp4dYfOyAEZHVF_R61datzGDLwh7bj
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcf02bZ0v0A4pok53cpObRDFHofo4KPoEe-ZPRKmb_Y5109aF
p5
0
<
mean
0.37
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcz2aIGm-rItY8S-hiHYbq-rg_7TsEo1xiC76ljtK7h3JGSNw
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdIUn4eAcYp7BIrDqCyvDUE62gy76opZmxNfRGXXYfrmWtebS
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdRZKg-oT_oOQAWX_exLBvakyFMs2lUCd8ZICjrEngWn2LhVh
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdi3nsHsDU4yaK2geLY3jZgW95WJ0V5Ut9yFCC-xSEaD8WFju
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdklLLM9DsZ2QyPw3ZEBo--fIq6OT6AqYLFXbTMPF9DGInHqV
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkffGL95xpMMchNZidpMlC-kC3Jc2pA5Beu0kVN3dASwr4oMPt
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgqRPKFdMEjY2Lykin3gOJku9yUL26AaFb1T5TJsQRqlOBMJV
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhEeKOQQy3bJABNkonSPXdg4XaCToh8_RIOOrCftPVxh4jTX8
p5
0
<
mean
0.45
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhKXv8pgpc9uiFRGRU9jTwEmNgcIfgpODlUiuw-cJV0KxiqY9
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhSx8_QS3UMGRu4MOC76g74OA1yGkVj-wMUGp3u4Vu6qQBCjM
p5
0
<
mean
0.42
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhbtc65AAx3zGB2f4EdCcp8_2g4r8lPc-jukuhGTaPskTOYSc
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhnEje5xFgIZu0nUqJPqZg7aPb5O78bTfJBYyH8uwk7jFTY9J
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiAOU0GUXLR6Ah2WJGABhxg2TQqN7DlxPiFTdbKJUxAfT0V6g
p5
0
<
mean
0.48
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiDA7DVZw1P0ln3Qtlku-Ms3yZPsn5gFEXC1CkRqkZWMjCx1x
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiZqA-RH4-Y7gjXF8Hr-wI0yw4HHHxdI-8SU_7EnO7b0M_rYQ
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiy5r0tr9MxXaA_w7nVajcItnRl2GQnupbNXXDbDNZavUG-z5
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkj3Xt4jU8IbEIpfZTNIWo4EqA_ne6grhTsh5J0Am2to7kyKxu
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkTBCnPqrKjt5_UKS4R1d5MgboZcOFBlt10Lgp8KwO_gXYggW
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkl793BZnNLh5qWGtc1HZg5WD18A5TIaQpTGmyK9zxY8Rmq0Tj
p5
0
<
mean
0.22
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkl_J40JiPVPpLJ1TOCmFb5mLP55_qzAD_dO9Y3bbRfRXfuTuE
p5
0
<
mean
0.39
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmAHRKwzTF05dIC88cn-oCf1CPhzCVc_XlgRWxTz1fZLZniFu
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpIQuYn28-zonoY37yuOzrWgASamvrFs7IwOcjRGA_uUp-Ue2
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpsda3GaRj9u_KByN_8-8mXWAM3gLv-_Rs89GqXasYbDrKGEW
p5
0
<
mean
0.03
SD
0.16
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkr4UtYhc4SaDaMpl6oWDJHcRXibDkcnDLuZ9-qjAkIlabWcCS
p5
0
<
mean
0.42
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrCrQNMUZZy8dKi1VEh8PXttMgyxDBOV3g9-Kwe55_YVBjAXW
p5
0
<
mean
0.43
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAksX6pzOEUJQvY53tJKoYWKF5OhOwN3W9JN_p8D3KEknEZHH4b
p5
0
<
mean
0.3
SD
0.46
<
p95
1
  • get_agent_activity_volatile_agent: uhCAksyeu1Thb1LhyOGwW1u5VeoYatBd8z3tlUq6_5LpCFKML7DB6
p5
0
<
mean
0.38
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAktHPbKSdXj_3HyWDDrQ9vg6VUFpzoYgjbQ8Rd3fExZOIgRMR7
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkth0WKSW26UlDmul5EZ-nn5u2NMilNJx3TIepgtvAwN6ss1BG
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAku1ile67NxYGyaSJOjHSCzfiZOHAzjBQywwp2VEC1OBvh5BEZ
p5
0
<
mean
0.43
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkvkskfeDLxMC0Cc9kpnJFlCz9x2GPayc1wMCeyyEjZFh3Mpnl
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkweI7dFOVoEDeho4ame33SDGczHJs68Nznv4UzPtAErdO_hx4
p5
0
<
mean
0.41
SD
0.49
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxcQbSXbim5WEUc7dC-MvHFE299ykdmFi71D-5Sgt5H9XVS6X
p5
0
<
mean
0.32
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkyqxCA9ODotWWsritDPCt1dXxEn2r0nRwC6kRa5nGOkAG_vrz
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkz50x4U1IfeP9dTS8MNMHSEtKsWZHT3Dx4gqaiBU--b_l_xvs
p5
0
<
mean
0.35
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzA69C8fXwwRmh4jpoXocfV8N5yyXrPZ81nHyGUYOflT3eYV-
p5
0
<
mean
0.36
SD
0.48
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzASVDy0gUh724MSlIMkPK-ClegZ9UR7dn9nM_xwJ1ugKVtum
p5
0
<
mean
0.48
SD
0.5
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzppuwiWVjR_UCeXoTLDSnmJTM8cMc79HXeyZbdJfANbObNhU
p5
0
<
mean
0.38
SD
0.48
<
p95
1
Error count
The number of errors accumulated.
1920
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.148343
s
SD
28.149189
s
p50
0.301176
s
<
p95
21.85086
s
<
p99
175.90453
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 1709.463867541s
mean rate
159.7
/s
std
1085.32
/s
p5
0
/s
p95
454.22
/s
peak
31179.33
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006353
s
SD
0.002052
s
p50
0.006342
s
<
p95
0.00936
s
<
p99
0.01129
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.005812
s
SD
0.002036
s
p50
0.006643
s
<
p95
0.009008
s
<
p99
0.010537
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.086957
s
SD
6.40319
s
p50
0.295062
s
<
p95
2.049231
s
<
p99
21.575288
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.008074
s
SD
1.070394
s
p50
0.00861
s
<
p95
2.162885
s
<
p99
7.91033
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005166
s
SD
0.001781
s
p50
0.005173
s
<
p95
0.007937
s
<
p99
0.009744
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004649
s
SD
0.001673
s
p50
0.005313
s
<
p95
0.00714
s
<
p99
0.008928
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.172233
s
SD
29.082559
s
p50
0.312982
s
<
p95
22.01478
s
<
p99
197.787248
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006695
s
SD
0.99017
s
p50
0.007134
s
<
p95
1.939746
s
<
p99
7.60942
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001465
s
SD
0.00056
s
p50
0.001646
s
<
p95
0.002216
s
<
p99
0.002348
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
4.6e-05
s
SD
3.2e-05
s
p50
3.9e-05
s
<
p95
9.1e-05
s
<
p99
0.000201
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000605
s
SD
0.000652
s
p50
0.000624
s
<
p95
0.001062
s
<
p99
0.004378
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000567
s
SD
0.000272
s
p50
0.000531
s
<
p95
0.000894
s
<
p99
0.001798
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.170158
s
SD
29.080948
s
p50
0.311513
s
<
p95
22.013784
s
<
p99
197.784597
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.0035
s
SD
0.989995
s
p50
0.001291
s
<
p95
1.937571
s
<
p99
7.604959
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
8e-06
s
p50
1.1e-05
s
<
p95
2.3e-05
s
<
p99
4.7e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002403
s
SD
0.000897
s
p50
0.002573
s
<
p95
0.00392
s
<
p99
0.005451
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000383
s
SD
0.00038
s
p50
0.000426
s
<
p95
0.000876
s
<
p99
0.001641
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
5.1
s
<
mean
323.88
s
SD
490.67
s
<
p95
1492.32
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
212.393743
s
SD
236.522441
s
p50
81.295453
s
<
p95
737.283471
s
<
p99
1060.513293
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.139854
s
SD
0.205134
s
p50
0.021215
s
<
p95
0.519355
s
<
p99
1.000446
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
1e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.055938
s
SD
0.187306
s
p50
0.019382
s
<
p95
0.409375
s
<
p99
0.965983
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.776822
s
SD
1.184842
s
p50
0.101218
s
<
p95
3.746892
s
<
p99
5.367277
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
34.301158
s
SD
49.202759
s
p50
46.296607
s
<
p95
136.456722
s
<
p99
221.416209
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
63.6382
s
SD
76.429702
s
p50
0.017589
s
<
p95
217.503963
s
<
p99
297.040354
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00508
s
SD
0.002674
s
p50
0.001106
s
<
p95
0.008305
s
<
p99
0.011601
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008074
s
SD
0.042444
s
p50
0.002809
s
<
p95
0.021928
s
<
p99
0.156607
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000556
s
SD
0.001131
s
p50
0.000611
s
<
p95
0.002901
s
<
p99
0.00624
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
5.581255
s
SD
28.014227
s
p50
0.935052
s
<
p95
60.00256
s
<
p99
60.004177
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.777046
s
SD
0.558289
s
p50
0.530434
s
<
p95
1.930332
s
<
p99
2.472057
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.258957
s
SD
21.8107
s
p50
0.435307
s
<
p95
60.001576
s
<
p99
60.001873
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.612269
s
SD
20.678778
s
p50
1.458315
s
<
p95
60.001647
s
<
p99
60.001871
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
20.75
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
5.30
KiB/s
<
mean
122.35
KiB/s
SD
234.64
KiB/s
<
p95
409.47
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
545
B/s
<
mean
31.94
KiB/s
SD
99.93
KiB/s
<
p95
136.31
KiB/s
Total bytes received
Total bytes received on primary network interface
count
29.73
GiB
mean
17.00
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
7.76
GiB
mean
4.44
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.5%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 20% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.52
%
<
mean
27.84
%
SD
30.35
%
<
p95
90.77
%
System
7.10
%
CPU percentiles
CPU usage percentiles
p50
11.74
%
p95
90.77
%
p99
97.55
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
86
hosts
mean time
346.70
s
Memory used percentage
Percentage of memory used
p5
6.55
%
<
mean
10.49
%
SD
8.12
%
<
p95
20.79
%
Memory available percentage
Percentage of available memory
p5
79.21
%
<
mean
89.51
%
SD
8.12
%
<
p95
93.45
%
Max host memory used
Maximum memory usage percentage across all hosts
max
89.84
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.66
%
Memory growth rate
Rate of memory growth over time
growth
0.26
MiB/s
Disk read throughput
Disk read throughput in MiB/s
1.32
MiB/s
Disk write throughput
Disk write throughput in MiB/s
47.17
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/118
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.10
5 min
1.33
15 min
1.67
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
20.41
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
9.4441
%
SD
17.7659
%
<
p95
50.89
%
60 second average
9.4726
%
300 second average
12.6388
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.9046
%
SD
5.3668
%
<
p95
10.31
%
60 second average
1.9175
%
300 second average
2.5898
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.0765
%
SD
3.4574
%
<
p95
5.47
%
60 second average
1.0767
%
300 second average
1.3127
%

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Thu, 15 Oct 2026 05:07:07 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_38671000696_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
210.33
Max
Highest chain head value observed for any single write agent.
907
Write agents observed
9
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
119.427928
s
SD
43.354976
s
Max
Highest per-partition mean latency
max mean
494.0738
s
Min
Lowest per-partition mean latency
min mean
8.585
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.14
/s
Max
Highest per-partition mean rate
max
5.89
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.488533
s
SD
0.047936
s
Max
Highest per-partition mean latency
max mean
11.527868
s
Min
Lowest per-partition mean latency
min mean
0.010245
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
32111
Agents affected
71 / 74
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
434
Max
Highest count in any single partition
780
Min
Lowest count in any single partition
0
Final error count
The total number of all types of error accumulated over the run by all agents.
130567
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.021244
s
SD
0.462162
s
p50
0.001813
s
<
p95
0.096289
s
<
p99
0.860493
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 775.68021319s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 888.015915267s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'
Metered usage count of the WASM ribosome for this zome function.
total
2390733050
over 888.015915267s
mean rate
3.30111953232e+09
/s
std
7.06942385378e+09
/s
p5
6444.08
/s
p95
1.43815686358e+10
/s
peak
1.59559120242e+11
/s
WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
20478155
over 888.693580486s
mean rate
1.8685406125e+08
/s
std
1.01239700542e+09
/s
p5
166416.71
/s
p95
5.1557351506e+08
/s
peak
3.21747948862e+10
/s
WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Metered usage count of the WASM ribosome for this zome function.
total
4442160
over 888.693580486s
mean rate
5.8194433093e+09
/s
std
2.02851944784e+10
/s
p5
0
/s
p95
3.11617194152e+10
/s
peak
2.62832260575e+11
/s
WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
21050
over 888.693580486s
mean rate
3.232187965e+07
/s
std
1.3621976745e+08
/s
p5
0
/s
p95
1.7412885707e+08
/s
peak
2.34613333221e+09
/s
WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
121183434
over 888.770824984s
mean rate
5.4415691785e+09
/s
std
2.4731580897e+10
/s
p5
3.96932885e+07
/s
p95
2.11038726042e+10
/s
peak
9.88475033738e+11
/s
WASM usage: 'validated_must_get_agent_activity_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
878424843
over 888.770824984s
mean rate
6.05813087724e+10
/s
std
1.20471592782e+12
/s
p5
4.42098416e+07
/s
p95
1.53392303943e+11
/s
peak
9.03823166473e+13
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007075
s
SD
0.007758
s
p50
0.005655
s
<
p95
0.012009
s
<
p99
0.055337
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.2689
s
SD
0.584703
s
p50
0.161733
s
<
p95
0.699769
s
<
p99
3.514306
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.011274
s
SD
0.007235
s
p50
0.008496
s
<
p95
0.023645
s
<
p99
0.039067
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.020814
s
SD
0.200726
s
p50
0.007372
s
<
p95
0.082225
s
<
p99
1.393005
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.030223
s
SD
0.828446
s
p50
0.008143
s
<
p95
0.522605
s
<
p99
1.46655
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005762
s
SD
0.005958
s
p50
0.004503
s
<
p95
0.010624
s
<
p99
0.042352
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.266798
s
SD
0.581832
s
p50
0.159729
s
<
p95
0.695252
s
<
p99
3.497317
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009306
s
SD
0.006386
s
p50
0.006736
s
<
p95
0.020147
s
<
p99
0.035179
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006554
s
SD
0.16635
s
p50
0.006603
s
<
p95
0.020012
s
<
p99
1.392401
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.027292
s
SD
0.828474
s
p50
0.006786
s
<
p95
0.522054
s
<
p99
1.464966
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001644
s
SD
0.002692
s
p50
0.001718
s
<
p95
0.003549
s
<
p99
0.014542
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.015506
s
SD
0.096167
s
p50
0.009744
s
<
p95
0.058834
s
<
p99
0.125601
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
7.8e-05
s
SD
0.000162
s
p50
3.8e-05
s
<
p95
0.000158
s
<
p99
0.000381
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.005125
s
SD
0.018458
s
p50
0.001275
s
<
p95
0.013256
s
<
p99
0.035973
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.006845
s
SD
0.020761
s
p50
0.000673
s
<
p95
0.015381
s
<
p99
0.084279
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001154
s
SD
0.603773
s
p50
0.000954
s
<
p95
0.453628
s
<
p99
1.390376
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.02272
s
SD
0.045217
s
p50
0.014974
s
<
p95
0.074134
s
<
p99
0.161017
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.120512
s
SD
0.113182
s
p50
0.073766
s
<
p95
0.258626
s
<
p99
0.551334
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
2.4e-05
s
SD
1.2e-05
s
p50
1.1e-05
s
<
p95
2.9e-05
s
<
p99
6.2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003316
s
SD
0.006267
s
p50
0.003064
s
<
p95
0.008054
s
<
p99
0.023008
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002045
s
SD
0.003943
s
p50
0.00041
s
<
p95
0.005509
s
<
p99
0.010837
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.68
s
<
mean
429.98
s
SD
249.57
s
<
p95
824.26
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3976
over 888.770624683s
mean rate
597321.7
/s
std
5.93174716e+06
/s
p5
2269.74
/s
p95
1.71306505e+06
/s
peak
3.2055293131e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
91.746117
s
SD
62.638552
s
p50
25.010144
s
<
p95
174.036372
s
<
p99
317.719652
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
1.884563
s
SD
1.055686
s
p50
1.95187
s
<
p95
3.74652
s
<
p99
5.709066
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
1.03545
s
SD
0.531983
s
p50
0.640469
s
<
p95
1.613271
s
<
p99
1.976464
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.681965
s
SD
0.605456
s
p50
0.122049
s
<
p95
1.179569
s
<
p99
3.288417
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
15.683162
s
SD
16.226184
s
p50
14.745386
s
<
p95
54.310156
s
<
p99
79.639784
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
10.970504
s
SD
27.76123
s
p50
0.071167
s
<
p95
81.018843
s
<
p99
140.508484
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004175
s
SD
0.004313
s
p50
0.004084
s
<
p95
0.011261
s
<
p99
0.02092
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004217
s
SD
0.004659
s
p50
0.00303
s
<
p95
0.008896
s
<
p99
0.029501
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.003391
s
SD
0.026015
s
p50
0.001346
s
<
p95
0.012315
s
<
p99
0.084143
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.136319
s
SD
28.844135
s
p50
0.707895
s
<
p95
60.003293
s
<
p99
60.013558
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.857265
s
SD
0.810496
s
p50
0.457335
s
<
p95
3.619069
s
≤
p99
3.619069
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
4.617884
s
SD
15.711633
s
p50
0.906413
s
<
p95
60.000623
s
≤
p99
60.000623
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.720946
s
SD
23.071397
s
p50
1.371738
s
<
p95
60.006449
s
<
p99
60.023975
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
50.73
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
53.06
KiB/s
<
mean
465.59
KiB/s
SD
407.53
KiB/s
<
p95
1.21
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
7.16
KiB/s
<
mean
105.66
KiB/s
SD
157.35
KiB/s
<
p95
409.78
KiB/s
Total bytes received
Total bytes received on primary network interface
count
56.92
GiB
mean
65.49
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
12.92
GiB
mean
14.86
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (78.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 78% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
20.22
%
<
mean
73.27
%
SD
22.88
%
<
p95
94.04
%
System
22.54
%
CPU percentiles
CPU usage percentiles
p50
82.05
%
p95
94.04
%
p99
98.15
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
138
hosts
mean time
522.65
s
Memory used percentage
Percentage of memory used
p5
7.87
%
<
mean
13.32
%
SD
7.96
%
<
p95
24.96
%
Memory available percentage
Percentage of available memory
p5
75.04
%
<
mean
86.68
%
SD
7.96
%
<
p95
92.13
%
Max host memory used
Maximum memory usage percentage across all hosts
max
89.16
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
78.24
%
Memory growth rate
Rate of memory growth over time
growth
0.57
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.76
MiB/s
Disk write throughput
Disk write throughput in MiB/s
147.10
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.80
5 min
2.82
15 min
1.99
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
77.55
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.9
%
<
mean
50.6654
%
SD
29.4621
%
<
p95
86.81
%
60 second average
47.1576
%
300 second average
34.2114
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0001
%
SD
0.0088
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0001
%
SD
0.0088
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
9.6681
%
SD
7.388
%
<
p95
21.35
%
60 second average
9.0817
%
300 second average
6.6938
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.4303
%
SD
3.7178
%
<
p95
10.87
%
60 second average
3.2814
%
300 second average
2.6091
%
Holochain process CPU usage
CPU usage by Holochain process
p5
11.65
%
<
mean
67.18
%
SD
23.69
%
<
p95
89.73
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
175.47
KiB
<
mean
407.82
KiB
SD
109.79
KiB
<
p95
558.71
KiB
Holochain process threads
Number of threads in Holochain process
p5
12
threads
<
mean
34.74
threads
SD
9.81
threads
<
p95
44
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
42
file descriptors
<
mean
80.27
file descriptors
SD
18.86
file descriptors
<
p95
92
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Thu, 15 Oct 2026 06:33:35 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_38671000696_1
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
45.85
/s
Max
Highest per-partition mean rate
max
84.74
/s
Min
Lowest per-partition mean rate
min
7.9
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
4.76634976782e+29
s
SD
2.51907625968e+30
s
Max
Highest per-partition mean latency
max mean
1.36071078744e+31
s
Min
Lowest per-partition mean latency
min mean
10.766726
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
45.72
/s
Max
Highest per-partition mean rate
max
960
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0007
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
249689
Agents affected
60 / 60
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4161
Max
Highest count in any single partition
7970
Min
Lowest count in any single partition
725
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
10118
Agents affected
49 / 59
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
171
Max
Highest count in any single partition
3251
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
23
<
mean
64.79
SD
48.23
<
p95
147
zero-arc
p5
1
<
mean
22.37
SD
13.77
<
p95
51
Error count
The number of errors accumulated across all nodes.
618
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.05374
s
SD
1.146124
s
p50
0.35198
s
<
p95
2.346306
s
<
p99
4.418248
s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
158364270
over 878.891903884s
mean rate
6.70189573457e+09
/s
std
2.57952416636e+11
/s
p5
3.09554624e+06
/s
p95
5.66357978629e+09
/s
peak
1.24973806078e+13
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
107664222
over 878.917475755s
mean rate
2.25760507734e+10
/s
std
9.56576227557e+10
/s
p5
1.322773488e+07
/s
p95
8.71202894139e+10
/s
peak
1.22918058383e+12
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
28683930
over 889.11617741s
mean rate
9.3250056157e+08
/s
std
2.35354301762e+10
/s
p5
0
/s
p95
2.58662136547e+09
/s
peak
1.96824602026e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.0093
s
SD
0.009939
s
p50
0.009775
s
<
p95
0.016911
s
<
p99
0.061811
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
1.65593
s
SD
20.930479
s
p50
1.901214
s
<
p95
19.986298
s
<
p99
131.117444
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008162
s
SD
0.008907
s
p50
0.008626
s
<
p95
0.014827
s
<
p99
0.055287
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
1.654677
s
SD
20.93027
s
p50
1.899858
s
<
p95
19.985587
s
<
p99
131.116439
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001465
s
SD
0.001389
s
p50
0.001585
s
<
p95
0.00259
s
<
p99
0.006247
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000847
s
SD
0.001809
s
p50
0.000821
s
<
p95
0.00221
s
<
p99
0.009129
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000778
s
SD
0.001685
s
p50
0.000741
s
<
p95
0.001829
s
<
p99
0.007134
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.051723
s
SD
1.249159
s
p50
0.34348
s
<
p95
2.08078
s
<
p99
5.281401
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.273756
s
SD
1.1961
s
p50
0.273015
s
<
p95
2.834609
s
<
p99
5.138441
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002383
s
SD
0.002318
s
p50
0.002578
s
<
p95
0.004192
s
<
p99
0.015347
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.00059
s
SD
0.001411
s
p50
0.000357
s
<
p95
0.001731
s
<
p99
0.005898
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.9
s
<
mean
432.14
s
SD
249.89
s
<
p95
824.77
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.116268971s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
18.300181
s
SD
33.947518
s
p50
0
s
<
p95
0.912669
s
<
p99
36.155841
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.263922
s
SD
0.24115
s
p50
0.001945
s
<
p95
0.531369
s
<
p99
1.178173
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.066153
s
SD
0.479565
s
p50
0.002028
s
<
p95
1.267952
s
<
p99
1.444357
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.167114
s
SD
0.112108
s
p50
0.030735
s
<
p95
0.247064
s
<
p99
0.46406
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
9.070019
s
SD
9.794341
s
p50
0.003132
s
<
p95
16.226931
s
<
p99
49.510886
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.978531
s
SD
3.417207
s
p50
0.00158
s
<
p95
0.113717
s
<
p99
6.788764
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002671
s
SD
0.00276
s
p50
0.001477
s
<
p95
0.007426
s
<
p99
0.011593
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006064
s
SD
0.011265
s
p50
0.002864
s
<
p95
0.012995
s
<
p99
0.033481
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000939
s
SD
0.006262
s
p50
0.000638
s
<
p95
0.004431
s
<
p99
0.025888
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.505555
s
SD
28.688759
s
p50
0.534675
s
<
p95
60.002062
s
<
p99
60.056714
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.182078
s
SD
20.573912
s
p50
0.15512
s
<
p95
60.001464
s
<
p99
60.006631
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.393742
s
SD
1.814194
s
p50
0.187998
s
<
p95
4.597137
s
<
p99
8.147358
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
35.61
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
5.18
KiB/s
<
mean
468.10
KiB/s
SD
492.67
KiB/s
<
p95
1.43
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.26
KiB/s
<
mean
97.34
KiB/s
SD
148.64
KiB/s
<
p95
373.44
KiB/s
Total bytes received
Total bytes received on primary network interface
count
57.80
GiB
mean
66.50
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
12.02
GiB
mean
13.83
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.7%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (76.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 52% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.22
%
<
mean
49.11
%
SD
37.28
%
<
p95
90.2
%
System
13.50
%
CPU percentiles
CPU usage percentiles
p50
60.74
%
p95
90.20
%
p99
94.67
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
94
hosts
mean time
573.24
s
Memory used percentage
Percentage of memory used
p5
7.58
%
<
mean
11.91
%
SD
7.58
%
<
p95
22.03
%
Memory available percentage
Percentage of available memory
p5
77.97
%
<
mean
88.09
%
SD
7.58
%
<
p95
92.42
%
Max host memory used
Maximum memory usage percentage across all hosts
max
87.89
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
76.26
%
Memory growth rate
Rate of memory growth over time
growth
0.41
MiB/s
Disk read throughput
Disk read throughput in MiB/s
1.06
MiB/s
Disk write throughput
Disk write throughput in MiB/s
89.44
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.92
5 min
1.95
15 min
2.04
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
51.70
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
22.6283
%
SD
22.9089
%
<
p95
59.1
%
60 second average
21.9360
%
300 second average
21.7677
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.3914
%
SD
5.1401
%
<
p95
13.36
%
60 second average
3.2852
%
300 second average
3.0930
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.5603
%
SD
2.9768
%
<
p95
5.92
%
60 second average
1.5106
%
300 second average
1.5166
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
42.84
%
SD
36.11
%
<
p95
86.5
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
206.33
KiB
<
mean
271.85
KiB
SD
65.25
KiB
<
p95
388.49
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
29.7
threads
SD
8.61
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
72.09
file descriptors
SD
20.44
file descriptors
<
p95
89
file descriptors

Zero-Arc Create Data

↑ Back to index
A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 15 Oct 2026 05:42:07 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data--high_zero_38671000696_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
50.96
/s
Max
Highest per-partition mean rate
max
87.94
/s
Min
Lowest per-partition mean rate
min
9.38
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
98.280638
s
SD
33.615873
s
Max
Highest per-partition mean latency
max mean
439.467566
s
Min
Lowest per-partition mean latency
min mean
6.24736
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
8.72
/s
Max
Highest per-partition mean rate
max
52.33
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0002
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
646999
Agents affected
140 / 140
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4621
Max
Highest count in any single partition
8158
Min
Lowest count in any single partition
913
Received count
Total number of entries received by full-arc nodes, by agent.
Total
1045
Agents affected
2 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
116
Max
Highest count in any single partition
759
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
4
<
mean
52.7
SD
37.65
<
p95
129
zero-arc
p5
2
<
mean
14.62
SD
11.95
<
p95
40
Error count
The number of errors accumulated across all nodes.
8
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.00869
s
SD
0.033702
s
p50
0.001539
s
<
p95
0.062258
s
<
p99
0.204516
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 789.425354371s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
334973870
over 875.446758723s
mean rate
1.52042359506e+10
/s
std
1.32201565564e+11
/s
p5
1.878684806e+07
/s
p95
2.65042044249e+10
/s
peak
5.44293644373e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
165581360
over 887.977308581s
mean rate
3.3645376808e+08
/s
std
4.2305024093e+08
/s
p5
1.073682007e+07
/s
p95
1.47490282236e+09
/s
peak
1.89870901375e+09
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
52817290
over 889.910426376s
mean rate
2.63378040153e+09
/s
std
3.09910171135e+10
/s
p5
2.62792355e+06
/s
p95
4.27689877897e+09
/s
peak
1.94134806325e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007421
s
SD
0.005576
s
p50
0.007691
s
<
p95
0.014193
s
<
p99
0.031419
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.026432
s
SD
0.479175
s
p50
0.021803
s
<
p95
1.284255
s
<
p99
2.675805
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006468
s
SD
0.004907
s
p50
0.00669
s
<
p95
0.01237
s
<
p99
0.02726
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.025451
s
SD
0.477304
s
p50
0.019833
s
<
p95
1.276892
s
<
p99
2.664273
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001287
s
SD
0.002227
s
p50
0.001383
s
<
p95
0.002671
s
<
p99
0.00791
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00044
s
SD
0.000553
s
p50
0.000444
s
<
p95
0.00082
s
<
p99
0.002204
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000433
s
SD
0.000542
s
p50
0.000415
s
<
p95
0.000842
s
<
p99
0.002146
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.012906
s
SD
0.041966
s
p50
0.002431
s
<
p95
0.100066
s
<
p99
0.226341
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001305
s
SD
0.015283
s
p50
0.001298
s
<
p95
0.038304
s
<
p99
0.067796
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002074
s
SD
0.004143
s
p50
0.002063
s
<
p95
0.007412
s
<
p99
0.01756
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000409
s
SD
0.000993
s
p50
0.000475
s
<
p95
0.001166
s
<
p99
0.006638
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
45
s
<
mean
431.5
s
SD
249.57
s
<
p95
824.51
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.910433153s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
1.921408
s
SD
20.104692
s
p50
0
s
≤
p95
0
s
<
p99
38.966139
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.218305
s
SD
0.275021
s
p50
0.001667
s
<
p95
0.156363
s
<
p99
2.053634
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.010042
s
SD
0.304245
s
p50
0.001655
s
<
p95
1.012587
s
<
p99
1.406171
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.13572
s
SD
0.122694
s
p50
0.119716
s
<
p95
0.239508
s
<
p99
0.486715
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.485334
s
SD
10.604485
s
p50
0.002618
s
<
p95
6.011863
s
<
p99
42.84924
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.086007
s
SD
4.913249
s
p50
0.00095
s
<
p95
0.009415
s
<
p99
0.198389
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.007452
s
SD
0.004377
s
p50
0.008049
s
<
p95
0.015348
s
<
p99
0.018238
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006764
s
SD
0.007142
s
p50
0.003651
s
<
p95
0.01485
s
<
p99
0.036417
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000388
s
SD
0.037434
s
p50
0.000348
s
<
p95
0.001824
s
<
p99
0.040471
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
8.612324
s
SD
27.818639
s
p50
5.811336
s
<
p95
60.206517
s
≤
p99
60.206517
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.907345
s
SD
19.010654
s
p50
0.534261
s
<
p95
61.219524
s
<
p99
61.313239
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
53.82
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.21
KiB/s
<
mean
172.60
KiB/s
SD
363.38
KiB/s
<
p95
523.72
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1014
B/s
<
mean
35.10
KiB/s
SD
61.60
KiB/s
<
p95
161.99
KiB/s
Total bytes received
Total bytes received on primary network interface
count
21.42
GiB
mean
24.64
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.36
GiB
mean
5.01
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (77.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 67% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
18.99
%
<
mean
69.95
%
SD
21.65
%
<
p95
89.04
%
System
16.12
%
CPU percentiles
CPU usage percentiles
p50
78.42
%
p95
89.04
%
p99
93.19
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
122
hosts
mean time
494.15
s
Memory used percentage
Percentage of memory used
p5
7.21
%
<
mean
11.61
%
SD
7.59
%
<
p95
22.2
%
Memory available percentage
Percentage of available memory
p5
77.8
%
<
mean
88.39
%
SD
7.59
%
<
p95
92.79
%
Max host memory used
Maximum memory usage percentage across all hosts
max
86.32
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
77.08
%
Memory growth rate
Rate of memory growth over time
growth
0.38
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.18
MiB/s
Disk write throughput
Disk write throughput in MiB/s
105.69
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.30
5 min
2.28
15 min
2.43
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
66.89
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.2
%
<
mean
24.2575
%
SD
15.7008
%
<
p95
51.37
%
60 second average
23.5213
%
300 second average
23.6512
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.6384
%
SD
4.2847
%
<
p95
10.97
%
60 second average
3.5449
%
300 second average
3.5741
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.3179
%
SD
3.4381
%
<
p95
5.36
%
60 second average
2.2472
%
300 second average
2.2029
%
Holochain process CPU usage
CPU usage by Holochain process
p5
11.22
%
<
mean
64.17
%
SD
23.03
%
<
p95
85.15
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
208.61
KiB
<
mean
276.82
KiB
SD
60.74
KiB
<
p95
376.17
KiB
Holochain process threads
Number of threads in Holochain process
p5
21
threads
<
mean
31.41
threads
SD
7.71
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
76.78
file descriptors
SD
17.75
file descriptors
<
p95
89
file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 15 Oct 2026 06:16:29 UTC
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data_validated--high_zero_38671000696_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
45.35
/s
Max
Highest per-partition mean rate
max
83.07
/s
Min
Lowest per-partition mean rate
min
27.4
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
129.953411
s
SD
4.117701
s
Max
Highest per-partition mean latency
max mean
366.353826
s
Min
Lowest per-partition mean latency
min mean
38.701839
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0002
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
553388
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4010
Max
Highest count in any single partition
7297
Min
Lowest count in any single partition
2306
Received count
Total number of entries received by full-arc nodes, by agent.
Total
885
Agents affected
3 / 8
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
111
Max
Highest count in any single partition
472
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
56.92
SD
49.25
<
p95
145
zero-arc
p5
4
<
mean
15.68
SD
12.12
<
p95
42
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.007908
s
SD
0.004959
s
p50
0.009094
s
<
p95
0.018375
s
<
p99
0.021788
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
6667
over 807.497851935s
mean rate
10741.2
/s
std
23384.38
/s
p5
15
/s
p95
72038.89
/s
peak
104563.03
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
270293760
over 872.485654406s
mean rate
5.96295288251e+09
/s
std
6.92670654237e+10
/s
p5
2.986301799e+07
/s
p95
1.50636174187e+10
/s
peak
3.46935605704e+12
/s
WASM usage: 'timed_and_validated::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
38145150
over 880.659817202s
mean rate
2.58647134987e+09
/s
std
3.41657352803e+09
/s
p5
6.19382518e+06
/s
p95
9.51459389327e+09
/s
peak
1.02705426698e+10
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
160042200
over 889.547330597s
mean rate
3.69425201255e+09
/s
std
3.96323281184e+10
/s
p5
1.184327718e+07
/s
p95
1.05543397212e+10
/s
peak
2.039344581e+12
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
1252172451
over 889.547330597s
mean rate
2.87739511697e+10
/s
std
3.1095547713e+11
/s
p5
1.2007456483e+08
/s
p95
8.00656698609e+10
/s
peak
1.59566184216e+13
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007726
s
SD
0.003344
s
p50
0.008165
s
<
p95
0.015209
s
<
p99
0.01611
s
Zome call duration: 'timed_and_validated::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.030078
s
SD
0.131242
s
p50
0.020229
s
<
p95
0.317107
s
<
p99
0.739303
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006741
s
SD
0.002908
s
p50
0.007101
s
<
p95
0.013279
s
<
p99
0.014058
s
WASM call duration: 'timed_and_validated::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.028838
s
SD
0.130514
s
p50
0.017135
s
<
p95
0.311398
s
<
p99
0.733793
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001303
s
SD
0.000762
s
p50
0.001387
s
<
p95
0.00251
s
<
p99
0.004798
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006048
s
SD
0.004338
s
p50
0.005447
s
<
p95
0.01106
s
<
p99
0.026481
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000433
s
SD
0.000188
s
p50
0.000452
s
<
p95
0.000841
s
<
p99
0.000884
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000423
s
SD
0.000194
s
p50
0.000418
s
<
p95
0.000877
s
<
p99
0.00093
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.005778
s
SD
0.006478
s
p50
0.004707
s
<
p95
0.023069
s
<
p99
0.027953
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000874
s
SD
0.001547
s
p50
0.001388
s
<
p95
0.005173
s
<
p99
0.00803
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.008668
s
SD
0.004737
s
p50
0.009424
s
<
p95
0.018439
s
<
p99
0.021666
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00227
s
SD
0.002458
s
p50
0.002168
s
<
p95
0.004417
s
<
p99
0.016472
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000512
s
SD
0.002006
s
p50
0.000624
s
<
p95
0.001268
s
<
p99
0.011076
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
44.65
s
<
mean
432.21
s
SD
249.83
s
<
p95
822.79
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.547308173s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
2.100259
s
SD
22.742408
s
p50
0
s
≤
p95
0
s
<
p99
55.56549
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
4.951807
s
SD
1.826047
s
p50
0.001699
s
<
p95
2.623914
s
<
p99
9.544112
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.009606
s
SD
0.252425
s
p50
0.002066
s
<
p95
0.795773
s
<
p99
1.250704
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.124536
s
SD
0.058097
s
p50
0.108101
s
<
p95
0.216973
s
<
p99
0.252785
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.103081
s
SD
4.301548
s
p50
0.003059
s
<
p95
1.369387
s
<
p99
21.131185
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.473146
s
SD
5.058711
s
p50
0.001362
s
<
p95
0.010709
s
<
p99
4.2896
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.006068
s
SD
0.003196
s
p50
0.006148
s
<
p95
0.011366
s
<
p99
0.013362
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006259
s
SD
0.004543
s
p50
0.003621
s
<
p95
0.013463
s
<
p99
0.024585
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000393
s
SD
0.004832
s
p50
0.000351
s
<
p95
0.000894
s
<
p99
0.022925
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
11.133395
s
SD
29.448631
s
p50
1.912628
s
<
p95
60.072997
s
<
p99
60.074918
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.190252
s
SD
15.81822
s
p50
0.120962
s
<
p95
60.075152
s
<
p99
60.108773
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
51.44
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
16.79
KiB/s
<
mean
214.36
KiB/s
SD
410.97
KiB/s
<
p95
622.37
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.98
KiB/s
<
mean
44.36
KiB/s
SD
53.95
KiB/s
<
p95
143.48
KiB/s
Total bytes received
Total bytes received on primary network interface
count
25.99
GiB
mean
29.90
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.38
GiB
mean
6.19
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.5%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (76.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 47% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
23.91
%
<
mean
68.81
%
SD
19.99
%
<
p95
90.27
%
System
17.37
%
CPU percentiles
CPU usage percentiles
p50
74.87
%
p95
90.27
%
p99
95.49
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
123
hosts
mean time
343.35
s
Memory used percentage
Percentage of memory used
p5
9.04
%
<
mean
13.5
%
SD
7.9
%
<
p95
25.23
%
Memory available percentage
Percentage of available memory
p5
74.77
%
<
mean
86.5
%
SD
7.9
%
<
p95
90.96
%
Max host memory used
Maximum memory usage percentage across all hosts
max
89.82
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
76.77
%
Memory growth rate
Rate of memory growth over time
growth
0.45
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.45
MiB/s
Disk write throughput
Disk write throughput in MiB/s
113.67
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /efi-boot
0/5
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/5
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.29
5 min
2.14
15 min
2.13
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
47.30
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.91
%
<
mean
25.3284
%
SD
15.5369
%
<
p95
55.76
%
60 second average
23.9014
%
300 second average
22.0150
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.0603
%
SD
4.058
%
<
p95
10.77
%
60 second average
2.9193
%
300 second average
2.7011
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.843
%
SD
3.2703
%
<
p95
4.7
%
60 second average
1.7748
%
300 second average
1.6949
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.5
%
<
mean
62.65
%
SD
22.53
%
<
p95
86.24
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
177.00
KiB
<
mean
383.16
KiB
SD
101.29
KiB
<
p95
518.33
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
31.98
threads
SD
8.66
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
8
file descriptors
<
mean
75.38
file descriptors
SD
20.26
file descriptors
<
p95
89
file descriptors