DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Thu, 17 Sep 2026 03:13:08 UTC
Duration
1800s
Peer count
150
Peer count at end
148
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_35174760384_1
Create rate
The average number of created records per agent.
Mean
mean
17.32
/s
Max
Highest per-partition mean rate
max
52.3
/s
Min
Lowest per-partition mean rate
min
5.36
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
348.98324
s
SD
207.724855
s
Max
Highest per-partition mean latency
max mean
1636.479277
s
Min
Lowest per-partition mean latency
min mean
27.94058
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
72.13
/s
Max
Highest per-partition mean rate
max
506.83
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0032
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
400581
Agents affected
126 / 126
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3179
Max
Highest count in any single partition
9771
Min
Lowest count in any single partition
1018
Received count
The total number of entries received by record_lag agents, by agent.
Total
29773
Agents affected
22 / 23
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1294
Max
Highest count in any single partition
4708
Min
Lowest count in any single partition
0
Error count
The number of errors encountered during the scenario.
138
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.0011
s
SD
0.002155
s
p50
0.00164
s
<
p95
0.006906
s
<
p99
0.008071
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 1731.953786522s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
207954124
over 1789.009518636s
mean rate
5.39570453337e+09
/s
std
2.52721274593e+10
/s
p5
3.360690332e+07
/s
p95
2.0881226998e+10
/s
peak
1.32548812981e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
6822331614
over 1786.327552444s
mean rate
2.18750287556e+11
/s
std
5.35246894565e+11
/s
p5
2.03271297014e+09
/s
p95
1.11560794002e+12
/s
peak
3.66870070177e+12
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
26515776
over 1789.009518636s
mean rate
1.41683399222e+09
/s
std
1.48823445053e+10
/s
p5
4.21504986e+06
/s
p95
3.40206858838e+09
/s
peak
1.01689416419e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.023038
s
SD
0.029435
s
p50
0.022588
s
<
p95
0.091673
s
<
p99
0.129291
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.1036
s
SD
0.362775
s
p50
0.093765
s
<
p95
0.847603
s
<
p99
2.182261
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.021474
s
SD
0.028503
s
p50
0.02044
s
<
p95
0.087784
s
<
p99
0.125084
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.102525
s
SD
0.362286
s
p50
0.092485
s
<
p95
0.845183
s
<
p99
2.178405
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001833
s
SD
0.001
s
p50
0.002133
s
<
p95
0.00417
s
<
p99
0.004942
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.005454
s
SD
0.009985
s
p50
0.004137
s
<
p95
0.028098
s
<
p99
0.043684
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.005334
s
SD
0.009918
s
p50
0.003979
s
<
p95
0.027965
s
<
p99
0.04267
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001093
s
SD
0.002154
s
p50
0.001901
s
<
p95
0.006912
s
<
p99
0.008186
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.004076
s
SD
0.013912
s
p50
0.003862
s
<
p95
0.045104
s
<
p99
0.072983
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003987
s
SD
0.003822
s
p50
0.004279
s
<
p95
0.013177
s
<
p99
0.016898
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001796
s
SD
0.002779
s
p50
0.00126
s
<
p95
0.007683
s
<
p99
0.013063
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
85.54
s
<
mean
879.58
s
SD
509.03
s
<
p95
1674.09
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
8973
over 1789.009447303s
mean rate
2.92583596e+06
/s
std
1.6284384541e+08
/s
p5
93.42
/s
p95
1.54742957e+06
/s
peak
1.55015923567e+10
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
162.024283
s
SD
130.745334
s
p50
21.418494
s
<
p95
381.169197
s
<
p99
553.198049
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.297597
s
SD
1.156675
s
p50
0.270717
s
<
p95
2.269484
s
<
p99
7.339708
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.146874
s
SD
0.44784
s
p50
0.082768
s
<
p95
1.325705
s
<
p99
1.80381
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
1.94362
s
SD
2.483174
s
p50
1.461769
s
<
p95
7.523648
s
<
p99
10.017402
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
26.972335
s
SD
52.992757
s
p50
32.626857
s
<
p95
160.559514
s
<
p99
244.818537
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
15.839888
s
SD
49.189862
s
p50
0.073271
s
<
p95
107.548228
s
<
p99
245.366291
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003754
s
SD
0.003453
s
p50
0.004609
s
<
p95
0.011586
s
<
p99
0.013131
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.010017
s
SD
0.008211
s
p50
0.004341
s
<
p95
0.024353
s
<
p99
0.030681
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.005083
s
SD
0.011846
s
p50
0.003095
s
<
p95
0.029322
s
<
p99
0.051853
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.935752
s
SD
27.753387
s
p50
2.918246
s
<
p95
60.082245
s
<
p99
61.594914
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.470177
s
SD
25.656749
s
p50
6.336644
s
<
p95
60.038136
s
<
p99
60.08437
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
53.52
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
54.58
KiB/s
<
mean
1.19
MiB/s
SD
777.56
KiB/s
<
p95
2.45
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
4.49
KiB/s
<
mean
121.17
KiB/s
SD
120.67
KiB/s
<
p95
347.26
KiB/s
Total bytes received
Total bytes received on primary network interface
count
301.66
GiB
mean
172.57
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
29.95
GiB
mean
17.13
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 80% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
35.91
%
<
mean
73.79
%
SD
18.13
%
<
p95
94.64
%
System
20.27
%
CPU percentiles
CPU usage percentiles
p50
76.80
%
p95
94.64
%
p99
98.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
132
hosts
mean time
911.67
s
Memory used percentage
Percentage of memory used
p5
8.01
%
<
mean
12.08
%
SD
4.86
%
<
p95
19.38
%
Memory available percentage
Percentage of available memory
p5
80.62
%
<
mean
87.92
%
SD
4.86
%
<
p95
91.99
%
Max host memory used
Maximum memory usage percentage across all hosts
max
38.73
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.73
%
Memory growth rate
Rate of memory growth over time
growth
0.31
MiB/s
Disk read throughput
Disk read throughput in MiB/s
2.61
MiB/s
Disk write throughput
Disk write throughput in MiB/s
123.63
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.63
5 min
3.18
15 min
2.27
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
79.59
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
2.32
%
<
mean
44.7543
%
SD
23.2118
%
<
p95
74.83
%
60 second average
43.5959
%
300 second average
38.3035
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
9.5836
%
SD
8.4354
%
<
p95
22.48
%
60 second average
9.3789
%
300 second average
8.1881
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.9314
%
SD
4.0989
%
<
p95
8.95
%
60 second average
2.8718
%
300 second average
2.5373
%
Holochain process CPU usage
CPU usage by Holochain process
p5
36.56
%
<
mean
70.05
%
SD
16.08
%
<
p95
90.68
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
216.56
KiB
<
mean
392.81
KiB
SD
130.63
KiB
<
p95
597.51
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
33.89
threads
SD
5.49
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
73
file descriptors
<
mean
82.98
file descriptors
SD
13.99
file descriptors
<
p95
89
file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 17 Sep 2026 06:51:31 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_35174760384_1
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
18.11
/s
Max
Highest per-partition mean rate
max
48.16
/s
Min
Lowest per-partition mean rate
min
7.94
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
37.889516
s
SD
2.147619
s
Max
Highest per-partition mean latency
max mean
1256.110613
s
Min
Lowest per-partition mean latency
min mean
0.163387
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
14.28
/s
Max
Highest per-partition mean rate
max
340
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0014
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
13244
Agents affected
8 / 8
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1656
Max
Highest count in any single partition
4408
Min
Lowest count in any single partition
681
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
2660
Agents affected
30 / 140
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
19
Max
Highest count in any single partition
509
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
62
<
mean
109.17
SD
33.76
<
p95
145
zero-arc
p5
0
<
mean
12.65
SD
15.76
<
p95
49
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
1891
Agents affected
138 / 140
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
14
Max
Highest count in any single partition
14
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
1891
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.405656
s
SD
3.594729
s
p50
0.813781
s
<
p95
9.627714
s
<
p99
17.153677
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4
over 769.438231448s
mean rate
3792.77
/s
std
36584.55
/s
p5
0
/s
p95
3933.19
/s
peak
650099.88
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.013715
s
SD
0.008225
s
p50
0.014738
s
<
p95
0.035237
s
<
p99
0.038134
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.032072
s
SD
99.462988
s
p50
0.019341
s
<
p95
255.817207
s
<
p99
548.662593
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012374
s
SD
0.007732
s
p50
0.013187
s
<
p95
0.033083
s
<
p99
0.035877
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.119245
s
SD
106.647391
s
p50
0.757757
s
<
p95
281.789566
s
<
p99
493.447259
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001171
s
SD
0.000637
s
p50
0.001771
s
<
p95
0.002794
s
<
p99
0.003314
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012676
s
SD
0.003359
s
p50
0.001832
s
<
p95
0.008236
s
<
p99
0.017233
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002381
s
SD
0.002888
s
p50
0.002253
s
<
p95
0.010736
s
<
p99
0.011421
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.001934
s
SD
0.00219
s
p50
0.001868
s
<
p95
0.008139
s
<
p99
0.009342
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
1.608229
s
SD
4.889363
s
p50
2.531841
s
<
p95
14.474518
s
<
p99
22.888531
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.014243
s
SD
2.632773
s
p50
0.241554
s
<
p95
5.722382
s
<
p99
13.359796
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.01375
s
SD
0.006118
s
p50
0.009796
s
<
p95
0.024866
s
<
p99
0.026675
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002036
s
SD
0.00114
s
p50
0.003095
s
<
p95
0.004882
s
<
p99
0.005698
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000943
s
SD
0.000615
s
p50
0.000232
s
<
p95
0.001848
s
<
p99
0.002568
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.36
s
<
mean
430.91
s
SD
249.45
s
<
p95
818.17
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.985070567s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
80.298801
s
SD
17.708346
s
p50
0
s
≤
p95
0
s
<
p99
85.894004
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.140983
s
SD
0.698667
s
p50
0.001871
s
<
p95
0.244255
s
<
p99
3.30843
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.192507
s
SD
0.130569
s
p50
0.002236
s
<
p95
0.011043
s
<
p99
0.790971
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.256357
s
SD
0.070772
s
p50
0.018359
s
<
p95
0.086147
s
<
p99
0.445406
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.069295
s
SD
3.438296
s
p50
0.00321
s
<
p95
0.651778
s
<
p99
14.76372
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.177161
s
SD
0.450551
s
p50
0.001469
s
<
p95
0.012152
s
<
p99
0.544297
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001733
s
SD
0.001375
s
p50
0.000161
s
<
p95
0.001905
s
<
p99
0.00816
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008034
s
SD
0.010194
s
p50
0.004235
s
<
p95
0.027921
s
<
p99
0.052487
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.001937
s
SD
0.000837
s
p50
0.000809
s
<
p95
0.00208
s
<
p99
0.004254
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
6.841115
s
SD
29.718742
s
p50
2.013808
s
<
p95
60.001695
s
<
p99
60.007495
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.974375
s
SD
24.328231
s
p50
0.818445
s
<
p95
60.001464
s
<
p99
60.001992
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.741538
s
SD
16.03894
s
p50
0.05988
s
<
p95
60.00066
s
<
p99
60.001103
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.67
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.75
KiB/s
<
mean
210.79
KiB/s
SD
477.33
KiB/s
<
p95
931.93
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
229
B/s
<
mean
18.06
KiB/s
SD
79.68
KiB/s
<
p95
58.81
KiB/s
Total bytes received
Total bytes received on primary network interface
count
25.78
GiB
mean
29.67
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.21
GiB
mean
2.54
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.5%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 5% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.48
%
<
mean
12.06
%
SD
23.1
%
<
p95
86.32
%
System
4.38
%
CPU percentiles
CPU usage percentiles
p50
3.75
%
p95
86.32
%
p99
95.54
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
93
hosts
mean time
86.50
s
Memory used percentage
Percentage of memory used
p5
7.19
%
<
mean
12.39
%
SD
4.54
%
<
p95
19.69
%
Memory available percentage
Percentage of available memory
p5
80.31
%
<
mean
87.61
%
SD
4.54
%
<
p95
92.81
%
Max host memory used
Maximum memory usage percentage across all hosts
max
38.21
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.91
%
Memory growth rate
Rate of memory growth over time
growth
0.53
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.07
MiB/s
Disk write throughput
Disk write throughput in MiB/s
33.98
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.52
5 min
0.60
15 min
0.86
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
5.48
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
4.6781
%
SD
17.3341
%
<
p95
33.8
%
60 second average
4.5886
%
300 second average
5.6104
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6921
%
SD
2.6923
%
<
p95
3.64
%
60 second average
0.6874
%
300 second average
0.8169
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2671
%
SD
1.2766
%
<
p95
1.6
%
60 second average
0.2672
%
300 second average
0.2700
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.48
%
<
mean
8.74
%
SD
22.39
%
<
p95
82.23
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
309.42
KiB
<
mean
370.84
KiB
SD
52.56
KiB
<
p95
432.68
KiB
Holochain process threads
Number of threads in Holochain process
p5
21
threads
<
mean
28
threads
SD
5.48
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
55
file descriptors
<
mean
64.39
file descriptors
SD
9.11
file descriptors
<
p95
83
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Thu, 17 Sep 2026 07:08:34 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_35174760384_1
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
391199
Agents affected
103 / 103
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3798
Max
Highest count in any single partition
11093
Min
Lowest count in any single partition
855
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1603
Max
Highest chain head value observed for any single write agent.
3736
Write agents observed
20
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
0.619203
s
SD
1.395709
s
Max
Highest per-partition mean latency
max mean
7.378946
s
Min
Lowest per-partition mean latency
min mean
-1.269112
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
9.39
/s
Max
Highest per-partition mean rate
max
92.91
/s
Min
Lowest per-partition mean rate
min
0
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
1.183181
s
SD
3.138847
s
Max
Highest per-partition mean latency
max mean
2.571145
s
Min
Lowest per-partition mean latency
min mean
0.046623
s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
125
Agents affected
44 / 44
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3
Max
Highest count in any single partition
8
Min
Lowest count in any single partition
1
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
34
<
mean
82.94
SD
36.99
<
p95
146
  • behaviour: zero_read
p5
17
<
mean
41.92
SD
19.28
<
p95
81
  • behaviour: zero_write
p5
6
<
mean
32.52
SD
16.79
<
p95
69
Final error count
The total number of errors accumulated across all agents over the run.
130
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.451751
s
SD
9.530555
s
p50
1.638542
s
<
p95
25.661322
s
<
p99
44.254149
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 813.793654692s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.008894
s
SD
0.003776
s
p50
0.008786
s
<
p95
0.015448
s
<
p99
0.020383
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009074
s
SD
0.012086
s
p50
0.008167
s
<
p95
0.040581
s
<
p99
0.055763
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.45842
s
SD
2.519226
s
p50
1.600243
s
<
p95
3.577603
s
<
p99
11.59687
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.350762
s
SD
12.09722
s
p50
1.52945
s
<
p95
38.281709
s
<
p99
44.260476
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007346
s
SD
0.003306
s
p50
0.007265
s
<
p95
0.012168
s
<
p99
0.01777
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007761
s
SD
0.011345
s
p50
0.006595
s
<
p95
0.037363
s
<
p99
0.051918
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.457248
s
SD
2.519211
s
p50
1.598429
s
<
p95
3.576028
s
<
p99
11.595545
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.38643
s
SD
11.908108
s
p50
3.786508
s
<
p95
38.280297
s
<
p99
44.259021
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001766
s
SD
0.000708
s
p50
0.001889
s
<
p95
0.003514
s
<
p99
0.004231
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
6.8e-05
s
SD
6.4e-05
s
p50
5e-05
s
<
p95
0.000179
s
<
p99
0.000258
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002504
s
SD
0.007614
s
p50
0.000964
s
<
p95
0.022215
s
<
p99
0.032877
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.00072
s
SD
0.000422
s
p50
0.000683
s
<
p95
0.001193
s
<
p99
0.002128
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.454956
s
SD
2.519245
s
p50
1.595985
s
<
p95
3.573591
s
<
p99
11.59269
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.382312
s
SD
11.907791
s
p50
3.781598
s
<
p95
38.273864
s
<
p99
44.25421
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
9e-06
s
p50
1.6e-05
s
<
p95
3.4e-05
s
<
p99
4.9e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003338
s
SD
0.002715
s
p50
0.003336
s
<
p95
0.009913
s
<
p99
0.014039
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001122
s
SD
0.002266
s
p50
0.000599
s
<
p95
0.00632
s
<
p99
0.010366
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.56
s
<
mean
431.05
s
SD
249.25
s
<
p95
818.06
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.165754792s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
5.697157
s
SD
20.91184
s
p50
0
s
<
p95
2.951191
s
<
p99
72.083943
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.243749
s
SD
0.709664
s
p50
0.003096
s
<
p95
1.220105
s
<
p99
3.60312
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.056309
s
SD
0.231572
s
p50
0.002615
s
<
p95
0.598053
s
<
p99
1.182092
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.274281
s
SD
0.194824
s
p50
0.179254
s
<
p95
0.547287
s
<
p99
0.758876
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
13.5247
s
SD
23.849299
s
p50
0.004334
s
<
p95
54.107989
s
<
p99
118.52599
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.263104
s
SD
11.474852
s
p50
0.001559
s
<
p95
0.083695
s
<
p99
17.590186
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00418
s
SD
0.002639
s
p50
0.002849
s
<
p95
0.007882
s
<
p99
0.010452
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00637
s
SD
0.003789
s
p50
0.003248
s
<
p95
0.012962
s
<
p99
0.018763
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002607
s
SD
0.006759
s
p50
0.000841
s
<
p95
0.01916
s
<
p99
0.032446
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
4.27134
s
SD
27.208214
s
p50
1.743893
s
<
p95
60.028531
s
<
p99
63.226254
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.136969
s
SD
12.505521
s
p50
0.640887
s
<
p95
38.270401
s
<
p99
60.001617
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.308714
s
SD
28.744603
s
p50
0.715276
s
<
p95
60.001688
s
<
p99
60.002028
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.192101
s
SD
18.08516
s
p50
0.482039
s
<
p95
60.043959
s
<
p99
60.120072
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
39.09
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.51
KiB/s
<
mean
1003.41
KiB/s
SD
772.61
KiB/s
<
p95
2.30
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
106.12
KiB/s
SD
170.42
KiB/s
<
p95
369.76
KiB/s
Total bytes received
Total bytes received on primary network interface
count
123.41
GiB
mean
142.00
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.05
GiB
mean
15.02
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.3%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 34% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.75
%
<
mean
53.9
%
SD
35.31
%
<
p95
93
%
System
14.81
%
CPU percentiles
CPU usage percentiles
p50
64.11
%
p95
93.00
%
p99
97.29
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
114
hosts
mean time
432.73
s
Memory used percentage
Percentage of memory used
p5
6.84
%
<
mean
11.39
%
SD
4.53
%
<
p95
18.59
%
Memory available percentage
Percentage of available memory
p5
81.41
%
<
mean
88.61
%
SD
4.53
%
<
p95
93.16
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.33
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.91
%
Memory growth rate
Rate of memory growth over time
growth
0.44
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.19
MiB/s
Disk write throughput
Disk write throughput in MiB/s
84.26
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.36
5 min
1.83
15 min
1.29
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
34.25
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
29.0064
%
SD
28.6586
%
<
p95
75.1
%
60 second average
27.5489
%
300 second average
21.1981
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
5.0118
%
SD
6.8208
%
<
p95
20.1
%
60 second average
4.7898
%
300 second average
3.6643
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.4939
%
SD
2.1925
%
<
p95
5.06
%
60 second average
1.4442
%
300 second average
1.1334
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.2
%
<
mean
50.57
%
SD
34.21
%
<
p95
88.02
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
201.77
KiB
<
mean
279.39
KiB
SD
77.82
KiB
<
p95
420.62
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
30.7
threads
SD
5.77
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
61
file descriptors
<
mean
77.41
file descriptors
SD
12.19
file descriptors
<
p95
89
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Thu, 17 Sep 2026 07:25:33 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_35174760384_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
291.73
Max
Highest chain head value observed for any single write agent.
1413
Write agents observed
15
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
247.937659
s
SD
146.130317
s
Max
Highest per-partition mean latency
max mean
676.081
s
Min
Lowest per-partition mean latency
min mean
-732.355
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
0.82
/s
Max
Highest per-partition mean rate
max
2.08
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
8.767462
s
SD
8.727667
s
Max
Highest per-partition mean latency
max mean
34.683971
s
Min
Lowest per-partition mean latency
min mean
0.085943
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
82536
Agents affected
43 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1919
Max
Highest count in any single partition
5615
Min
Lowest count in any single partition
15
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
39
<
mean
83.28
SD
36.91
<
p95
146
  • behaviour: zero_must_get_agent_activity
p5
10
<
mean
34.24
SD
15.4
<
p95
55
  • behaviour: zero_write
p5
10
<
mean
38.02
SD
20.57
<
p95
79
Final error count
The total number of all types of error accumulated over the run by all agents.
101228
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.262996
s
SD
5.837887
s
p50
0.289471
s
<
p95
18.68996
s
<
p99
22.872338
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 809.432355179s
mean rate
27.89
/s
std
83.39
/s
p5
0
/s
p95
163.3
/s
peak
683.76
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.008577
s
SD
0.003288
s
p50
0.008346
s
<
p95
0.01332
s
<
p99
0.024511
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.189335
s
SD
0.186633
s
p50
0.088016
s
<
p95
0.547314
s
<
p99
0.892741
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007934
s
SD
0.00108
s
p50
0.007796
s
<
p95
0.009068
s
<
p99
0.010572
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
1.882112
s
SD
5.108662
s
p50
0.322156
s
<
p95
16.346253
s
<
p99
22.24674
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.248571
s
SD
7.817825
s
p50
0.60752
s
<
p95
21.983601
s
<
p99
22.882146
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007041
s
SD
0.002899
s
p50
0.006743
s
<
p95
0.01116
s
<
p99
0.019567
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.187274
s
SD
0.18566
s
p50
0.086435
s
<
p95
0.543658
s
<
p99
0.885607
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006492
s
SD
0.000901
s
p50
0.006417
s
<
p95
0.007404
s
<
p99
0.00906
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.501289
s
SD
5.738636
s
p50
0.55006
s
<
p95
11.241198
s
<
p99
28.939858
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.248068
s
SD
7.817718
s
p50
0.606231
s
<
p95
21.982137
s
<
p99
22.88036
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002095
s
SD
0.000789
s
p50
0.001973
s
<
p95
0.003606
s
<
p99
0.004964
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.024938
s
SD
0.093014
s
p50
0.006283
s
<
p95
0.125976
s
<
p99
0.518226
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.8e-05
s
SD
5.1e-05
s
p50
4.6e-05
s
<
p95
0.00017
s
<
p99
0.000272
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.003309
s
SD
0.007705
s
p50
0.00067
s
<
p95
0.017496
s
<
p99
0.041281
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.005159
s
SD
0.007472
s
p50
0.000688
s
<
p95
0.015685
s
<
p99
0.043049
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.464509
s
SD
7.045416
s
p50
0.583864
s
<
p95
21.975471
s
<
p99
28.930989
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.211394
s
SD
0.538681
s
p50
0.167178
s
<
p95
1.029394
s
<
p99
1.981825
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.066278
s
SD
0.062155
s
p50
0.024157
s
<
p95
0.17806
s
<
p99
0.206711
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
5e-06
s
SD
6e-06
s
p50
1.8e-05
s
<
p95
2.3e-05
s
<
p99
4.5e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.004201
s
SD
0.002344
s
p50
0.003335
s
<
p95
0.008413
s
<
p99
0.014922
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001756
s
SD
0.002126
s
p50
0.000626
s
<
p95
0.006301
s
<
p99
0.01009
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.27
s
<
mean
430.88
s
SD
249.22
s
<
p95
817.87
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.801495479s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
95.629766
s
SD
40.055785
s
p50
0
s
<
p95
22.655042
s
<
p99
186.255271
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.846668
s
SD
2.345052
s
p50
0.002916
s
<
p95
5.560875
s
<
p99
9.056869
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
1e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.585333
s
SD
0.383951
s
p50
0.002736
s
<
p95
1.247955
s
<
p99
1.39074
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.190489
s
SD
0.172867
s
p50
0.03632
s
<
p95
0.514566
s
<
p99
0.657712
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
11.291672
s
SD
13.618137
s
p50
0.004399
s
<
p95
31.361004
s
<
p99
69.697672
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.403633
s
SD
15.552544
s
p50
0.001561
s
<
p95
2.921273
s
<
p99
32.884366
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002089
s
SD
0.00155
s
p50
0.000217
s
<
p95
0.00438
s
<
p99
0.006014
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004015
s
SD
0.005723
s
p50
0.003357
s
<
p95
0.017968
s
<
p99
0.028904
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.003397
s
SD
0.007117
s
p50
0.000597
s
<
p95
0.015899
s
<
p99
0.039242
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.969056
s
SD
27.981894
s
p50
1.022115
s
<
p95
60.003774
s
<
p99
60.017598
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.178647
s
SD
21.249366
s
p50
0.326977
s
<
p95
60.00166
s
<
p99
60.002063
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.194932
s
SD
28.093735
s
p50
0.183827
s
<
p95
60.002034
s
<
p99
60.002358
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
2.365753
s
SD
25.285678
s
p50
2.162663
s
<
p95
60.008817
s
<
p99
60.052726
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
24.15
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
1.65
KiB/s
<
mean
1.00
MiB/s
SD
867.28
KiB/s
<
p95
2.35
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
89.72
KiB/s
SD
86.38
KiB/s
<
p95
243.94
KiB/s
Total bytes received
Total bytes received on primary network interface
count
126.02
GiB
mean
144.99
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
11.03
GiB
mean
12.70
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.3%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 34% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.26
%
<
mean
35.76
%
SD
33.49
%
<
p95
88.15
%
System
11.61
%
CPU percentiles
CPU usage percentiles
p50
16.13
%
p95
88.15
%
p99
93.31
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
120
hosts
mean time
245.86
s
Memory used percentage
Percentage of memory used
p5
7.29
%
<
mean
12.66
%
SD
4.59
%
<
p95
19.41
%
Memory available percentage
Percentage of available memory
p5
80.59
%
<
mean
87.34
%
SD
4.59
%
<
p95
92.71
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.90
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.91
%
Memory growth rate
Rate of memory growth over time
growth
0.36
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
75.09
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.45
5 min
1.59
15 min
1.59
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
33.56
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.11
%
<
mean
17.312
%
SD
22.5164
%
<
p95
57.58
%
60 second average
16.7675
%
300 second average
18.3796
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1205
%
SD
6.4783
%
<
p95
16.49
%
60 second average
3.9767
%
300 second average
3.9481
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.7873
%
SD
3.0033
%
<
p95
7.86
%
60 second average
1.7219
%
300 second average
1.5670
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.5
%
<
mean
32.07
%
SD
32.22
%
<
p95
81.34
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
236.64
KiB
<
mean
372.70
KiB
SD
81.68
KiB
<
p95
487.97
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
30.47
threads
SD
6.07
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
61
file descriptors
<
mean
75.28
file descriptors
SD
13.18
file descriptors
<
p95
89
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Thu, 17 Sep 2026 02:39:20 UTC
Duration
900s
Peer count
150
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_35174760384_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-0.5237
s
SD
14.453288
s
Max
Highest per-partition mean latency
max mean
0.164139
s
Min
Lowest per-partition mean latency
min mean
-3.597415
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.123512
s
SD
0.227146
s
Max
Highest per-partition mean latency
max mean
0.410811
s
Min
Lowest per-partition mean latency
min mean
0.046266
s
Error count
The total number of errors accumulated during the run.
4387
Holochain MetricsClick to toggle visibility.
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
88018713
over 889.049562496s
mean rate
2.14807399678e+10
/s
std
8.68542692089e+11
/s
p5
7.98356698e+06
/s
p95
3.24988637856e+10
/s
peak
6.79368514369e+13
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
22189602
over 889.049562496s
mean rate
6.5772355266e+08
/s
std
1.87937681713e+10
/s
p5
233505.09
/s
p95
1.36273851115e+09
/s
peak
1.4388652424e+12
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.049562496s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.134891
s
SD
0.402051
s
p50
0.163022
s
<
p95
0.512378
s
<
p99
1.356268
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.00362
s
SD
0.000935
s
p50
0.003967
s
<
p95
0.004682
s
<
p99
0.005484
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.237755
s
SD
0.44049
s
p50
0.408081
s
<
p95
0.924052
s
<
p99
1.475722
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002284
s
SD
0.000601
s
p50
0.002519
s
<
p95
0.003137
s
<
p99
0.003647
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005042
s
SD
0.003151
s
p50
0.004544
s
<
p95
0.009433
s
<
p99
0.02028
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.234897
s
SD
0.440532
s
p50
0.405614
s
<
p95
0.921721
s
<
p99
1.473653
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000784
s
SD
0.00052
s
p50
0.000737
s
<
p95
0.001411
s
<
p99
0.002844
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.3e-05
s
SD
6e-06
s
p50
1.2e-05
s
<
p95
2.3e-05
s
<
p99
2.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000351
s
SD
0.000885
s
p50
0.000375
s
<
p95
0.001156
s
<
p99
0.002256
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000487
s
SD
0.000852
s
p50
0.000236
s
<
p95
0.001834
s
<
p99
0.00287
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.62
s
<
mean
431.02
s
SD
249.22
s
<
p95
817.84
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1896
over 889.049446045s
mean rate
21343.34
/s
std
376717.17
/s
p5
0
/s
p95
37663.01
/s
peak
2.850983562e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
11.585187
s
SD
6.412953
s
p50
9.32781
s
<
p95
24.469568
s
<
p99
36.179192
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.010752
s
SD
0.010468
s
p50
0.010508
s
<
p95
0.035269
s
<
p99
0.051306
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
3e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.083435
s
SD
0.044528
s
p50
0.082768
s
<
p95
0.157557
s
<
p99
0.218869
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.017294
s
SD
0.007035
s
p50
0.017453
s
<
p95
0.029173
s
<
p99
0.035665
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.566548
s
SD
3.252814
s
p50
4.14788
s
<
p95
10.860315
s
<
p99
14.261091
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.194516
s
SD
3.60952
s
p50
0.44886
s
<
p95
10.350391
s
<
p99
19.056078
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000404
s
SD
0.000154
s
p50
0.000365
s
<
p95
0.000659
s
<
p99
0.000889
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003405
s
SD
0.001756
s
p50
0.003071
s
<
p95
0.00702
s
<
p99
0.009219
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000569
s
SD
0.000197
s
p50
0.000553
s
<
p95
0.000832
s
<
p99
0.001131
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.134292
s
SD
22.385422
s
p50
0.095177
s
<
p95
60.002013
s
<
p99
60.003877
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.125375
s
SD
0.093317
s
p50
0.059697
s
<
p95
0.301642
s
<
p99
0.353029
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.131269
s
SD
10.275488
s
p50
0.16225
s
<
p95
0.906168
s
<
p99
60.001243
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
7.02
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
19.26
KiB/s
<
mean
215.97
KiB/s
SD
448.30
KiB/s
<
p95
542.03
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
5.87
KiB/s
<
mean
14.26
KiB/s
SD
13.05
KiB/s
<
p95
39.99
KiB/s
Total bytes received
Total bytes received on primary network interface
count
26.60
GiB
mean
30.27
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.76
GiB
mean
2.00
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.49
%
<
mean
12.25
%
SD
10.96
%
<
p95
33.54
%
System
5.23
%
CPU percentiles
CPU usage percentiles
p50
8.96
%
p95
33.54
%
p99
68.98
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
39
hosts
mean time
11.93
s
Memory used percentage
Percentage of memory used
p5
5.82
%
<
mean
9.72
%
SD
4.3
%
<
p95
15.94
%
Memory available percentage
Percentage of available memory
p5
84.06
%
<
mean
90.28
%
SD
4.3
%
<
p95
94.18
%
Max host memory used
Maximum memory usage percentage across all hosts
max
33.67
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.70
%
Memory growth rate
Rate of memory growth over time
growth
0.17
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.16
MiB/s
Disk write throughput
Disk write throughput in MiB/s
21.49
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.37
5 min
0.29
15 min
0.17
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
2.8897
%
SD
2.7789
%
<
p95
8.16
%
60 second average
2.7964
%
300 second average
2.1658
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3435
%
SD
1.5969
%
<
p95
1.4
%
60 second average
0.3509
%
300 second average
0.2921
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2538
%
SD
1.4976
%
<
p95
0.98
%
60 second average
0.2559
%
300 second average
0.2111
%
Holochain process CPU usage
CPU usage by Holochain process
p5
2.05
%
<
mean
9.09
%
SD
9.49
%
<
p95
19.37
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
206.54
KiB
<
mean
230.78
KiB
SD
27.57
KiB
<
p95
273.51
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
31.33
threads
SD
4.09
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
79.16
file descriptors
SD
6.63
file descriptors
<
p95
87
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Thu, 17 Sep 2026 02:56:12 UTC
Duration
900s
Peer count
300
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_signals_35174760384_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.616845
s
SD
2.396959
s
p50
0.097947
s
<
p95
2.578008
s
<
p99
12.394466
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
326
over 1338.850743698s
mean rate
49255.35
/s
std
4.71891688e+06
/s
p5
0
/s
p95
7806.69
/s
peak
5e+08
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0014
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 820.007921445s
mean rate
0.24
/s
std
1.42
/s
p5
0
/s
p95
0.56
/s
peak
10.14
/s
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.203613719s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
23061982
over 889.203613719s
mean rate
1.68011260311e+11
/s
std
1.73758837324e+13
/s
p5
4.04378192e+06
/s
p95
1.08047076557e+10
/s
peak
1.8785925e+15
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
12337560
over 889.203613719s
mean rate
9.671254688e+07
/s
std
1.96095600173e+09
/s
p5
0
/s
p95
1.171892743e+08
/s
peak
2.00404167982e+11
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.004005
s
SD
0.001032
s
p50
0.00412
s
<
p95
0.005421
s
<
p99
0.006466
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.003844
s
SD
0.000802
s
p50
0.003871
s
<
p95
0.005032
s
<
p99
0.005525
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008617
s
SD
0.044673
s
p50
0.00412
s
<
p95
0.012369
s
<
p99
0.079791
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002529
s
SD
0.000648
s
p50
0.002647
s
<
p95
0.003438
s
<
p99
0.004011
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002355
s
SD
0.000506
s
p50
0.002386
s
<
p95
0.00322
s
<
p99
0.003561
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00146
s
SD
0.009249
s
p50
0.000696
s
<
p95
0.002033
s
<
p99
0.007527
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.2e-05
s
SD
1.9e-05
s
p50
7.5e-05
s
<
p95
0.0001
s
<
p99
0.00011
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
6.7e-05
s
SD
2.6e-05
s
p50
6.9e-05
s
<
p95
0.0001
s
<
p99
0.00012
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
4e-06
s
p50
8e-06
s
<
p95
1.6e-05
s
<
p99
2.1e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003704
s
SD
0.043655
s
p50
0.000495
s
<
p95
0.002093
s
<
p99
0.067502
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1299
over 889.203617737s
mean rate
298829.34
/s
std
7.86552904e+06
/s
p5
147.3
/s
p95
491749.66
/s
peak
7.7513227513e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1377
over 889.203615751s
mean rate
249285.91
/s
std
6.63116809e+06
/s
p5
155.99
/s
p95
430459.59
/s
peak
4.9545454545e+08
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000591
s
SD
0.001381
s
p50
0.000245
s
<
p95
0.002546
s
<
p99
0.004261
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.85
s
<
mean
430.18
s
SD
248.4
s
<
p95
817.46
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3808
over 889.203653572s
mean rate
550199.3
/s
std
6.47908469e+06
/s
p5
0
/s
p95
1.48717622e+06
/s
peak
5e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
14.240822
s
SD
8.117993
s
p50
12.322534
s
<
p95
30.347659
s
<
p99
42.389419
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.018534
s
SD
0.072889
s
p50
0.019471
s
<
p95
0.065231
s
<
p99
0.099608
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.102289
s
SD
0.057474
s
p50
0.106629
s
<
p95
0.20786
s
<
p99
0.276894
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.035677
s
SD
0.039746
s
p50
0.027498
s
<
p95
0.08747
s
<
p99
0.168546
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.890808
s
SD
6.51162
s
p50
5.627347
s
<
p95
20.104532
s
<
p99
28.882821
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
4.30259
s
SD
5.776011
s
p50
4.000088
s
<
p95
14.292339
s
<
p99
27.572734
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00081
s
SD
0.000463
s
p50
0.000682
s
<
p95
0.001499
s
<
p99
0.002205
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005066
s
SD
0.00318
s
p50
0.004287
s
<
p95
0.01037
s
<
p99
0.015328
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000871
s
SD
0.000852
s
p50
0.000769
s
<
p95
0.001616
s
<
p99
0.003258
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.318698
s
SD
24.481104
s
p50
0.344244
s
<
p95
60.002341
s
<
p99
60.006684
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.331684
s
SD
0.414528
s
p50
0.368777
s
<
p95
1.015009
s
<
p99
2.094105
s
P2P received remote signals
Count of remote signals received.
total
1301
over 889.203743629s
mean rate
438008.15
/s
std
1.052691455e+07
/s
p5
140.02
/s
p95
544339.47
/s
peak
7.5409836066e+08
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
12.60
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
24.54
KiB/s
<
mean
492.17
KiB/s
SD
522.48
KiB/s
<
p95
1.36
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
14.41
KiB/s
<
mean
39.26
KiB/s
SD
28.18
KiB/s
<
p95
91.88
KiB/s
Total bytes received
Total bytes received on primary network interface
count
60.50
GiB
mean
69.61
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.83
GiB
mean
5.55
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
5.72
%
<
mean
19.61
%
SD
16.58
%
<
p95
50.25
%
System
7.00
%
CPU percentiles
CPU usage percentiles
p50
14.59
%
p95
50.25
%
p99
95.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
125
hosts
mean time
28.19
s
Memory used percentage
Percentage of memory used
p5
7.33
%
<
mean
12.6
%
SD
4.58
%
<
p95
18.74
%
Memory available percentage
Percentage of available memory
p5
81.26
%
<
mean
87.4
%
SD
4.58
%
<
p95
92.67
%
Max host memory used
Maximum memory usage percentage across all hosts
max
38.07
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.70
%
Memory growth rate
Rate of memory growth over time
growth
0.32
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.02
MiB/s
Disk write throughput
Disk write throughput in MiB/s
31.03
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.73
5 min
0.64
15 min
0.44
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.06
%
<
mean
8.4143
%
SD
12.394
%
<
p95
28.21
%
60 second average
8.2985
%
300 second average
7.3348
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8662
%
SD
2.5207
%
<
p95
3.49
%
60 second average
0.8794
%
300 second average
0.7624
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.522
%
SD
2.0843
%
<
p95
2.02
%
60 second average
0.5295
%
300 second average
0.4630
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.82
%
<
mean
8.19
%
SD
8.38
%
<
p95
25.64
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
124.54
KiB
<
mean
221.27
KiB
SD
45.20
KiB
<
p95
268.73
KiB
Holochain process threads
Number of threads in Holochain process
p5
26
threads
<
mean
30.91
threads
SD
5.72
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
70
file descriptors
<
mean
78.23
file descriptors
SD
13.46
file descriptors
<
p95
89
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Thu, 17 Sep 2026 03:45:36 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
validation_receipts_35174760384_1
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
38.569815
s
SD
33.495808
s
Max
Highest per-partition mean latency
max mean
265.96308
s
Min
Lowest per-partition mean latency
min mean
16.162651
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
1
/s
Max
Highest per-partition mean rate
max
1.2
/s
Min
Lowest per-partition mean rate
min
1
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
1143840
over 889.067564492s
mean rate
2.550676687e+07
/s
std
4.3488512115e+08
/s
p5
0
/s
p95
5.711937462e+07
/s
peak
3.40655675212e+10
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
11376160
over 889.067564492s
mean rate
1.5330632224e+08
/s
std
1.34455862744e+09
/s
p5
131124.78
/s
p95
2.8449255732e+08
/s
peak
5.06261833521e+10
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.067564492s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
140864
over 889.067564492s
mean rate
3.0869991e+06
/s
std
5.303590428e+07
/s
p5
0
/s
p95
6.94606151e+06
/s
peak
4.19517773754e+09
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.00686
s
SD
0.001765
s
p50
0.006953
s
<
p95
0.009504
s
<
p99
0.01064
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.018833
s
SD
0.013656
s
p50
0.01116
s
<
p95
0.041863
s
<
p99
0.075481
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005602
s
SD
0.001448
s
p50
0.00573
s
<
p95
0.007842
s
<
p99
0.008787
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.017319
s
SD
0.013412
s
p50
0.009688
s
<
p95
0.039592
s
<
p99
0.073043
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.003668
s
SD
0.002284
s
p50
0.003393
s
<
p95
0.005662
s
<
p99
0.014322
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001835
s
SD
0.000435
s
p50
0.001868
s
<
p95
0.002479
s
<
p99
0.002905
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000648
s
SD
0.001202
s
p50
0.000704
s
<
p95
0.00164
s
<
p99
0.003006
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.015051
s
SD
0.013246
s
p50
0.007506
s
<
p95
0.036843
s
<
p99
0.070063
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002867
s
SD
0.000773
s
p50
0.002921
s
<
p95
0.004042
s
<
p99
0.004559
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000616
s
SD
0.000475
s
p50
0.000544
s
<
p95
0.001347
s
<
p99
0.002151
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.64
s
<
mean
428.78
s
SD
247.99
s
<
p95
814.76
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
10428
over 889.067572505s
mean rate
91606.49
/s
std
466103.9
/s
p5
426.08
/s
p95
331120.16
/s
peak
1.315208323e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
10.6677
s
SD
7.084648
s
p50
8.591738
s
<
p95
26.929489
s
<
p99
35.860369
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.00461
s
SD
0.02518
s
p50
0.006338
s
<
p95
0.062961
s
<
p99
0.135462
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.04171
s
SD
0.071213
s
p50
0.055494
s
<
p95
0.228191
s
<
p99
0.317999
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.075506
s
SD
0.054432
s
p50
0.040779
s
<
p95
0.166854
s
<
p99
0.281362
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.307136
s
SD
5.129854
s
p50
0.561707
s
<
p95
14.183735
s
<
p99
20.678609
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
21.028171
s
SD
20.18014
s
p50
24.29326
s
<
p95
64.750515
s
<
p99
83.706529
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001173
s
SD
0.000594
s
p50
0.000841
s
<
p95
0.002264
s
<
p99
0.002909
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004207
s
SD
0.002229
s
p50
0.003216
s
<
p95
0.007231
s
<
p99
0.009043
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000602
s
SD
0.000233
s
p50
0.000579
s
<
p95
0.001041
s
<
p99
0.001302
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.25642
s
SD
27.582302
s
p50
0.207117
s
<
p95
60.003245
s
<
p99
60.005939
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.171203
s
SD
0.146103
s
p50
0.173912
s
<
p95
0.428522
s
<
p99
0.693785
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
14.51
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
21.09
KiB/s
<
mean
272.11
KiB/s
SD
479.12
KiB/s
<
p95
643.55
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.40
KiB/s
<
mean
20.12
KiB/s
SD
18.48
KiB/s
<
p95
57.36
KiB/s
Total bytes received
Total bytes received on primary network interface
count
33.43
GiB
mean
38.47
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.47
GiB
mean
2.84
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 3% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
5.14
%
<
mean
21.61
%
SD
15.47
%
<
p95
54.11
%
System
7.10
%
CPU percentiles
CPU usage percentiles
p50
16.91
%
p95
54.11
%
p99
76.06
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
62
hosts
mean time
15.04
s
Memory used percentage
Percentage of memory used
p5
5.98
%
<
mean
10.3
%
SD
4.31
%
<
p95
17.17
%
Memory available percentage
Percentage of available memory
p5
82.83
%
<
mean
89.7
%
SD
4.31
%
<
p95
94.02
%
Max host memory used
Maximum memory usage percentage across all hosts
max
34.51
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.77
%
Memory growth rate
Rate of memory growth over time
growth
0.20
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.40
MiB/s
Disk write throughput
Disk write throughput in MiB/s
30.31
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.71
5 min
1.28
15 min
2.10
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
2.74
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.03
%
<
mean
6.3475
%
SD
7.2274
%
<
p95
19.83
%
60 second average
6.3490
%
300 second average
13.7065
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.8759
%
SD
4.0637
%
<
p95
5.15
%
60 second average
1.8473
%
300 second average
3.2749
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.386
%
SD
3.7219
%
<
p95
3.08
%
60 second average
1.3258
%
300 second average
1.5040
%
Holochain process CPU usage
CPU usage by Holochain process
p5
3.75
%
<
mean
18.76
%
SD
14.65
%
<
p95
50.84
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
218.27
KiB
<
mean
256.00
KiB
SD
34.74
KiB
<
p95
300.26
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
31.02
threads
SD
4.34
threads
<
p95
34
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
79.27
file descriptors
SD
8.39
file descriptors
<
p95
85
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 17 Sep 2026 04:02:35 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_35174760384_1
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2180
Max
Highest chain head value observed for any single write agent.
7249
Write agents observed
25
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
1.090785
s
SD
2.075495
s
Max
Highest per-partition mean latency
max mean
5.026945
s
Min
Lowest per-partition mean latency
min mean
0.036582
s
Error count
The number of errors accumulated.
329
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.092923
s
SD
4.241512
s
p50
0.418
s
<
p95
11.339081
s
<
p99
25.328105
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
5
over 813.474929125s
mean rate
9.54
/s
std
46.37
/s
p5
0
/s
p95
54.14
/s
peak
972.62
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007139
s
SD
0.002022
s
p50
0.007195
s
<
p95
0.009966
s
<
p99
0.013485
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008785
s
SD
0.006457
s
p50
0.009367
s
<
p95
0.022709
s
<
p99
0.033386
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.22737
s
SD
2.7331
s
p50
0.744639
s
<
p95
5.188283
s
<
p99
13.359816
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007389
s
SD
4.941887
s
p50
0.007576
s
<
p95
11.272582
s
<
p99
28.062965
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005917
s
SD
0.001625
s
p50
0.006093
s
<
p95
0.008199
s
<
p99
0.01118
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007498
s
SD
0.0059
s
p50
0.007811
s
<
p95
0.020195
s
<
p99
0.030553
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.238951
s
SD
2.73524
s
p50
0.765006
s
<
p95
5.219427
s
<
p99
13.359054
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.008691
s
SD
5.213621
s
p50
0.008463
s
<
p95
11.502941
s
<
p99
25.335605
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001495
s
SD
0.000487
s
p50
0.001726
s
<
p95
0.002557
s
<
p99
0.002959
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.3e-05
s
SD
3e-05
s
p50
4.1e-05
s
<
p95
0.000113
s
<
p99
0.000156
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002619
s
SD
0.003699
s
p50
0.002392
s
<
p95
0.01045
s
<
p99
0.017828
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000577
s
SD
0.000169
s
p50
0.000558
s
<
p95
0.000864
s
<
p99
0.000918
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.237038
s
SD
2.735078
s
p50
0.762641
s
<
p95
5.216528
s
<
p99
13.357046
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.004853
s
SD
5.213333
s
p50
0.002653
s
<
p95
11.496136
s
<
p99
25.329599
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
9e-06
s
SD
1.2e-05
s
p50
1.2e-05
s
<
p95
2.4e-05
s
<
p99
8e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002513
s
SD
0.001278
s
p50
0.002836
s
<
p95
0.005344
s
<
p99
0.008142
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001092
s
SD
0.000963
s
p50
0.000251
s
<
p95
0.002489
s
<
p99
0.004845
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.59
s
<
mean
430.91
s
SD
249.14
s
<
p95
817.7
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
24617
over 888.675619096s
mean rate
1.46281214e+06
/s
std
3.885244396e+07
/s
p5
271.33
/s
p95
1.86319087e+06
/s
peak
2.58917010006e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
76.747562
s
SD
57.371933
s
p50
23.337729
s
<
p95
174.30572
s
<
p99
297.73261
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.08396
s
SD
0.307611
s
p50
0.109991
s
<
p95
0.693555
s
<
p99
1.809933
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.07927
s
SD
0.367382
s
p50
0.228808
s
<
p95
1.188408
s
<
p99
1.362527
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.882852
s
SD
0.745763
s
p50
0.119112
s
<
p95
2.272967
s
<
p99
3.484428
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
12.075959
s
SD
22.008087
s
p50
25.334771
s
<
p95
71.213537
s
<
p99
100.927825
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
12.673498
s
SD
23.185694
s
p50
0.26192
s
<
p95
68.088448
s
<
p99
111.397473
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004475
s
SD
0.002826
s
p50
0.003654
s
<
p95
0.009082
s
<
p99
0.010825
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006189
s
SD
0.003649
s
p50
0.003884
s
<
p95
0.012104
s
<
p99
0.017926
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002649
s
SD
0.003511
s
p50
0.000862
s
<
p95
0.008749
s
<
p99
0.017544
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.463221
s
SD
29.409511
s
p50
2.00284
s
<
p95
60.009184
s
<
p99
60.04071
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
2.067357
s
SD
9.414732
s
p50
0.839074
s
<
p95
19.216676
s
<
p99
60.001832
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.211518
s
SD
29.380665
s
p50
0.665898
s
<
p95
60.009013
s
<
p99
60.023413
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.015871
s
SD
26.352216
s
p50
3.052433
s
<
p95
60.012932
s
<
p99
60.043021
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
43.20
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
11.13
KiB/s
<
mean
981.33
KiB/s
SD
710.67
KiB/s
<
p95
2.06
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
58
B/s
<
mean
117.26
KiB/s
SD
214.89
KiB/s
<
p95
361.78
KiB/s
Total bytes received
Total bytes received on primary network interface
count
120.67
GiB
mean
138.84
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
14.42
GiB
mean
16.59
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 55% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
8.82
%
<
mean
61.68
%
SD
24.37
%
<
p95
91.8
%
System
18.47
%
CPU percentiles
CPU usage percentiles
p50
66.82
%
p95
91.80
%
p99
94.15
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
123
hosts
mean time
274.90
s
Memory used percentage
Percentage of memory used
p5
6.42
%
<
mean
11.04
%
SD
4.59
%
<
p95
17.62
%
Memory available percentage
Percentage of available memory
p5
82.38
%
<
mean
88.96
%
SD
4.59
%
<
p95
93.58
%
Max host memory used
Maximum memory usage percentage across all hosts
max
36.39
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.77
%
Memory growth rate
Rate of memory growth over time
growth
0.41
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.58
MiB/s
Disk write throughput
Disk write throughput in MiB/s
84.24
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.95
5 min
2.25
15 min
1.92
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
55.48
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.16
%
<
mean
37.4093
%
SD
25.9882
%
<
p95
79.63
%
60 second average
34.9237
%
300 second average
26.1670
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
10.1564
%
SD
8.5318
%
<
p95
23.66
%
60 second average
9.5357
%
300 second average
7.1943
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.6383
%
SD
4.9621
%
<
p95
12.87
%
60 second average
4.4026
%
300 second average
3.5268
%
Holochain process CPU usage
CPU usage by Holochain process
p5
11.61
%
<
mean
58.89
%
SD
22.04
%
<
p95
86.82
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
206.20
KiB
<
mean
309.71
KiB
SD
77.30
KiB
<
p95
416.83
KiB
Holochain process threads
Number of threads in Holochain process
p5
27
threads
<
mean
31.84
threads
SD
5.4
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
70
file descriptors
<
mean
79.95
file descriptors
SD
12.44
file descriptors
<
p95
89
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 17 Sep 2026 04:19:23 UTC
Duration
1800s
Peer count
150
Peer count at end
121
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_35174760384_1
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
287.28
/s
Max
Highest per-partition mean rate
max
936.09
/s
Min
Lowest per-partition mean rate
min
129.24
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.029829
s
SD
0.227435
s
Max
Highest per-partition mean latency
max mean
0.055891
s
Min
Lowest per-partition mean latency
min mean
0.007209
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
54
conductors
<
mean
72.67
conductors
SD
12.04
conductors
<
p95
91
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
827.500901
s
SD
404.741393
s
Max
Highest per-partition mean latency
max mean
1508.37037
s
Min
Lowest per-partition mean latency
min mean
449.444444
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
39.150266
s
SD
34.039694
s
Max
Highest per-partition mean latency
max mean
143.002732
s
Min
Lowest per-partition mean latency
min mean
26.394501
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
32.801709
s
SD
34.213882
s
Max
Highest per-partition mean latency
max mean
53.603285
s
Min
Lowest per-partition mean latency
min mean
7.801109
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk-g-CTpBqcOyBrP9NgZgA04eCG1OF3yLlN6ct1BajcTDZi3Q3
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0Rc6Ddrs0MuhiPe5GKugoZAxUO04LrAmikV_mh_l4mGTC99x
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1Bg98-5F_HnWk6HIAbeGN-QcCwZ0xC_dMDvy7Scel6GMaoGo
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1MNdJ0elWLgeAFclDi68_O5_yYp0uJrETisZn1dPy_rLFqzY
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1fQtTWGFJ3vcjZYgmOlqB5Zj63TZntA2o4R5b5WE-M67bR4t
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1t_61nXzNkbOmSipgCxygAiWJLCKA_h-9qzafwyCX_yoeJ9j
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2kU_Zwfln0-HnLB97wsJ9ubMsUWP2Sfimd6NRC_8LM_sLidk
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2mH6bDUodBfbGIHlS_UANLHojKwk5R_xeCejDX_UYcjOY_yZ
p5
0
<
mean
0.25
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3fuA_ZwQkj25PXak0kYCJ7PHgM6-89vXhPD0wF4tl_No5Nj4
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3mcWme4RlczbbMY5e5miffLocnJVw9lg8nXzTbNVUmyYpYu7
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3pK32PS1XaNOu6ytpR8-zgwkzEPzCgtEwZkxuLraauzMJCWo
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk5BKQcVc5XhClxLX8nYKbbK6f1fpES_fiXraqqtrhA7dBd4Fe
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk6-KHwb5xbL0Jf2vq9U9Nz-XoT_Y716Ok9mdWOkxzE24-rRgq
p5
0
<
mean
0.18
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk6ooMvBNlfe_4DWrH3lkw7eqHc9hjHuea3GjE0_1uUb8NEvWZ
p5
0
<
mean
0.24
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk7elpy2IVpZvMq6MDtThJV9OtEseqUrR-Kp2Exz8Rf5VFJ7Jw
p5
0
<
mean
0.19
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk7u9cRRioQdm3JQnlDxbVik2ZIX7H_dC69JKpTPOXIPhsxRk5
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk8T_g9pQJS92c2BjLAmcyuZvzqvhypLaeTlyCGq1XlovDvJZ2
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkAfHEy--RVUbXzxj_A_-gl4hjTFTLYti8ECfTYhjZECi53ixp
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkAliXK0WRhDdYEeGTdvokK6GFVfNhwy2GdanGHO_cVn_ZdEum
p5
0
<
mean
0.13
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBWRHWVUx-2TsejOW-ccVhHPNJEHeviTt2_iE_pR2yJBYHLaF
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkB_1yQ_QiceX4Z7Dkf7p7VejEbTt8fghgb9EvkaOs7AJokkI1
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkCitjAhntLFEu0pKrXYOybZ1gugX0ZGt-GaNtF2PWCbejV-AO
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkCvvUYzEuDHmcG3Dl1ygYv_9KY9qcP814u3bqXdfUhHdsuHGD
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkD3ltMR1ff7gSCIaGAvJ7toOMKcwO4o52Q6BzRoouA1iH8skn
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkDRNO3QXG1R3YeP-51p0jk082eCbt89ARpE2POgkYrjfeQy1t
p5
0
<
mean
0.33
SD
0.47
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkErsKi90WxjRx_v9igRJHqzAby8aMdVHfCOtJbI8TuzMaSyCH
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkG7qLZpvYbAparo5ffN7toIEqew5OKjGkFhA4mS922JAQe6-a
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkGjzFctcqBwjeN8wC_Ll-Giu07Sp5CxmJkQV1K3aR0zauiJTi
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkGphG3nMl1iWmpLN-ATrwew5vl6oe21hW73FsMaFCEG-oFhQK
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHH2ri8bJj2dmQXId7_wMKpZcUSy9XFfdEeMMfKMAANQ9Jbbp
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHWJqg9PEmYGXrGLSMiYLQ3Pxf8AEc5bh3lVlXb-tyhqc5cZd
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIY3x12o48JDW8B56BK2Lfi1RpanL-J4rEyJPIy-N-RX0BwON
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkJ4A8cgkjbzK4O9OvK6zV_TprKmEjB6XHgGQ7IB9nU8MvHC97
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkJ9B9y4CrMBLwIZ0ZbU_WNoiDIE3OUASIbMQyz0ESR1CxHqkh
p5
0
<
mean
0.15
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkKiRu-lsXE5PLFJrt8iDuM1j9GdktGo5ftvBqTYrR3POLELOv
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkL-1azwk-z6OUvNB6QRA8Ztcd1hLFgsr49Tp_rzgksHlk-BEB
p5
0
<
mean
0.22
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMZKTT7jdgZMRl9ahCDh08bemWtb90LMaLW7liHvkmI30B4Ld
p5
0
<
mean
0.2
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMmJU1px2AERlRwQ1WWY-ChN1dRLtOlSvXiUSMlBL4kIoLIq0
p5
0
<
mean
0.24
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMvVG9JlDRJGVI_psnUflglLd6_0p4mtQD7s-B3XubNf7-OPg
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkOt8OWYb_f4D9yp0BZ6MqnqBSWuFq_3rtpfDQGAo5k8W32IE7
p5
0
<
mean
0.24
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkP4rC158530dbjavdllGSD4TucijuCVASVEBPB0yGLWrEEZkD
p5
0
<
mean
0.2
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkP_JMNkJs0zXsB4DMhnJTITnJbUzSe3huidmmHxGJyuWrqS-5
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQ6O1cPLUzjZuvvuoGJa3SD7Q_EKJoQ-Ne842BAsnOPFET7nS
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQ93ucWl0MMNTH-9CiY9Q4VRiOdkivfBFyJm3qK6fkrtNMhzO
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQzBkUSOSYq0WvJ8VacEvg43-FqtaVzxLpaFRvhRXtGvvtQ1k
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRHWdDMJ63VXsTrizm1j835ziPbWv-z1qXuTcrb8T0wX7OBC5
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRRPd_WJdYXgsuCzCRgSOsleB_KBgHLY58yQEhVfv9VExaJyh
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRokM_yQi8Qe6I4AE-OQcjIr2FTmVlI6Y93yf-JcRJQzor4tV
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkS0d1fb2rEmLbrEb3e0i97cY1wDHUXBI7vNiN09_hWdQRsuXM
p5
0
<
mean
0.29
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkS1pczsiSxXR9kRxaqryi8EWSK_Vd8zDSD_WM6wNWAmXzSAgx
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSJkBkDT6MazlXUL4IOHngnV1tHeRKwKRcls23lrdjbLIzhhg
p5
0
<
mean
0.16
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkU9SSfYhmemahRLLAlgCWPRfMmLYQkzwGpE4JQfNf6Erxu2Ey
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVCZO0-4HENpYhgYOiz5aop9c_oQmpdiDV15zZhMe2Fgvru1j
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVezuP62bsBWOePEONIBLB31JgM9m2_SR5IhNKh0jTbtAmzHn
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVlQE9ukfzvC4sQsZEzBcivz8DmFTegDWhh57hQTgLYMjEwRE
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVln4kDVHY-ZfrEuvSirgHyk_1lvR_dSkbNxiZ9T1NTuBulcP
p5
0
<
mean
0.24
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVuIAUGAfzOtGg8_o06D701jhxT89LnvMhXfzfcptBi-yFJ1N
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkWeL9sP9IoQfjBfyGXwpI8AT0vWtdVWIat1ALo9WdRa-QGSjb
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkX-L72HSV9250sxgelvUNLwVnie8EHUuBLQYg0CAxGQvJHljr
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXLYK1d9gK1_YUJVZeh8QqqpWEHGDtYQh7uv8gTVktaD-jm8u
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXpNKAlCsasC-V7ino8beZ3NWcUr6YKc4Xnu_nNDiy2lNLQQD
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkY1BWZHRHRoDexHVJQcoDCWSMi05clZvuoKkCi-uzm3up68Zu
p5
0
<
mean
0.19
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_r6MQzFBff-eblokqZHXn9W5xPPC4_wWpULVL_8A0M13xTDM
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkau-niwm7vG4OXjeE0Ez3_e-OMoFM2Cy0nBkNa2Hun8fv5bLz
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbVpl6zCnPA5alF-zFsB6OrUcCRXtNSZOeQ4PN7NdlOmPYvaw
p5
0
<
mean
0.03
SD
0.16
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkc2Q3OYmlFwbMPKvY2SOZ6soEUUxpDvzqKtu0R0zuKiJzqbhC
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcEC8Mb2clTGzYNdw39avvHroTstDLJ7kCBbmJQ5jeZJ2J5Pq
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkclEXq69ipAIjiALeF61feQWIiRxBOihhbFmyhXqUn96tPh-Q
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdyHTJPgKdAUR2cmc4eLs77tjKsTWhgdYcJeY5EQZMTscw3g5
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAke0FfwxU0QIA9t52G3BAn38fLM_h1HXqbft0_4cHvSQ9vCiyE
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkezWan_vwB_0tNBYZmLJLeBP-nM7NB4CAtSSmRpajTXMKdQQr
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkfwCuWhWtRMeWXoUqqK9FoxK5GfOBVKVt3DWvJvHei-9DvrJC
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkg5Mjp0ohEl3Vk1QjCTi2QVMilowvk6w2OW9ODcLfylqyGngs
p5
0
<
mean
0.22
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgkSAlMYPzlhlvkQHNr8TUZaaDWOFTPAK3rEC5-Wb-IVe4yx-
p5
0
<
mean
0.23
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkhTgLuiY3x3CUd_mcHkEMWxWKU3QYpoEBpYPtTYTU-XZZFtCm
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAki-9K71RZeX-FXBvMArpe2wzJqYcgtWvXckEFccxcS7UKfCBx
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkiN--p6b8Xa9ZdJ3oj-K8R0A85CpiEwA19CK_rUbcvzDs27l7
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkisQ4-1VE43ouFStvhTe-oD1YCsNXRXLYsb8x352VY8g9Rb9m
p5
0
<
mean
0.18
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkk95ho33tUSvzsJpq_yTW5Dpy5KI2gYAjASONVM43VRedI6Xz
p5
0
<
mean
0.15
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkJeGEyC1LL56R3OET_fIWeR9y-n0KTmiOGZlYoCJOZSMi8EK
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkV5hvdBEHANQUXAKnlQ2ISmFceREweOuZP4g5bIM5_LTYHjS
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkl-Kn7RaFtbMOnWCHGdsBDvZVGa_o_bPWc8AQQWN8BThr4sYE
p5
0
<
mean
0.22
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAklw45hJKb70hNuTwpEOPPt6Yyk3jKvlT6AqfepdUx1aAdAogH
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmq_HMfkQxVqHkHkeF4zOqdvn8hnBArgiwNwEGb9O-DgcAkTn
p5
0
<
mean
0.28
SD
0.45
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmziwUbB3fvrxHit5EkHby3BXAL7I7PSWqdLkBOuyXlXyjDSu
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkn0_LlWWpdYbj9n3T2fwHwXKczRdMELCjI-wvb5SohHL2-oj0
p5
0
<
mean
0.19
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAknXY-1PZLQQaO7bn20_eyOskxTNbnoTkWhAiFT42u0lShXoxl
p5
0
<
mean
0.14
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkp7TGD1qXigb3hfXMnuuxlipWAlVE1Buy1XidNszxyiNJdSY_
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpXjpGIdPC3BzGahgpaVH4n2ilYBfZOzMRtF7Ws0ZGDuR7a4h
p5
0
<
mean
0.17
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkqB7abqqmkDfDQduCPE3ULsURAMFBOg8FdXPVknJsQ8yIFD7m
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAksIymvkckRd0UsP1uuVNqQqpDY0NQYGpxBlbScG3Bp3plwQHR
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkswvulMwpkpBY4WtGtZvlKpx2wRgNk8k4y0xLj06nzCB6GNri
p5
0
<
mean
0.2
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAktyaLsuAGaCwlsiyPFC2_L-OAAqzqRyxmnn9lICpLpxIsxIyq
p5
0
<
mean
0.22
SD
0.42
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkvqvauTEdj4Uc-cw9HzEYsONmluuv1ooaigllrDh1uWPq4quD
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkvs2XkCiDV__5PxQwRsyhFfwQvcShDoH5SpqC5KgMHGoIeavx
p5
0
<
mean
0.26
SD
0.44
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkwJz-y1qYw1LR_f5kVk7qGv1nc97XBGo_1C62PQx5f4cOX4by
p5
0
<
mean
0.21
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxVVp8kXBLo9J1vUeGF7tXXkO52d_sfLRg5yej8VVl3AAjoFU
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxedCsrE1wOiUdqj_AQzWtsZejs2EEDwJXB2xIRlmX-HjqOt6
p5
0
<
mean
0.24
SD
0.43
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxouMQTwuQDpKa2ufxH9Z7_c_syxwOMlN8JBIY3wvKxj9Tnss
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxzccCLRPIKYgL6id2kt131Usk2tchngGbDHVDBvOOrKPZMG4
p5
0
<
mean
0.22
SD
0.41
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkybGVvHz__uZ-BvX4ubWrmHzsx1808i5BB0G3aX43z0PouKDH
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkytW_8zK3yXMI8PiEbED-cTIh3xXhZUbSCYxGvlAykmjwyDNS
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkz7dPCmvAu3TGv4kW7ET3-Qkd79a4uAUDqfvg5cZv-d8qzIvT
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzXVLDDrd6wuxx9gmrXcbYmtrHYto0vdB6LnBdqkhJJvFSE30
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzk6_IJE5H7Z0S1Gx96KKZQvxMQdlVYSsbl_MKMwg78-4WJai
p5
0
<
mean
0.15
SD
0.35
<
p95
1
Error count
The number of errors accumulated.
782
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.016905
s
SD
18.230609
s
p50
0.002066
s
<
p95
4.031406
s
<
p99
43.753008
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
6
over 1713.577306779s
mean rate
80.34
/s
std
242.39
/s
p5
0
/s
p95
399.73
/s
peak
5868.63
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007044
s
SD
0.003656
s
p50
0.006704
s
<
p95
0.010047
s
<
p99
0.026634
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006427
s
SD
0.00138
s
p50
0.006686
s
<
p95
0.009039
s
<
p99
0.009555
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.012861
s
SD
4.756067
s
p50
0.009368
s
<
p95
1.788132
s
<
p99
22.170284
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007534
s
SD
4.196054
s
p50
0.585653
s
<
p95
9.277932
s
<
p99
22.018972
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005793
s
SD
0.003342
s
p50
0.005516
s
<
p95
0.007857
s
<
p99
0.025428
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005186
s
SD
0.001118
s
p50
0.005421
s
<
p95
0.00718
s
<
p99
0.00758
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.018994
s
SD
18.654919
s
p50
0.004365
s
<
p95
3.815562
s
<
p99
43.907772
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007823
s
SD
4.802028
s
p50
0.583023
s
<
p95
10.772363
s
<
p99
22.020237
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001728
s
SD
0.000398
s
p50
0.001827
s
<
p95
0.002299
s
<
p99
0.002888
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.2e-05
s
SD
3.1e-05
s
p50
4.4e-05
s
<
p95
0.000126
s
<
p99
0.000141
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000564
s
SD
0.000147
s
p50
0.000581
s
<
p95
0.00088
s
<
p99
0.000955
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000527
s
SD
0.000189
s
p50
0.000496
s
<
p95
0.000897
s
<
p99
0.001102
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.017532
s
SD
18.654862
s
p50
0.001726
s
<
p95
3.813213
s
<
p99
43.906099
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.005116
s
SD
4.801723
s
p50
0.577942
s
<
p95
10.765994
s
<
p99
22.015177
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
6e-06
s
SD
9e-06
s
p50
1.5e-05
s
<
p95
2.5e-05
s
<
p99
6.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002686
s
SD
0.00081
s
p50
0.002798
s
<
p95
0.003756
s
<
p99
0.005631
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000355
s
SD
0.000287
s
p50
0.000373
s
<
p95
0.000666
s
<
p99
0.000881
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
3.34
s
<
mean
344.97
s
SD
509.47
s
<
p95
1517.11
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
26886
over 1788.100510107s
mean rate
1.13009938e+06
/s
std
2.855948905e+07
/s
p5
0
/s
p95
1.09004275e+06
/s
peak
2.17679100059e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
95.184054
s
SD
90.957855
s
p50
46.812356
s
<
p95
266.134945
s
<
p99
438.206569
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.122606
s
SD
0.165617
s
p50
0.031907
s
<
p95
0.425331
s
<
p99
0.792746
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
2e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.02407
s
SD
0.354297
s
p50
0.015081
s
<
p95
0.395768
s
<
p99
1.562517
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
1.257256
s
SD
1.39899
s
p50
0.12515
s
<
p95
4.149251
s
<
p99
6.873311
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
29.331105
s
SD
32.080074
s
p50
16.873802
s
<
p95
89.572132
s
<
p99
147.422715
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
12.043668
s
SD
19.26354
s
p50
0.021862
s
<
p95
31.842819
s
<
p99
117.659067
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004765
s
SD
0.002902
s
p50
0.000703
s
<
p95
0.008973
s
<
p99
0.01171
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007647
s
SD
0.118952
s
p50
0.003258
s
<
p95
0.028165
s
<
p99
0.282792
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000449
s
SD
0.001329
s
p50
0.000815
s
<
p95
0.003137
s
<
p99
0.006644
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.490531
s
SD
28.459184
s
p50
1.300647
s
<
p95
60.002439
s
<
p99
60.003065
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.412774
s
SD
2.507088
s
p50
0.784945
s
<
p95
9.140017
s
<
p99
11.219234
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.102164
s
SD
13.101541
s
p50
0.342874
s
<
p95
60.000173
s
<
p99
60.001993
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.188586
s
SD
28.26104
s
p50
0.857661
s
<
p95
60.001722
s
<
p99
60.003854
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
27.90
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
7.17
KiB/s
<
mean
226.29
KiB/s
SD
343.55
KiB/s
<
p95
671.62
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
55
B/s
<
mean
19.86
KiB/s
SD
46.31
KiB/s
<
p95
97.29
KiB/s
Total bytes received
Total bytes received on primary network interface
count
56.04
GiB
mean
32.06
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.92
GiB
mean
2.81
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 18% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.16
%
<
mean
37.56
%
SD
28.63
%
<
p95
88.25
%
System
9.66
%
CPU percentiles
CPU usage percentiles
p50
36.37
%
p95
88.25
%
p99
95.99
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
107
hosts
mean time
255.77
s
Memory used percentage
Percentage of memory used
p5
6.35
%
<
mean
9.54
%
SD
4.64
%
<
p95
16.94
%
Memory available percentage
Percentage of available memory
p5
83.06
%
<
mean
90.46
%
SD
4.64
%
<
p95
93.65
%
Max host memory used
Maximum memory usage percentage across all hosts
max
36.05
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.76
%
Memory growth rate
Rate of memory growth over time
growth
0.10
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.19
MiB/s
Disk write throughput
Disk write throughput in MiB/s
41.31
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.24
5 min
1.44
15 min
1.69
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
17.69
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
12.7806
%
SD
15.8829
%
<
p95
45.56
%
60 second average
12.7736
%
300 second average
15.4015
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2074
%
SD
2.4261
%
<
p95
5.47
%
60 second average
1.3187
%
300 second average
2.5229
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.7222
%
SD
1.6867
%
<
p95
3.23
%
60 second average
0.7670
%
300 second average
1.2607
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
35.4
%
SD
29.87
%
<
p95
86.8
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
49.96
KiB
<
mean
176.11
KiB
SD
98.80
KiB
<
p95
341.74
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
27.18
threads
SD
8.45
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
12
file descriptors
<
mean
73.63
file descriptors
SD
21.02
file descriptors
<
p95
88
file descriptors

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Thu, 17 Sep 2026 04:51:24 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_35174760384_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
249.2
Max
Highest chain head value observed for any single write agent.
1215
Write agents observed
10
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
182.061442
s
SD
41.588111
s
Max
Highest per-partition mean latency
max mean
1028.687385
s
Min
Lowest per-partition mean latency
min mean
6.247923
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.31
/s
Max
Highest per-partition mean rate
max
9
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.146854
s
SD
0.068293
s
Max
Highest per-partition mean latency
max mean
1.222664
s
Min
Lowest per-partition mean latency
min mean
0.011202
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
39638
Agents affected
85 / 85
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
466
Max
Highest count in any single partition
807
Min
Lowest count in any single partition
4
Final error count
The total number of all types of error accumulated over the run by all agents.
133383
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.0167
s
SD
2.616719
s
p50
0.003058
s
<
p95
3.263214
s
<
p99
16.492336
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 813.957534536s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 887.667350203s
Not enough time data to show a trend.
WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'
Metered usage count of the WASM ribosome for this zome function.
total
1154668603
over 887.667350203s
mean rate
3.21706754931e+10
/s
std
2.95166352887e+11
/s
p5
3782.35
/s
p95
3.81317103859e+10
/s
peak
9.02243834399e+12
/s
WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
19451398
over 888.572455521s
mean rate
3.0126315189e+08
/s
std
1.45459950012e+09
/s
p5
380174.09
/s
p95
1.3508849881e+09
/s
peak
5.27792160777e+10
/s
WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Metered usage count of the WASM ribosome for this zome function.
total
11262
over 889.259896242s
mean rate
1.3731460078e+10
/s
std
4.24994052819e+11
/s
p5
0
/s
p95
3.13297328606e+10
/s
peak
2.82196679952e+13
/s
WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
1175324
over 889.259896242s
mean rate
1.5248099694e+08
/s
std
1.3045105098e+09
/s
p5
32730.02
/s
p95
4.9134596912e+08
/s
peak
5.22510892694e+10
/s
WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
97966980
over 889.259896242s
mean rate
1.65157339974e+10
/s
std
4.24989217924e+11
/s
p5
1.811261616e+07
/s
p95
2.81865715163e+10
/s
peak
2.75614869725e+13
/s
WASM usage: 'validated_must_get_agent_activity_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
69444579
over 889.259896242s
mean rate
1.03550045353e+11
/s
std
1.65318833215e+12
/s
p5
4.868927314e+07
/s
p95
2.60898619772e+11
/s
peak
1.06916508893e+14
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007378
s
SD
0.00262
s
p50
0.007002
s
<
p95
0.010781
s
<
p99
0.018788
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.325308
s
SD
0.232151
s
p50
0.230707
s
<
p95
0.72188
s
<
p99
0.931303
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.014653
s
SD
0.008451
s
p50
0.009253
s
<
p95
0.029668
s
<
p99
0.042772
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.232818
s
SD
3.542424
s
p50
0.007537
s
<
p95
4.610267
s
<
p99
23.69682
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.180458
s
SD
2.73359
s
p50
0.008953
s
<
p95
8.708444
s
<
p99
11.009984
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006208
s
SD
0.002502
s
p50
0.005852
s
<
p95
0.0087
s
<
p99
0.017978
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.322622
s
SD
0.231061
s
p50
0.22829
s
<
p95
0.717654
s
<
p99
0.926501
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012395
s
SD
0.007559
s
p50
0.007448
s
<
p95
0.025973
s
<
p99
0.038499
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011759
s
SD
3.465007
s
p50
0.007037
s
<
p95
4.202181
s
<
p99
19.002326
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.229752
s
SD
3.236973
s
p50
0.007369
s
<
p95
9.163919
s
<
p99
13.200633
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002105
s
SD
0.000935
s
p50
0.002033
s
<
p95
0.003914
s
<
p99
0.004758
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.01588
s
SD
0.026961
s
p50
0.011888
s
<
p95
0.0521
s
<
p99
0.072928
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
9.6e-05
s
SD
6.6e-05
s
p50
4.5e-05
s
<
p95
0.000223
s
<
p99
0.000294
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.007315
s
SD
0.006932
s
p50
0.001573
s
<
p95
0.019765
s
<
p99
0.031955
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.010895
s
SD
0.009048
s
p50
0.000683
s
<
p95
0.025463
s
<
p99
0.037666
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.009942
s
SD
3.359869
s
p50
0.000887
s
<
p95
7.587305
s
<
p99
18.996222
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.017206
s
SD
0.093229
s
p50
0.013301
s
<
p95
0.059094
s
<
p99
0.089383
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.112894
s
SD
0.074548
s
p50
0.083249
s
<
p95
0.216602
s
<
p99
0.254843
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.9e-05
s
SD
8e-06
s
p50
1.3e-05
s
<
p95
2.9e-05
s
<
p99
6.6e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00449
s
SD
0.002594
s
p50
0.003477
s
<
p95
0.00989
s
<
p99
0.013521
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002489
s
SD
0.001771
s
p50
0.000504
s
<
p95
0.004654
s
<
p99
0.00841
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.56
s
<
mean
430.96
s
SD
249.19
s
<
p95
817.84
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
4207
over 889.25988931s
mean rate
1.03962172e+06
/s
std
1.390560611e+07
/s
p5
698.74
/s
p95
2.47187627e+06
/s
peak
9.5991012299e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
81.162417
s
SD
52.999468
s
p50
24.294237
s
<
p95
146.016651
s
<
p99
276.130729
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
1.915647
s
SD
0.961952
s
p50
1.95439
s
<
p95
3.516859
s
<
p99
4.54336
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
1e-06
s
SD
1e-05
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
1.012492
s
SD
0.52114
s
p50
0.533953
s
<
p95
1.490641
s
<
p99
1.649571
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.61203
s
SD
0.505658
s
p50
0.107637
s
<
p95
1.371566
s
<
p99
2.249926
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
13.285081
s
SD
17.048493
s
p50
15.875872
s
<
p95
51.210891
s
<
p99
69.88283
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
11.031117
s
SD
20.18249
s
p50
1.941601
s
<
p95
47.366404
s
<
p99
89.669643
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004937
s
SD
0.003543
s
p50
0.004384
s
<
p95
0.011342
s
<
p99
0.013191
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006283
s
SD
0.003701
s
p50
0.004094
s
<
p95
0.012408
s
<
p99
0.016398
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.005108
s
SD
0.006243
s
p50
0.0016
s
<
p95
0.016885
s
<
p99
0.028965
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.388799
s
SD
28.321041
s
p50
0.987579
s
<
p95
60.004815
s
<
p99
60.013337
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
3.31256
s
SD
4.005366
s
p50
1.868841
s
<
p95
10.381902
s
<
p99
20.312111
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
1.019323
s
SD
2.722342
s
p50
0.71029
s
<
p95
10.592913
s
≤
p99
10.592913
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.028563
s
SD
26.447494
s
p50
2.092309
s
<
p95
60.016613
s
<
p99
60.04252
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
44.76
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
159
B/s
<
mean
825.75
KiB/s
SD
606.54
KiB/s
<
p95
1.70
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
78.37
KiB/s
SD
91.93
KiB/s
<
p95
290.62
KiB/s
Total bytes received
Total bytes received on primary network interface
count
101.46
GiB
mean
116.74
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.63
GiB
mean
11.08
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 85% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
10.99
%
<
mean
66.26
%
SD
23.78
%
<
p95
91.52
%
System
21.50
%
CPU percentiles
CPU usage percentiles
p50
72.12
%
p95
91.52
%
p99
97.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
137
hosts
mean time
347.60
s
Memory used percentage
Percentage of memory used
p5
7.43
%
<
mean
12.33
%
SD
4.89
%
<
p95
19.87
%
Memory available percentage
Percentage of available memory
p5
80.13
%
<
mean
87.67
%
SD
4.89
%
<
p95
92.57
%
Max host memory used
Maximum memory usage percentage across all hosts
max
39.15
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.76
%
Memory growth rate
Rate of memory growth over time
growth
0.34
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.52
MiB/s
Disk write throughput
Disk write throughput in MiB/s
83.97
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.55
5 min
2.81
15 min
2.13
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
84.93
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.88
%
<
mean
47.5454
%
SD
28.3897
%
<
p95
80.45
%
60 second average
44.4424
%
300 second average
34.4783
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
10.5716
%
SD
8.6492
%
<
p95
24.52
%
60 second average
9.9500
%
300 second average
7.2224
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.907
%
SD
3.8979
%
<
p95
11.2
%
60 second average
3.7245
%
300 second average
2.8510
%
Holochain process CPU usage
CPU usage by Holochain process
p5
12.43
%
<
mean
63.43
%
SD
21.69
%
<
p95
87.43
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
90.79
KiB
<
mean
364.42
KiB
SD
117.64
KiB
<
p95
516.93
KiB
Holochain process threads
Number of threads in Holochain process
p5
13
threads
<
mean
33.94
threads
SD
6.95
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
42
file descriptors
<
mean
79.54
file descriptors
SD
15.98
file descriptors
<
p95
89
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Thu, 17 Sep 2026 06:17:09 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_35174760384_1
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
47.69
/s
Max
Highest per-partition mean rate
max
85.77
/s
Min
Lowest per-partition mean rate
min
33.1
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
1.1963123477499999e+29
s
SD
1.52617621106e+30
s
Max
Highest per-partition mean latency
max mean
2.34802057854e+30
s
Min
Lowest per-partition mean latency
min mean
3.132065
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
39.52
/s
Max
Highest per-partition mean rate
max
257.77
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0014
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
244506
Agents affected
57 / 57
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4290
Max
Highest count in any single partition
7921
Min
Lowest count in any single partition
2927
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
18314
Agents affected
46 / 55
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
333
Max
Highest count in any single partition
3361
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
28
<
mean
70.52
SD
41.3
<
p95
146
zero-arc
p5
3
<
mean
23.15
SD
14.2
<
p95
51
Error count
The number of errors accumulated across all nodes.
752
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.101307
s
SD
1.064657
s
p50
0.337487
s
<
p95
1.870458
s
<
p99
3.583016
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 760.012055641s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
442004719
over 880.684219154s
mean rate
7.146943374e+08
/s
std
1.56225071527e+09
/s
p5
5.43758966e+06
/s
p95
5.12560209308e+09
/s
peak
9.61673417555e+09
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
132609934
over 878.766468807s
mean rate
1.2046206873e+10
/s
std
4.63538411075e+10
/s
p5
9.82431578e+06
/s
p95
5.74099920512e+10
/s
peak
5.39524805949e+11
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
82536
over 888.766612847s
mean rate
1.62727732009e+10
/s
std
6.46252290362e+11
/s
p5
0
/s
p95
3.64869147637e+09
/s
peak
4.50242823529e+13
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009019
s
SD
0.003303
s
p50
0.009199
s
<
p95
0.014636
s
<
p99
0.021017
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
12.32608
s
SD
74.04
s
p50
42.460789
s
<
p95
215.969584
s
<
p99
341.735634
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007874
s
SD
0.002934
s
p50
0.007986
s
<
p95
0.012788
s
<
p99
0.01882
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
12.324858
s
SD
74.040004
s
p50
42.459204
s
<
p95
215.968189
s
<
p99
341.73418
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001496
s
SD
0.000479
s
p50
0.001774
s
<
p95
0.002513
s
<
p99
0.002998
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000669
s
SD
0.000645
s
p50
0.000599
s
<
p95
0.001534
s
<
p99
0.003661
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000654
s
SD
0.000528
s
p50
0.000581
s
<
p95
0.001543
s
<
p99
0.002904
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.099928
s
SD
1.151646
s
p50
0.305609
s
<
p95
1.787623
s
<
p99
3.098967
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.419702
s
SD
2.591292
s
p50
0.555459
s
<
p95
5.967673
s
<
p99
13.026939
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002358
s
SD
0.000782
s
p50
0.002864
s
<
p95
0.004158
s
<
p99
0.004764
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000481
s
SD
0.000544
s
p50
0.000326
s
<
p95
0.001558
s
<
p99
0.003029
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.49
s
<
mean
430.24
s
SD
248.76
s
<
p95
815.48
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.766698342s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
35.424736
s
SD
38.72022
s
p50
0
s
<
p95
25.898012
s
<
p99
177.40961
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.080004
s
SD
0.167441
s
p50
0.002484
s
<
p95
0.142183
s
<
p99
0.376714
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.031187
s
SD
0.328624
s
p50
0.002213
s
<
p95
1.103478
s
<
p99
1.241188
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.321589
s
SD
0.194188
s
p50
0.03019
s
<
p95
0.552917
s
<
p99
0.692337
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.379006
s
SD
7.786777
s
p50
0.003577
s
<
p95
16.893482
s
<
p99
32.465067
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
6.198415
s
SD
14.271193
s
p50
0.001245
s
<
p95
6.26467
s
<
p99
70.303431
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00244
s
SD
0.00175
s
p50
0.001182
s
<
p95
0.004826
s
<
p99
0.006031
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00413
s
SD
0.004482
s
p50
0.003801
s
<
p95
0.01064
s
<
p99
0.02841
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000918
s
SD
0.004132
s
p50
0.0007
s
<
p95
0.007127
s
<
p99
0.033155
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.3141
s
SD
29.593508
s
p50
0.881782
s
<
p95
60.002747
s
<
p99
60.025778
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.158689
s
SD
12.875886
s
p50
0.218716
s
<
p95
7.418739
s
<
p99
60.00135
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.410419
s
SD
15.057976
s
p50
0.51361
s
<
p95
60.000818
s
<
p99
60.006987
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
31.34
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
9.15
KiB/s
<
mean
519.49
KiB/s
SD
570.50
KiB/s
<
p95
1.38
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
602
B/s
<
mean
56.19
KiB/s
SD
118.76
KiB/s
<
p95
232.64
KiB/s
Total bytes received
Total bytes received on primary network interface
count
63.29
GiB
mean
72.81
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
6.85
GiB
mean
7.88
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 92.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 32% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.52
%
<
mean
43.18
%
SD
34.25
%
<
p95
86.89
%
System
11.84
%
CPU percentiles
CPU usage percentiles
p50
58.52
%
p95
86.89
%
p99
92.16
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
111
hosts
mean time
265.54
s
Memory used percentage
Percentage of memory used
p5
6.8
%
<
mean
11.04
%
SD
4.67
%
<
p95
17.84
%
Memory available percentage
Percentage of available memory
p5
82.16
%
<
mean
88.96
%
SD
4.67
%
<
p95
93.2
%
Max host memory used
Maximum memory usage percentage across all hosts
max
37.18
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.76
%
Memory growth rate
Rate of memory growth over time
growth
0.48
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.46
MiB/s
Disk write throughput
Disk write throughput in MiB/s
105.97
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.57
5 min
1.69
15 min
1.87
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
32.19
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.02
%
<
mean
15.2848
%
SD
18.3057
%
<
p95
51.88
%
60 second average
15.1246
%
300 second average
17.3310
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.6799
%
SD
5.1876
%
<
p95
14.61
%
60 second average
2.6433
%
300 second average
2.8387
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.3745
%
SD
3.2966
%
<
p95
7.02
%
60 second average
1.3553
%
300 second average
1.5367
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.1
%
<
mean
38.76
%
SD
31.8
%
<
p95
80.04
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
201.19
KiB
<
mean
267.24
KiB
SD
70.62
KiB
<
p95
395.19
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
29.69
threads
SD
5.53
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
60
file descriptors
<
mean
74.21
file descriptors
SD
12.82
file descriptors
<
p95
89
file descriptors

Zero-Arc Create Data

↑ Back to index
A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 17 Sep 2026 05:25:47 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data--high_zero_35174760384_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
47.55
/s
Max
Highest per-partition mean rate
max
87.54
/s
Min
Lowest per-partition mean rate
min
32.16
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
173.868224
s
SD
52.750117
s
Max
Highest per-partition mean latency
max mean
639.225589
s
Min
Lowest per-partition mean latency
min mean
1.580378
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
91.25
/s
Max
Highest per-partition mean rate
max
365
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0005
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
578030
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4189
Max
Highest count in any single partition
7799
Min
Lowest count in any single partition
2833
Received count
Total number of entries received by full-arc nodes, by agent.
Total
2425
Agents affected
4 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
269
Max
Highest count in any single partition
2047
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
15
<
mean
45.98
SD
18.62
<
p95
76
zero-arc
p5
1
<
mean
19.56
SD
18.16
<
p95
52
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.004146
s
SD
0.001615
s
p50
0.001181
s
<
p95
0.005313
s
<
p99
0.006343
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 590.005350889s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
310295898
over 833.665162432s
mean rate
7.77707881623e+09
/s
std
1.32859769412e+11
/s
p5
9.50412362e+06
/s
p95
1.19450560472e+10
/s
peak
9.7612593056e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
3039720736
over 842.549648088s
mean rate
9.3895709653e+08
/s
std
1.24932704873e+09
/s
p5
3.87469436e+06
/s
p95
3.43418384506e+09
/s
peak
5.41704540107e+09
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
39738912
over 889.667134736s
mean rate
9.9493608675e+08
/s
std
1.60140031977e+10
/s
p5
0
/s
p95
1.98884631424e+09
/s
peak
1.24463684704e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008511
s
SD
0.002854
s
p50
0.008982
s
<
p95
0.013899
s
<
p99
0.01459
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.047279
s
SD
0.066265
s
p50
0.016042
s
<
p95
0.206149
s
<
p99
0.253762
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007419
s
SD
0.002468
s
p50
0.007842
s
<
p95
0.012091
s
<
p99
0.012725
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.045644
s
SD
0.066303
s
p50
0.01424
s
<
p95
0.204643
s
<
p99
0.252252
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001515
s
SD
0.000505
s
p50
0.001717
s
<
p95
0.002368
s
<
p99
0.003139
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000495
s
SD
0.000186
s
p50
0.000537
s
<
p95
0.00083
s
<
p99
0.000917
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000487
s
SD
0.000196
s
p50
0.000516
s
<
p95
0.00085
s
<
p99
0.000924
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.004469
s
SD
0.001534
s
p50
0.003325
s
<
p95
0.005998
s
<
p99
0.006526
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002326
s
SD
0.002365
s
p50
0.001235
s
<
p95
0.00793
s
<
p99
0.009306
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002423
s
SD
0.001258
s
p50
0.002679
s
<
p95
0.003949
s
<
p99
0.008821
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000414
s
SD
0.000273
s
p50
0.000476
s
<
p95
0.000962
s
<
p99
0.001169
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.81
s
<
mean
431.14
s
SD
249.28
s
<
p95
818.09
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.667055875s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
3.230617
s
SD
20.473439
s
p50
0
s
<
p95
0.866667
s
<
p99
30.097413
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.110218
s
SD
0.190372
s
p50
0.002107
s
<
p95
0.059054
s
<
p99
0.361294
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.008834
s
SD
0.257365
s
p50
0.00193
s
<
p95
0.755978
s
<
p99
1.28531
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.224302
s
SD
0.127745
s
p50
0.226354
s
<
p95
0.337862
s
<
p99
0.377014
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.431857
s
SD
7.063245
s
p50
0.003134
s
<
p95
2.355412
s
<
p99
15.453282
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.570426
s
SD
3.166041
s
p50
0.001334
s
<
p95
0.009152
s
<
p99
0.448965
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.007197
s
SD
0.004394
s
p50
0.007209
s
<
p95
0.014803
s
<
p99
0.016733
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.008217
s
SD
0.004791
s
p50
0.004106
s
<
p95
0.015051
s
<
p99
0.016864
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000401
s
SD
0.002339
s
p50
0.000461
s
<
p95
0.000963
s
<
p99
0.003015
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
6.765792
s
SD
28.573349
s
p50
2.332963
s
<
p95
60.029322
s
≤
p99
60.029322
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
5.092913
s
SD
22.545904
s
p50
2.221456
s
<
p95
60.069508
s
<
p99
60.084554
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
53.43
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
1.49
KiB/s
<
mean
343.68
KiB/s
SD
303.90
KiB/s
<
p95
844.69
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
39.54
KiB/s
SD
55.18
KiB/s
<
p95
145.93
KiB/s
Total bytes received
Total bytes received on primary network interface
count
42.01
GiB
mean
48.34
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.83
GiB
mean
5.56
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.6%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 51% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
17.21
%
<
mean
69.41
%
SD
20.89
%
<
p95
88.11
%
System
15.97
%
CPU percentiles
CPU usage percentiles
p50
75.68
%
p95
88.11
%
p99
94.56
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
113
hosts
mean time
462.16
s
Memory used percentage
Percentage of memory used
p5
6.54
%
<
mean
10.81
%
SD
4.47
%
<
p95
17.5
%
Memory available percentage
Percentage of available memory
p5
82.5
%
<
mean
89.19
%
SD
4.47
%
<
p95
93.46
%
Max host memory used
Maximum memory usage percentage across all hosts
max
35.44
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.76
%
Memory growth rate
Rate of memory growth over time
growth
0.38
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
76.39
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.23
5 min
2.11
15 min
2.29
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
50.68
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.62
%
<
mean
24.1761
%
SD
13.8546
%
<
p95
46.95
%
60 second average
23.0696
%
300 second average
22.1079
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.5026
%
SD
4.5643
%
<
p95
12.88
%
60 second average
4.2970
%
300 second average
3.9784
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.9834
%
SD
3.2819
%
<
p95
7.21
%
60 second average
2.8369
%
300 second average
2.5112
%
Holochain process CPU usage
CPU usage by Holochain process
p5
26.84
%
<
mean
66.96
%
SD
18.72
%
<
p95
84.63
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
204.82
KiB
<
mean
270.86
KiB
SD
53.86
KiB
<
p95
364.17
KiB
Holochain process threads
Number of threads in Holochain process
p5
25
threads
<
mean
31.57
threads
SD
3.76
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
67
file descriptors
<
mean
78.74
file descriptors
SD
8
file descriptors
<
p95
87
file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 17 Sep 2026 06:00:01 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data_validated--high_zero_35174760384_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
42.19
/s
Max
Highest per-partition mean rate
max
81.83
/s
Min
Lowest per-partition mean rate
min
27.4
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
111.95516
s
SD
4.504264
s
Max
Highest per-partition mean latency
max mean
381.148639
s
Min
Lowest per-partition mean latency
min mean
5.168723
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0002
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
499958
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3623
Max
Highest count in any single partition
7036
Min
Lowest count in any single partition
2357
Received count
Total number of entries received by full-arc nodes, by agent.
Total
1059
Agents affected
6 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
118
Max
Highest count in any single partition
528
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
13
<
mean
48.76
SD
30.15
<
p95
97
zero-arc
p5
2
<
mean
20.17
SD
17.84
<
p95
52
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.007744
s
SD
0.00521
s
p50
0.009958
s
<
p95
0.01914
s
<
p99
0.020796
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
380
over 771.874613077s
mean rate
26225.4
/s
std
30215.89
/s
p5
10.84
/s
p95
100397.18
/s
peak
121943.01
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
201522678
over 828.355086477s
mean rate
2.24297989014e+09
/s
std
9.46035320166e+09
/s
p5
6.29963667e+06
/s
p95
9.1979422206e+09
/s
peak
5.22676385451e+11
/s
WASM usage: 'timed_and_validated::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
340998145
over 884.158903441s
mean rate
8.606765349e+08
/s
std
1.3437346264e+09
/s
p5
1.023963314e+07
/s
p95
4.00223237511e+09
/s
peak
6.0011724138e+09
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
111709584
over 888.398883771s
mean rate
1.30638583102e+09
/s
std
4.97341664801e+09
/s
p5
0
/s
p95
5.33198757068e+09
/s
peak
2.8672772997e+11
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
934661272
over 888.355706941s
mean rate
1.25296819757e+10
/s
std
4.43523572199e+10
/s
p5
1.242335169e+07
/s
p95
5.09516024286e+10
/s
peak
2.39980614824e+12
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009051
s
SD
0.003257
s
p50
0.009311
s
<
p95
0.015419
s
<
p99
0.016606
s
Zome call duration: 'timed_and_validated::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.032482
s
SD
0.30465
s
p50
0.018271
s
<
p95
0.981595
s
<
p99
1.230058
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007876
s
SD
0.002819
s
p50
0.00807
s
<
p95
0.013381
s
<
p99
0.014394
s
WASM call duration: 'timed_and_validated::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.031389
s
SD
0.30406
s
p50
0.016138
s
<
p95
0.978185
s
<
p99
1.226541
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001523
s
SD
0.000617
s
p50
0.001652
s
<
p95
0.002504
s
<
p99
0.004227
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006723
s
SD
0.003531
s
p50
0.006462
s
<
p95
0.010961
s
<
p99
0.021443
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000502
s
SD
0.000203
s
p50
0.000512
s
<
p95
0.000913
s
<
p99
0.001043
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000495
s
SD
0.000213
s
p50
0.000485
s
<
p95
0.000932
s
<
p99
0.00103
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.005038
s
SD
0.001782
s
p50
0.003594
s
<
p95
0.007331
s
<
p99
0.007684
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001631
s
SD
0.016549
s
p50
0.001966
s
<
p95
0.05464
s
<
p99
0.060615
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.009364
s
SD
0.004995
s
p50
0.010434
s
<
p95
0.019303
s
<
p99
0.020919
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002559
s
SD
0.001842
s
p50
0.002587
s
<
p95
0.004521
s
<
p99
0.011651
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000556
s
SD
0.00075
s
p50
0.000642
s
<
p95
0.00137
s
<
p99
0.001533
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
43.4
s
<
mean
430.38
s
SD
248.91
s
<
p95
816.44
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.398788461s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
3.103016
s
SD
25.390502
s
p50
0
s
<
p95
0.680851
s
<
p99
88.731959
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.84756
s
SD
1.79518
s
p50
0.002197
s
<
p95
1.260838
s
<
p99
10.010119
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.009284
s
SD
0.227146
s
p50
0.002366
s
<
p95
0.423428
s
<
p99
1.140743
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.161176
s
SD
0.072845
s
p50
0.140357
s
<
p95
0.245798
s
<
p99
0.288511
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.653132
s
SD
8.437499
s
p50
0.003548
s
<
p95
3.881622
s
<
p99
35.966582
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.849911
s
SD
10.993693
s
p50
0.001338
s
<
p95
0.013268
s
<
p99
7.041307
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.005712
s
SD
0.003275
s
p50
0.006059
s
<
p95
0.011052
s
<
p99
0.012302
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007138
s
SD
0.004152
s
p50
0.004216
s
<
p95
0.013101
s
<
p99
0.018328
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000552
s
SD
0.005669
s
p50
0.000431
s
<
p95
0.001056
s
<
p99
0.037704
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
35.737469
s
SD
34.989526
s
p50
18.007296
s
<
p95
63.053746
s
<
p99
154.548777
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.229968
s
SD
1.199709
s
p50
0.034467
s
<
p95
3.64179
s
≤
p99
3.64179
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
48.11
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
127
B/s
<
mean
389.07
KiB/s
SD
346.32
KiB/s
<
p95
1.02
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
36.33
KiB/s
SD
45.67
KiB/s
<
p95
117.61
KiB/s
Total bytes received
Total bytes received on primary network interface
count
47.86
GiB
mean
55.06
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.47
GiB
mean
5.14
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 42% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
16.24
%
<
mean
65.86
%
SD
19.27
%
<
p95
83.76
%
System
17.75
%
CPU percentiles
CPU usage percentiles
p50
71.69
%
p95
83.76
%
p99
89.33
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
120
hosts
mean time
189.55
s
Memory used percentage
Percentage of memory used
p5
7.82
%
<
mean
12.51
%
SD
4.84
%
<
p95
19.88
%
Memory available percentage
Percentage of available memory
p5
80.12
%
<
mean
87.49
%
SD
4.84
%
<
p95
92.18
%
Max host memory used
Maximum memory usage percentage across all hosts
max
39.22
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.76
%
Memory growth rate
Rate of memory growth over time
growth
0.38
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.02
MiB/s
Disk write throughput
Disk write throughput in MiB/s
73.50
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/2
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.11
5 min
2.02
15 min
2.08
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
41.78
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.84
%
<
mean
23.83
%
SD
13.5554
%
<
p95
40.89
%
60 second average
22.8175
%
300 second average
22.1810
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.5568
%
SD
3.9398
%
<
p95
9.77
%
60 second average
3.4203
%
300 second average
3.3058
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.2141
%
SD
2.7616
%
<
p95
5.36
%
60 second average
2.1150
%
300 second average
1.9746
%
Holochain process CPU usage
CPU usage by Holochain process
p5
27.49
%
<
mean
63.46
%
SD
17.31
%
<
p95
79.79
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
173.26
KiB
<
mean
372.46
KiB
SD
92.22
KiB
<
p95
492.18
KiB
Holochain process threads
Number of threads in Holochain process
p5
25
threads
<
mean
32.82
threads
SD
4.76
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
65
file descriptors
<
mean
78.69
file descriptors
SD
10.43
file descriptors
<
p95
87
file descriptors