DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Thu, 10 Sep 2026 12:27:55 UTC
Duration
1800s
Peer count
150
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_34428490451_2
Create rate
The average number of created records per agent.
Mean
mean
15.38
/s
Max
Highest per-partition mean rate
max
39.52
/s
Min
Lowest per-partition mean rate
min
4.14
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
498.633833
s
SD
335.004344
s
Max
Highest per-partition mean latency
max mean
1262.945516
s
Min
Lowest per-partition mean latency
min mean
13.556744
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
69.94
/s
Max
Highest per-partition mean rate
max
1015.5
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0038
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
357383
Agents affected
125 / 125
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2859
Max
Highest count in any single partition
7337
Min
Lowest count in any single partition
733
Received count
The total number of entries received by record_lag agents, by agent.
Total
30970
Agents affected
23 / 23
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1347
Max
Highest count in any single partition
6899
Min
Lowest count in any single partition
1
Error count
The number of errors encountered during the scenario.
69
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.0012
s
SD
0.001939
s
p50
0.001674
s
<
p95
0.006125
s
<
p99
0.008983
s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
277930823
over 1790.354868453s
mean rate
8.10475431503e+09
/s
std
5.64342534176e+10
/s
p5
5.023786249e+07
/s
p95
2.04744798935e+10
/s
peak
3.17657195902e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
23326689876
over 1788.710526743s
mean rate
1.18735198163e+11
/s
std
8.03783923845e+11
/s
p5
2.191882917e+08
/s
p95
3.90732459596e+11
/s
peak
3.30109885218e+13
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
35438352
over 1790.354868453s
mean rate
2.7696892896e+09
/s
std
5.8783594213e+10
/s
p5
7.43857332e+06
/s
p95
4.6891209564e+09
/s
peak
4.35500675676e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.024427
s
SD
0.040604
s
p50
0.02143
s
<
p95
0.122195
s
<
p99
0.173284
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.12846
s
SD
0.171945
s
p50
0.111217
s
<
p95
0.492171
s
<
p99
0.908532
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.022912
s
SD
0.03922
s
p50
0.019524
s
<
p95
0.117582
s
<
p99
0.166445
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.127366
s
SD
0.171952
s
p50
0.110041
s
<
p95
0.490536
s
<
p99
0.906607
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001858
s
SD
0.001316
s
p50
0.002015
s
<
p95
0.005059
s
<
p99
0.006625
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.006092
s
SD
0.013919
s
p50
0.004115
s
<
p95
0.039491
s
<
p99
0.058301
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.006055
s
SD
0.01427
s
p50
0.003919
s
<
p95
0.040769
s
<
p99
0.058983
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001194
s
SD
0.002105
s
p50
0.001749
s
<
p95
0.006406
s
<
p99
0.009956
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.005012
s
SD
0.006873
s
p50
0.005011
s
<
p95
0.016581
s
<
p99
0.041606
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.004092
s
SD
0.004887
s
p50
0.00394
s
<
p95
0.01587
s
<
p99
0.0221
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002034
s
SD
0.004183
s
p50
0.001124
s
<
p95
0.012042
s
<
p99
0.017692
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
86.55
s
<
mean
874.69
s
SD
506.2
s
<
p95
1664.88
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
16496
over 1790.354871707s
mean rate
605301.88
/s
std
7.49415292e+06
/s
p5
6.38
/s
p95
1.19957117e+06
/s
peak
5.6753536858e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
166.199415
s
SD
141.825668
s
p50
13.767548
s
<
p95
404.668983
s
<
p99
647.953468
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.530492
s
SD
1.364005
s
p50
0.356571
s
<
p95
2.566124
s
<
p99
7.242339
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.212899
s
SD
0.486992
s
p50
0.155148
s
<
p95
1.368221
s
<
p99
1.762745
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
2.109838
s
SD
3.112577
s
p50
1.704873
s
<
p95
9.008136
s
<
p99
12.803222
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
48.946965
s
SD
57.984159
s
p50
36.511585
s
<
p95
175.867897
s
<
p99
240.673247
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
13.880655
s
SD
43.531641
s
p50
0.050855
s
<
p95
93.649472
s
<
p99
224.163965
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003232
s
SD
0.003751
s
p50
0.003895
s
<
p95
0.012336
s
<
p99
0.014917
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006745
s
SD
0.00856
s
p50
0.004299
s
<
p95
0.026021
s
<
p99
0.037007
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.006292
s
SD
0.015669
s
p50
0.001772
s
<
p95
0.044223
s
<
p99
0.075074
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
4.189642
s
SD
28.500522
s
p50
5.081388
s
<
p95
60.059322
s
<
p99
60.216029
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.73974
s
SD
25.044391
s
p50
1.364631
s
<
p95
60.037907
s
<
p99
60.084075
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
56.08
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
73.34
KiB/s
<
mean
830.85
KiB/s
SD
670.00
KiB/s
<
p95
2.06
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
4.25
KiB/s
<
mean
170.11
KiB/s
SD
253.72
KiB/s
<
p95
525.68
KiB/s
Total bytes received
Total bytes received on primary network interface
count
206.16
GiB
mean
117.93
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
41.20
GiB
mean
23.57
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/18 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (12.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 80% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
38.08
%
<
mean
76.62
%
SD
19.16
%
<
p95
95.31
%
System
20.53
%
CPU percentiles
CPU usage percentiles
p50
84.00
%
p95
95.31
%
p99
98.25
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
137
hosts
mean time
1158.64
s
Memory used percentage
Percentage of memory used
p5
6.88
%
<
mean
12.82
%
SD
8.48
%
<
p95
28.72
%
Memory available percentage
Percentage of available memory
p5
71.28
%
<
mean
87.18
%
SD
8.48
%
<
p95
93.12
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.80
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
12.71
%
Memory growth rate
Rate of memory growth over time
growth
0.39
MiB/s
Disk read throughput
Disk read throughput in MiB/s
6.44
MiB/s
Disk write throughput
Disk write throughput in MiB/s
235.87
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/18
hosts
Mount Point /etc/hosts
1/18
hosts
Mount Point /etc/nomad.d/client-meta.json
0/21
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/18
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.91
5 min
3.42
15 min
2.42
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
79.59
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.36
%
<
mean
44.2435
%
SD
23.2168
%
<
p95
74.06
%
60 second average
43.0460
%
300 second average
37.6457
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
7.9066
%
SD
7.6104
%
<
p95
18.4
%
60 second average
7.7003
%
300 second average
6.7010
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.4673
%
SD
4.0529
%
<
p95
8.35
%
60 second average
2.3979
%
300 second average
2.1177
%
Holochain process CPU usage
CPU usage by Holochain process
p5
41.23
%
<
mean
73.14
%
SD
15.83
%
<
p95
92.04
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
228.90
KiB
<
mean
424.43
KiB
SD
155.02
KiB
<
p95
707.60
KiB
Holochain process threads
Number of threads in Holochain process
p5
30
threads
<
mean
35.43
threads
SD
5.87
threads
<
p95
48
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
78
file descriptors
<
mean
84.95
file descriptors
SD
11.18
file descriptors
<
p95
91
file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 10 Sep 2026 16:12:46 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_34428490451_2
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
38.81
/s
Max
Highest per-partition mean rate
max
56.47
/s
Min
Lowest per-partition mean rate
min
21.33
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
304.148147
s
SD
3.069914
s
Max
Highest per-partition mean latency
max mean
1261.491675
s
Min
Lowest per-partition mean latency
min mean
14.619473
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0003
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
30972
Agents affected
9 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3441
Max
Highest count in any single partition
5208
Min
Lowest count in any single partition
1795
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
1223
Agents affected
5 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
9
Max
Highest count in any single partition
511
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
4
<
mean
40.66
SD
21.89
<
p95
79
zero-arc
p5
1
<
mean
25.92
SD
19.77
<
p95
51
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
857
Agents affected
101 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
6
Max
Highest count in any single partition
14
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
860
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.013858
s
SD
13.54641
s
p50
0.004055
s
<
p95
42.858709
s
<
p99
60.001858
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
11
over 798.248853578s
mean rate
8066.35
/s
std
65093.06
/s
p5
0
/s
p95
924.41
/s
peak
820507.51
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.009025
s
SD
0.004192
s
p50
0.0086
s
<
p95
0.017385
s
<
p99
0.017648
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.006207
s
SD
0.033077
s
p50
0.005572
s
<
p95
0.010515
s
<
p99
0.099429
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007934
s
SD
0.003637
s
p50
0.00748
s
<
p95
0.015296
s
<
p99
0.015451
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.018096
s
SD
23.008511
s
p50
0.007672
s
<
p95
71.11083
s
<
p99
91.624448
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001617
s
SD
0.000612
s
p50
0.001526
s
<
p95
0.00257
s
<
p99
0.003448
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009069
s
SD
0.003095
s
p50
0.00192
s
<
p95
0.007098
s
<
p99
0.013865
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000511
s
SD
0.000311
s
p50
0.000469
s
<
p95
0.001202
s
<
p99
0.001228
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000497
s
SD
0.000312
s
p50
0.000502
s
<
p95
0.001175
s
<
p99
0.001209
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.400534
s
SD
8.542719
s
p50
3.672131
s
<
p95
14.959525
s
<
p99
47.223752
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.0098
s
SD
17.244489
s
p50
0.002292
s
<
p95
54.662501
s
<
p99
60.004373
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.015348
s
SD
0.008844
s
p50
0.011794
s
<
p95
0.034228
s
<
p99
0.037681
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002567
s
SD
0.001037
s
p50
0.002354
s
<
p95
0.004523
s
<
p99
0.005066
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000539
s
SD
0.000551
s
p50
0.000242
s
<
p95
0.00156
s
<
p99
0.003261
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.79
s
<
mean
426.2
s
SD
246.45
s
<
p95
808.18
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.639083276s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
120.572535
s
SD
23.053393
s
p50
0
s
≤
p95
0
s
<
p99
188.521419
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.746663
s
SD
0.81212
s
p50
0.00206
s
<
p95
0.005814
s
<
p99
4.846445
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.015814
s
SD
0.00735
s
p50
0.002337
s
<
p95
0.0063
s
<
p99
0.040218
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.135863
s
SD
0.036748
s
p50
0.010296
s
<
p95
0.078881
s
<
p99
0.202652
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
4.006973
s
SD
3.871063
s
p50
0.003204
s
<
p95
0.014493
s
<
p99
24.882356
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.058075
s
SD
0.063754
s
p50
0.001311
s
<
p95
0.00545
s
<
p99
0.066053
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000326
s
SD
0.001884
s
p50
0.000102
s
<
p95
0.003522
s
<
p99
0.011307
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006828
s
SD
0.006924
s
p50
0.002741
s
<
p95
0.008516
s
<
p99
0.031021
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000409
s
SD
0.000491
s
p50
0.000744
s
<
p95
0.001688
s
<
p99
0.002059
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
50.68587
s
SD
29.973221
s
p50
12.68243
s
<
p95
60.001763
s
<
p99
62.803459
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
52.26686
s
SD
22.455087
s
p50
60.000673
s
<
p95
60.001454
s
<
p99
60.001923
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
56.028857
s
SD
63.024245
s
p50
15.284316
s
<
p95
152.810409
s
<
p99
245.619619
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
17.25
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.69
KiB/s
<
mean
92.38
KiB/s
SD
328.57
KiB/s
<
p95
248.68
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
390
B/s
<
mean
8.87
KiB/s
SD
31.73
KiB/s
<
p95
29.80
KiB/s
Total bytes received
Total bytes received on primary network interface
count
11.31
GiB
mean
13.02
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.09
GiB
mean
1.25
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 90.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (16.2% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 7% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.12
%
<
mean
31.4
%
SD
25.42
%
<
p95
74.21
%
System
14.15
%
CPU percentiles
CPU usage percentiles
p50
44.28
%
p95
74.21
%
p99
90.94
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
90
hosts
mean time
49.15
s
Memory used percentage
Percentage of memory used
p5
7.01
%
<
mean
12.78
%
SD
8.23
%
<
p95
28.44
%
Memory available percentage
Percentage of available memory
p5
71.56
%
<
mean
87.22
%
SD
8.23
%
<
p95
92.99
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.70
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
16.21
%
Memory growth rate
Rate of memory growth over time
growth
0.51
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.03
MiB/s
Disk write throughput
Disk write throughput in MiB/s
56.89
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.90
5 min
0.87
15 min
1.00
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
6.85
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
7.585
%
SD
8.8804
%
<
p95
16.27
%
60 second average
7.3357
%
300 second average
7.4088
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.0023
%
SD
2.1024
%
<
p95
2.77
%
60 second average
0.9610
%
300 second average
0.9242
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8351
%
SD
1.9207
%
<
p95
2.07
%
60 second average
0.7837
%
300 second average
0.6024
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.25
%
<
mean
26.78
%
SD
23.92
%
<
p95
71.25
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
156.43
KiB
<
mean
338.64
KiB
SD
92.80
KiB
<
p95
442.19
KiB
Holochain process threads
Number of threads in Holochain process
p5
20
threads
<
mean
28.15
threads
SD
6.73
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
55
file descriptors
<
mean
70.49
file descriptors
SD
13.57
file descriptors
<
p95
85
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Thu, 10 Sep 2026 16:30:36 UTC
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_34428490451_2
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
415908
Agents affected
105 / 105
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3961
Max
Highest count in any single partition
11331
Min
Lowest count in any single partition
657
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2436.42
Max
Highest chain head value observed for any single write agent.
4388
Write agents observed
19
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
0.450492
s
SD
2.830703
s
Max
Highest per-partition mean latency
max mean
8.494449
s
Min
Lowest per-partition mean latency
min mean
-0.670703
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
3.48
/s
Max
Highest per-partition mean rate
max
15.25
/s
Min
Lowest per-partition mean rate
min
0
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
2.403109
s
SD
4.917763
s
Max
Highest per-partition mean latency
max mean
14.323518
s
Min
Lowest per-partition mean latency
min mean
0.082979
s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
93
Agents affected
33 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2
Max
Highest count in any single partition
9
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
37
<
mean
102.11
SD
35.91
<
p95
147
  • behaviour: zero_read
p5
13
<
mean
47.77
SD
19.35
<
p95
89
  • behaviour: zero_write
p5
8
<
mean
46.66
SD
20.12
<
p95
82
Final error count
The total number of errors accumulated across all agents over the run.
97
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.568423
s
SD
3.909913
s
p50
1.474346
s
<
p95
11.326486
s
<
p99
17.566559
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 802.968280292s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009181
s
SD
0.004406
s
p50
0.008645
s
<
p95
0.016191
s
<
p99
0.028732
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008556
s
SD
0.016313
s
p50
0.008594
s
<
p95
0.05543
s
<
p99
0.077809
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.575941
s
SD
3.113021
s
p50
1.885668
s
<
p95
8.07395
s
<
p99
14.534765
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.263733
s
SD
3.861762
s
p50
0.691128
s
<
p95
11.089536
s
<
p99
17.575858
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007332
s
SD
0.003279
s
p50
0.007132
s
<
p95
0.010876
s
<
p99
0.022185
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007333
s
SD
0.01538
s
p50
0.00691
s
<
p95
0.051465
s
<
p99
0.072759
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.575736
s
SD
3.151406
s
p50
1.899053
s
<
p95
8.294022
s
<
p99
14.532319
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.361349
s
SD
4.516459
s
p50
0.830424
s
<
p95
12.048515
s
<
p99
17.573368
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001633
s
SD
0.000888
s
p50
0.001855
s
<
p95
0.003782
s
<
p99
0.005002
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
7e-05
s
SD
5.4e-05
s
p50
5e-05
s
<
p95
0.000221
s
<
p99
0.000307
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002471
s
SD
0.010671
s
p50
0.001042
s
<
p95
0.033105
s
<
p99
0.048389
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000896
s
SD
0.001258
s
p50
0.000706
s
<
p95
0.001552
s
<
p99
0.003954
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.572965
s
SD
3.151104
s
p50
1.894416
s
<
p95
8.290343
s
<
p99
14.526518
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.355434
s
SD
4.516688
s
p50
0.825529
s
<
p95
12.043529
s
<
p99
17.566663
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.2e-05
s
SD
6e-06
s
p50
1.6e-05
s
<
p95
2.5e-05
s
<
p99
3.2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003087
s
SD
0.002913
s
p50
0.003464
s
<
p95
0.009416
s
<
p99
0.017958
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001117
s
SD
0.003291
s
p50
0.000509
s
<
p95
0.009369
s
<
p99
0.016578
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.81
s
<
mean
426.42
s
SD
246.66
s
<
p95
811.56
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.628408987s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
8.702015
s
SD
22.421966
s
p50
0
s
<
p95
7.493598
s
<
p99
94.6529
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.178969
s
SD
0.552227
s
p50
0.002787
s
<
p95
0.89648
s
<
p99
2.503481
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.055586
s
SD
0.260557
s
p50
0.002749
s
<
p95
0.702733
s
<
p99
1.2302
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.299542
s
SD
0.234027
s
p50
0.19963
s
<
p95
0.639623
s
<
p99
0.931468
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
9.281435
s
SD
20.461488
s
p50
0.004112
s
<
p95
52.800728
s
<
p99
113.057661
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.474788
s
SD
5.97907
s
p50
0.001659
s
<
p95
0.492058
s
<
p99
12.428179
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003908
s
SD
0.002699
s
p50
0.002567
s
<
p95
0.008492
s
<
p99
0.010602
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005701
s
SD
0.004121
s
p50
0.002992
s
<
p95
0.013157
s
<
p99
0.01989
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002478
s
SD
0.009021
s
p50
0.000781
s
<
p95
0.02557
s
<
p99
0.045504
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
7.418136
s
SD
25.757584
s
p50
5.204308
s
<
p95
60.049113
s
<
p99
61.724679
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.309417
s
SD
2.576595
s
p50
0.655317
s
<
p95
6.694849
s
<
p99
12.040104
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.442085
s
SD
28.230145
s
p50
1.544883
s
<
p95
60.002155
s
<
p99
60.002912
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.493026
s
SD
22.883908
s
p50
1.171412
s
<
p95
60.049252
s
<
p99
68.176855
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
37.21
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
133
B/s
<
mean
729.14
KiB/s
SD
662.33
KiB/s
<
p95
1.85
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
119.17
KiB/s
SD
172.59
KiB/s
<
p95
400.61
KiB/s
Total bytes received
Total bytes received on primary network interface
count
89.59
GiB
mean
103.08
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
14.76
GiB
mean
16.98
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (16.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 37% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.16
%
<
mean
52.18
%
SD
36.39
%
<
p95
92.06
%
System
14.98
%
CPU percentiles
CPU usage percentiles
p50
63.54
%
p95
92.06
%
p99
94.16
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
106
hosts
mean time
471.03
s
Memory used percentage
Percentage of memory used
p5
6.61
%
<
mean
11.97
%
SD
8.03
%
<
p95
26.27
%
Memory available percentage
Percentage of available memory
p5
73.73
%
<
mean
88.03
%
SD
8.03
%
<
p95
93.39
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.09
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
16.81
%
Memory growth rate
Rate of memory growth over time
growth
0.57
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.05
MiB/s
Disk write throughput
Disk write throughput in MiB/s
119.02
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/23
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.30
5 min
1.85
15 min
1.40
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
36.73
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
27.7209
%
SD
28.5072
%
<
p95
72.98
%
60 second average
26.0813
%
300 second average
20.3410
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.0938
%
SD
5.8469
%
<
p95
15.79
%
60 second average
3.8748
%
300 second average
3.0375
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.229
%
SD
2.0771
%
<
p95
4.74
%
60 second average
1.1687
%
300 second average
1.0476
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.9
%
<
mean
49.22
%
SD
34.7
%
<
p95
87.76
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
207.39
KiB
<
mean
288.25
KiB
SD
89.37
KiB
<
p95
454.00
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
31.15
threads
SD
6.04
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
76.94
file descriptors
SD
11.82
file descriptors
<
p95
89
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Thu, 10 Sep 2026 16:48:12 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_34428490451_2
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
780.5
Max
Highest chain head value observed for any single write agent.
1798
Write agents observed
14
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
199.007431
s
SD
107.144109
s
Max
Highest per-partition mean latency
max mean
1031.926556
s
Min
Lowest per-partition mean latency
min mean
6.998433
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.28
/s
Max
Highest per-partition mean rate
max
2.82
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
2.494501
s
SD
3.305736
s
Max
Highest per-partition mean latency
max mean
24.201332
s
Min
Lowest per-partition mean latency
min mean
0.06162
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
79354
Agents affected
41 / 41
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1935
Max
Highest count in any single partition
25575
Min
Lowest count in any single partition
58
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
42
<
mean
86.09
SD
33.33
<
p95
146
  • behaviour: zero_must_get_agent_activity
p5
0
<
mean
39.56
SD
19.94
<
p95
84
  • behaviour: zero_write
p5
8
<
mean
36.3
SD
18.39
<
p95
81
Final error count
The total number of all types of error accumulated over the run by all agents.
92388
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.082352
s
SD
4.604348
s
p50
0.341325
s
<
p95
11.902744
s
<
p99
21.981638
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 791.037879906s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.00854
s
SD
0.003459
s
p50
0.00861
s
<
p95
0.012914
s
<
p99
0.013992
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.196216
s
SD
0.229592
s
p50
0.085283
s
<
p95
0.756007
s
<
p99
0.956964
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007183
s
SD
0.002375
s
p50
0.008074
s
<
p95
0.01134
s
<
p99
0.013841
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.550708
s
SD
2.839272
s
p50
0.309766
s
<
p95
5.656396
s
<
p99
18.766537
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.151229
s
SD
5.015088
s
p50
0.874403
s
<
p95
14.377036
s
<
p99
20.175178
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00683
s
SD
0.002068
s
p50
0.007039
s
<
p95
0.010039
s
<
p99
0.011411
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.194279
s
SD
0.228598
s
p50
0.08348
s
<
p95
0.751717
s
<
p99
0.952866
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005932
s
SD
0.002018
s
p50
0.006687
s
<
p95
0.00969
s
<
p99
0.011704
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.796088
s
SD
6.087979
s
p50
1.026179
s
<
p95
21.981457
s
<
p99
24.818069
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.195429
s
SD
5.760277
s
p50
0.905052
s
<
p95
14.894281
s
<
p99
21.987853
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001959
s
SD
0.001106
s
p50
0.002161
s
<
p95
0.004707
s
<
p99
0.005729
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.021972
s
SD
0.103406
s
p50
0.007821
s
<
p95
0.185002
s
<
p99
0.444905
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.5e-05
s
SD
5e-05
s
p50
4.4e-05
s
<
p95
0.000156
s
<
p99
0.000266
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.004106
s
SD
0.00971
s
p50
0.000692
s
<
p95
0.029691
s
<
p99
0.045136
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.006103
s
SD
0.009397
s
p50
0.000716
s
<
p95
0.026196
s
<
p99
0.048755
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.700952
s
SD
5.949508
s
p50
0.960732
s
<
p95
20.168522
s
<
p99
24.809549
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.054561
s
SD
0.497824
s
p50
0.054462
s
<
p95
1.064153
s
<
p99
2.39287
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.061324
s
SD
0.063988
s
p50
0.022217
s
<
p95
0.187137
s
<
p99
0.223405
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
9e-06
s
p50
1.7e-05
s
<
p95
2.6e-05
s
<
p99
6.4e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003974
s
SD
0.002839
s
p50
0.003454
s
<
p95
0.010358
s
<
p99
0.016463
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001892
s
SD
0.002778
s
p50
0.000583
s
<
p95
0.008611
s
<
p99
0.011476
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.6
s
<
mean
425.75
s
SD
246.52
s
<
p95
808.19
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
115.941941
s
SD
37.869735
s
p50
0
s
<
p95
62.47275
s
<
p99
204.236673
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.079489
s
SD
1.186437
s
p50
0.002956
s
<
p95
3.058215
s
<
p99
3.960618
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
1e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.466785
s
SD
0.343724
s
p50
0.002834
s
<
p95
1.171807
s
<
p99
1.355229
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.177456
s
SD
0.185365
s
p50
0.035173
s
<
p95
0.564577
s
<
p99
0.709077
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
10.501347
s
SD
12.334444
s
p50
0.004308
s
<
p95
30.713309
s
<
p99
50.29454
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
3.675872
s
SD
11.57995
s
p50
0.001737
s
<
p95
5.980461
s
<
p99
75.197344
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002452
s
SD
0.002002
s
p50
0.000256
s
<
p95
0.005803
s
<
p99
0.007646
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004416
s
SD
0.006714
s
p50
0.003493
s
<
p95
0.018317
s
<
p99
0.033977
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.004306
s
SD
0.008883
s
p50
0.000617
s
<
p95
0.026939
s
<
p99
0.041381
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
3.167675
s
SD
27.827787
s
p50
1.64258
s
<
p95
60.007145
s
<
p99
60.13575
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.311108
s
SD
18.87661
s
p50
0.59092
s
<
p95
60.001394
s
<
p99
60.002363
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.246889
s
SD
24.268465
s
p50
0.309624
s
<
p95
60.002024
s
<
p99
60.002934
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.334933
s
SD
25.846025
s
p50
1.188255
s
<
p95
60.014226
s
<
p99
60.072719
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
23.10
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
127
B/s
<
mean
566.25
KiB/s
SD
540.49
KiB/s
<
p95
1.65
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
85.71
KiB/s
SD
89.11
KiB/s
<
p95
240.55
KiB/s
Total bytes received
Total bytes received on primary network interface
count
69.67
GiB
mean
80.16
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.55
GiB
mean
12.13
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/17 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (17.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.44
%
<
mean
34.58
%
SD
33.53
%
<
p95
89.21
%
System
11.48
%
CPU percentiles
CPU usage percentiles
p50
16.29
%
p95
89.21
%
p99
97.19
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
102
hosts
mean time
293.04
s
Memory used percentage
Percentage of memory used
p5
7.2
%
<
mean
13.21
%
SD
8.2
%
<
p95
27.4
%
Memory available percentage
Percentage of available memory
p5
72.6
%
<
mean
86.79
%
SD
8.2
%
<
p95
92.8
%
Max host memory used
Maximum memory usage percentage across all hosts
max
79.10
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
17.55
%
Memory growth rate
Rate of memory growth over time
growth
0.62
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.14
MiB/s
Disk write throughput
Disk write throughput in MiB/s
113.96
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/17
hosts
Mount Point /etc/hosts
1/17
hosts
Mount Point /etc/nomad.d/client-meta.json
0/23
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/17
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.52
5 min
1.60
15 min
1.62
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
29.25
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
16.8898
%
SD
23.9261
%
<
p95
64.41
%
60 second average
16.1854
%
300 second average
17.4536
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0008
%
SD
0.0478
%
≤
p95
0
%
60 second average
0.0003
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0005
%
SD
0.0326
%
≤
p95
0
%
60 second average
0.0002
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.6479
%
SD
6.5398
%
<
p95
16.37
%
60 second average
3.5002
%
300 second average
3.3929
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.4316
%
SD
2.985
%
<
p95
7.32
%
60 second average
1.3849
%
300 second average
1.2883
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.35
%
<
mean
31.19
%
SD
32.01
%
<
p95
82.51
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
255.89
KiB
<
mean
379.75
KiB
SD
82.61
KiB
<
p95
501.53
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
31.65
threads
SD
6.36
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
61
file descriptors
<
mean
75.99
file descriptors
SD
11.91
file descriptors
<
p95
89
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Thu, 10 Sep 2026 11:52:37 UTC
Duration
900s
Peer count
150
Peer count at end
141
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_34428490451_2
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-1.766808
s
SD
28.822674
s
Max
Highest per-partition mean latency
max mean
3.440483
s
Min
Lowest per-partition mean latency
min mean
-19.424771
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.157508
s
SD
0.294808
s
Max
Highest per-partition mean latency
max mean
0.492811
s
Min
Lowest per-partition mean latency
min mean
0.066685
s
Error count
The total number of errors accumulated during the run.
4562
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 730.008723714s
Not enough time data to show a trend.
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
152763276
over 888.354625699s
mean rate
3.27797612287e+09
/s
std
6.93960852804e+10
/s
p5
8.67580474e+06
/s
p95
8.24113543237e+09
/s
peak
5.23393500413e+12
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
13177189
over 888.354625699s
mean rate
4.7139791135e+08
/s
std
5.96584285988e+09
/s
p5
359145.44
/s
p95
1.30202468176e+09
/s
peak
2.89544340402e+11
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 888.354625699s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.166531
s
SD
0.460403
s
p50
0.175807
s
<
p95
0.69485
s
<
p99
2.562408
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.004135
s
SD
0.001101
s
p50
0.00407
s
<
p95
0.005494
s
<
p99
0.00671
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.36536
s
SD
0.610965
s
p50
0.447539
s
<
p95
1.417386
s
<
p99
3.179206
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002624
s
SD
0.00076
s
p50
0.002594
s
<
p95
0.003628
s
<
p99
0.004527
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005913
s
SD
0.012229
s
p50
0.004359
s
<
p95
0.009173
s
<
p99
0.077439
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.362254
s
SD
0.610924
s
p50
0.444695
s
<
p95
1.413833
s
<
p99
3.175427
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.001126
s
SD
0.00487
s
p50
0.000689
s
<
p95
0.001242
s
<
p99
0.009773
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.4e-05
s
SD
6e-06
s
p50
1.3e-05
s
<
p95
2.5e-05
s
<
p99
2.7e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000365
s
SD
0.004756
s
p50
0.000433
s
<
p95
0.001098
s
<
p99
0.002142
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000624
s
SD
0.001446
s
p50
0.000231
s
<
p95
0.002646
s
<
p99
0.008969
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.21
s
<
mean
422.23
s
SD
244.53
s
<
p95
802.65
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1828
over 888.354634093s
mean rate
14664.56
/s
std
97291.47
/s
p5
0
/s
p95
40769.6
/s
peak
3.47170926e+06
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
15.336145
s
SD
8.216307
s
p50
12.916829
s
<
p95
31.069857
s
<
p99
38.660183
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.010413
s
SD
0.016077
s
p50
0.011148
s
<
p95
0.049264
s
<
p99
0.073001
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.076568
s
SD
0.054359
s
p50
0.080646
s
<
p95
0.183145
s
<
p99
0.273821
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.018548
s
SD
0.012304
s
p50
0.018015
s
<
p95
0.034228
s
<
p99
0.053928
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.345157
s
SD
4.977902
s
p50
4.028861
s
<
p95
15.681766
s
<
p99
22.757995
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.46124
s
SD
3.650751
s
p50
0.770894
s
<
p95
9.126445
s
<
p99
21.038563
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000582
s
SD
0.001053
s
p50
0.000439
s
<
p95
0.001775
s
<
p99
0.00476
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003756
s
SD
0.002625
s
p50
0.003015
s
<
p95
0.007314
s
<
p99
0.012275
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000595
s
SD
0.000316
s
p50
0.000575
s
<
p95
0.000888
s
<
p99
0.001397
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.308037
s
SD
29.163719
s
p50
0.30253
s
<
p95
60.003477
s
<
p99
60.006401
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.152348
s
SD
14.300019
s
p50
0.090504
s
<
p95
60.001059
s
<
p99
60.001837
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.206079
s
SD
28.964862
s
p50
0.292674
s
<
p95
60.001858
s
<
p99
71.366423
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
6.49
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
11.90
KiB/s
<
mean
133.01
KiB/s
SD
289.93
KiB/s
<
p95
467.89
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.33
KiB/s
<
mean
15.58
KiB/s
SD
14.11
KiB/s
<
p95
40.41
KiB/s
Total bytes received
Total bytes received on primary network interface
count
16.12
GiB
mean
18.55
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.89
GiB
mean
2.17
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/17 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.32
%
<
mean
11.52
%
SD
10.83
%
<
p95
32.03
%
System
5.03
%
CPU percentiles
CPU usage percentiles
p50
8.94
%
p95
32.03
%
p99
60.59
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
66
hosts
mean time
11.75
s
Memory used percentage
Percentage of memory used
p5
5.65
%
<
mean
10.39
%
SD
8.1
%
<
p95
24.81
%
Memory available percentage
Percentage of available memory
p5
75.19
%
<
mean
89.61
%
SD
8.1
%
<
p95
94.35
%
Max host memory used
Maximum memory usage percentage across all hosts
max
77.55
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
7.69
%
Memory growth rate
Rate of memory growth over time
growth
0.19
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.39
MiB/s
Disk write throughput
Disk write throughput in MiB/s
29.33
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/107
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/17
hosts
Mount Point /etc/hosts
1/17
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/17
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.36
5 min
0.29
15 min
0.17
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
2.6609
%
SD
2.4694
%
<
p95
7.28
%
60 second average
2.5787
%
300 second average
1.9881
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0002
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0002
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.4107
%
SD
2.8429
%
<
p95
1.21
%
60 second average
0.5840
%
300 second average
0.6439
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3232
%
SD
2.6201
%
<
p95
0.87
%
60 second average
0.4852
%
300 second average
0.5542
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.18
%
<
mean
8.37
%
SD
9.06
%
<
p95
17.65
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
209.08
KiB
<
mean
233.75
KiB
SD
28.70
KiB
<
p95
277.25
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
32.74
threads
SD
5.78
threads
<
p95
47
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
80.37
file descriptors
SD
7.63
file descriptors
<
p95
92
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Thu, 10 Sep 2026 12:10:24 UTC
Duration
900s
Peer count
300
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_signals_34428490451_2
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.514743
s
SD
2.084866
s
p50
0.11565
s
<
p95
1.226118
s
<
p99
9.337281
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
393
over 1404.535009943s
mean rate
241285.52
/s
std
2.175416801e+07
/s
p5
2.99
/s
p95
27914.92
/s
peak
2.68181818182e+09
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0018
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 797.709241271s
Not enough time data to show a trend.
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 890.310361496s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
7255554
over 890.310361496s
mean rate
7.91547028454e+09
/s
std
1.83025171804e+11
/s
p5
5.31750595e+06
/s
p95
1.65303193922e+10
/s
peak
1.50572773109e+13
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
11501640
over 890.310361496s
mean rate
9.565156383e+07
/s
std
2.19984522668e+09
/s
p5
0
/s
p95
1.8319412669e+08
/s
peak
1.828128355e+11
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.004268
s
SD
0.001242
s
p50
0.004289
s
<
p95
0.00565
s
<
p99
0.006873
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.004102
s
SD
0.000976
s
p50
0.004214
s
<
p95
0.005312
s
<
p99
0.006132
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005707
s
SD
0.007092
s
p50
0.003967
s
<
p95
0.013481
s
<
p99
0.040403
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002723
s
SD
0.000836
s
p50
0.002767
s
<
p95
0.003641
s
<
p99
0.004555
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002547
s
SD
0.000615
s
p50
0.002634
s
<
p95
0.003473
s
<
p99
0.004017
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000929
s
SD
0.001771
s
p50
0.000645
s
<
p95
0.001812
s
<
p99
0.008787
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.5e-05
s
SD
2.7e-05
s
p50
7.8e-05
s
<
p95
0.000104
s
<
p99
0.000117
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.2e-05
s
SD
2.6e-05
s
p50
7.6e-05
s
<
p95
0.000107
s
<
p99
0.000123
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1e-05
s
SD
1e-05
s
p50
1e-05
s
<
p95
1.8e-05
s
<
p99
2.8e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.001279
s
SD
0.005436
s
p50
0.000459
s
<
p95
0.002637
s
<
p99
0.033312
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
465
over 890.310362823s
mean rate
269925.05
/s
std
6.72017787e+06
/s
p5
200.4
/s
p95
718414.45
/s
peak
6.568627451e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
820
over 890.310363494s
mean rate
226297.79
/s
std
4.66902019e+06
/s
p5
178.57
/s
p95
632584.83
/s
peak
4.7368421053e+08
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000588
s
SD
0.001194
s
p50
0.000249
s
<
p95
0.002487
s
<
p99
0.004994
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.77
s
<
mean
422.73
s
SD
244.62
s
<
p95
802.78
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3819
over 890.310287314s
mean rate
1.05832396e+06
/s
std
1.771049705e+07
/s
p5
0
/s
p95
2.22005631e+06
/s
peak
1.70491803279e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
17.231584
s
SD
9.855587
s
p50
14.812056
s
<
p95
36.040196
s
<
p99
46.560483
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.019836
s
SD
0.034938
s
p50
0.019629
s
<
p95
0.094687
s
<
p99
0.187295
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.103677
s
SD
0.074197
s
p50
0.107854
s
<
p95
0.251835
s
<
p99
0.339024
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.024798
s
SD
0.015444
s
p50
0.024132
s
<
p95
0.052091
s
<
p99
0.071306
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
4.778141
s
SD
7.961912
s
p50
6.50439
s
<
p95
25.808772
s
<
p99
37.747883
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.158427
s
SD
4.430248
s
p50
1.348812
s
<
p95
9.216907
s
<
p99
25.913988
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000789
s
SD
0.000465
s
p50
0.000586
s
<
p95
0.001613
s
<
p99
0.001999
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004841
s
SD
0.002732
s
p50
0.003644
s
<
p95
0.009394
s
<
p99
0.01166
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000876
s
SD
0.000957
s
p50
0.00075
s
<
p95
0.001823
s
<
p99
0.003399
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.284512
s
SD
25.139563
s
p50
0.259912
s
<
p95
60.002411
s
<
p99
60.006361
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.192083
s
SD
4.623167
s
p50
0.139772
s
<
p95
0.65217
s
<
p99
1.176817
s
P2P received remote signals
Count of remote signals received.
total
465
over 890.310366044s
mean rate
363186.52
/s
std
1.553538097e+07
/s
p5
176.16
/s
p95
664031.75
/s
peak
1.71929824561e+09
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
11.85
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
25.63
KiB/s
<
mean
269.30
KiB/s
SD
289.23
KiB/s
<
p95
836.30
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
18.58
KiB/s
<
mean
38.31
KiB/s
SD
28.40
KiB/s
<
p95
84.94
KiB/s
Total bytes received
Total bytes received on primary network interface
count
33.35
GiB
mean
38.37
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.74
GiB
mean
5.46
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.6%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/17 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.19
%
<
mean
18.9
%
SD
16.76
%
<
p95
48.71
%
System
7.05
%
CPU percentiles
CPU usage percentiles
p50
14.67
%
p95
48.71
%
p99
96.56
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
110
hosts
mean time
30.26
s
Memory used percentage
Percentage of memory used
p5
6.09
%
<
mean
12.97
%
SD
8.23
%
<
p95
28.25
%
Memory available percentage
Percentage of available memory
p5
71.75
%
<
mean
87.03
%
SD
8.23
%
<
p95
93.91
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.72
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
8.84
%
Memory growth rate
Rate of memory growth over time
growth
0.43
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.07
MiB/s
Disk write throughput
Disk write throughput in MiB/s
46.66
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/17
hosts
Mount Point /etc/hosts
1/17
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/17
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.72
5 min
0.63
15 min
0.43
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
7.818
%
SD
12.0859
%
<
p95
26.14
%
60 second average
7.6952
%
300 second average
6.6701
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6759
%
SD
1.7961
%
<
p95
2.93
%
60 second average
0.6860
%
300 second average
0.6105
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3953
%
SD
1.3898
%
<
p95
1.63
%
60 second average
0.4012
%
300 second average
0.3701
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.11
%
<
mean
8.01
%
SD
8.6
%
<
p95
25.54
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
189.23
KiB
<
mean
225.41
KiB
SD
38.45
KiB
<
p95
273.36
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
32.71
threads
SD
6.44
threads
<
p95
49
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
74
file descriptors
<
mean
80.15
file descriptors
SD
10.61
file descriptors
<
p95
94
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Thu, 10 Sep 2026 13:00:38 UTC
Duration
900s
Peer count
150
Peer count at end
144
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
validation_receipts_34428490451_2
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
45.743266
s
SD
35.29109
s
Max
Highest per-partition mean latency
max mean
724.219057
s
Min
Lowest per-partition mean latency
min mean
12.707541
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
0.99
/s
Max
Highest per-partition mean rate
max
1.22
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 690.00722264s
Not enough time data to show a trend.
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
1501290
over 889.429277557s
mean rate
1.815410352e+07
/s
std
1.8920686632e+08
/s
p5
0
/s
p95
5.595273473e+07
/s
peak
1.35856195149e+10
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
9989773
over 889.429277557s
mean rate
1.0057502412e+08
/s
std
7.7435183327e+08
/s
p5
211173.73
/s
p95
3.0866145698e+08
/s
peak
4.26396374004e+10
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.429277557s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
184884
over 889.429277557s
mean rate
2.19529599e+06
/s
std
2.305481773e+07
/s
p5
0
/s
p95
6.81614113e+06
/s
peak
1.67307027849e+09
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.00782
s
SD
0.002028
s
p50
0.007811
s
<
p95
0.010423
s
<
p99
0.011666
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.023038
s
SD
0.018432
s
p50
0.012837
s
<
p95
0.051156
s
<
p99
0.103724
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006367
s
SD
0.001633
s
p50
0.006426
s
<
p95
0.008519
s
<
p99
0.00945
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.021342
s
SD
0.018024
s
p50
0.011089
s
<
p95
0.048829
s
<
p99
0.100457
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005726
s
SD
0.019055
s
p50
0.003311
s
<
p95
0.006818
s
<
p99
0.071761
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002058
s
SD
0.000493
s
p50
0.002093
s
<
p95
0.002634
s
<
p99
0.003043
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000818
s
SD
0.009519
s
p50
0.000734
s
<
p95
0.001809
s
<
p99
0.009018
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.018863
s
SD
0.017815
s
p50
0.008433
s
<
p95
0.045907
s
<
p99
0.097197
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003267
s
SD
0.000868
s
p50
0.00328
s
<
p95
0.004455
s
<
p99
0.005058
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000742
s
SD
0.000579
s
p50
0.000569
s
<
p95
0.001713
s
<
p99
0.002466
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.51
s
<
mean
425
s
SD
246.19
s
<
p95
808.1
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
10673
over 889.429204213s
mean rate
85385.16
/s
std
1.09101792e+06
/s
p5
474.43
/s
p95
233553.9
/s
peak
7.628960311e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
14.010898
s
SD
8.150249
s
p50
11.436427
s
<
p95
30.927975
s
<
p99
38.38139
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005644
s
SD
0.044371
s
p50
0.008946
s
<
p95
0.144478
s
<
p99
0.186074
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
3e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.047837
s
SD
0.104293
s
p50
0.07801
s
<
p95
0.336887
s
<
p99
0.397889
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.103461
s
SD
0.076203
s
p50
0.049834
s
<
p95
0.216959
s
<
p99
0.378331
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.392159
s
SD
6.154354
s
p50
1.166841
s
<
p95
15.727998
s
<
p99
25.096368
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
19.670638
s
SD
24.482915
s
p50
16.185911
s
<
p95
76.904747
s
<
p99
107.857519
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001469
s
SD
0.000799
s
p50
0.001037
s
<
p95
0.002793
s
<
p99
0.003557
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005145
s
SD
0.002692
s
p50
0.003557
s
<
p95
0.008548
s
<
p99
0.011666
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000749
s
SD
0.000401
s
p50
0.000638
s
<
p95
0.001492
s
<
p99
0.001889
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.302302
s
SD
28.790372
s
p50
0.269971
s
<
p95
60.003831
s
<
p99
60.00942
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.181748
s
SD
28.400106
s
p50
0.21978
s
<
p95
60.001733
s
<
p99
60.002236
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
15.64
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
14.40
KiB/s
<
mean
156.01
KiB/s
SD
286.29
KiB/s
<
p95
388.37
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
5.69
KiB/s
<
mean
20.72
KiB/s
SD
19.25
KiB/s
<
p95
57.43
KiB/s
Total bytes received
Total bytes received on primary network interface
count
18.85
GiB
mean
21.69
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.50
GiB
mean
2.88
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (15.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 9% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.18
%
<
mean
23.59
%
SD
17.92
%
<
p95
64.6
%
System
7.95
%
CPU percentiles
CPU usage percentiles
p50
20.64
%
p95
64.60
%
p99
82.95
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
77
hosts
mean time
24.44
s
Memory used percentage
Percentage of memory used
p5
5.93
%
<
mean
10.86
%
SD
7.81
%
<
p95
24.52
%
Memory available percentage
Percentage of available memory
p5
75.48
%
<
mean
89.14
%
SD
7.81
%
<
p95
94.07
%
Max host memory used
Maximum memory usage percentage across all hosts
max
74.69
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
15.31
%
Memory growth rate
Rate of memory growth over time
growth
0.26
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.07
MiB/s
Disk write throughput
Disk write throughput in MiB/s
40.07
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/109
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/20
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/4
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.82
5 min
1.42
15 min
2.33
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
8.97
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
7.7946
%
SD
10.682
%
<
p95
29.7
%
60 second average
7.6277
%
300 second average
14.7195
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0002
%
SD
0.0124
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
0.0002
%
SD
0.0124
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.1317
%
SD
4.1119
%
<
p95
6.65
%
60 second average
2.0731
%
300 second average
3.1030
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.4856
%
SD
3.5947
%
<
p95
3.42
%
60 second average
1.4225
%
300 second average
1.4725
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.89
%
<
mean
20.69
%
SD
17.14
%
<
p95
60.59
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
220.88
KiB
<
mean
255.77
KiB
SD
32.25
KiB
<
p95
299.30
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
32.06
threads
SD
5.7
threads
<
p95
50
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
76
file descriptors
<
mean
80.02
file descriptors
SD
9.19
file descriptors
<
p95
95
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 10 Sep 2026 13:18:10 UTC
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_34428490451_2
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
3053.1
Max
Highest chain head value observed for any single write agent.
7119
Write agents observed
29
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
1.463219
s
SD
3.54883
s
Max
Highest per-partition mean latency
max mean
6.784619
s
Min
Lowest per-partition mean latency
min mean
0.023979
s
Error count
The number of errors accumulated.
2110
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.073372
s
SD
7.334532
s
p50
0.635631
s
<
p95
20.894634
s
<
p99
35.984639
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4
over 826.838474611s
mean rate
430.19
/s
std
1619.34
/s
p5
0
/s
p95
2759.08
/s
peak
15723.27
/s
WASM usage: 'agent_activity::announce_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.380739877s
Not enough time data to show a trend.
WASM usage: 'agent_activity::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
153164772
over 889.380739877s
mean rate
5.3958004048e+08
/s
std
1.21760890064e+09
/s
p5
1.072929917e+07
/s
p95
2.65532371801e+09
/s
peak
1.9881813595e+10
/s
WASM usage: 'agent_activity::get_agent_activity_full'
Metered usage count of the WASM ribosome for this zome function.
total
67357224
over 888.324719204s
mean rate
1.08493125661e+10
/s
std
2.82438111967e+10
/s
p5
6.25023277e+06
/s
p95
6.98066203206e+10
/s
peak
6.4876801811e+11
/s
WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'
Metered usage count of the WASM ribosome for this zome function.
total
31777403
over 888.995301154s
mean rate
4.74048228923e+10
/s
std
1.39343768574e+12
/s
p5
0
/s
p95
5.08399165837e+10
/s
peak
6.69166521012e+13
/s
WASM usage: 'agent_activity_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
18756537
over 889.380739877s
mean rate
2.91001238071e+09
/s
std
8.08760161052e+10
/s
p5
0
/s
p95
4.12970862616e+09
/s
peak
4.75158619866e+12
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.014997
s
SD
0.048058
s
p50
0.007392
s
<
p95
0.011838
s
<
p99
0.323818
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.005443
s
SD
0.002952
s
p50
0.0058
s
<
p95
0.011936
s
<
p99
0.016558
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.429137
s
SD
5.400172
s
p50
1.679445
s
<
p95
12.658359
s
<
p99
25.527049
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007699
s
SD
7.002895
s
p50
0.008439
s
<
p95
14.974735
s
<
p99
28.971342
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.013857
s
SD
0.04811
s
p50
0.006116
s
<
p95
0.009571
s
<
p99
0.323129
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004491
s
SD
0.00246
s
p50
0.00479
s
<
p95
0.009949
s
<
p99
0.013944
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.441427
s
SD
5.453333
s
p50
1.748079
s
<
p95
13.209657
s
<
p99
25.525377
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.014417
s
SD
8.549053
s
p50
0.019307
s
<
p95
22.042263
s
<
p99
37.165076
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001496
s
SD
0.00052
s
p50
0.00168
s
<
p95
0.002386
s
<
p99
0.002715
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.6e-05
s
SD
4.5e-05
s
p50
4.4e-05
s
<
p95
0.000119
s
<
p99
0.00025
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000746
s
SD
0.001039
s
p50
0.000611
s
<
p95
0.003085
s
<
p99
0.005704
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000567
s
SD
0.000203
s
p50
0.000511
s
<
p95
0.000845
s
<
p99
0.001256
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.439374
s
SD
5.45321
s
p50
1.745167
s
<
p95
13.2072
s
<
p99
25.521551
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.010326
s
SD
8.548431
s
p50
0.014533
s
<
p95
22.035342
s
<
p99
37.159654
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.1e-05
s
SD
2.1e-05
s
p50
1.3e-05
s
<
p95
2.9e-05
s
<
p99
3.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002293
s
SD
0.023229
s
p50
0.00268
s
<
p95
0.004301
s
<
p99
0.020619
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000413
s
SD
0.00115
s
p50
0.000267
s
<
p95
0.00147
s
<
p99
0.003094
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.67
s
<
mean
426.32
s
SD
246.52
s
<
p95
808.47
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
4304
over 889.380810476s
mean rate
373711.6
/s
std
1.53919286e+06
/s
p5
24.3
/s
p95
1.57360775e+06
/s
peak
4.872861639e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
89.977691
s
SD
59.788837
s
p50
38.356958
s
<
p95
173.86773
s
<
p99
284.234569
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.066911
s
SD
0.22749
s
p50
0.103853
s
<
p95
0.601957
s
<
p99
0.940699
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.035974
s
SD
0.395279
s
p50
0.187916
s
<
p95
1.136277
s
<
p99
1.535541
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.781189
s
SD
0.70409
s
p50
0.074568
s
<
p95
1.883345
s
<
p99
3.542747
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
11.163013
s
SD
24.644754
s
p50
21.412279
s
<
p95
72.868631
s
<
p99
98.377208
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
10.7857
s
SD
23.812122
s
p50
0.512284
s
<
p95
66.318453
s
<
p99
122.306624
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002872
s
SD
0.002282
s
p50
0.002012
s
<
p95
0.007089
s
<
p99
0.010076
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003748
s
SD
0.002787
s
p50
0.003183
s
<
p95
0.009242
s
<
p99
0.014925
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000725
s
SD
0.001095
s
p50
0.000565
s
<
p95
0.002101
s
<
p99
0.00532
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.691254
s
SD
29.134753
s
p50
2.052507
s
<
p95
60.004765
s
<
p99
60.207274
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
7.451438
s
SD
18.201754
s
p50
1.354897
s
<
p95
60.001268
s
<
p99
60.002645
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.2542
s
SD
29.253367
s
p50
0.773012
s
<
p95
60.003366
s
<
p99
60.010299
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.623765
s
SD
26.562964
s
p50
2.010782
s
<
p95
60.002313
s
<
p99
60.040139
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
38.95
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.37
KiB/s
<
mean
434.60
KiB/s
SD
461.43
KiB/s
<
p95
1.36
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
104.83
KiB/s
SD
223.28
KiB/s
<
p95
366.49
KiB/s
Total bytes received
Total bytes received on primary network interface
count
53.89
GiB
mean
61.31
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.00
GiB
mean
14.79
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.5%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 25% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
4.35
%
<
mean
53.86
%
SD
29.88
%
<
p95
93.29
%
System
14.91
%
CPU percentiles
CPU usage percentiles
p50
59.30
%
p95
93.29
%
p99
96.53
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
122
hosts
mean time
280.83
s
Memory used percentage
Percentage of memory used
p5
6.33
%
<
mean
11.15
%
SD
7.44
%
<
p95
25.21
%
Memory available percentage
Percentage of available memory
p5
74.79
%
<
mean
88.85
%
SD
7.44
%
<
p95
93.67
%
Max host memory used
Maximum memory usage percentage across all hosts
max
76.68
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
6.33
%
Memory growth rate
Rate of memory growth over time
growth
0.64
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.16
MiB/s
Disk write throughput
Disk write throughput in MiB/s
188.18
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/109
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/16
hosts
Mount Point /etc/hosts
0/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/16
hosts
Mount Point /nix/store
0/4
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.42
5 min
1.82
15 min
1.74
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
25.17
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.12
%
<
mean
27.2838
%
SD
27.9072
%
<
p95
84.44
%
60 second average
24.9148
%
300 second average
18.1213
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.9906
%
SD
6.517
%
<
p95
16.12
%
60 second average
4.5944
%
300 second average
3.5598
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.4165
%
SD
4.0637
%
<
p95
9.41
%
60 second average
2.2718
%
300 second average
1.9215
%
Holochain process CPU usage
CPU usage by Holochain process
p5
2
%
<
mean
51.42
%
SD
27.96
%
<
p95
89
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
207.53
KiB
<
mean
306.74
KiB
SD
86.70
KiB
<
p95
459.73
KiB
Holochain process threads
Number of threads in Holochain process
p5
26
threads
<
mean
31.92
threads
SD
6.71
threads
<
p95
48
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
70
file descriptors
<
mean
79.11
file descriptors
SD
11.65
file descriptors
<
p95
92
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 10 Sep 2026 13:35:32 UTC
Duration
1800s
Peer count
150
Peer count at end
120
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_34428490451_2
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
355.96
/s
Max
Highest per-partition mean rate
max
1074.82
/s
Min
Lowest per-partition mean rate
min
111.31
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.027777
s
SD
0.250703
s
Max
Highest per-partition mean latency
max mean
0.064815
s
Min
Lowest per-partition mean latency
min mean
0.006396
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
57
conductors
<
mean
73.42
conductors
SD
12.19
conductors
<
p95
93
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
995.123897
s
SD
475.869886
s
Max
Highest per-partition mean latency
max mean
1748.076923
s
Min
Lowest per-partition mean latency
min mean
522
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
42.847223
s
SD
45.142953
s
Max
Highest per-partition mean latency
max mean
72.336872
s
Min
Lowest per-partition mean latency
min mean
26.434054
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
35.210272
s
SD
45.38003
s
Max
Highest per-partition mean latency
max mean
50.50181
s
Min
Lowest per-partition mean latency
min mean
19.053819
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk-vnG4AicyIYb-_ZdNUaE6fD2ii8HdXqMdfzXobuq2L_bHxn_
p5
0
<
mean
0.18
SD
0.38
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0GdwdYE8S6z7CD7kVVnxCTs8aB2JAMILbeCW0DSLGfC5MCXA
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0i9Zne2MJ8rzM-D__sV9q8fqy2hZmejDExOnzdH8k9uVXWFQ
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1-xKlXDuoFzVmG7ZzB3wQ5u2DAEPKeRatnmYd9wS6mxEshzY
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1rvOXFycQCoMs4CHUS2v-sn66zZB2tt8gmAVmZ2HSIJzSRXB
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2D_wrjZSNsX6YgKu2BXbpCSPAJTGj3pmXDq8k8Avddhb6x-g
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2W7aR2dzq10V6bNv75d2pcvKZzq5iI_6ks19O4nfTQNPx9Ph
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk3H0MYvQIigXQ57Hg8zC4QXp-79GkX1efCgk69oAx8VAAbFgt
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4mROuT3xtJ80X1WbO09kyQhhyW5C4Kby8asYLnSMiLCwQ6eS
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk5VKOPDHA2yVtFhzICwwuuyOG25YUeddpQ8JeVW9HSbaq_vRy
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk60mg7ycrekxPmzS7viD7WQ_YcCuL-Yp79gUds2vd1ahecJyT
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk7PCtg_nb7zpGFkKPfdZvYV4O7EegcNFXDVLmgT8oeWaHC9g4
p5
0
<
mean
0.06
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk80s5UlgId4us8qgGUeijBaEbQzTttcpdbvN0R04j4pGtew7K
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk8EpT4tAOrd6y9aCnkJo3oRrJUQR6VWeNQJp__-Tjud8p7hqm
p5
0
<
mean
0.03
SD
0.18
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk8MEcfaojNHNdgMZRbRNN2d6EPstDHGYIGZZyaXiMhsShsrfq
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk8uNzrQ5OuNTDVlihqd4wUibD1bneWyKJI1nJZsR1UHqi8yOg
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk9JyYWeoDeFwJFZwpPculj2ajJo9Pib3q67FpOk1J99_NYiB8
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk9w-bh8dk4pu_NMZh9rptZCicaTLb3dvs1WWgbjeA5amticnj
p5
0
<
mean
0.03
SD
0.16
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkAQm0v7_X8lcoiRCtKocLm-iivpDd-M7kdmt1D5MEwwgAjUXH
p5
0
<
mean
0.14
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkB6wYTlcHAJGdEVmR6E-VZSJP57ApNWc6mc5N2p2YKKmAs-EQ
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkCnpmB9gTK18InLstvj0gfkJlzpYqhOpLsWDztuINVGu73JjQ
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkDMVJKUcUNN5ihL7NmiiFrvIoKtfeOTbTlyCJZomXW5eOKBzs
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkDvxQEwX9oVuoyA0z6LfZ0eJwQAXHN31pfzaTGuMqB4Ir3m5N
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkESrNolqM7AXJyoYCiRi_3yt1UzP17xlSUEJiYDEcNBUkJdgX
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkEoBPzui0l-X1fKQoftAcP7fKTKW5bVJBOTLnvtZqfKapN32_
p5
0
<
mean
0.06
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFBJyg5vqcX50MR0cENwCvXrBCpK4DlTk1Hs1mhVNEqPPGcSs
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFSxB4Mbs19BEgo7EUDeBBjVWfiMaiBD3d7HVIXuXVNKY_U1x
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFhjRmDVEliGofKUrmxGHQ-J8O5k2fHt8fZOcNUi9W_A7AkAg
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkH3oksHQHVIHxBTiodvaaEXGYMxlVVhNU1kdiWS7vJIA_qFSk
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHRSmydb0MRqlqAo6O3VaDbb9ahmwT_YFo1xVQTH8-HCYhpGv
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkH__fuBoXQvh5KoL9tXNsmXCzZerTGBOnBN09m68mHJTlGdAq
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHimQi03AR8cVIoVvQbcM7oyAsvC97oNxMjyc7CoUOliYCf85
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHzpEM5ax5ZQD0OCjwLrY2a6I6sTOHF33QQz-DWX-9d_5oX1_
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkI1OeydF5U57m6XdQ5TwyC5DgmPJcMOkK2x9czepCkQP0oW-s
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkICQ0UxFYmeQew4-dQeye7f90huPI-JDozz-3Cc0oYmffnl8Q
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIKTtfFEGPffp_ynop0PEc-OJSgHtVL6aIuSGz-X3hjwwNbYb
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIQoP43I3Ebaa_Fdu2sEy96fnGnlzu38NgeQKlEHBu1K4Nl2n
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkIftOVRd4wx27C4NBU0RuRYf5w2wHtNENk1jiZvbaW7iEE5Zd
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkJWSKkqAm2q2Z3xLq2Zoz_ZfOMQQZemifPO1MQueO3GYK9r7k
p5
0
<
mean
0.03
SD
0.16
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkKQAG_wqUtcu0MGoX-mktbQvjgtZQ8_3SRSxqwnUgYWvh_bv3
p5
0
<
mean
0.03
SD
0.18
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkLGVz4xxecuwXYCM9O4Q9l-H2Bq4Av6zjqw8fkA0k9Irg4gZr
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMe9i_iiMP8l2kpLyl82oLsmmd9cGe8OOb3zR6flcHjC1_6I4
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMw1NBiZ3EYyy7f5xtUGsJwz5jaLh9KlGBbC-IzVT0TbpHJ1a
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkNGEO4pdfvtllpTjjBF6lUgtgUnt4gqNl2eTdNx0mKA_T6xLr
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQ8Mh_uPT2BnEndf2ZmWBhaioUcl6Pc4_u9OwuyJAug6rGXT-
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQEwWIBfCSLfQQmzSG28-cA7o21umwW_mBqCq9YI5Pbz6mTtg
p5
0
<
mean
0.06
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQnDkclmxKtjF29eD05qa262B_oarvmbUbXslvanf5EdGeiFP
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRi3PJnwIGxxgc30eQJHWes21ruKddqGj8cspnTHUuP_jpll3
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSUmtBs2KlX6_-o7zUld9BfyShv1x0Q9DM6MRoJjEgs1KVFfO
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSmnO03iCAlh0DXvbs2cVBlf6L5yDFcOGqYQOLz9TE9fMan0-
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTH1Whs6h-yYRWBUdxvIjX84xCSS13hXerhlcfssiIkIA9EEG
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTHAvKh5m7X32eciAK7DGhdkGOAFkquDX_gPNcFf-c-Xno_WJ
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkTM6yrC2gFGWbTKnqAT4DVnZfEtPXLf_uXjbeHljRk7gyegOl
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVBf2F7UlE1m_-YQdpyn8KgJ_gUDxOWITGwKRuSHOOiAIqVGU
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVZdWIidj-K5tE7MldktGiRhOxU8MHXWih4_UWlDZWu0Fx0E7
p5
0
<
mean
0.06
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkXIqc_bsYiAAJ73CpB3Smby8CxXH3t4M9XjnA-HtXxSl9L-Io
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYAVQWEKnwVxY51ZgOwiz49G90n146n5qm7gOIG6JtcLsASZD
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYZvUUP43wcvviTY-aUq4m4hvBZcjM3h6-y1CDsRxTRm7xtec
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkZFRvRKU76eeoj79QZybAmer8pJ8s_lLaymJ2uhzOB4B2SO0j
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkZsq1HY4emmDMn_bIkGrx2O0S0a7sDZJNzVxmfmm3PDKf4MlW
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_2wrTbKjEquZQi_pdU1_nUAV-A4tssgv4gpq09d-AzevMmPX
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_LPkhxbhzEX62iRO8QMUPjpmfelh0ocUfuJv8XZuz0Gf2w1-
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_TCWamFQfGx8tJhYiA3OF2YqdPVdScpmMhgXV_CkjS4AYtc3
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk_ULEfTEnKEunQ7WsHtvigdDDv_f5voQZaoaydtM6iyPy5W0p
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkaWwF5RF0mf5CQN-SvPl7pbwNOg56T8Y_SEzXCjhTuVJEVx0P
p5
0
<
mean
0.03
SD
0.17
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkaw4l4ZTinYND3fg1n4St7vJxJ05cwnHSdD-bik9j4wQxvhDM
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkbZcU-FjNCkqVo3ShKMOVs1vg5LrSl05nMsGSgWNQknH6gtUm
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkciIA2afuowZtOQaaWvab_27CgeE7SEPxnIiIXX6YkqEHq87y
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkeQsTFDnDIhP9LY9-ukyAFuOs4uCFpazPIi18tSrGti8-0E5p
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkfyaQcgr18TSC7tobXM9DhA4eH14061phJ5QJ-ffonPbrRM6a
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgBcG_x8nltJd2SNdiJLWbkI_atjAkiu9RMnn2Io8_dkzt-JI
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkghDfzqwYWw0q4cyqAejFi6BG4HuosBltDJ5jtYD6kVpzf67U
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkgkRcNvs2xs0pzkilI_BbQrvjEIuaSPVxCvc4-FZ67oHHzjVx
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkh-fluiWKGjEiwXHLpzlkATdN6JoLNdVVa6pJXyJNmJJFRQ1n
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkieDQurBcJSEKP06idoY54zMOU3046ggSjlFKYETA9wf6lkfj
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjFveySDroaz6hHMoXZ0CDO5P90a7iutZU4h7IAQwI_M7tjYQ
p5
0
<
mean
0.16
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjX10Pd2D6OuJu6W2b9DFrx3p_yI0A3UgYPgdlqqDxYzx2MNn
p5
0
<
mean
0.15
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjdRHnhINsmAJyjOX8MqIbLamB0FSFIIHHoT7fGkldiWVZ-T4
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjow6CbcEzUDsp9ukLTsLxhx_HqIhGGF19iGuyt8ajLBchtuB
p5
0
<
mean
0.12
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkRsUIcMRQmLX7wAdEkkqrNXmAW9ACUNV-j1cSLlOXhmr0T0M
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkkvsgoUjhkdC0XXjIbz-_ADHRATw4DZj81lo9hTPfFPqMiULu
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmTOh1mrbfwx93xTKOJnMqxMaupfxeJ3Rk4ICyvuLspKWbDP7
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmuwByySom3auhYwdTHrGzwjH2Wew8Txo3IfEb2-Usf9CZq6a
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkn3__r9iWCPmYsL7is25sPimRAWli0yfWTG-itS8vGTB-Jmji
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAknUl8nYCskhGLKirQF5PTAwmKM8EiSruvb2pXdfaN33MjYkck
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAknjmE3pnbYTr2NoaDkEjQE3ZT7tY5RnCt__F9j-iFxKVWfuHx
p5
0
<
mean
0.11
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkp5GtUAG-2MCze38yDEfH8bODaQeCG9MD9iUMVnuQj96RnPCM
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkp9z54qnnLdRHiG5J5GWFBsxulxYeaLe8IZbrazXF-GPTpkxD
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpFuxbh7FSCPh_rp1-a-5T9wQouMucfPec-NZoo-Srv0jgr_d
p5
0
<
mean
0.05
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpHX9tlC-j0-q2-zAhRZGYh0PLgGmqvPgcgKVsX7Lc9Sqgohd
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkqP5-qAzicIgFFEpVsiLS7Kxy200pV7uDBOpq9jADQJakkXRH
p5
0
<
mean
0.04
SD
0.19
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkqd-qNjbFGlNXhbXDzm036KD4sRSSGmCJuS_5g5fqGSST5Dc-
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrOA2deNnzUuxmzvWzgzxeF7vOo4AcKEvnyfJb0b4Q1ylkGq_
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrTZBGaOTc6CRVLuGmINTavUZ7x5Kbocvx_1Xgi-sFXcsELAs
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrkkkZL87ybCHCB90D2fDor67j-Hn0nlQ0xzIhSUdOt2U-i2T
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkssbZ5jl46F4DIkEssTnHANCkCuLsd2PUH9CFcnLzfmYZOXKL
p5
0
<
mean
0.03
SD
0.18
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAku5epW3Hh_5gauR-YeJMrz_-HNfw_6uoYZpEh9-qy5lTEkrpu
p5
0
<
mean
0.09
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkugAUyNV4yK3GNztphlsf3m0Ozmn6eSJXhpmRBkamDq03nM8f
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkwDj_yK31RKaf0RnVx54pKTeEI9OaZtMtzziXI6ZkaKWuJ7h3
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkwJdK4hyGomVbWXXKEjKodk4CZmIkHpjlOgEZYZKZxL2SO5sR
p5
0
<
mean
0.15
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkx-1fkBsW08il7s4bYLBHlojtfPyyFut9ebPrbtaMlpmDOY1D
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkxVAlrs0zcxIFrl1H8br7IZN1ef_Eq_CMb4zsMn3d6gmFF6Wd
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkzP_j_sJSdv1-Vl39bG9S5H3lmTfm09zzYLYU6Bsdi4Kt4AXf
p5
0
<
mean
0.08
SD
0.27
<
p95
1
Error count
The number of errors accumulated.
762
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.020644
s
SD
25.513607
s
p50
0.001148
s
<
p95
22.007567
s
<
p99
153.703316
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
2
over 1700.434864464s
mean rate
112.84
/s
std
833.66
/s
p5
0
/s
p95
379.26
/s
peak
38158.71
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007384
s
SD
0.00189
s
p50
0.007265
s
<
p95
0.010041
s
<
p99
0.012213
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006685
s
SD
0.001637
s
p50
0.007365
s
<
p95
0.009046
s
<
p99
0.009289
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.010885
s
SD
7.826889
s
p50
0.006663
s
<
p95
7.622331
s
<
p99
22.733189
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.011808
s
SD
5.971593
s
p50
0.228406
s
<
p95
16.272495
s
<
p99
22.324483
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006091
s
SD
0.001538
s
p50
0.005794
s
<
p95
0.008682
s
<
p99
0.008898
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005388
s
SD
0.001295
s
p50
0.006033
s
<
p95
0.007191
s
<
p99
0.00744
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.022523
s
SD
26.143641
s
p50
0.003536
s
<
p95
22.013296
s
<
p99
153.780671
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.013301
s
SD
9.577602
s
p50
0.17281
s
<
p95
18.272162
s
<
p99
43.769697
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001773
s
SD
0.000463
s
p50
0.002024
s
<
p95
0.002336
s
<
p99
0.002771
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.8e-05
s
SD
3e-05
s
p50
4.8e-05
s
<
p95
0.000121
s
<
p99
0.00015
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000617
s
SD
0.000181
s
p50
0.000647
s
<
p95
0.000898
s
<
p99
0.000925
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000596
s
SD
0.000178
s
p50
0.000594
s
<
p95
0.000836
s
<
p99
0.001233
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.02118
s
SD
26.14362
s
p50
0.001198
s
<
p95
22.011168
s
<
p99
153.779525
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.010003
s
SD
9.577066
s
p50
0.168567
s
<
p95
18.268019
s
<
p99
43.765778
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
1.1e-05
s
p50
1.5e-05
s
<
p95
3.6e-05
s
<
p99
6.9e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002782
s
SD
0.000889
s
p50
0.003024
s
<
p95
0.004377
s
<
p99
0.004976
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000382
s
SD
0.000331
s
p50
0.000398
s
<
p95
0.000675
s
<
p99
0.000785
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
3.43
s
<
mean
337.71
s
SD
496.73
s
<
p95
1497.34
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
26791
over 1789.641276434s
mean rate
356521.76
/s
std
2.51345239e+06
/s
p5
0
/s
p95
1.1962838e+06
/s
peak
1.183575017e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
125.674057
s
SD
111.078028
s
p50
42.056838
s
<
p95
307.713095
s
<
p99
546.501002
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.087373
s
SD
0.232027
s
p50
0.019518
s
<
p95
0.59687
s
<
p99
1.250717
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
2e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.032774
s
SD
0.300926
s
p50
0.01385
s
<
p95
0.357845
s
<
p99
1.285433
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
1.262089
s
SD
1.155883
s
p50
0.088704
s
<
p95
3.222733
s
<
p99
4.776273
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
17.853929
s
SD
45.509698
s
p50
12.750144
s
<
p95
120.056685
s
<
p99
252.375199
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
16.424306
s
SD
35.208324
s
p50
0.015532
s
<
p95
54.026633
s
<
p99
214.511256
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004987
s
SD
0.003233
s
p50
0.000487
s
<
p95
0.009734
s
<
p99
0.013306
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007732
s
SD
0.104556
s
p50
0.003369
s
<
p95
0.022537
s
<
p99
0.275394
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000498
s
SD
0.001358
s
p50
0.000762
s
<
p95
0.003737
s
<
p99
0.006657
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
2.660027
s
SD
28.605557
s
p50
2.950373
s
<
p95
60.002775
s
<
p99
60.003954
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
3.016681
s
SD
5.255424
s
p50
1.220355
s
<
p95
12.389775
s
<
p99
21.043906
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.885275
s
SD
21.895142
s
p50
1.540378
s
<
p95
60.001402
s
<
p99
60.00196
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.672993
s
SD
27.299726
s
p50
4.047356
s
<
p95
60.001756
s
<
p99
60.003156
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
26.13
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.86
KiB/s
<
mean
102.23
KiB/s
SD
199.57
KiB/s
<
p95
327.50
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
192
B/s
<
mean
16.34
KiB/s
SD
44.08
KiB/s
<
p95
68.74
KiB/s
Total bytes received
Total bytes received on primary network interface
count
25.20
GiB
mean
14.41
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.03
GiB
mean
2.30
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.5%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 16% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.55
%
<
mean
35.81
%
SD
30.59
%
<
p95
90.45
%
System
9.69
%
CPU percentiles
CPU usage percentiles
p50
32.03
%
p95
90.45
%
p99
96.48
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
92
hosts
mean time
368.25
s
Memory used percentage
Percentage of memory used
p5
5.84
%
<
mean
9.91
%
SD
7.14
%
<
p95
21.87
%
Memory available percentage
Percentage of available memory
p5
78.13
%
<
mean
90.09
%
SD
7.14
%
<
p95
94.16
%
Max host memory used
Maximum memory usage percentage across all hosts
max
74.81
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
7.19
%
Memory growth rate
Rate of memory growth over time
growth
0.12
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.41
MiB/s
Disk write throughput
Disk write throughput in MiB/s
53.47
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/109
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/16
hosts
Mount Point /etc/hosts
0/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/21
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/16
hosts
Mount Point /nix/store
0/4
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.25
5 min
1.39
15 min
1.59
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
16.44
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
12.3457
%
SD
16.8306
%
<
p95
49.47
%
60 second average
12.1842
%
300 second average
14.0271
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.3589
%
SD
2.7568
%
<
p95
6.22
%
60 second average
1.3870
%
300 second average
1.9360
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.7969
%
SD
1.8892
%
<
p95
3.57
%
60 second average
0.8024
%
300 second average
1.0148
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0
%
<
mean
33
%
SD
31.87
%
<
p95
87.68
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
53.68
KiB
<
mean
175.34
KiB
SD
100.14
KiB
<
p95
345.94
KiB
Holochain process threads
Number of threads in Holochain process
p5
5
threads
<
mean
27.34
threads
SD
8.92
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
23
file descriptors
<
mean
73.49
file descriptors
SD
20.49
file descriptors
<
p95
89
file descriptors

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Thu, 10 Sep 2026 14:07:58 UTC
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_34428490451_2
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
508.56
Max
Highest chain head value observed for any single write agent.
1171
Write agents observed
9
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
228.37155
s
SD
66.285541
s
Max
Highest per-partition mean latency
max mean
1194.167079
s
Min
Lowest per-partition mean latency
min mean
9.722
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
2.18
/s
Max
Highest per-partition mean rate
max
7.25
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.593171
s
SD
0.087086
s
Max
Highest per-partition mean latency
max mean
15.292342
s
Min
Lowest per-partition mean latency
min mean
0.01914
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
40084
Agents affected
86 / 87
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
461
Max
Highest count in any single partition
788
Min
Lowest count in any single partition
0
Final error count
The total number of all types of error accumulated over the run by all agents.
120431
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.022278
s
SD
0.270657
s
p50
0.001872
s
<
p95
0.058165
s
<
p99
0.598651
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
11
over 738.970143943s
mean rate
2.34
/s
std
2.82
/s
p5
0
/s
p95
8.42
/s
peak
8.46
/s
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007387
s
SD
0.002598
s
p50
0.007136
s
<
p95
0.011444
s
<
p99
0.014708
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.269495
s
SD
0.272569
s
p50
0.15404
s
<
p95
0.832025
s
<
p99
1.296972
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.014233
s
SD
0.010938
s
p50
0.010046
s
<
p95
0.037373
s
<
p99
0.053261
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.02468
s
SD
1.060869
s
p50
0.009082
s
<
p95
0.023187
s
<
p99
10.344665
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.02184
s
SD
0.434124
s
p50
0.009031
s
<
p95
0.574565
s
<
p99
0.889648
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006075
s
SD
0.002092
s
p50
0.006044
s
<
p95
0.009342
s
<
p99
0.011708
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.26732
s
SD
0.271347
s
p50
0.151962
s
<
p95
0.826538
s
<
p99
1.291624
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.012086
s
SD
0.009878
s
p50
0.008129
s
<
p95
0.033068
s
<
p99
0.048979
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009408
s
SD
0.227494
s
p50
0.007632
s
<
p95
0.02394
s
<
p99
0.106841
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.019904
s
SD
0.43425
s
p50
0.007373
s
<
p95
0.57382
s
<
p99
0.888012
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001813
s
SD
0.001253
s
p50
0.00192
s
<
p95
0.00449
s
<
p99
0.006116
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.016215
s
SD
0.048172
s
p50
0.012357
s
<
p95
0.056271
s
<
p99
0.094581
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
7e-05
s
SD
4.5e-05
s
p50
4.1e-05
s
<
p95
0.000155
s
<
p99
0.000206
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.006058
s
SD
0.00914
s
p50
0.001639
s
<
p95
0.020721
s
<
p99
0.054586
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.007801
s
SD
0.010301
s
p50
0.00075
s
<
p95
0.023045
s
<
p99
0.064207
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002966
s
SD
0.349345
s
p50
0.000963
s
<
p95
0.052345
s
<
p99
0.67742
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.02329
s
SD
0.049419
s
p50
0.015354
s
<
p95
0.060214
s
<
p99
0.112527
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.102529
s
SD
0.07911
s
p50
0.058318
s
<
p95
0.238141
s
<
p99
0.295629
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.8e-05
s
SD
1e-05
s
p50
1.3e-05
s
<
p95
2.3e-05
s
<
p99
3.1e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003632
s
SD
0.00306
s
p50
0.003602
s
<
p95
0.0105
s
<
p99
0.015828
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002298
s
SD
0.002569
s
p50
0.000401
s
<
p95
0.006794
s
<
p99
0.012423
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.75
s
<
mean
424.59
s
SD
245.7
s
<
p95
805.23
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
106
over 888.539238757s
mean rate
700163.12
/s
std
1.916706802e+07
/s
p5
1712.52
/s
p95
1.26356273e+06
/s
peak
1.40043398431e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
77.476745
s
SD
49.219502
s
p50
23.937865
s
<
p95
140.708248
s
<
p99
217.042865
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.207542
s
SD
1.170285
s
p50
2.147671
s
<
p95
4.288736
s
<
p99
5.560071
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
2e-06
s
SD
1.9e-05
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.978018
s
SD
0.503987
s
p50
0.569184
s
<
p95
1.548861
s
<
p99
1.776658
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.670254
s
SD
0.56112
s
p50
0.136292
s
<
p95
1.32015
s
<
p99
2.752507
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
14.337357
s
SD
18.57494
s
p50
17.092559
s
<
p95
53.036147
s
<
p99
101.16937
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
10.333798
s
SD
23.653851
s
p50
0.215049
s
<
p95
40.359357
s
<
p99
125.585731
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.00478
s
SD
0.003701
s
p50
0.004429
s
<
p95
0.011901
s
<
p99
0.014367
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005228
s
SD
0.003421
s
p50
0.003765
s
<
p95
0.011022
s
<
p99
0.016205
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.004545
s
SD
0.007646
s
p50
0.001444
s
<
p95
0.017805
s
<
p99
0.04149
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.18202
s
SD
27.999832
s
p50
0.758967
s
<
p95
60.003797
s
<
p99
60.01167
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.355359
s
SD
2.223841
s
p50
0.348167
s
<
p95
3.074131
s
<
p99
10.332862
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.994532
s
SD
0.955148
s
p50
0.326437
s
<
p95
2.577832
s
≤
p99
2.577832
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.508633
s
SD
27.390452
s
p50
3.485032
s
<
p95
60.011604
s
<
p99
60.03288
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
48.66
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
116
B/s
<
mean
548.93
KiB/s
SD
550.86
KiB/s
<
p95
1.65
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
90.28
KiB/s
SD
119.78
KiB/s
<
p95
316.51
KiB/s
Total bytes received
Total bytes received on primary network interface
count
66.49
GiB
mean
76.50
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.93
GiB
mean
12.58
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 79% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
15.56
%
<
mean
70.53
%
SD
23.7
%
<
p95
93.28
%
System
21.88
%
CPU percentiles
CPU usage percentiles
p50
79.87
%
p95
93.28
%
p99
98.20
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
133
hosts
mean time
488.65
s
Memory used percentage
Percentage of memory used
p5
6.69
%
<
mean
12.6
%
SD
7.58
%
<
p95
26.7
%
Memory available percentage
Percentage of available memory
p5
73.3
%
<
mean
87.4
%
SD
7.58
%
<
p95
93.31
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.27
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
7.23
%
Memory growth rate
Rate of memory growth over time
growth
0.51
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.10
MiB/s
Disk write throughput
Disk write throughput in MiB/s
141.76
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/15
hosts
Mount Point /etc/hosts
0/15
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
0/15
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.75
5 min
2.95
15 min
2.20
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
78.62
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.19
%
<
mean
47.4561
%
SD
30.3447
%
<
p95
85.92
%
60 second average
44.3056
%
300 second average
34.1787
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.2297
%
SD
7.1939
%
<
p95
19.14
%
60 second average
7.7017
%
300 second average
5.6703
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.109
%
SD
3.6128
%
<
p95
10.25
%
60 second average
2.9471
%
300 second average
2.3160
%
Holochain process CPU usage
CPU usage by Holochain process
p5
22.92
%
<
mean
68.66
%
SD
20.02
%
<
p95
89.74
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
144.93
KiB
<
mean
392.35
KiB
SD
110.47
KiB
<
p95
538.85
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
35.38
threads
SD
6.79
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
73
file descriptors
<
mean
81.69
file descriptors
SD
13.73
file descriptors
<
p95
90
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Thu, 10 Sep 2026 15:35:41 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_34428490451_2
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
46.02
/s
Max
Highest per-partition mean rate
max
81.13
/s
Min
Lowest per-partition mean rate
min
28.39
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
123.423907
s
SD
37.681048
s
Max
Highest per-partition mean latency
max mean
593.157709
s
Min
Lowest per-partition mean latency
min mean
3.316528
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
24.81
/s
Max
Highest per-partition mean rate
max
332
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0004
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
246816
Agents affected
59 / 59
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4183
Max
Highest count in any single partition
7400
Min
Lowest count in any single partition
2550
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
5106
Agents affected
48 / 58
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
88
Max
Highest count in any single partition
769
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
47.66
SD
36.27
<
p95
145
zero-arc
p5
2
<
mean
31.66
SD
23
<
p95
77
Error count
The number of errors accumulated across all nodes.
765
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.476964
s
SD
1.919402
s
p50
0.690674
s
<
p95
3.694597
s
<
p99
10.507323
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 800.306666022s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
271321534
over 883.683403505s
mean rate
1.90032272263e+09
/s
std
4.32872183857e+09
/s
p5
6.6259585e+06
/s
p95
9.86489330724e+09
/s
peak
3.49248304851e+10
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
32721210
over 879.886699025s
mean rate
1.16074807677e+09
/s
std
3.00726985236e+09
/s
p5
5.2644882e+06
/s
p95
7.43779829458e+09
/s
peak
6.16849581908e+10
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
65160
over 889.138395102s
mean rate
1.73085709574e+09
/s
std
1.83824485776e+10
/s
p5
0
/s
p95
4.14412680043e+09
/s
peak
1.14285863874e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008709
s
SD
0.004161
s
p50
0.00895
s
<
p95
0.016853
s
<
p99
0.019356
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
5.12332
s
SD
70.242103
s
p50
13.920366
s
<
p95
179.132485
s
<
p99
329.61025
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007627
s
SD
0.003659
s
p50
0.007851
s
<
p95
0.014865
s
<
p99
0.017192
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
5.121852
s
SD
70.242037
s
p50
13.91901
s
<
p95
179.131752
s
<
p99
329.6084
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001467
s
SD
0.000612
s
p50
0.00181
s
<
p95
0.002702
s
<
p99
0.003243
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000663
s
SD
0.000613
s
p50
0.000591
s
<
p95
0.002288
s
<
p99
0.002772
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000639
s
SD
0.000552
s
p50
0.000567
s
<
p95
0.002006
s
<
p99
0.00245
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.477049
s
SD
2.307232
s
p50
0.619723
s
<
p95
2.97308
s
<
p99
6.208821
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.475873
s
SD
2.737541
s
p50
1.586135
s
<
p95
7.692958
s
<
p99
13.258687
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002304
s
SD
0.001002
s
p50
0.002813
s
<
p95
0.004342
s
<
p99
0.004953
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.00045
s
SD
0.000504
s
p50
0.000246
s
<
p95
0.001395
s
<
p99
0.002194
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.75
s
<
mean
426.29
s
SD
246.57
s
<
p95
808.47
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
15.440171
s
SD
16.759933
s
p50
0
s
<
p95
8.563972
s
<
p99
58.189073
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.101554
s
SD
0.116976
s
p50
0.00231
s
<
p95
0.167522
s
<
p99
0.502793
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.036785
s
SD
0.348674
s
p50
0.002383
s
<
p95
1.121805
s
<
p99
1.315195
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.272771
s
SD
0.16631
s
p50
0.026701
s
<
p95
0.421105
s
<
p99
0.547217
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.9932
s
SD
8.070835
s
p50
0.003662
s
<
p95
21.56876
s
<
p99
43.142509
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.598258
s
SD
3.519165
s
p50
0.00185
s
<
p95
1.165486
s
<
p99
11.498828
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002574
s
SD
0.00196
s
p50
0.001102
s
<
p95
0.005696
s
<
p99
0.007004
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004212
s
SD
0.00418
s
p50
0.003483
s
<
p95
0.01384
s
<
p99
0.017372
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000786
s
SD
0.005403
s
p50
0.000645
s
<
p95
0.002174
s
<
p99
0.011293
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.450641
s
SD
29.239333
s
p50
1.666717
s
<
p95
60.002596
s
<
p99
60.026486
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.198156
s
SD
14.873916
s
p50
0.515553
s
<
p95
60.000908
s
<
p99
60.002124
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.575042
s
SD
18.209825
s
p50
0.210391
s
<
p95
60.00072
s
<
p99
60.000958
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
31.27
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
2.44
KiB/s
<
mean
549.75
KiB/s
SD
530.75
KiB/s
<
p95
1.52
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
98.11
KiB/s
SD
169.73
KiB/s
<
p95
385.94
KiB/s
Total bytes received
Total bytes received on primary network interface
count
68.08
GiB
mean
78.33
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
12.15
GiB
mean
13.98
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 92.3%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (16.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 35% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.27
%
<
mean
43.55
%
SD
35.67
%
<
p95
89.08
%
System
12.28
%
CPU percentiles
CPU usage percentiles
p50
40.86
%
p95
89.08
%
p99
92.32
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
89
hosts
mean time
446.09
s
Memory used percentage
Percentage of memory used
p5
6.5
%
<
mean
11.55
%
SD
8.06
%
<
p95
26.36
%
Memory available percentage
Percentage of available memory
p5
73.64
%
<
mean
88.45
%
SD
8.06
%
<
p95
93.5
%
Max host memory used
Maximum memory usage percentage across all hosts
max
77.90
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
16.05
%
Memory growth rate
Rate of memory growth over time
growth
0.43
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.86
MiB/s
Disk write throughput
Disk write throughput in MiB/s
130.18
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/109
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/4
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.74
5 min
1.73
15 min
1.88
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
35.37
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
17.9604
%
SD
20.9749
%
<
p95
55.86
%
60 second average
17.1698
%
300 second average
16.7168
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.6474
%
SD
4.706
%
<
p95
13.98
%
60 second average
2.5237
%
300 second average
2.2971
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.3396
%
SD
2.7868
%
<
p95
6.89
%
60 second average
1.2738
%
300 second average
1.1946
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.95
%
<
mean
39.67
%
SD
33.92
%
<
p95
85.14
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
205.94
KiB
<
mean
276.56
KiB
SD
72.28
KiB
<
p95
420.88
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
31.41
threads
SD
5.51
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
74.99
file descriptors
SD
11.17
file descriptors
<
p95
89
file descriptors

Zero-Arc Create Data

↑ Back to index
A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 10 Sep 2026 14:42:57 UTC
Duration
900s
Peer count
150
Peer count at end
143
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data--high_zero_34428490451_2
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
50.21
/s
Max
Highest per-partition mean rate
max
85.55
/s
Min
Lowest per-partition mean rate
min
28.99
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
554.72099
s
SD
224.799035
s
Max
Highest per-partition mean latency
max mean
667.672484
s
Min
Lowest per-partition mean latency
min mean
323.348542
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
50.44
/s
Max
Highest per-partition mean rate
max
142
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0004
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
620582
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4497
Max
Highest count in any single partition
7771
Min
Lowest count in any single partition
2566
Received count
Total number of entries received by full-arc nodes, by agent.
Total
2079
Agents affected
4 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
231
Max
Highest count in any single partition
1384
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
9
<
mean
34.49
SD
29.45
<
p95
126
zero-arc
p5
2
<
mean
24.69
SD
20.31
<
p95
63
Error count
The number of errors accumulated across all nodes.
7
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.001576
s
SD
0.001818
s
p50
0.000609
s
<
p95
0.005748
s
<
p99
0.006463
s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
490654537
over 860.638693601s
mean rate
3.31430511878e+09
/s
std
1.61186809763e+10
/s
p5
1.345642307e+07
/s
p95
1.3097105646e+10
/s
peak
7.08510524109e+11
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
196116588
over 876.268692236s
mean rate
2.56169735012e+10
/s
std
4.66795029685e+10
/s
p5
8.390538798e+07
/s
p95
1.33138122567e+11
/s
peak
2.72681603506e+11
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
63213888
over 890.007700094s
mean rate
7.7365220516e+08
/s
std
5.23915040654e+09
/s
p5
87482.39
/s
p95
2.46981319521e+09
/s
peak
2.66890769147e+11
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007805
s
SD
0.003846
s
p50
0.008029
s
<
p95
0.016327
s
<
p99
0.017363
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.016128
s
SD
0.0377
s
p50
0.010581
s
<
p95
0.108018
s
<
p99
0.185058
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006807
s
SD
0.003317
s
p50
0.00697
s
<
p95
0.014231
s
<
p99
0.015052
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.014934
s
SD
0.037536
s
p50
0.00917
s
<
p95
0.106509
s
<
p99
0.18348
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001424
s
SD
0.000633
s
p50
0.001442
s
<
p95
0.002697
s
<
p99
0.002821
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000447
s
SD
0.000248
s
p50
0.000459
s
<
p95
0.001006
s
<
p99
0.001148
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000435
s
SD
0.000257
s
p50
0.000424
s
<
p95
0.00104
s
<
p99
0.00114
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001866
s
SD
0.002211
s
p50
0.004345
s
<
p95
0.006514
s
<
p99
0.006681
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.000696
s
SD
0.001318
s
p50
0.000664
s
<
p95
0.005022
s
<
p99
0.006074
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002229
s
SD
0.001187
s
p50
0.002192
s
<
p95
0.004576
s
<
p99
0.006166
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000401
s
SD
0.000374
s
p50
0.000486
s
<
p95
0.001156
s
<
p99
0.001392
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.69
s
<
mean
426.07
s
SD
246.47
s
<
p95
808.19
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 890.007626002s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
0.610542
s
SD
14.727059
s
p50
0
s
<
p95
0.434543
s
<
p99
37.324642
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.08583
s
SD
0.122553
s
p50
0.001952
s
<
p95
0.024204
s
<
p99
0.687816
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.006909
s
SD
0.264894
s
p50
0.001805
s
<
p95
0.104417
s
<
p99
1.294313
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.133394
s
SD
0.100385
s
p50
0.117311
s
<
p95
0.247851
s
<
p99
0.286444
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
4.742838
s
SD
6.257414
s
p50
0.002939
s
<
p95
2.862009
s
<
p99
38.599559
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.038584
s
SD
4.879034
s
p50
0.001539
s
<
p95
0.006122
s
<
p99
0.036727
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.007638
s
SD
0.004323
s
p50
0.008273
s
<
p95
0.015325
s
<
p99
0.017151
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006589
s
SD
0.004733
s
p50
0.004055
s
<
p95
0.014848
s
<
p99
0.019698
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.00037
s
SD
0.006906
s
p50
0.000371
s
<
p95
0.001055
s
<
p99
0.036153
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
17.398143
s
SD
26.853793
s
p50
12.709663
s
<
p95
61.980344
s
≤
p99
61.980344
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
6.33397
s
SD
25.183923
s
p50
1.911675
s
<
p95
60.0622
s
<
p99
60.072445
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
50.19
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.25
KiB/s
<
mean
190.19
KiB/s
SD
291.73
KiB/s
<
p95
527.44
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
470
B/s
<
mean
36.41
KiB/s
SD
40.62
KiB/s
<
p95
122.05
KiB/s
Total bytes received
Total bytes received on primary network interface
count
23.30
GiB
mean
26.81
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.46
GiB
mean
5.13
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.4%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (15.3% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 45% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
16.66
%
<
mean
66.22
%
SD
22.47
%
<
p95
88.24
%
System
16.02
%
CPU percentiles
CPU usage percentiles
p50
74.54
%
p95
88.24
%
p99
94.43
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
111
hosts
mean time
427.26
s
Memory used percentage
Percentage of memory used
p5
6.39
%
<
mean
11.26
%
SD
7.96
%
<
p95
25.75
%
Memory available percentage
Percentage of available memory
p5
74.25
%
<
mean
88.74
%
SD
7.96
%
<
p95
93.61
%
Max host memory used
Maximum memory usage percentage across all hosts
max
76.01
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
15.29
%
Memory growth rate
Rate of memory growth over time
growth
0.26
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.05
MiB/s
Disk write throughput
Disk write throughput in MiB/s
125.81
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.21
5 min
2.13
15 min
2.31
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
45.21
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.05
%
<
mean
21.2903
%
SD
15.0883
%
<
p95
38.58
%
60 second average
20.7186
%
300 second average
20.2631
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.9665
%
SD
3.8349
%
<
p95
7.15
%
60 second average
2.8832
%
300 second average
2.8354
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.9885
%
SD
3.0475
%
<
p95
4.77
%
60 second average
1.9200
%
300 second average
1.7935
%
Holochain process CPU usage
CPU usage by Holochain process
p5
26.34
%
<
mean
64.24
%
SD
18.61
%
<
p95
83.8
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
201.87
KiB
<
mean
272.39
KiB
SD
59.12
KiB
<
p95
376.43
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
32.1
threads
SD
4.14
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
67
file descriptors
<
mean
78.39
file descriptors
SD
9.82
file descriptors
<
p95
87
file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 10 Sep 2026 15:18:04 UTC
Duration
900s
Peer count
150
Peer count at end
145
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data_validated--high_zero_34428490451_2
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
44.62
/s
Max
Highest per-partition mean rate
max
79.93
/s
Min
Lowest per-partition mean rate
min
26.67
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
95.404304
s
SD
5.63924
s
Max
Highest per-partition mean latency
max mean
458.689296
s
Min
Lowest per-partition mean latency
min mean
7.721631
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0002
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
536250
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3886
Max
Highest count in any single partition
6802
Min
Lowest count in any single partition
2266
Received count
Total number of entries received by full-arc nodes, by agent.
Total
970
Agents affected
4 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
108
Max
Highest count in any single partition
769
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
8
<
mean
46.43
SD
28.08
<
p95
92
zero-arc
p5
2
<
mean
24.82
SD
22.23
<
p95
66
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.004842
s
SD
0.005273
s
p50
0.009059
s
<
p95
0.018949
s
<
p99
0.021363
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
4980
over 750.387046936s
mean rate
8248.46
/s
std
11628.39
/s
p5
17.09
/s
p95
38443.19
/s
peak
46662.95
/s
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
176466192
over 841.413813899s
mean rate
4.87381926283e+09
/s
std
3.76245263222e+10
/s
p5
1.90936976e+07
/s
p95
1.32038428239e+10
/s
peak
1.49185141765e+12
/s
WASM usage: 'timed_and_validated::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
559245297
over 883.484174324s
mean rate
4.05201309127e+09
/s
std
6.58783840203e+09
/s
p5
2.0167792585e+08
/s
p95
2.1174165257e+10
/s
peak
2.50544533363e+10
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
98097912
over 888.952984159s
mean rate
2.67113355171e+09
/s
std
1.9453719763e+10
/s
p5
0
/s
p95
7.60313025507e+09
/s
peak
8.17419050653e+11
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
820773748
over 888.952984159s
mean rate
2.45355480649e+10
/s
std
1.67378328772e+11
/s
p5
1.37420907e+07
/s
p95
6.68508811225e+10
/s
peak
6.83926498885e+12
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008266
s
SD
0.003808
s
p50
0.008843
s
<
p95
0.016003
s
<
p99
0.016547
s
Zome call duration: 'timed_and_validated::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.02483
s
SD
0.033527
s
p50
0.025386
s
<
p95
0.126399
s
<
p99
0.144331
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007217
s
SD
0.003292
s
p50
0.007705
s
<
p95
0.013939
s
<
p99
0.01445
s
WASM call duration: 'timed_and_validated::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.023663
s
SD
0.033295
s
p50
0.023029
s
<
p95
0.124421
s
<
p99
0.142338
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001441
s
SD
0.00077
s
p50
0.00151
s
<
p95
0.002633
s
<
p99
0.00439
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006433
s
SD
0.004581
s
p50
0.006027
s
<
p95
0.011506
s
<
p99
0.025408
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00045
s
SD
0.000218
s
p50
0.000484
s
<
p95
0.000879
s
<
p99
0.00092
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000437
s
SD
0.000226
s
p50
0.00045
s
<
p95
0.000909
s
<
p99
0.00096
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001113
s
SD
0.002186
s
p50
0.003449
s
<
p95
0.008108
s
<
p99
0.008453
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.001414
s
SD
0.001635
s
p50
0.002426
s
<
p95
0.005772
s
<
p99
0.006352
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.008384
s
SD
0.005115
s
p50
0.009479
s
<
p95
0.019173
s
<
p99
0.021511
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002445
s
SD
0.002398
s
p50
0.002391
s
<
p95
0.004584
s
<
p99
0.013058
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000525
s
SD
0.000813
s
p50
0.000659
s
<
p95
0.001367
s
<
p99
0.005452
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.75
s
<
mean
426.05
s
SD
246.43
s
<
p95
808.18
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.952946234s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
1.146407
s
SD
22.5022
s
p50
0
s
≤
p95
0
s
<
p99
24.047619
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
4.367903
s
SD
2.680069
s
p50
0.002052
s
<
p95
1.70362
s
<
p99
14.227888
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.01136
s
SD
0.262791
s
p50
0.002176
s
<
p95
0.952331
s
<
p99
1.249628
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.121152
s
SD
0.064088
s
p50
0.102731
s
<
p95
0.22308
s
<
p99
0.251247
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
7.054511
s
SD
8.891682
s
p50
0.003235
s
<
p95
7.862575
s
<
p99
36.645614
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
0.734873
s
SD
1.771556
s
p50
0.001378
s
<
p95
0.011079
s
<
p99
15.165102
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.005249
s
SD
0.00354
s
p50
0.005891
s
<
p95
0.012317
s
<
p99
0.014805
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005937
s
SD
0.004152
s
p50
0.003993
s
<
p95
0.012986
s
<
p99
0.01648
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000358
s
SD
0.003601
s
p50
0.000387
s
<
p95
0.001139
s
<
p99
0.008586
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
30.825238
s
SD
27.465596
s
p50
21.549104
s
<
p95
60.080609
s
<
p99
60.09948
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.971185
s
SD
22.410277
s
p50
0.811726
s
<
p95
60.032254
s
<
p99
60.141193
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
46.41
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
6.34
KiB/s
<
mean
162.95
KiB/s
SD
282.22
KiB/s
<
p95
510.05
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.29
KiB/s
<
mean
36.87
KiB/s
SD
54.79
KiB/s
<
p95
163.42
KiB/s
Total bytes received
Total bytes received on primary network interface
count
20.03
GiB
mean
23.04
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.53
GiB
mean
5.21
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 91.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
Detected

Critical Disk /etc/resolv.conf: 1/16 hosts above 90% capacity

Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (16.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 38% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
10.84
%
<
mean
63.52
%
SD
21.67
%
<
p95
87.59
%
System
17.11
%
CPU percentiles
CPU usage percentiles
p50
71.31
%
p95
87.59
%
p99
91.93
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
114
hosts
mean time
211.15
s
Memory used percentage
Percentage of memory used
p5
6.61
%
<
mean
13.04
%
SD
8.32
%
<
p95
28.08
%
Memory available percentage
Percentage of available memory
p5
71.92
%
<
mean
86.96
%
SD
8.32
%
<
p95
93.39
%
Max host memory used
Maximum memory usage percentage across all hosts
max
78.87
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
16.08
%
Memory growth rate
Rate of memory growth over time
growth
0.42
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.24
MiB/s
Disk write throughput
Disk write throughput in MiB/s
129.94
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/108
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
1/16
hosts
Mount Point /etc/hosts
1/16
hosts
Mount Point /etc/nomad.d/client-meta.json
0/22
hosts
Mount Point /etc/nomad.d/meta.json
0/1
hosts
Mount Point /etc/resolv.conf
1/16
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.17
5 min
2.06
15 min
2.09
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
37.67
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.05
%
<
mean
21.6906
%
SD
15.9773
%
<
p95
54.11
%
60 second average
21.0008
%
300 second average
20.1404
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.514
%
SD
3.4412
%
<
p95
7.16
%
60 second average
2.4422
%
300 second average
2.4397
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.454
%
SD
2.4737
%
<
p95
3.63
%
60 second average
1.4045
%
300 second average
1.3636
%
Holochain process CPU usage
CPU usage by Holochain process
p5
24.09
%
<
mean
61.7
%
SD
18.11
%
<
p95
81.95
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
179.09
KiB
<
mean
387.69
KiB
SD
98.69
KiB
<
p95
538.05
KiB
Holochain process threads
Number of threads in Holochain process
p5
27
threads
<
mean
33.71
threads
SD
4.51
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
66
file descriptors
<
mean
79.24
file descriptors
SD
8.92
file descriptors
<
p95
87
file descriptors