DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
  • record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.
Started
Thu, 03 Sep 2026 02:52:12 UTC
Duration
1800s
Peer count
150
Peer count at end
147
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
dht_sync_lag_33706506143_1
Create rate
The average number of created records per agent.
Mean
mean
14.78
/s
Max
Highest per-partition mean rate
max
58.01
/s
Min
Lowest per-partition mean rate
min
4.65
/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean
438.585889
s
SD
261.474241
s
Max
Highest per-partition mean latency
max mean
1470.272162
s
Min
Lowest per-partition mean latency
min mean
27.19229
s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean
48.14
/s
Max
Highest per-partition mean rate
max
345
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all record_lag agents.
0.0021
Sent count
The total number of entries sent (created) by write agents, by agent.
Total
345469
Agents affected
125 / 125
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2764
Max
Highest count in any single partition
10620
Min
Lowest count in any single partition
841
Received count
The total number of entries received by record_lag agents, by agent.
Total
16339
Agents affected
21 / 23
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
710
Max
Highest count in any single partition
2769
Min
Lowest count in any single partition
0
Error count
The number of errors encountered during the scenario.
50
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.003702
s
SD
0.002323
s
p50
0.003471
s
<
p95
0.008296
s
<
p99
0.009455
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
1
over 1701.92681377s
mean rate
13.67
/s
std
48.32
/s
p5
0
/s
p95
128.64
/s
peak
254.03
/s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
295101347
over 1789.134408313s
mean rate
1.54915799804e+10
/s
std
1.91856397045e+11
/s
p5
7.734777712e+07
/s
p95
3.4409902028e+10
/s
peak
9.12539763477e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
9767551162
over 1788.863684434s
mean rate
5.89283444838e+10
/s
std
2.18911557343e+11
/s
p5
3.3977574449e+08
/s
p95
4.65688355228e+11
/s
peak
3.18123335224e+12
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
41824032
over 1789.280390159s
mean rate
2.257422084e+09
/s
std
2.32135141219e+10
/s
p5
7.55427055e+06
/s
p95
4.9685407097e+09
/s
peak
1.16289233605e+12
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.02965
s
SD
0.034857
s
p50
0.028911
s
<
p95
0.11475
s
<
p99
0.14816
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.158334
s
SD
0.268155
s
p50
0.158789
s
<
p95
0.866005
s
<
p99
1.133264
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.028095
s
SD
0.033845
s
p50
0.027104
s
<
p95
0.110888
s
<
p99
0.142719
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.157097
s
SD
0.267634
s
p50
0.157632
s
<
p95
0.864788
s
<
p99
1.130109
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002105
s
SD
0.001254
s
p50
0.0022
s
<
p95
0.004898
s
<
p99
0.006584
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.007887
s
SD
0.01203
s
p50
0.007246
s
<
p95
0.03714
s
<
p99
0.049124
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.007677
s
SD
0.012018
s
p50
0.006871
s
<
p95
0.037382
s
<
p99
0.050051
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.003691
s
SD
0.002212
s
p50
0.003658
s
<
p95
0.008304
s
<
p99
0.009275
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.006505
s
SD
0.012748
s
p50
0.00602
s
<
p95
0.037983
s
<
p99
0.059739
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.004949
s
SD
0.004404
s
p50
0.004996
s
<
p95
0.015524
s
<
p99
0.019682
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002383
s
SD
0.0034
s
p50
0.001612
s
<
p95
0.010466
s
<
p99
0.014056
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
85.59
s
<
mean
870.23
s
SD
505.43
s
<
p95
1662.25
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
50
over 1789.280381295s
mean rate
442520.2
/s
std
8.81067747e+06
/s
p5
11.76
/s
p95
761253.16
/s
peak
7.2856621671e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
125.283268
s
SD
139.314081
s
p50
9.415101
s
<
p95
343.967606
s
<
p99
768.331925
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.257481
s
SD
0.935509
s
p50
0.227603
s
<
p95
2.738002
s
<
p99
4.247621
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.267227
s
SD
0.426054
s
p50
0.190565
s
<
p95
1.329817
s
<
p99
1.766025
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
2.153398
s
SD
2.883697
s
p50
1.488234
s
<
p95
8.198695
s
<
p99
11.91351
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
22.788994
s
SD
64.814936
s
p50
30.722274
s
<
p95
203.268467
s
<
p99
316.716402
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
24.747434
s
SD
72.434713
s
p50
0.370797
s
<
p95
143.566025
s
<
p99
380.007577
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004551
s
SD
0.003464
s
p50
0.004833
s
<
p95
0.011913
s
<
p99
0.014924
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00871
s
SD
0.007188
s
p50
0.004567
s
<
p95
0.023331
s
<
p99
0.031153
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.008037
s
SD
0.013776
s
p50
0.005639
s
<
p95
0.039798
s
<
p99
0.06051
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
5.032919
s
SD
28.450776
s
p50
4.8513
s
<
p95
60.054537
s
<
p99
60.116015
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.326144
s
SD
26.130884
s
p50
2.267606
s
<
p95
60.041343
s
<
p99
60.089095
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
60.20
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
67.95
KiB/s
<
mean
688.03
KiB/s
SD
550.81
KiB/s
<
p95
1.77
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.66
KiB/s
<
mean
144.70
KiB/s
SD
186.45
KiB/s
<
p95
440.16
KiB/s
Total bytes received
Total bytes received on primary network interface
count
165.02
GiB
mean
94.40
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
34.70
GiB
mean
19.85
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 94% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
43.89
%
<
mean
81.94
%
SD
17.25
%
<
p95
95.83
%
System
21.74
%
CPU percentiles
CPU usage percentiles
p50
87.78
%
p95
95.83
%
p99
98.00
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
137
hosts
mean time
1402.50
s
Memory used percentage
Percentage of memory used
p5
8.47
%
<
mean
12.96
%
SD
6
%
<
p95
27.9
%
Memory available percentage
Percentage of available memory
p5
72.1
%
<
mean
87.04
%
SD
6
%
<
p95
91.53
%
Max host memory used
Maximum memory usage percentage across all hosts
max
41.94
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
3.12
%
Memory growth rate
Rate of memory growth over time
growth
0.37
MiB/s
Disk read throughput
Disk read throughput in MiB/s
1.11
MiB/s
Disk write throughput
Disk write throughput in MiB/s
137.87
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/118
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/12
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.89
5 min
3.41
15 min
2.43
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
93.75
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
2.92
%
<
mean
48.4934
%
SD
19.6614
%
<
p95
72.84
%
60 second average
47.2085
%
300 second average
41.3323
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
8.966
%
SD
5.8897
%
<
p95
17.11
%
60 second average
8.7550
%
300 second average
7.6544
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.5576
%
SD
3.2849
%
<
p95
7.83
%
60 second average
2.5036
%
300 second average
2.2428
%
Holochain process CPU usage
CPU usage by Holochain process
p5
50.02
%
<
mean
78.13
%
SD
13.32
%
<
p95
92.95
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
227.18
KiB
<
mean
443.78
KiB
SD
153.98
KiB
<
p95
697.98
KiB
Holochain process threads
Number of threads in Holochain process
p5
30
threads
<
mean
35.17
threads
SD
5.72
threads
<
p95
39
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
79
file descriptors
<
mean
85.39
file descriptors
SD
12.41
file descriptors
<
p95
90
file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 03 Sep 2026 06:33:42 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_33706506143_1
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean
22.24
/s
Max
Highest per-partition mean rate
max
37.87
/s
Min
Lowest per-partition mean rate
min
11.12
/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean
69.453871
s
SD
10.52342
s
Max
Highest per-partition mean latency
max mean
1438.475895
s
Min
Lowest per-partition mean latency
min mean
0.493422
s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean
18.32
/s
Max
Highest per-partition mean rate
max
115
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc nodes.
0.0032
Entry created count
Total number of entries created by full-arc nodes, by agent.
Total
17731
Agents affected
9 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1970
Max
Highest count in any single partition
3293
Min
Lowest count in any single partition
975
Received count
Total number of entries fetched by zero-arc nodes, by agent.
Total
7897
Agents affected
28 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
57
Max
Highest count in any single partition
942
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
35
<
mean
69.84
SD
43.15
<
p95
147
zero-arc
p5
0
<
mean
22.41
SD
22.1
<
p95
63
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
1852
Agents affected
138 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
13
Max
Highest count in any single partition
14
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
1855
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.156988
s
SD
1.347582
s
p50
0.157911
s
<
p95
3.362553
s
<
p99
6.667938
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 809.176298718s
Not enough time data to show a trend.
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
119999866
over 873.918707905s
mean rate
7.363924941e+07
/s
std
6.349170629e+07
/s
p5
4.72704974e+06
/s
p95
2.4102719794e+08
/s
peak
2.7706461327e+08
/s
WASM usage: 'timed_and_validated::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
166522734
over 889.60456787s
mean rate
7.7451305153e+10
/s
std
1.59209543741e+12
/s
p5
6.32265087e+07
/s
p95
8.63977300673e+10
/s
peak
9.22337193285e+13
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
3016124
over 889.60456787s
mean rate
6.5525502648e+09
/s
std
1.29405983448e+11
/s
p5
1.42302975e+06
/s
p95
8.63967160436e+09
/s
peak
8.02597503229e+12
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.60456787s
Not enough time data to show a trend.
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.013776
s
SD
0.011583
s
p50
0.012179
s
<
p95
0.04245
s
<
p99
0.053335
s
Zome call duration: 'timed_and_validated::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.023771
s
SD
43.421193
s
p50
0.007302
s
<
p95
12.088351
s
<
p99
227.633481
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.01256
s
SD
0.011057
s
p50
0.010828
s
<
p95
0.040287
s
<
p99
0.050163
s
WASM call duration: 'timed_and_validated::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.183221
s
SD
60.214041
s
p50
0.185785
s
<
p95
144.407542
s
<
p99
245.852837
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001467
s
SD
0.00065
s
p50
0.001757
s
<
p95
0.00272
s
<
p99
0.003269
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011021
s
SD
0.004664
s
p50
0.00191
s
<
p95
0.009838
s
<
p99
0.024852
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00217
s
SD
0.003843
s
p50
0.001238
s
<
p95
0.012266
s
<
p99
0.015346
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.002122
s
SD
0.003921
s
p50
0.001188
s
<
p95
0.012281
s
<
p99
0.015577
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.238555
s
SD
4.247589
s
p50
1.110944
s
<
p95
8.054516
s
<
p99
20.054159
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.017551
s
SD
2.471111
s
p50
0.021173
s
<
p95
1.521494
s
<
p99
15.769978
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.017306
s
SD
0.009251
s
p50
0.012939
s
<
p95
0.031462
s
<
p99
0.043112
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002495
s
SD
0.001282
s
p50
0.002793
s
<
p95
0.004774
s
<
p99
0.006935
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000749
s
SD
0.000541
s
p50
0.000274
s
<
p95
0.001973
s
<
p99
0.002401
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.68
s
<
mean
423.89
s
SD
245.44
s
<
p95
806.28
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.6045803s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
84.025758
s
SD
17.532377
s
p50
0
s
<
p95
0.00552
s
<
p99
92.735818
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.943572
s
SD
1.975958
s
p50
0.001959
s
<
p95
0.76164
s
<
p99
11.197389
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.097629
s
SD
0.059606
s
p50
0.002208
s
<
p95
0.019537
s
<
p99
0.349967
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.211554
s
SD
0.056058
s
p50
0.021663
s
<
p95
0.082381
s
<
p99
0.347674
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.471048
s
SD
9.082509
s
p50
0.00334
s
<
p95
3.382013
s
<
p99
42.120993
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.578811
s
SD
5.122017
s
p50
0.001213
s
<
p95
0.009212
s
<
p99
16.758438
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001683
s
SD
0.001448
s
p50
0.000148
s
<
p95
0.00316
s
<
p99
0.008026
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.009036
s
SD
0.012572
s
p50
0.004377
s
<
p95
0.032024
s
<
p99
0.059798
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.001573
s
SD
0.001142
s
p50
0.00077
s
<
p95
0.001625
s
<
p99
0.005634
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
5.316923
s
SD
29.870064
s
p50
0.700559
s
<
p95
60.001672
s
<
p99
60.002966
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
2.436422
s
SD
25.161533
s
p50
0.729745
s
<
p95
60.001616
s
<
p99
60.002328
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.37471
s
SD
2.386836
s
p50
0.38413
s
<
p95
1.052559
s
<
p99
15.295846
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
8.40
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
3.12
KiB/s
<
mean
154.88
KiB/s
SD
321.72
KiB/s
<
p95
804.00
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
198
B/s
<
mean
26.56
KiB/s
SD
123.00
KiB/s
<
p95
71.21
KiB/s
Total bytes received
Total bytes received on primary network interface
count
19.06
GiB
mean
21.93
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.27
GiB
mean
3.76
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.1% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 6% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.67
%
<
mean
13.26
%
SD
23.19
%
<
p95
83.86
%
System
4.86
%
CPU percentiles
CPU usage percentiles
p50
4.89
%
p95
83.86
%
p99
95.98
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
105
hosts
mean time
70.41
s
Memory used percentage
Percentage of memory used
p5
7.01
%
<
mean
12.69
%
SD
5.65
%
<
p95
25.29
%
Memory available percentage
Percentage of available memory
p5
74.71
%
<
mean
87.31
%
SD
5.65
%
<
p95
92.99
%
Max host memory used
Maximum memory usage percentage across all hosts
max
46.51
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.06
%
Memory growth rate
Rate of memory growth over time
growth
0.49
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.03
MiB/s
Disk write throughput
Disk write throughput in MiB/s
42.74
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.61
5 min
0.68
15 min
0.94
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
6.16
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
4.9089
%
SD
16.3697
%
<
p95
39.13
%
60 second average
4.7806
%
300 second average
5.7621
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.7797
%
SD
3.4496
%
<
p95
4.16
%
60 second average
0.7709
%
300 second average
0.9096
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.2389
%
SD
1.0731
%
<
p95
1.17
%
60 second average
0.2414
%
300 second average
0.2701
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.5
%
<
mean
9.58
%
SD
22.18
%
<
p95
78.48
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
276.83
KiB
<
mean
372.27
KiB
SD
78.28
KiB
<
p95
473.05
KiB
Holochain process threads
Number of threads in Holochain process
p5
20
threads
<
mean
26.36
threads
SD
7.41
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
54
file descriptors
<
mean
63.39
file descriptors
SD
11
file descriptors
<
p95
79
file descriptors

Mixed Arc get_agent_activity

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Thu, 03 Sep 2026 06:51:20 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_get_agent_activity_33706506143_1
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
425278
Agents affected
104 / 104
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4089
Max
Highest count in any single partition
13172
Min
Lowest count in any single partition
694
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1879.4
Max
Highest chain head value observed for any single write agent.
4431
Write agents observed
20
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean
-0.011154
s
SD
2.654175
s
Max
Highest per-partition mean latency
max mean
4.935689
s
Min
Lowest per-partition mean latency
min mean
-10.59392
s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean
3.83
/s
Max
Highest per-partition mean rate
max
29.33
/s
Min
Lowest per-partition mean rate
min
0
/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean
2.221258
s
SD
5.396652
s
Max
Highest per-partition mean latency
max mean
6.608193
s
Min
Lowest per-partition mean latency
min mean
0.056481
s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
135
Agents affected
41 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3
Max
Highest count in any single partition
11
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5
33
<
mean
98.69
SD
36.43
<
p95
146
  • behaviour: zero_read
p5
0
<
mean
43.95
SD
22.97
<
p95
88
  • behaviour: zero_write
p5
6
<
mean
42.23
SD
20.46
<
p95
88
Final error count
The total number of errors accumulated across all agents over the run.
138
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.746245
s
SD
6.554687
s
p50
1.481109
s
<
p95
14.984613
s
<
p99
33.376195
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 800.156777357s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009126
s
SD
0.005247
s
p50
0.008649
s
<
p95
0.018716
s
<
p99
0.035365
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.00893
s
SD
0.016047
s
p50
0.008966
s
<
p95
0.05313
s
<
p99
0.073973
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.794273
s
SD
6.934275
s
p50
1.79301
s
<
p95
19.699535
s
<
p99
37.365718
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.128142
s
SD
5.974094
s
p50
0.583036
s
<
p95
14.991233
s
<
p99
24.69938
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007668
s
SD
0.004592
s
p50
0.007129
s
<
p95
0.01581
s
<
p99
0.030834
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007776
s
SD
0.015091
s
p50
0.007387
s
<
p95
0.049413
s
<
p99
0.069146
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.793291
s
SD
6.934401
s
p50
1.792096
s
<
p95
19.698758
s
<
p99
37.364926
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.154805
s
SD
6.147941
s
p50
0.581698
s
<
p95
14.989905
s
<
p99
24.697962
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001612
s
SD
0.000869
s
p50
0.001867
s
<
p95
0.003657
s
<
p99
0.004734
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
6.6e-05
s
SD
4e-05
s
p50
5.2e-05
s
<
p95
0.000163
s
<
p99
0.0002
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.002901
s
SD
0.010586
s
p50
0.001356
s
<
p95
0.032042
s
<
p99
0.046996
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000755
s
SD
0.00056
s
p50
0.000694
s
<
p95
0.001248
s
<
p99
0.003866
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.791236
s
SD
6.934646
s
p50
1.789893
s
<
p95
19.696578
s
<
p99
37.363323
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.15321
s
SD
6.147605
s
p50
0.573919
s
<
p95
14.984676
s
<
p99
24.692948
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
3e-06
s
SD
1.1e-05
s
p50
1.6e-05
s
<
p95
2.4e-05
s
<
p99
7.2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003122
s
SD
0.003363
s
p50
0.003505
s
<
p95
0.010463
s
<
p99
0.018687
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001212
s
SD
0.003264
s
p50
0.00051
s
<
p95
0.00987
s
<
p99
0.014655
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.38
s
<
mean
422.57
s
SD
244.91
s
<
p95
805.15
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
2
over 889.142499104s
mean rate
84957.03
/s
std
5.85048646e+06
/s
p5
0
/s
p95
20162.33
/s
peak
5.6550183501e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
5.297323
s
SD
17.22781
s
p50
0
s
<
p95
7.151495
s
<
p99
50.882177
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.159482
s
SD
0.886738
s
p50
0.002649
s
<
p95
1.436343
s
<
p99
4.337101
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.054083
s
SD
0.264704
s
p50
0.002816
s
<
p95
0.77955
s
<
p99
1.210222
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.197148
s
SD
0.17111
s
p50
0.140615
s
<
p95
0.474434
s
<
p99
0.661409
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
7.8977
s
SD
23.927643
s
p50
0.004971
s
<
p95
63.388572
s
<
p99
134.279969
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.891801
s
SD
12.913681
s
p50
0.001597
s
<
p95
0.117997
s
<
p99
40.421392
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.003595
s
SD
0.002614
s
p50
0.002427
s
<
p95
0.008225
s
<
p99
0.010247
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006654
s
SD
0.00385
s
p50
0.003103
s
<
p95
0.012808
s
<
p99
0.019105
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.002975
s
SD
0.009255
s
p50
0.000979
s
<
p95
0.027629
s
<
p99
0.042404
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
7.429502
s
SD
27.326056
s
p50
7.867746
s
<
p95
60.065508
s
<
p99
65.859575
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.939278
s
SD
5.185381
s
p50
0.566874
s
<
p95
13.36442
s
<
p99
24.689036
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.550431
s
SD
28.570359
s
p50
1.29294
s
<
p95
60.001636
s
<
p99
60.002733
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
2.381052
s
SD
17.855134
s
p50
0.227183
s
<
p95
60.079484
s
<
p99
60.196039
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
39.00
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
4.96
KiB/s
<
mean
655.23
KiB/s
SD
562.77
KiB/s
<
p95
1.67
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
135.83
KiB/s
SD
192.16
KiB/s
<
p95
525.44
KiB/s
Total bytes received
Total bytes received on primary network interface
count
80.63
GiB
mean
92.77
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
16.72
GiB
mean
19.23
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.7%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.5% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 38% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.91
%
<
mean
54.13
%
SD
36.72
%
<
p95
92.78
%
System
15.13
%
CPU percentiles
CPU usage percentiles
p50
65.37
%
p95
92.78
%
p99
97.65
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
111
hosts
mean time
504.64
s
Memory used percentage
Percentage of memory used
p5
6.85
%
<
mean
11.75
%
SD
5.59
%
<
p95
24.13
%
Memory available percentage
Percentage of available memory
p5
75.87
%
<
mean
88.25
%
SD
5.59
%
<
p95
93.15
%
Max host memory used
Maximum memory usage percentage across all hosts
max
46.13
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.49
%
Memory growth rate
Rate of memory growth over time
growth
0.46
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.55
MiB/s
Disk write throughput
Disk write throughput in MiB/s
90.67
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.33
5 min
1.82
15 min
1.32
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
37.67
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
28.8503
%
SD
28.0071
%
<
p95
70.28
%
60 second average
27.4212
%
300 second average
21.1423
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1808
%
SD
5.8832
%
<
p95
15.54
%
60 second average
3.9477
%
300 second average
3.0114
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2738
%
SD
2.3781
%
<
p95
4.68
%
60 second average
1.1975
%
300 second average
0.9112
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.7
%
<
mean
51.06
%
SD
34.64
%
<
p95
88.55
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
200.74
KiB
<
mean
286.41
KiB
SD
87.09
KiB
<
p95
456.17
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
31.01
threads
SD
7.9
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
59
file descriptors
<
mean
77.1
file descriptors
SD
13.64
file descriptors
<
p95
89
file descriptors

Mixed Arc must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Thu, 03 Sep 2026 07:08:43 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
mixed_arc_must_get_agent_activity_33706506143_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
206.38
Max
Highest chain head value observed for any single write agent.
742
Write agents observed
13
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
267.177679
s
SD
196.543467
s
Max
Highest per-partition mean latency
max mean
462.488
s
Min
Lowest per-partition mean latency
min mean
127.72
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.13
/s
Max
Highest per-partition mean rate
max
1.88
/s
Min
Lowest per-partition mean rate
min
1
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
3.507203
s
SD
4.330499
s
Max
Highest per-partition mean latency
max mean
13.532804
s
Min
Lowest per-partition mean latency
min mean
0.129965
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
62407
Agents affected
43 / 43
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1451
Max
Highest count in any single partition
4734
Min
Lowest count in any single partition
274
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5
27
<
mean
86.09
SD
43.14
<
p95
146
  • behaviour: zero_must_get_agent_activity
p5
4
<
mean
41.42
SD
20.99
<
p95
85
  • behaviour: zero_write
p5
10
<
mean
37.69
SD
18.06
<
p95
82
Final error count
The total number of all types of error accumulated over the run by all agents.
77795
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.138932
s
SD
3.948901
s
p50
0.229628
s
<
p95
7.365103
s
<
p99
24.703728
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 769.690983272s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.009339
s
SD
0.005408
s
p50
0.008883
s
<
p95
0.020622
s
<
p99
0.029471
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.189833
s
SD
0.237324
s
p50
0.082932
s
<
p95
0.76677
s
<
p99
1.005528
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007854
s
SD
0.001105
s
p50
0.008083
s
<
p95
0.009238
s
<
p99
0.00943
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
1.102042
s
SD
5.544954
s
p50
0.73792
s
<
p95
10.721646
s
<
p99
35.420721
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.909015
s
SD
5.34391
s
p50
0.522427
s
<
p95
12.479696
s
<
p99
24.710855
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007743
s
SD
0.004722
s
p50
0.00734
s
<
p95
0.018672
s
<
p99
0.026622
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.187983
s
SD
0.236367
s
p50
0.081289
s
<
p95
0.762586
s
<
p99
1.001287
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006521
s
SD
0.000891
s
p50
0.006766
s
<
p95
0.007572
s
<
p99
0.007732
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.559809
s
SD
5.432686
s
p50
0.585338
s
<
p95
6.840358
s
<
p99
35.419033
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.907818
s
SD
5.343758
s
p50
0.520846
s
<
p95
12.478403
s
<
p99
24.709519
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001991
s
SD
0.00106
s
p50
0.002067
s
<
p95
0.004566
s
<
p99
0.005791
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.024719
s
SD
0.074092
s
p50
0.006742
s
<
p95
0.178514
s
<
p99
0.299062
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.2e-05
s
SD
7.4e-05
s
p50
4.2e-05
s
<
p95
0.00018
s
<
p99
0.000256
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.003642
s
SD
0.010416
s
p50
0.000652
s
<
p95
0.029948
s
<
p99
0.050411
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.005784
s
SD
0.010275
s
p50
0.000731
s
<
p95
0.028657
s
<
p99
0.054107
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.558207
s
SD
5.40401
s
p50
0.54861
s
<
p95
10.71531
s
<
p99
35.412366
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.102061
s
SD
0.326304
s
p50
0.072531
s
<
p95
0.716513
s
<
p99
1.210627
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.066526
s
SD
0.0697
s
p50
0.023262
s
<
p95
0.203653
s
<
p99
0.262234
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1.4e-05
s
SD
1.2e-05
s
p50
1.9e-05
s
<
p95
3.4e-05
s
<
p99
8.9e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003944
s
SD
0.003027
s
p50
0.003409
s
<
p95
0.011011
s
<
p99
0.017723
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001648
s
SD
0.002759
s
p50
0.000701
s
<
p95
0.009093
s
<
p99
0.011969
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.38
s
<
mean
421.52
s
SD
244.91
s
<
p95
805.01
s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
78.348745
s
SD
39.628611
s
p50
0
s
<
p95
40.40387
s
<
p99
260.853048
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.487912
s
SD
1.659856
s
p50
0.003153
s
<
p95
3.378103
s
<
p99
4.991133
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.582438
s
SD
0.38666
s
p50
0.002965
s
<
p95
1.219185
s
<
p99
1.349394
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.187527
s
SD
0.190963
s
p50
0.032005
s
<
p95
0.572137
s
<
p99
0.77467
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
8.139222
s
SD
12.420863
s
p50
0.00455
s
<
p95
23.547177
s
<
p99
56.201313
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
6.315174
s
SD
14.803714
s
p50
0.001816
s
<
p95
6.248442
s
<
p99
94.307481
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001996
s
SD
0.001769
s
p50
0.000218
s
<
p95
0.005378
s
<
p99
0.006698
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.002747
s
SD
0.005891
s
p50
0.003379
s
<
p95
0.017645
s
<
p99
0.03083
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.003587
s
SD
0.01046
s
p50
0.00058
s
<
p95
0.02952
s
<
p99
0.050746
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.395833
s
SD
28.762547
s
p50
0.491059
s
<
p95
60.003653
s
<
p99
60.050862
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.206051
s
SD
21.494965
s
p50
0.30161
s
<
p95
60.001755
s
<
p99
60.002074
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
0.249439
s
SD
28.351732
s
p50
0.280933
s
<
p95
60.002301
s
<
p99
60.002542
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.762273
s
SD
22.088142
s
p50
1.278465
s
<
p95
60.002412
s
<
p99
60.009218
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
23.41
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
133
B/s
<
mean
369.66
KiB/s
SD
302.75
KiB/s
<
p95
776.29
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
82.30
KiB/s
SD
92.03
KiB/s
<
p95
288.56
KiB/s
Total bytes received
Total bytes received on primary network interface
count
44.74
GiB
mean
51.48
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.96
GiB
mean
11.46
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.1%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 28% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.19
%
<
mean
34.53
%
SD
34.2
%
<
p95
90.44
%
System
11.12
%
CPU percentiles
CPU usage percentiles
p50
15.03
%
p95
90.44
%
p99
97.10
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
118
hosts
mean time
248.82
s
Memory used percentage
Percentage of memory used
p5
7.23
%
<
mean
12.87
%
SD
5.77
%
<
p95
27.11
%
Memory available percentage
Percentage of available memory
p5
72.89
%
<
mean
87.13
%
SD
5.77
%
<
p95
92.77
%
Max host memory used
Maximum memory usage percentage across all hosts
max
46.59
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.68
%
Memory growth rate
Rate of memory growth over time
growth
0.49
MiB/s
Disk read throughput
Disk read throughput in MiB/s
2.15
MiB/s
Disk write throughput
Disk write throughput in MiB/s
163.50
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/118
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/15
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.53
5 min
1.60
15 min
1.60
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
28.28
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.05
%
<
mean
17.0978
%
SD
24.0518
%
<
p95
64.28
%
60 second average
16.4002
%
300 second average
17.7150
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
3.2194
%
SD
5.9183
%
<
p95
15.23
%
60 second average
3.1062
%
300 second average
3.1598
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2062
%
SD
2.6993
%
<
p95
6.34
%
60 second average
1.1758
%
300 second average
1.1558
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.4
%
<
mean
31.17
%
SD
32.49
%
<
p95
83.99
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
225.42
KiB
<
mean
370.14
KiB
SD
106.02
KiB
<
p95
488.60
KiB
Holochain process threads
Number of threads in Holochain process
p5
23
threads
<
mean
32
threads
SD
11.69
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
62
file descriptors
<
mean
76.97
file descriptors
SD
16.97
file descriptors
<
p95
89
file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Thu, 03 Sep 2026 02:17:30 UTC
Duration
900s
Peer count
150
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_call_rate_33706506143_1
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean
-1.409436
s
SD
28.743466
s
Max
Highest per-partition mean latency
max mean
57.324174
s
Min
Lowest per-partition mean latency
min mean
-7.169582
s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean
0.118719
s
SD
0.22522
s
Max
Highest per-partition mean latency
max mean
0.636703
s
Min
Lowest per-partition mean latency
min mean
0.053448
s
Error count
The total number of errors accumulated during the run.
3799
Holochain MetricsClick to toggle visibility.
WASM usage: 'remote_call::call_echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
262764759
over 888.765472619s
mean rate
3.58475811007e+10
/s
std
1.03040814305e+12
/s
p5
1.375330714e+07
/s
p95
1.89784303584e+10
/s
peak
6.49944336983e+13
/s
WASM usage: 'remote_call::echo_timestamp'
Metered usage count of the WASM ribosome for this zome function.
total
40248233
over 888.765472619s
mean rate
9.6922260438e+08
/s
std
8.08139171246e+09
/s
p5
354793.4
/s
p95
2.53800715904e+09
/s
peak
5.07115204694e+11
/s
WASM usage: 'remote_call::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 888.765472619s
Not enough time data to show a trend.
Zome call duration: 'remote_call::call_echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.118436
s
SD
0.200111
s
p50
0.128757
s
<
p95
0.323152
s
<
p99
0.6844
s
Zome call duration: 'remote_call::echo_timestamp'
Duration of this zome call as measured by Holochain internally.
mean
0.004046
s
SD
0.001156
s
p50
0.004122
s
<
p95
0.005538
s
<
p99
0.005866
s
WASM call duration: 'remote_call::call_echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.131618
s
SD
0.261278
s
p50
0.13417
s
<
p95
0.600312
s
<
p99
1.098276
s
WASM call duration: 'remote_call::echo_timestamp'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002517
s
SD
0.00075
s
p50
0.002563
s
<
p95
0.003679
s
<
p99
0.00395
s
WASM call duration: 'remote_call::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005281
s
SD
0.006165
s
p50
0.004051
s
<
p95
0.012231
s
<
p99
0.031508
s
Host function call duration: '__hc__call_1'
Duration of this host function call invoked from within WASM.
mean
0.128584
s
SD
0.26142
s
p50
0.130725
s
<
p95
0.597837
s
<
p99
1.095106
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000974
s
SD
0.001942
s
p50
0.000706
s
<
p95
0.00141
s
<
p99
0.014891
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1.4e-05
s
SD
7e-06
s
p50
1.3e-05
s
<
p95
2.4e-05
s
<
p99
2.5e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000396
s
SD
0.002318
s
p50
0.000447
s
<
p95
0.001035
s
<
p99
0.007
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000769
s
SD
0.002514
s
p50
0.000222
s
<
p95
0.002243
s
<
p99
0.013978
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.32
s
<
mean
422.7
s
SD
244.8
s
<
p95
805
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
1858
over 888.765661373s
mean rate
24398.4
/s
std
235098.88
/s
p5
0
/s
p95
49889.15
/s
peak
1.139270954e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
10.508301
s
SD
6.308993
s
p50
8.047888
s
<
p95
22.287483
s
<
p99
36.944429
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.012594
s
SD
0.015512
s
p50
0.013341
s
<
p95
0.048178
s
<
p99
0.070652
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.077135
s
SD
0.060051
s
p50
0.081584
s
<
p95
0.188052
s
<
p99
0.318403
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.019625
s
SD
0.010888
s
p50
0.018716
s
<
p95
0.038654
s
<
p99
0.049265
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
3.964863
s
SD
4.35971
s
p50
5.382201
s
<
p95
12.148145
s
<
p99
19.537622
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.279315
s
SD
4.279441
s
p50
0.533811
s
<
p95
11.383052
s
<
p99
18.765357
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.000442
s
SD
0.000255
s
p50
0.000416
s
<
p95
0.000781
s
<
p99
0.001311
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.003736
s
SD
0.001898
s
p50
0.003159
s
<
p95
0.006851
s
<
p99
0.007927
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000626
s
SD
0.000241
s
p50
0.000611
s
<
p95
0.000902
s
<
p99
0.001208
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.182682
s
SD
24.812698
s
p50
0.117175
s
<
p95
60.002794
s
<
p99
60.004407
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.122546
s
SD
0.081919
s
p50
0.065222
s
<
p95
0.261664
s
<
p99
0.309026
s
P2P request roundtrip: 'call_remote'
Time spent sending a call_remote request and awaiting its response.
mean
0.114615
s
SD
12.035332
s
p50
0.127735
s
<
p95
0.743296
s
<
p99
60.001931
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
10.16
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
16.86
KiB/s
<
mean
120.58
KiB/s
SD
258.61
KiB/s
<
p95
348.06
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
6.65
KiB/s
<
mean
15.52
KiB/s
SD
14.07
KiB/s
<
p95
38.52
KiB/s
Total bytes received
Total bytes received on primary network interface
count
14.78
GiB
mean
17.00
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.90
GiB
mean
2.19
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.01
%
<
mean
17.92
%
SD
11.26
%
<
p95
37.06
%
System
7.76
%
CPU percentiles
CPU usage percentiles
p50
17.58
%
p95
37.06
%
p99
61.95
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
55
hosts
mean time
11.36
s
Memory used percentage
Percentage of memory used
p5
6.04
%
<
mean
10.27
%
SD
5.75
%
<
p95
23.79
%
Memory available percentage
Percentage of available memory
p5
76.21
%
<
mean
89.73
%
SD
5.75
%
<
p95
93.96
%
Max host memory used
Maximum memory usage percentage across all hosts
max
44.07
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.62
%
Memory growth rate
Rate of memory growth over time
growth
0.27
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.07
MiB/s
Disk write throughput
Disk write throughput in MiB/s
25.15
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.55
5 min
0.42
15 min
0.24
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
5.426
%
SD
3.7944
%
<
p95
11.27
%
60 second average
5.1732
%
300 second average
3.8425
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6689
%
SD
2.8641
%
<
p95
1.75
%
60 second average
0.9726
%
300 second average
1.0494
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.474
%
SD
2.7051
%
<
p95
1.14
%
60 second average
0.7675
%
300 second average
0.8782
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.55
%
<
mean
14.31
%
SD
9.71
%
<
p95
24.93
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
200.61
KiB
<
mean
230.64
KiB
SD
37.07
KiB
<
p95
274.77
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
33.5
threads
SD
10.59
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
82.11
file descriptors
SD
11.23
file descriptors
<
p95
89
file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started
Thu, 03 Sep 2026 02:35:01 UTC
Duration
900s
Peer count
300
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
remote_signals_33706506143_1
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean
0.798565
s
SD
4.138029
s
p50
0.106739
s
<
p95
1.77625
s
<
p99
20.197503
s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
total
471
over 1652.445235236s
mean rate
7451.69
/s
std
130367.49
/s
p5
0
/s
p95
13607.48
/s
peak
9.44206009e+06
/s
Timeout rate
The proportion of remote signal round-trips that timed out.
0.0022
Holochain MetricsClick to toggle visibility.
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 780.008348545s
Not enough time data to show a trend.
WASM usage: 'remote_signal::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.058010706s
Not enough time data to show a trend.
WASM usage: 'remote_signal::recv_remote_signal'
Metered usage count of the WASM ribosome for this zome function.
total
28879422
over 889.058010706s
mean rate
3.99807102563e+09
/s
std
5.11933398007e+10
/s
p5
3.36212794e+06
/s
p95
1.10771183136e+10
/s
peak
4.82655733333e+12
/s
WASM usage: 'remote_signal::signal_request'
Metered usage count of the WASM ribosome for this zome function.
total
13219920
over 889.058010706s
mean rate
1.0086030997e+08
/s
std
1.52391795277e+09
/s
p5
0
/s
p95
1.1262154287e+08
/s
peak
9.91399817017e+10
/s
Zome call duration: 'remote_signal::recv_remote_signal'
Duration of this zome call as measured by Holochain internally.
mean
0.004066
s
SD
0.001454
s
p50
0.004393
s
<
p95
0.005745
s
<
p99
0.006861
s
Zome call duration: 'remote_signal::signal_request'
Duration of this zome call as measured by Holochain internally.
mean
0.003891
s
SD
0.001247
s
p50
0.004275
s
<
p95
0.005424
s
<
p99
0.005829
s
WASM call duration: 'remote_signal::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004786
s
SD
0.004973
s
p50
0.003858
s
<
p95
0.01134
s
<
p99
0.025883
s
WASM call duration: 'remote_signal::recv_remote_signal'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002582
s
SD
0.000963
s
p50
0.002814
s
<
p95
0.003747
s
<
p99
0.004506
s
WASM call duration: 'remote_signal::signal_request'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.002392
s
SD
0.000772
s
p50
0.002646
s
<
p95
0.003374
s
<
p99
0.003732
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000738
s
SD
0.000828
s
p50
0.000626
s
<
p95
0.00147
s
<
p99
0.002925
s
Host function call duration: '__hc__emit_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.6e-05
s
SD
2.8e-05
s
p50
8.2e-05
s
<
p95
0.000111
s
<
p99
0.000129
s
Host function call duration: '__hc__send_remote_signal_1'
Duration of this host function call invoked from within WASM.
mean
7.3e-05
s
SD
4e-05
s
p50
7.9e-05
s
<
p95
0.000113
s
<
p99
0.000149
s
Host function call duration: '__hc__sys_time_1'
Duration of this host function call invoked from within WASM.
mean
1e-05
s
SD
6e-06
s
p50
1e-05
s
<
p95
1.9e-05
s
<
p99
2.5e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.000761
s
SD
0.002141
s
p50
0.000398
s
<
p95
0.001192
s
<
p99
0.01377
s
Emitted local signals
Count of local signals emitted via the emit_signal host function.
total
1599
over 889.057991576s
mean rate
225372.03
/s
std
3.43736364e+06
/s
p5
100.14
/s
p95
488215.03
/s
peak
2.20657277e+08
/s
Sent remote signals
Count of remote signals sent via the send_remote_signal host function.
total
1633
over 889.057996879s
mean rate
171994.13
/s
std
1.76583362e+06
/s
p5
76.9
/s
p95
469005.7
/s
peak
1.1021009952e+08
/s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000499
s
SD
0.000966
s
p50
0.000236
s
<
p95
0.001805
s
<
p99
0.004595
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
40.56
s
<
mean
421.93
s
SD
245.82
s
<
p95
805.19
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
3707
over 889.058009559s
mean rate
2.55398373e+06
/s
std
8.442806827e+07
/s
p5
61.81
/s
p95
2.07121018e+06
/s
peak
6.36538461538e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
18.483615
s
SD
9.817623
s
p50
16.759316
s
<
p95
36.771046
s
<
p99
46.108279
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.019992
s
SD
0.032983
s
p50
0.020378
s
<
p95
0.082636
s
<
p99
0.127889
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.096586
s
SD
0.066054
s
p50
0.098814
s
<
p95
0.226834
s
<
p99
0.318156
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.023954
s
SD
0.014324
s
p50
0.02316
s
<
p95
0.04718
s
<
p99
0.068942
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.938477
s
SD
7.052765
s
p50
7.715044
s
<
p95
23.284138
s
<
p99
32.330557
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
4.38183
s
SD
6.524977
s
p50
3.090873
s
<
p95
15.301849
s
<
p99
31.571381
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.0007
s
SD
0.000462
s
p50
0.000572
s
<
p95
0.001256
s
<
p99
0.002153
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00413
s
SD
0.002434
s
p50
0.003456
s
<
p95
0.007685
s
<
p99
0.011578
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000828
s
SD
0.000952
s
p50
0.000781
s
<
p95
0.001655
s
<
p99
0.003779
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.486984
s
SD
29.467116
s
p50
0.757426
s
<
p95
60.003757
s
<
p99
60.006556
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.310213
s
SD
23.233452
s
p50
0.28589
s
<
p95
60.001562
s
<
p99
60.001937
s
P2P received remote signals
Count of remote signals received.
total
1599
over 889.058025798s
mean rate
269209.03
/s
std
7.13514826e+06
/s
p5
82.07
/s
p95
525282.62
/s
peak
6.146953405e+08
/s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
12.49
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
29.38
KiB/s
<
mean
229.27
KiB/s
SD
287.60
KiB/s
<
p95
641.11
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
16.45
KiB/s
<
mean
35.73
KiB/s
SD
28.60
KiB/s
<
p95
83.80
KiB/s
Total bytes received
Total bytes received on primary network interface
count
27.37
GiB
mean
31.50
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
4.27
GiB
mean
4.91
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.0%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (75.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
CPU usage
Total CPU usage and kernel CPU usage
Total
p5
3.67
%
<
mean
19.87
%
SD
16.35
%
<
p95
44.74
%
System
7.37
%
CPU percentiles
CPU usage percentiles
p50
16.19
%
p95
44.74
%
p99
96.00
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
121
hosts
mean time
43.15
s
Memory used percentage
Percentage of memory used
p5
6.11
%
<
mean
13.04
%
SD
6.08
%
<
p95
27.44
%
Memory available percentage
Percentage of available memory
p5
72.56
%
<
mean
86.96
%
SD
6.08
%
<
p95
93.89
%
Max host memory used
Maximum memory usage percentage across all hosts
max
44.81
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
75.58
%
Memory growth rate
Rate of memory growth over time
growth
0.55
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.01
MiB/s
Disk write throughput
Disk write throughput in MiB/s
32.46
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.68
5 min
0.63
15 min
0.48
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.03
%
<
mean
8.0367
%
SD
11.4926
%
<
p95
23.23
%
60 second average
7.7872
%
300 second average
7.0795
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.6062
%
SD
1.9083
%
<
p95
2.21
%
60 second average
0.6168
%
300 second average
0.6093
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.3598
%
SD
1.6001
%
<
p95
1.32
%
60 second average
0.3681
%
300 second average
0.3868
%
Holochain process CPU usage
CPU usage by Holochain process
p5
1.15
%
<
mean
8.33
%
SD
8.28
%
<
p95
22.84
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
186.19
KiB
<
mean
225.23
KiB
SD
43.73
KiB
<
p95
270.88
KiB
Holochain process threads
Number of threads in Holochain process
p5
28
threads
<
mean
32.72
threads
SD
10.99
threads
<
p95
37
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
73
file descriptors
<
mean
80.52
file descriptors
SD
13.98
file descriptors
<
p95
89
file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Thu, 03 Sep 2026 03:24:59 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • default (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
validation_receipts_33706506143_1
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean
39.430725
s
SD
24.50193
s
Max
Highest per-partition mean latency
max mean
426.211474
s
Min
Lowest per-partition mean latency
min mean
15.318758
s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean
1
/s
Max
Highest per-partition mean rate
max
1.13
/s
Min
Lowest per-partition mean rate
min
0
/s
Error count
The total number of errors accumulated during the run.
0
Holochain MetricsClick to toggle visibility.
WASM usage: 'crud::create_sample_entry'
Metered usage count of the WASM ribosome for this zome function.
total
1036605
over 889.63391587s
mean rate
1.586418927e+07
/s
std
8.312502704e+07
/s
p5
0
/s
p95
4.752645067e+07
/s
peak
3.05831721966e+09
/s
WASM usage: 'crud::get_sample_entry_validation_receipts'
Metered usage count of the WASM ribosome for this zome function.
total
12177360
over 889.63391587s
mean rate
1.5415909816e+08
/s
std
1.2507877058e+09
/s
p5
548718.83
/s
p95
3.8673003e+08
/s
peak
7.51906945704e+10
/s
WASM usage: 'crud::init'
Metered usage count of the WASM ribosome for this zome function.
total
0
over 889.63391587s
Not enough time data to show a trend.
WASM usage: 'crud_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
127658
over 889.63391587s
mean rate
1.9289056e+06
/s
std
1.014509258e+07
/s
p5
0
/s
p95
5.80984057e+06
/s
peak
3.7663204367e+08
/s
Zome call duration: 'crud::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007515
s
SD
0.003586
s
p50
0.007896
s
<
p95
0.011947
s
<
p99
0.016036
s
Zome call duration: 'crud::get_sample_entry_validation_receipts'
Duration of this zome call as measured by Holochain internally.
mean
0.022444
s
SD
0.016929
s
p50
0.014171
s
<
p95
0.053888
s
<
p99
0.088298
s
WASM call duration: 'crud::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006159
s
SD
0.003141
s
p50
0.006505
s
<
p95
0.009924
s
<
p99
0.014709
s
WASM call duration: 'crud::get_sample_entry_validation_receipts'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.020842
s
SD
0.016604
s
p50
0.012485
s
<
p95
0.051645
s
<
p99
0.085916
s
WASM call duration: 'crud::init'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.004518
s
SD
0.00675
s
p50
0.003409
s
<
p95
0.009589
s
<
p99
0.056613
s
WASM call duration: 'crud_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001963
s
SD
0.000811
s
p50
0.002092
s
<
p95
0.003004
s
<
p99
0.004259
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000858
s
SD
0.004507
s
p50
0.000772
s
<
p95
0.00293
s
<
p99
0.015249
s
Host function call duration: '__hc__get_validation_receipts_1'
Duration of this host function call invoked from within WASM.
mean
0.018473
s
SD
0.016311
s
p50
0.009881
s
<
p95
0.048633
s
<
p99
0.082889
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003119
s
SD
0.001432
s
p50
0.003299
s
<
p95
0.005052
s
<
p99
0.006838
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000834
s
SD
0.000891
s
p50
0.000676
s
<
p95
0.00254
s
<
p99
0.004067
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.63
s
<
mean
423.96
s
SD
245.43
s
<
p95
805.61
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
10409
over 889.633890456s
mean rate
101231.5
/s
std
1.4862294e+06
/s
p5
235.69
/s
p95
296551.29
/s
peak
8.507175618e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
17.834302
s
SD
8.761077
s
p50
15.851109
s
<
p95
32.786094
s
<
p99
39.39823
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.005989
s
SD
0.053392
s
p50
0.010311
s
<
p95
0.152667
s
<
p99
0.245077
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.045918
s
SD
0.127348
s
p50
0.076818
s
<
p95
0.391704
s
<
p99
0.541836
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.082034
s
SD
0.064304
s
p50
0.050297
s
<
p95
0.201459
s
<
p99
0.315437
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
0.443015
s
SD
6.828735
s
p50
1.220072
s
<
p95
19.070572
s
<
p99
26.821926
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
23.471891
s
SD
21.494794
s
p50
24.490118
s
<
p95
67.015925
s
<
p99
90.979011
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.001494
s
SD
0.000913
s
p50
0.001132
s
<
p95
0.003058
s
<
p99
0.003875
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004975
s
SD
0.002441
s
p50
0.003476
s
<
p95
0.008433
s
<
p99
0.012518
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000709
s
SD
0.000364
s
p50
0.000632
s
<
p95
0.001322
s
<
p99
0.001716
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.409163
s
SD
29.60299
s
p50
0.293429
s
<
p95
60.004374
s
<
p99
60.006778
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.154711
s
SD
0.060514
s
p50
0.142287
s
<
p95
0.261223
s
<
p99
0.329725
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
16.52
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
18.15
KiB/s
<
mean
142.29
KiB/s
SD
292.92
KiB/s
<
p95
393.65
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
5.59
KiB/s
<
mean
21.59
KiB/s
SD
26.80
KiB/s
<
p95
68.15
KiB/s
Total bytes received
Total bytes received on primary network interface
count
17.46
GiB
mean
20.09
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.65
GiB
mean
3.05
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 92.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.9% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 9% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
4.14
%
<
mean
24.56
%
SD
20.01
%
<
p95
73.51
%
System
8.04
%
CPU percentiles
CPU usage percentiles
p50
19.61
%
p95
73.51
%
p99
92.91
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
92
hosts
mean time
55.80
s
Memory used percentage
Percentage of memory used
p5
6.15
%
<
mean
11.11
%
SD
5.87
%
<
p95
24.76
%
Memory available percentage
Percentage of available memory
p5
75.24
%
<
mean
88.89
%
SD
5.87
%
<
p95
93.85
%
Max host memory used
Maximum memory usage percentage across all hosts
max
47.14
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.88
%
Memory growth rate
Rate of memory growth over time
growth
0.76
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.05
MiB/s
Disk write throughput
Disk write throughput in MiB/s
34.50
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.90
5 min
1.46
15 min
2.29
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
8.84
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
8.9894
%
SD
14.047
%
<
p95
36.22
%
60 second average
9.0087
%
300 second average
16.5149
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.9531
%
SD
2.7787
%
<
p95
6.91
%
60 second average
1.9490
%
300 second average
3.1952
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2313
%
SD
1.8841
%
<
p95
3.22
%
60 second average
1.1891
%
300 second average
1.2985
%
Holochain process CPU usage
CPU usage by Holochain process
p5
2.62
%
<
mean
19.72
%
SD
16
%
<
p95
54.98
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
220.10
KiB
<
mean
262.57
KiB
SD
44.99
KiB
<
p95
309.64
KiB
Holochain process threads
Number of threads in Holochain process
p5
29
threads
<
mean
32.25
threads
SD
11.15
threads
<
p95
34
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
75
file descriptors
<
mean
80.56
file descriptors
SD
14.08
file descriptors
<
p95
87
file descriptors

Write/get_agent_activity

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 03 Sep 2026 03:42:25 UTC
Duration
900s
Peer count
150
Peer count at end
148
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_33706506143_1
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1943.35
Max
Highest chain head value observed for any single write agent.
7182
Write agents observed
17
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.667599
s
SD
1.577288
s
Max
Highest per-partition mean latency
max mean
5.540491
s
Min
Lowest per-partition mean latency
min mean
0.016555
s
Error count
The number of errors accumulated.
318
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.101777
s
SD
2.876945
s
p50
0.184868
s
<
p95
8.507965
s
<
p99
14.197159
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 791.149328975s
Not enough time data to show a trend.
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007225
s
SD
0.002429
s
p50
0.007206
s
<
p95
0.010771
s
<
p99
0.014512
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.01096
s
SD
0.011343
s
p50
0.01004
s
<
p95
0.038723
s
<
p99
0.050592
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.198862
s
SD
1.588398
s
p50
0.335305
s
<
p95
3.607647
s
<
p99
8.285853
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006548
s
SD
3.67918
s
p50
0.008429
s
<
p95
11.096165
s
<
p99
16.149015
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.00599
s
SD
0.002117
s
p50
0.00595
s
<
p95
0.008999
s
<
p99
0.013025
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.009598
s
SD
0.010424
s
p50
0.008577
s
<
p95
0.035239
s
<
p99
0.045536
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.198669
s
SD
1.587874
s
p50
0.334303
s
<
p95
3.607067
s
<
p99
8.284033
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005651
s
SD
3.679146
s
p50
0.007112
s
<
p95
11.095193
s
<
p99
16.147689
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001535
s
SD
0.000691
s
p50
0.001667
s
<
p95
0.002953
s
<
p99
0.003786
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
6.6e-05
s
SD
4.7e-05
s
p50
5e-05
s
<
p95
0.000163
s
<
p99
0.000209
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.004152
s
SD
0.006896
s
p50
0.003042
s
<
p95
0.021253
s
<
p99
0.028528
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000677
s
SD
0.000366
s
p50
0.000669
s
<
p95
0.001094
s
<
p99
0.002012
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.196645
s
SD
1.587638
s
p50
0.332065
s
<
p95
3.606357
s
<
p99
8.281361
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.00178
s
SD
3.678778
s
p50
0.001029
s
<
p95
11.09038
s
<
p99
16.141179
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
1e-05
s
SD
9e-06
s
p50
1.3e-05
s
<
p95
2.7e-05
s
<
p99
3.2e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002736
s
SD
0.002038
s
p50
0.002919
s
<
p95
0.007241
s
<
p99
0.011851
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.001637
s
SD
0.001776
s
p50
0.000286
s
<
p95
0.004991
s
<
p99
0.008635
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.63
s
<
mean
423.05
s
SD
245.42
s
<
p95
805.2
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
28755
over 888.585981869s
mean rate
1.63805087e+06
/s
std
1.4750447e+07
/s
p5
1666.91
/s
p95
3.27831748e+06
/s
peak
6.2719521723e+08
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
103.364498
s
SD
59.543637
s
p50
39.97306
s
<
p95
176.071247
s
<
p99
280.833343
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.135166
s
SD
0.313368
s
p50
0.15094
s
<
p95
0.935088
s
<
p99
1.512261
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.101451
s
SD
0.429096
s
p50
0.305294
s
<
p95
1.270546
s
<
p99
1.467406
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.766189
s
SD
0.860082
s
p50
0.108285
s
<
p95
2.395258
s
<
p99
4.270773
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
15.663196
s
SD
26.307061
s
p50
24.984429
s
<
p95
82.550656
s
<
p99
108.439401
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
16.589183
s
SD
29.947037
s
p50
0.29485
s
<
p95
85.218197
s
<
p99
127.103194
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004695
s
SD
0.003205
s
p50
0.003458
s
<
p95
0.010277
s
<
p99
0.0126
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005425
s
SD
0.003859
s
p50
0.0037
s
<
p95
0.012529
s
<
p99
0.019591
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.003962
s
SD
0.006407
s
p50
0.000961
s
<
p95
0.01613
s
<
p99
0.033627
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.889352
s
SD
29.302255
s
p50
1.647495
s
<
p95
60.02392
s
<
p99
60.060264
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
2.085387
s
SD
3.199326
s
p50
0.894947
s
<
p95
9.357734
s
<
p99
14.190451
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.171485
s
SD
29.286318
s
p50
0.441735
s
<
p95
60.008611
s
<
p99
60.019403
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.818084
s
SD
25.97525
s
p50
1.230764
s
<
p95
60.012538
s
<
p99
60.048397
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
48.03
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
133
B/s
<
mean
588.00
KiB/s
SD
468.61
KiB/s
<
p95
1.41
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
121.03
KiB/s
SD
144.85
KiB/s
<
p95
392.93
KiB/s
Total bytes received
Total bytes received on primary network interface
count
70.88
GiB
mean
81.55
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
14.59
GiB
mean
16.79
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1.30 MiB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 53% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
6.71
%
<
mean
66.07
%
SD
27.93
%
<
p95
94.63
%
System
18.04
%
CPU percentiles
CPU usage percentiles
p50
77.39
%
p95
94.63
%
p99
98.25
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
132
hosts
mean time
447.70
s
Memory used percentage
Percentage of memory used
p5
6.31
%
<
mean
11.69
%
SD
6.01
%
<
p95
26.26
%
Memory available percentage
Percentage of available memory
p5
73.74
%
<
mean
88.31
%
SD
6.01
%
<
p95
93.69
%
Max host memory used
Maximum memory usage percentage across all hosts
max
44.80
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.81
%
Memory growth rate
Rate of memory growth over time
growth
1.30
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.02
MiB/s
Disk write throughput
Disk write throughput in MiB/s
108.33
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.14
5 min
2.38
15 min
2.04
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
53.42
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.23
%
<
mean
40.3932
%
SD
28.839
%
<
p95
85.15
%
60 second average
37.6322
%
300 second average
27.9827
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
6.813
%
SD
6.1595
%
<
p95
16.39
%
60 second average
6.3621
%
300 second average
4.8272
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.5221
%
SD
3.2428
%
<
p95
9.07
%
60 second average
2.3848
%
300 second average
1.9737
%
Holochain process CPU usage
CPU usage by Holochain process
p5
11.91
%
<
mean
63.18
%
SD
24.64
%
<
p95
90.73
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
209.86
KiB
<
mean
318.16
KiB
SD
89.03
KiB
<
p95
461.86
KiB
Holochain process threads
Number of threads in Holochain process
p5
26
threads
<
mean
32.46
threads
SD
10.11
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
70
file descriptors
<
mean
80.39
file descriptors
SD
15.35
file descriptors
<
p95
89
file descriptors

Write/get_agent_activity with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Thu, 03 Sep 2026 03:59:37 UTC
Duration
1800s
Peer count
150
Peer count at end
127
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_get_agent_activity_volatile_33706506143_1
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean
273.08
/s
Max
Highest per-partition mean rate
max
1101.5
/s
Min
Lowest per-partition mean rate
min
79
/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean
0.032741
s
SD
0.383911
s
Max
Highest per-partition mean latency
max mean
0.074334
s
Min
Lowest per-partition mean latency
min mean
0.006114
s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5
66
conductors
<
mean
84.06
conductors
SD
12.77
conductors
<
p95
98
conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean
1224.999604
s
SD
752.860937
s
Max
Highest per-partition mean latency
max mean
2283.375
s
Min
Lowest per-partition mean latency
min mean
633.733333
s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean
83.75247
s
SD
70.702463
s
Max
Highest per-partition mean latency
max mean
128.008908
s
Min
Lowest per-partition mean latency
min mean
41.863697
s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean
70.976726
s
SD
77.687053
s
Max
Highest per-partition mean latency
max mean
122.673774
s
Min
Lowest per-partition mean latency
min mean
38.2334
s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk-5Sb5SgrxhYtAJpfpovfwTOhas2xezIcacv440Ex5juxOfnG
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk0agXpF9YsHkG3jwVHaU1FeVsOuD6EBeA9vi1AtfdGPJ0akjy
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk0wRmBa72SHrqQ6Y9ftsB1ad3M1D5rHTi0TDaeKRFMGWeAOAB
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk1wx7isDvM10fIQetnvW3C0jRcWQrUWmPhmMWssdy9z4Q9vmt
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk27Vna3hgzUtOzdWHb3B3JhO463r-tXuBVOjBfBcvNgf-FWmD
p5
0
<
mean
0.17
SD
0.37
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk2YLRdNgpHzzguN6QNjjUuTSaM4sS_1w4bVBWP3QEn9BmvJwH
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk37D4UQksmd9voEukGTTkpm-ggnrmGYh94V65O11CU6WLaQvG
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk3CZtJ0aFRS4V6iltMymejF-JhZapJvxTba1WHzW12hPIX1RK
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk3DMpEselxy1uK37OLvJQHhoOtes-LqioAzLKou2OtBNXLHDv
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk3xFyPfNZFDRtarsyZH023JshkqdqkUvETlKLuLsKWS3oz_Cm
p5
0
<
mean
0.2
SD
0.4
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk4jnYeqhZrJ9LRcuIFeHN8fM2t4-KNdSburrQwRe1T5zLw2G-
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk5-_xEtJq00nWbOge3xdgfvotzngbI7_RICca8UQKWtz8vSSL
p5
0
<
mean
0.14
SD
0.35
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk57aTV-reU2gZDB5z5nUYZ-OZHigTpdDdDU5E7azPYBvrXMSD
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk5IS2iOlbbIG6f3EZsnNUeHshDE0n7o9DqeCXB3k9P12yYv2y
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk6TUOiXNbMM2D4gHnoDHdoswOw-a-uUxgk9-ifFFVN4NiUIBc
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk6b-nLGKr9YyZwGfh2G8DfEPnaKLyqTiZQ7M71GffC_wCzaWw
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk8Kd2YrPWEGnbBZlSh-cPGMcIuCqdfRSIDJEcKopazcfV3GtB
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk9LAtCMc23wfDlycCPc6qfPwN-7tbC6Ifm31aPZK7Lgk-o1MG
p5
0
<
mean
0.19
SD
0.39
<
p95
1
  • get_agent_activity_volatile_agent: uhCAk9TYaZiudKNI5MH1YY2WtWoxlKI4oHV_6HmgTn0D_rud8ePo5
p5
0
<
mean
0.04
SD
0.19
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk9seUG9BzyskSdxHxH1ObOhX1gw-t6aYbtThJSCWU7Sbh6P-f
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBOmxcY0nrMoOhqxEvlfZGLSs-nKRyMHfX7_XOgU0X-3xG6nh
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkBVKE8R3rf0_h9xfbPb1BoJWGWH4YYTDOsZ-IaHUgi4nDiBvR
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkBZc6mA66vsIMQJGLZARJDEcHGnkBwOymMaQ1qDvJ5W5ockLt
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkB_zGLJUAb-hDIRzTJhgKVtjvA6jUtII3oYaMxKCLt-fiNY3I
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkCKFkBh38m3nieyoQGs6eX0S8SVQkOy44R29n0Xr1gqXxW8TJ
p5
0
<
mean
0.07
SD
0.26
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkCiJgrczM9nrRgWCVFi905-wT_K6G3R_JriPMbucQSAOUhB2u
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkD5Libjovszy18WVgj7Qaw49QiPYCrWctYuxpjZGp-hzzdkpW
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkDLLX_WTvCnji2l7Dt4K2vrL4wyJhqXBCCKSs9JQJbn72yOOh
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkDhpXUkMScPh0c_OPAU-V0pSohB3wcnJupL0enc3HfqgorsLw
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkEDNzVoSE8I6iCDLVrJRci7Se9kKHp807Fh_rYg28OLUphIqo
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkEo-Rq16pXJA4r-VBpLupVZZzbx7vbwkxGBv2mq90zEawEfl_
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkEoxf2bIrJjnBZVmhRY_KX5BPA65pYeoTqLohxIBVzHL2tnZF
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFTchlCi5OZ23ONFudz6E1p4xr40TBLxUggOXf2WTcSxDdeqR
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkFjUXM5mhPIwuOnWoIaKyGzg4SBWV7MZOtN5y4nXFw4IzEdJV
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkGXnYLDIA7CgFJ12egY5yfrlDNVhx7LhIIOapVtILlG5qfhuB
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkGYDf1vNYOo0-vF945DSnLqSL_kDmBbT86b5eKlaTbLy1i4SV
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkHG3AC_a3HyVH5XAYRCrJXCNPF5VoRbPH55f2Gnf3A1Oelrn0
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkHReqGJS4i81HnUpx834dSVNi8o6loY8Lgkn2hudcpnLO5aMf
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkHRhUogCEbmQeJJs_lf7BUTDhOHWhXlhG4yVYSrOAlk-K2BWk
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkK2lQ0gBMAn9NoIE3lb3Lyu1-l3-_jwXZUnHR0QCyQY3yKfVZ
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkL9MBy9NbX2mGGvNCrco9erkMxPoXQjlN6a4FqEBYDyEC3won
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkLnObIHVa3gzOfriujiwNBy99Ny5PNvyZGB176QHn4OZUey88
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkLyDFzWC7IXwm9quJH_AYvwxJg4FYhEvLG4WPKHE4Cpl5EAHI
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkMNvVbm-0KMLKQfh8rZ8jDXiPF0F5goPKi23oXsHdzXCXuV3Q
p5
0
<
mean
0.1
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkP7hjjYDBxihYUjyc0GrriQieES5ramHleogCgqaqEnD3xcAk
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkPQcGEheQDOnyN2WEJkdFaIej0FNZK4-DhbtOTSPv-mUGSlcK
p5
0
<
mean
0.04
SD
0.19
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkPoj61s_qE45_7k5BmHi2ncTT2GRtMdqD3pgKJEkKgTFTv8TV
p5
0
<
mean
0.04
SD
0.19
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkQ1KdCWoeBuzD22nwx3dWsxp1k_Jd15EZTddoX_jE6wFk7QL8
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQ88BhPOZ4_laPQYUn-bOR0s-dt3KNurWmZNsJz2w8eVYRmxI
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQQ9-yr3h4MLpmEWObILaubArmaffRQPaHbOx3zZZJ3LB0gAK
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkQuzyNv4Ks430Nf33jPUbG2gsS5ELbnbS9BPqRHnmfrhga2DX
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkR-3lvgNVWs2WCbYqUoVr0uv0hvIAgmCxJnFWZWV_gTEe16RK
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkRJ2oDkJBlQ0dxEj3tBe9yUCrT8_8oLKpN24Tk3oXwxZiaFmR
p5
0
<
mean
0.1
SD
0.3
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRYsOzVRX-bCO48MxTczBoacmwPuMYUlhojh8KrGgUxqo_GGL
p5
0
<
mean
0.16
SD
0.36
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkRj4g5vRjHqJqJhO44lGVjtRX5g2IROgLgQGEe2CePDMzRFh6
p5
0
<
mean
0.06
SD
0.23
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkSmqBCKbjNoodBYwStXSePMwzzJhb1mL3oPsx4VwCLmobGfER
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkT77FC2fDPV6K3Ty7ffzpxL4W-cYiqcXy6s544JuMXBXAWlcM
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkVRPIK8PNSAPGpVYFOvgq1-ZYWhaScMrEbY-QmDaqS0SA_iWz
p5
0
<
mean
0.09
SD
0.29
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVvp9d7ei-PdwYjDOBuAg_KpNlf7e8xgxw0KsdpyOTqEMOfS1
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkVzQQKWz4SBWUI95I3lV_vSH50vt5oQFnSCQIiJ641VGWnzit
p5
0
<
mean
0.13
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkWoybwVR_x692V8q5aJse-eVFNCt2H3E6ZkAXZrshR0ygqE6t
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkXUpbH9y_ymPopZRgWdqHItpmOceBhr0nZ0aFTzznOV3r3ljD
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkXtgCLdsKRlxwn0OonOFJQoF3EP-Noy5bVnJSscT0vJEjSvKy
p5
0
<
mean
0.14
SD
0.34
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkYIoB7W4_ulv8vJ8go6HpXlvuKvimYGZvitr1WL5zNpI8gVWc
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkZOcHj5lEGsIALGn1-TDvpxG3tTj2cbWXZCL2WZbH39YjM6wh
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAk_hC0DxVN-bZz_-LBcScLOxTgvUCU-5t45rhou7Ca5OBKUhaf
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkaekP_E12AXZbxnOWJMi__xJ7tIs3K_46-OjdctocBEGjzBBG
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkb_DO4le5tWHgSydfclcY8dtkUYY6px57itMRvkFG0jhaI763
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkbqkvQTX3496mYePLAmDOXacMbwJ9fTpZjNaxXKGFA_rdZy7c
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcZXv4KY5BtnE6f7F2ZbXX0px1OmblPcf7_8PC95puNWN80YH
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkcp6jTG9h5hmmyTRLE9FoRoTSkdaCcgPEYepj3LvebSrVJ9QP
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdb2DqGwGVNE7NTBuWkipQhfjE6AT6l8xVWJrwivRAxARYCE1
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkdcibl3RWBaofjc6E9zXQjbUw2gYC4Oz2MBjwQ2aD6HcYyZED
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkdlchnZuk2NvZ0tQEOY6HU4hTByYOn7qAcl4Tc2kpEAH1r1eV
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAke2ej4hqvqpg8z60EsVjYxgmmRLDRag8dE7UikEvCw3vqYmaH
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkecEB6F5-L4kjZ2-TVmBO_sKU3igjPNrx_Rs2l5SQCJZR0MDm
p5
0
<
mean
0.07
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkeejOgWbN2jdGqwh-9oPkaPmMT84WyEKlj2N8WBP2AJX35MrF
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkezpAwr4wlvrwP_socpVLTEOZICy8OSsuyk9_r86d-BKXDxow
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkiAEc9wVUHUekiWh9XDY7_tMqTyl7vAQOC5S3jf3GObneql3f
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkiAH2Moz801odcT8IwNsm9iLAB8aB3PNp9cGYek-VB1JRWsLN
p5
0
<
mean
0.08
SD
0.28
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkj6bcvRuoPg3m3QnJ0ucvhCHaysci4t9FbJIAjjSKTlVSNKLT
p5
0
<
mean
0.06
SD
0.25
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkjrrceGC5BcS41MfrGUaltmPhT2tP1hJL-rtZK6bBeMFchU1_
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkk2JeolfUkRV6_9Y8DbjkEZ7vHB4patf8TuzEyrp9VkkXd9uN
p5
0
<
mean
0.13
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAklGoQZ9K9xs8VSoyvlu0MTPktb75PsVBwkDYvbUgJldItpDd2
p5
0
<
mean
0.05
SD
0.21
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAklJiYKUFyY1YcbovSER4uzBggCMiq-ifl912DKcsv_e2Bqtto
p5
0
<
mean
0.08
SD
0.27
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmIHS0YYuBR76qf0W_M0c9IkN5MFgNElvqz4UvekeUa5n1Eh_
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkmkQrIzppQhG8GdLXEBlnmVpNAaTSI0XQ9vQSU15x83-yS3AJ
p5
0
<
mean
0.13
SD
0.33
<
p95
1
  • get_agent_activity_volatile_agent: uhCAknOwOkTWmsP8a4FtIH-vkgdNx3QjWOEL7KDpxR5XYYr5KsFC6
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAknkPnulbhgzmLA9u__xW3JJWUTj6IFtDrp4_dUjkY4neI3T4I
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkpemvtRaA5wQiXuHNJOJMrA9UMTdDQlaaTt1e3cfaor4DeLx2
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkqN9tFcVqFPThmnLdPr85ifysnkbtJJAlcAGmXk6En5T6zU4h
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkr0hjuYq0Z6N82QZ1Nv5EVbwlvruBW3cHLY6w7lTodFz0wIM3
p5
0
<
mean
0.06
SD
0.24
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkrG8M8hiA5OnXAm16SWH9K1K9ivU95ZvavwqbRMh3b-H7pMSC
p5
0
<
mean
0.04
SD
0.19
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAksb0B5J5bZmAY3gv1SLqiNr6EzoCIh8OV0lJ6hoUYdbqGKhEW
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAktH9enY4gF29oUEFyvEnPknLC5VZdFpTzsGzbVnHsAaTT8yrz
p5
0
<
mean
0.12
SD
0.32
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkurp9Rnw7OU49nHDqSaNf0HDNcsQsFTsz84UOXq5Q3iii4ddK
p5
0
<
mean
0.11
SD
0.31
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkuyLnFAQeVgncV5PysYe8MoGHKfXqWekE_QznvKkeo0fVu_cw
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkvMbk6YYSyj630peENUJlD7u7ioHcTDSmQp0k_81i9SItINzD
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkvwGmjhk3q7GQHaun2j-Q1bwvfbW4Tc9jloBsrvCdlCDOPShH
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkwD8Bc6NGj2-_9QvwxyS5Rt3QOB6rGlApA1xWokERi56CDnTW
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkwTzpMfsgjBejHed1WXo4tPJ6XuUnLXsB9u9nCVfeY4S_2ZGC
p5
0
<
mean
0.05
SD
0.22
<
p95
1
  • get_agent_activity_volatile_agent: uhCAkwoiJKKyBo08wIVcAVdujpMoZtnkd8j9l8XMflzmPTrlCMvnD
p5
0
<
mean
0.04
SD
0.2
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkycBV8wkdyCgsk7fYcMVwBOTFdSfx_yxSpG6uq1Vstqshgnkg
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkyjQ5Q8_O-3oE9MoHvUILQtTm-zkpyvwCfkXKJiMzr4jqGc3Q
p5
0
≤
mean
0
SD
0
≤
p95
0
  • get_agent_activity_volatile_agent: uhCAkzcfS6oeJZnKVzDL-3MnH85ZMqUMHzFdLU0jBAq1SZXQ2akhH
p5
0
<
mean
0.23
SD
0.42
<
p95
1
Error count
The number of errors accumulated.
3002
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.079818
s
SD
50.740542
s
p50
0.173332
s
<
p95
65.518013
s
<
p99
285.43218
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
6
over 1701.540842329s
mean rate
138.7
/s
std
789.73
/s
p5
0
/s
p95
488.66
/s
peak
25481.24
/s
Zome call duration: 'agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.006954
s
SD
0.002644
s
p50
0.007559
s
<
p95
0.011202
s
<
p99
0.012227
s
Zome call duration: 'agent_activity::create_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.006176
s
SD
0.002144
s
p50
0.006971
s
<
p95
0.009383
s
<
p99
0.011509
s
Zome call duration: 'agent_activity::get_agent_activity_full'
Duration of this zome call as measured by Holochain internally.
mean
0.018865
s
SD
11.934114
s
p50
0.037932
s
<
p95
4.026548
s
<
p99
44.466353
s
Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.032565
s
SD
3.765021
s
p50
1.056412
s
<
p95
8.329667
s
<
p99
13.994909
s
WASM call duration: 'agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005758
s
SD
0.002182
s
p50
0.006352
s
<
p95
0.009084
s
<
p99
0.010743
s
WASM call duration: 'agent_activity::create_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.005004
s
SD
0.001695
s
p50
0.005669
s
<
p95
0.007556
s
<
p99
0.008925
s
WASM call duration: 'agent_activity::get_agent_activity_full'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.081929
s
SD
52.185091
s
p50
0.13884
s
<
p95
87.674778
s
<
p99
285.441859
s
WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.048595
s
SD
8.196273
s
p50
0.747111
s
<
p95
13.993258
s
<
p99
43.953076
s
WASM call duration: 'agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001648
s
SD
0.000541
s
p50
0.001881
s
<
p95
0.002393
s
<
p99
0.002649
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
5.9e-05
s
SD
3.9e-05
s
p50
5e-05
s
<
p95
0.000143
s
<
p99
0.000156
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000585
s
SD
0.000315
s
p50
0.000623
s
<
p95
0.001314
s
<
p99
0.001798
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000634
s
SD
0.000455
s
p50
0.000569
s
<
p95
0.001241
s
<
p99
0.00295
s
Host function call duration: '__hc__get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.080539
s
SD
52.185041
s
p50
0.136212
s
<
p95
87.672839
s
<
p99
285.439668
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.045972
s
SD
8.195698
s
p50
0.740092
s
<
p95
13.97057
s
<
p99
43.946631
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
8e-06
s
SD
6.9e-05
s
p50
1.7e-05
s
<
p95
4.8e-05
s
<
p99
7.3e-05
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002586
s
SD
0.00108
s
p50
0.002945
s
<
p95
0.00418
s
<
p99
0.00638
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000359
s
SD
0.000287
s
p50
0.000385
s
<
p95
0.000955
s
<
p99
0.001271
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
5.2
s
<
mean
339.73
s
SD
477.28
s
<
p95
1485.2
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
18538
over 1789.505256884s
mean rate
303801.03
/s
std
1.88273616e+06
/s
p5
0
/s
p95
979606.94
/s
peak
8.039920774e+07
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
170.058416
s
SD
201.009675
s
p50
50.214875
s
<
p95
648.41287
s
<
p99
845.406746
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.106916
s
SD
0.181267
s
p50
0.027913
s
<
p95
0.401365
s
<
p99
0.900306
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
9e-06
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.067844
s
SD
0.526659
s
p50
0.022692
s
<
p95
0.622347
s
<
p99
2.664416
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.243361
s
SD
0.494051
s
p50
0.102982
s
<
p95
0.442833
s
<
p99
1.877446
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
23.255444
s
SD
54.873898
s
p50
14.664408
s
<
p95
164.06437
s
<
p99
274.083411
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
11.315337
s
SD
32.640788
s
p50
0.012588
s
<
p95
43.372862
s
<
p99
189.797945
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004219
s
SD
0.002442
s
p50
0.000844
s
<
p95
0.006948
s
<
p99
0.011234
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.00541
s
SD
0.162179
s
p50
0.003364
s
<
p95
0.062179
s
<
p99
0.475545
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000492
s
SD
0.001328
s
p50
0.000967
s
<
p95
0.003238
s
<
p99
0.006977
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
5.087848
s
SD
28.491456
s
p50
0.852966
s
<
p95
60.002507
s
<
p99
60.003436
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
2.275213
s
SD
3.346829
s
p50
0.66727
s
<
p95
8.26289
s
<
p99
13.944129
s
P2P request roundtrip: 'get_agent_activity'
Time spent sending a get_agent_activity request and awaiting its response.
mean
0.586105
s
SD
23.600902
s
p50
0.548
s
<
p95
60.001729
s
<
p99
60.002108
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.312496
s
SD
23.820876
s
p50
0.556831
s
<
p95
60.001584
s
<
p99
60.002365
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
22.12
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
7.30
KiB/s
<
mean
106.05
KiB/s
SD
227.37
KiB/s
<
p95
326.16
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
690
B/s
<
mean
26.72
KiB/s
SD
71.92
KiB/s
<
p95
167.55
KiB/s
Total bytes received
Total bytes received on primary network interface
count
26.44
GiB
mean
15.13
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
6.66
GiB
mean
3.81
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.1%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 17% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
1.75
%
<
mean
30.18
%
SD
31.38
%
<
p95
89.82
%
System
8.06
%
CPU percentiles
CPU usage percentiles
p50
15.12
%
p95
89.82
%
p99
97.08
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
115
hosts
mean time
298.33
s
Memory used percentage
Percentage of memory used
p5
6.41
%
<
mean
10.16
%
SD
6.39
%
<
p95
26.68
%
Memory available percentage
Percentage of available memory
p5
73.32
%
<
mean
89.84
%
SD
6.39
%
<
p95
93.59
%
Max host memory used
Maximum memory usage percentage across all hosts
max
44.67
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.78
%
Memory growth rate
Rate of memory growth over time
growth
0.18
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.60
MiB/s
Disk write throughput
Disk write throughput in MiB/s
69.02
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/15
hosts
Mount Point /etc/hosts
0/15
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/15
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.24
5 min
1.49
15 min
1.81
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
16.89
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
10.6383
%
SD
17.784
%
<
p95
52.62
%
60 second average
10.7948
%
300 second average
14.4212
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.8186
%
SD
4.366
%
<
p95
12.39
%
60 second average
1.8582
%
300 second average
2.5148
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
0.8873
%
SD
2.2518
%
<
p95
5.74
%
60 second average
0.8974
%
300 second average
1.0907
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.35
%
<
mean
26.37
%
SD
30.74
%
<
p95
84.47
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
81.97
KiB
<
mean
182.14
KiB
SD
114.12
KiB
<
p95
392.01
KiB
Holochain process threads
Number of threads in Holochain process
p5
8
threads
<
mean
27.89
threads
SD
9.94
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
23
file descriptors
<
mean
75.06
file descriptors
SD
18.79
file descriptors
<
p95
88
file descriptors

Write Validated must_get_agent_activity

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Thu, 03 Sep 2026 04:32:00 UTC
Duration
900s
Peer count
150
Peer count at end
149
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
write_validated_must_get_agent_activity_33706506143_1
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
240.89
Max
Highest chain head value observed for any single write agent.
654
Write agents observed
9
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean
251.440691
s
SD
36.680905
s
Max
Highest per-partition mean latency
max mean
1064.581857
s
Min
Lowest per-partition mean latency
min mean
19.54
s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean
1.04
/s
Max
Highest per-partition mean rate
max
6.71
/s
Min
Lowest per-partition mean rate
min
0
/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean
0.267913
s
SD
0.081479
s
Max
Highest per-partition mean latency
max mean
2.314687
s
Min
Lowest per-partition mean latency
min mean
0.022428
s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
36995
Agents affected
77 / 77
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
480
Max
Highest count in any single partition
789
Min
Lowest count in any single partition
14
Final error count
The total number of all types of error accumulated over the run by all agents.
118967
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.026882
s
SD
1.077616
s
p50
0.001863
s
<
p95
0.099531
s
<
p99
1.640835
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 750.305778389s
Not enough time data to show a trend.
Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.007844
s
SD
0.005652
s
p50
0.0067
s
<
p95
0.013176
s
<
p99
0.04039
s
Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.276434
s
SD
0.235785
s
p50
0.158013
s
<
p95
0.772933
s
<
p99
0.957059
s
Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.013525
s
SD
0.009915
s
p50
0.009166
s
<
p95
0.033376
s
<
p99
0.053547
s
Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of this zome call as measured by Holochain internally.
mean
0.029606
s
SD
0.317221
s
p50
0.008823
s
<
p95
0.028428
s
<
p99
1.654994
s
Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of this zome call as measured by Holochain internally.
mean
0.060227
s
SD
1.640026
s
p50
0.009342
s
<
p95
0.608645
s
<
p99
13.889146
s
WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006256
s
SD
0.00463
s
p50
0.005418
s
<
p95
0.009825
s
<
p99
0.033816
s
WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.274307
s
SD
0.234643
s
p50
0.156274
s
<
p95
0.767658
s
<
p99
0.952526
s
WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.011146
s
SD
0.008755
s
p50
0.007325
s
<
p95
0.028583
s
<
p99
0.048043
s
WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007189
s
SD
0.279556
s
p50
0.007077
s
<
p95
0.025759
s
<
p99
1.652932
s
WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.179999
s
SD
1.930508
s
p50
0.007835
s
<
p95
1.037027
s
<
p99
13.888266
s
WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001638
s
SD
0.001023
s
p50
0.001812
s
<
p95
0.003995
s
<
p99
0.005246
s
WASM call duration: 'validated_must_get_agent_activity_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.015829
s
SD
0.03114
s
p50
0.009857
s
<
p95
0.049207
s
<
p99
0.073407
s
Host function call duration: '__hc__agent_info_1'
Duration of this host function call invoked from within WASM.
mean
7.8e-05
s
SD
7e-05
s
p50
4.5e-05
s
<
p95
0.000201
s
<
p99
0.000347
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.005497
s
SD
0.007415
s
p50
0.0013
s
<
p95
0.020308
s
<
p99
0.037938
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.00761
s
SD
0.009397
s
p50
0.00077
s
<
p95
0.023918
s
<
p99
0.046112
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.004264
s
SD
1.394542
s
p50
0.001098
s
<
p95
0.065213
s
<
p99
3.30266
s
Host function call duration: '__hc__must_get_agent_activity_1'
Duration of this host function call invoked from within WASM.
mean
0.029009
s
SD
0.138595
s
p50
0.019168
s
<
p95
0.100295
s
<
p99
0.175754
s
Host function call duration: '__hc__query_1'
Duration of this host function call invoked from within WASM.
mean
0.118656
s
SD
0.082303
s
p50
0.077141
s
<
p95
0.248724
s
<
p99
0.285872
s
Host function call duration: '__hc__random_bytes_1'
Duration of this host function call invoked from within WASM.
mean
2.6e-05
s
SD
3.1e-05
s
p50
1.3e-05
s
<
p95
3.5e-05
s
<
p99
0.000202
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.003212
s
SD
0.002799
s
p50
0.003401
s
<
p95
0.009181
s
<
p99
0.015228
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.002406
s
SD
0.002733
s
p50
0.000416
s
<
p95
0.006491
s
<
p99
0.012944
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.58
s
<
mean
422.86
s
SD
245.26
s
<
p95
805.66
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
5823
over 889.7080461s
mean rate
883940.03
/s
std
1.617605415e+07
/s
p5
941.98
/s
p95
2.18312688e+06
/s
peak
1.20714285714e+09
/s
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
93.11456
s
SD
61.562041
s
p50
19.411778
s
<
p95
136.005151
s
<
p99
339.738336
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.280489
s
SD
1.092738
s
p50
2.449506
s
<
p95
3.935594
s
<
p99
5.189173
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
<
p99
1e-06
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
1.04879
s
SD
0.513794
s
p50
0.520632
s
<
p95
1.569616
s
<
p99
1.752736
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.74456
s
SD
0.704876
s
p50
0.115303
s
<
p95
1.169451
s
<
p99
4.412612
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
14.976299
s
SD
16.305232
s
p50
16.832181
s
<
p95
47.481479
s
<
p99
79.537196
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
8.804825
s
SD
26.943744
s
p50
0.075216
s
<
p95
62.822052
s
<
p99
126.389631
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.004198
s
SD
0.003381
s
p50
0.004397
s
<
p95
0.010886
s
<
p99
0.013152
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.004387
s
SD
0.003298
s
p50
0.003472
s
<
p95
0.00998
s
<
p99
0.014081
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.004373
s
SD
0.006345
s
p50
0.001262
s
<
p95
0.016435
s
<
p99
0.03373
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
1.45878
s
SD
28.066019
s
p50
0.903472
s
<
p95
60.002686
s
<
p99
60.023333
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
1.622497
s
SD
4.160063
s
p50
0.31487
s
<
p95
13.842333
s
<
p99
13.947209
s
P2P request roundtrip: 'must_get_agent_activity'
Time spent sending a must_get_agent_activity request and awaiting its response.
mean
3.270901
s
SD
16.24119
s
p50
0.364449
s
<
p95
60.00095
s
≤
p99
60.00095
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.347194
s
SD
22.708565
s
p50
2.035547
s
<
p95
60.004227
s
<
p99
60.017098
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
47.60
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
127
B/s
<
mean
475.12
KiB/s
SD
428.07
KiB/s
<
p95
1.37
MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
0
B/s
<
mean
97.93
KiB/s
SD
151.31
KiB/s
<
p95
404.81
KiB/s
Total bytes received
Total bytes received on primary network interface
count
57.72
GiB
mean
65.67
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
11.90
GiB
mean
13.54
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 79% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
10.69
%
<
mean
70.15
%
SD
24.2
%
<
p95
93.78
%
System
22.55
%
CPU percentiles
CPU usage percentiles
p50
78.41
%
p95
93.78
%
p99
98.35
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
138
hosts
mean time
431.57
s
Memory used percentage
Percentage of memory used
p5
6.6
%
<
mean
12.97
%
SD
6.35
%
<
p95
28
%
Memory available percentage
Percentage of available memory
p5
72
%
<
mean
87.03
%
SD
6.35
%
<
p95
93.4
%
Max host memory used
Maximum memory usage percentage across all hosts
max
46.34
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.75
%
Memory growth rate
Rate of memory growth over time
growth
0.69
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.91
MiB/s
Disk write throughput
Disk write throughput in MiB/s
131.85
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/120
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/14
hosts
Mount Point /etc/hosts
0/14
hosts
Mount Point /etc/nomad.d/client-meta.json
0/13
hosts
Mount Point /etc/resolv.conf
0/14
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.56
5 min
2.76
15 min
2.08
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
78.77
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.4
%
<
mean
47.2138
%
SD
27.7896
%
<
p95
84.38
%
60 second average
44.0248
%
300 second average
33.0027
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
9.4543
%
SD
7.1401
%
<
p95
19.32
%
60 second average
8.8714
%
300 second average
6.6205
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
4.1786
%
SD
4.3249
%
<
p95
12.02
%
60 second average
3.9661
%
300 second average
3.0807
%
Holochain process CPU usage
CPU usage by Holochain process
p5
20
%
<
mean
67.2
%
SD
20.71
%
<
p95
90.5
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
123.92
KiB
<
mean
393.59
KiB
SD
130.97
KiB
<
p95
533.52
KiB
Holochain process threads
Number of threads in Holochain process
p5
12
threads
<
mean
34.91
threads
SD
12.41
threads
<
p95
40
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
42
file descriptors
<
mean
80.66
file descriptors
SD
19.61
file descriptors
<
p95
89
file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Thu, 03 Sep 2026 05:58:48 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_and_read_33706506143_1
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean
49.76
/s
Max
Highest per-partition mean rate
max
78.3
/s
Min
Lowest per-partition mean rate
min
30.14
/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean
286.265792
s
SD
67.227138
s
Max
Highest per-partition mean latency
max mean
921.18105
s
Min
Lowest per-partition mean latency
min mean
8.338507
s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean
24.1
/s
Max
Highest per-partition mean rate
max
419
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully fetched by all zero-arc read nodes.
0.0006
Entry created count
Total number of entries created by zero-arc write nodes, by agent.
Total
263759
Agents affected
59 / 59
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4470
Max
Highest count in any single partition
7057
Min
Lowest count in any single partition
2673
Received count
Total number of entries fetched by zero-arc read nodes, by agent.
Total
9514
Agents affected
51 / 57
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
167
Max
Highest count in any single partition
817
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
32.76
SD
24.9
<
p95
55
zero-arc
p5
2
<
mean
19.82
SD
16.1
<
p95
53
Error count
The number of errors accumulated across all nodes.
744
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.212777
s
SD
0.975811
s
p50
0.462915
s
<
p95
2.528373
s
<
p99
4.068867
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 766.498641726s
Not enough time data to show a trend.
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
269277424
over 879.074133541s
mean rate
3.03483129265e+09
/s
std
1.18035815616e+10
/s
p5
7.6402784e+06
/s
p95
1.16368020569e+10
/s
peak
2.41029043495e+11
/s
WASM usage: 'timed::get_timed_entries_network'
Metered usage count of the WASM ribosome for this zome function.
total
42316921
over 879.663027682s
mean rate
6.46318697756e+10
/s
std
2.25054033441e+12
/s
p5
3.34936876e+06
/s
p95
2.09041202881e+10
/s
peak
1.08247431017e+14
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
34778064
over 889.74999662s
mean rate
1.95524479682e+09
/s
std
1.77055854817e+10
/s
p5
0
/s
p95
3.57545991895e+09
/s
peak
6.784844287e+11
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008032
s
SD
0.00306
s
p50
0.007909
s
<
p95
0.014247
s
<
p99
0.016139
s
Zome call duration: 'timed::get_timed_entries_network'
Duration of this zome call as measured by Holochain internally.
mean
0.882895
s
SD
52.575637
s
p50
28.701248
s
<
p95
147.303089
s
<
p99
233.315417
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007034
s
SD
0.002658
s
p50
0.006938
s
<
p95
0.012423
s
<
p99
0.014071
s
WASM call duration: 'timed::get_timed_entries_network'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.889709
s
SD
52.810783
s
p50
28.699685
s
<
p95
145.837834
s
<
p99
233.314029
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001389
s
SD
0.000598
s
p50
0.001829
s
<
p95
0.002595
s
<
p99
0.003544
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.00054
s
SD
0.000297
s
p50
0.000492
s
<
p95
0.001147
s
<
p99
0.001527
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000517
s
SD
0.00027
s
p50
0.000461
s
<
p95
0.001117
s
<
p99
0.001288
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.237061
s
SD
1.259631
s
p50
0.47246
s
<
p95
2.659908
s
<
p99
4.693639
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.01731
s
SD
1.372992
s
p50
0.168486
s
<
p95
2.757795
s
<
p99
6.563076
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002156
s
SD
0.000914
s
p50
0.002958
s
<
p95
0.004125
s
<
p99
0.004545
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000435
s
SD
0.000912
s
p50
0.000341
s
<
p95
0.00182
s
<
p99
0.005926
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.59
s
<
mean
423.61
s
SD
245.33
s
<
p95
806.51
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.749863304s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
20.796275
s
SD
21.508559
s
p50
0
s
<
p95
24.85942
s
<
p99
137.219473
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.131597
s
SD
0.329193
s
p50
0.002374
s
<
p95
0.182843
s
<
p99
1.519218
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.022735
s
SD
0.33255
s
p50
0.002295
s
<
p95
1.108712
s
<
p99
1.436869
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.186338
s
SD
0.138303
s
p50
0.029378
s
<
p95
0.258625
s
<
p99
0.362504
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
11.34554
s
SD
13.409155
s
p50
0.003363
s
<
p95
21.761286
s
<
p99
67.392804
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
3.144973
s
SD
10.989797
s
p50
0.0015
s
<
p95
0.231327
s
<
p99
48.006144
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.002685
s
SD
0.002747
s
p50
0.000991
s
<
p95
0.008092
s
<
p99
0.010373
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.005613
s
SD
0.002992
s
p50
0.003593
s
<
p95
0.01083
s
<
p99
0.014083
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000633
s
SD
0.002301
s
p50
0.000679
s
<
p95
0.001961
s
<
p99
0.010433
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
0.776207
s
SD
29.64275
s
p50
1.506049
s
<
p95
60.004577
s
<
p99
60.02905
s
P2P request roundtrip: 'get_links'
Time spent sending a get_links request and awaiting its response.
mean
0.072501
s
SD
10.330088
s
p50
0.128304
s
<
p95
4.48897
s
<
p99
60.000851
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.745409
s
SD
2.745257
s
p50
0.328063
s
<
p95
7.182212
s
<
p99
12.140583
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
34.26
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
9.68
KiB/s
<
mean
299.24
KiB/s
SD
373.32
KiB/s
<
p95
816.70
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
609
B/s
<
mean
77.24
KiB/s
SD
176.73
KiB/s
<
p95
362.83
KiB/s
Total bytes received
Total bytes received on primary network interface
count
36.76
GiB
mean
42.30
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.49
GiB
mean
10.92
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.7%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 37% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
2.18
%
<
mean
46.99
%
SD
36.62
%
<
p95
88.65
%
System
12.73
%
CPU percentiles
CPU usage percentiles
p50
62.39
%
p95
88.65
%
p99
94.74
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
108
hosts
mean time
387.34
s
Memory used percentage
Percentage of memory used
p5
6.66
%
<
mean
11.31
%
SD
5.55
%
<
p95
22.95
%
Memory available percentage
Percentage of available memory
p5
77.05
%
<
mean
88.69
%
SD
5.55
%
<
p95
93.34
%
Max host memory used
Maximum memory usage percentage across all hosts
max
45.73
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.67
%
Memory growth rate
Rate of memory growth over time
growth
0.43
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.26
MiB/s
Disk write throughput
Disk write throughput in MiB/s
86.30
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.74
5 min
1.75
15 min
1.89
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
36.73
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0
%
<
mean
19.1143
%
SD
18.6243
%
<
p95
48.04
%
60 second average
18.4782
%
300 second average
18.3039
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.3267
%
SD
4.0402
%
<
p95
11.43
%
60 second average
2.2625
%
300 second average
2.2177
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.2221
%
SD
2.3225
%
<
p95
5.39
%
60 second average
1.1898
%
300 second average
1.2147
%
Holochain process CPU usage
CPU usage by Holochain process
p5
0.9
%
<
mean
42.45
%
SD
34.55
%
<
p95
85.18
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
203.83
KiB
<
mean
266.73
KiB
SD
64.45
KiB
<
p95
382.62
KiB
Holochain process threads
Number of threads in Holochain process
p5
22
threads
<
mean
30.01
threads
SD
7.07
threads
<
p95
35
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
57
file descriptors
<
mean
73.81
file descriptors
SD
13.43
file descriptors
<
p95
87
file descriptors

Zero-Arc Create Data

↑ Back to index
A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 03 Sep 2026 05:06:33 UTC
Duration
900s
Peer count
150
Peer count at end
146
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data--high_zero_33706506143_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
50.46
/s
Max
Highest per-partition mean rate
max
103.85
/s
Min
Lowest per-partition mean rate
min
29.05
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
355.621837
s
SD
134.363035
s
Max
Highest per-partition mean latency
max mean
723.9246
s
Min
Lowest per-partition mean latency
min mean
59.455892
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
107.1
/s
Max
Highest per-partition mean rate
max
535.5
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0009
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
611268
Agents affected
138 / 138
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4429
Max
Highest count in any single partition
9142
Min
Lowest count in any single partition
2549
Received count
Total number of entries received by full-arc nodes, by agent.
Total
4807
Agents affected
5 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
534
Max
Highest count in any single partition
3402
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
0
<
mean
41.33
SD
37.19
<
p95
129
zero-arc
p5
2
<
mean
26.27
SD
21.73
<
p95
62
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.001252
s
SD
0.003652
s
p50
0.00258
s
<
p95
0.010483
s
<
p99
0.011622
s
WASM usage: 'timed::created_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
469668341
over 813.173926337s
mean rate
9.03389265843e+09
/s
std
4.7384315064e+10
/s
p5
1.55306087e+07
/s
p95
3.7469565385e+10
/s
peak
2.71742836041e+12
/s
WASM usage: 'timed::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
2249547312
over 879.229906263s
mean rate
9.72410310661e+09
/s
std
1.5699511707e+10
/s
p5
1.037586553e+07
/s
p95
4.9131296037e+10
/s
peak
6.39568569989e+10
/s
WASM usage: 'timed_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
60242592
over 889.073605598s
mean rate
1.08765459008e+09
/s
std
5.65075495054e+09
/s
p5
0
/s
p95
4.74664909932e+09
/s
peak
3.46493353028e+11
/s
Zome call duration: 'timed::created_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.007829
s
SD
0.003547
s
p50
0.007791
s
<
p95
0.014844
s
<
p99
0.016823
s
Zome call duration: 'timed::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.062504
s
SD
0.233847
s
p50
0.04273
s
<
p95
0.635662
s
<
p99
0.717096
s
WASM call duration: 'timed::created_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006829
s
SD
0.003069
s
p50
0.006786
s
<
p95
0.012917
s
<
p99
0.01466
s
WASM call duration: 'timed::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.061073
s
SD
0.232362
s
p50
0.03951
s
<
p95
0.631512
s
<
p99
0.709579
s
WASM call duration: 'timed_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001392
s
SD
0.001007
s
p50
0.001412
s
<
p95
0.002728
s
<
p99
0.006337
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000451
s
SD
0.000238
s
p50
0.000438
s
<
p95
0.000931
s
<
p99
0.001192
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000436
s
SD
0.000236
s
p50
0.000404
s
<
p95
0.000932
s
<
p99
0.001121
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.001243
s
SD
0.003625
s
p50
0.007332
s
<
p95
0.010632
s
<
p99
0.011682
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.002872
s
SD
0.010951
s
p50
0.002666
s
<
p95
0.029927
s
<
p99
0.031795
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.002203
s
SD
0.00344
s
p50
0.002104
s
<
p95
0.004612
s
<
p99
0.0212
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000427
s
SD
0.003032
s
p50
0.000436
s
<
p95
0.001273
s
<
p99
0.008634
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.68
s
<
mean
424.05
s
SD
245.4
s
<
p95
805.62
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 889.073629315s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
3.799345
s
SD
23.218112
s
p50
0
s
<
p95
0.866667
s
<
p99
63.065902
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
0.163091
s
SD
0.259106
s
p50
0.001813
s
<
p95
0.126028
s
<
p99
1.912296
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.006747
s
SD
0.275415
s
p50
0.001601
s
<
p95
0.800016
s
<
p99
1.422972
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.157782
s
SD
0.079851
s
p50
0.152205
s
<
p95
0.271018
s
<
p99
0.315631
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
7.88104
s
SD
10.742418
s
p50
0.002921
s
<
p95
5.012741
s
<
p99
56.716171
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
1.855158
s
SD
14.126109
s
p50
0.001221
s
<
p95
0.004348
s
<
p99
0.013319
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.006388
s
SD
0.003989
s
p50
0.007566
s
<
p95
0.014141
s
<
p99
0.01541
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.007272
s
SD
0.004622
s
p50
0.004102
s
<
p95
0.014868
s
<
p99
0.021913
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000396
s
SD
0.012583
s
p50
0.000344
s
<
p95
0.001666
s
<
p99
0.087586
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
11.181788
s
SD
25.140176
s
p50
15.625541
s
<
p95
60.127512
s
<
p99
60.695356
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
1.171692
s
SD
27.012566
s
p50
9.384217
s
<
p95
60.133188
s
<
p99
60.268104
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
49.85
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
10.00
KiB/s
<
mean
214.06
KiB/s
SD
336.07
KiB/s
<
p95
1002.29
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
1.01
KiB/s
<
mean
43.61
KiB/s
SD
63.79
KiB/s
<
p95
170.12
KiB/s
Total bytes received
Total bytes received on primary network interface
count
26.31
GiB
mean
30.28
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.36
GiB
mean
6.17
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.6%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 42% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
7.14
%
<
mean
66.35
%
SD
22.56
%
<
p95
90.21
%
System
16.50
%
CPU percentiles
CPU usage percentiles
p50
70.95
%
p95
90.21
%
p99
95.57
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
130
hosts
mean time
349.47
s
Memory used percentage
Percentage of memory used
p5
6.52
%
<
mean
11.12
%
SD
5.55
%
<
p95
24
%
Memory available percentage
Percentage of available memory
p5
76
%
<
mean
88.88
%
SD
5.55
%
<
p95
93.48
%
Max host memory used
Maximum memory usage percentage across all hosts
max
45.56
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.72
%
Memory growth rate
Rate of memory growth over time
growth
0.66
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.00
MiB/s
Disk write throughput
Disk write throughput in MiB/s
89.50
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/12
hosts
Mount Point /etc/hosts
0/12
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/12
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.12
5 min
2.07
15 min
2.24
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
41.78
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
0.94
%
<
mean
22.1046
%
SD
16.8196
%
<
p95
56.19
%
60 second average
21.6483
%
300 second average
21.5460
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.8298
%
SD
3.9596
%
<
p95
8.74
%
60 second average
2.7911
%
300 second average
2.8393
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.7315
%
SD
2.9651
%
<
p95
4.89
%
60 second average
1.7048
%
300 second average
1.6863
%
Holochain process CPU usage
CPU usage by Holochain process
p5
22.13
%
<
mean
63.53
%
SD
19.35
%
<
p95
85.38
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
194.89
KiB
<
mean
271.16
KiB
SD
63.80
KiB
<
p95
354.19
KiB
Holochain process threads
Number of threads in Holochain process
p5
24
threads
<
mean
31.72
threads
SD
6.6
threads
<
p95
36
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
65
file descriptors
<
mean
78.32
file descriptors
SD
11.67
file descriptors
<
p95
89
file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Thu, 03 Sep 2026 05:41:19 UTC
Duration
900s
Peer count
150
Peer count at end
147
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.7.0
Wind Tunnel version
0.7.0
Run ID
zero_arc_create_data_validated--high_zero_33706506143_1
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean
44.17
/s
Max
Highest per-partition mean rate
max
94.56
/s
Min
Lowest per-partition mean rate
min
28.88
/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean
43.508362
s
SD
11.156977
s
Max
Highest per-partition mean latency
max mean
100.561333
s
Min
Lowest per-partition mean latency
min mean
6.400973
s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean
0
/s
Max
Highest per-partition mean rate
max
0
/s
Min
Lowest per-partition mean rate
min
0
/s
Delivery ratio
The proportion of created entries successfully received by all full-arc nodes.
0.0002
Entry created count
Total number of entries created by zero-arc nodes, by agent.
Total
539422
Agents affected
139 / 139
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3881
Max
Highest count in any single partition
8625
Min
Lowest count in any single partition
2434
Received count
Total number of entries received by full-arc nodes, by agent.
Total
855
Agents affected
4 / 9
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
95
Max
Highest count in any single partition
399
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors.
full-arc
p5
11
<
mean
57.62
SD
31.84
<
p95
142
zero-arc
p5
2
<
mean
23.88
SD
21.39
<
p95
64
Error count
The number of errors accumulated across all nodes.
0
Holochain MetricsClick to toggle visibility.
Cascade duration
Time taken to execute a cascade (get) query inside Holochain.
mean
0.006155
s
SD
0.004893
s
p50
0.009229
s
<
p95
0.018182
s
<
p99
0.020401
s
Cascade fetch errors
Count of network fetch errors during cascade calls.
total
0
over 800.03626254s
Not enough time data to show a trend.
WASM usage: 'timed_and_validated::create_timed_entry'
Metered usage count of the WASM ribosome for this zome function.
total
273122836
over 873.420350434s
mean rate
9.11643356838e+09
/s
std
6.64302275107e+10
/s
p5
1.878551198e+07
/s
p95
3.1470428522e+10
/s
peak
3.3760375899e+12
/s
WASM usage: 'timed_and_validated::get_timed_entries_local'
Metered usage count of the WASM ribosome for this zome function.
total
80282319
over 880.03866568s
mean rate
1.20187341378e+10
/s
std
2.27355652992e+10
/s
p5
1.829051989e+07
/s
p95
7.43643212856e+10
/s
peak
9.57684269461e+10
/s
WASM usage: 'timed_and_validated_integrity::entry_defs'
Metered usage count of the WASM ribosome for this zome function.
total
149711008
over 888.515401577s
mean rate
7.31359368555e+09
/s
std
1.01286986265e+11
/s
p5
8.27973908e+06
/s
p95
2.54422199693e+10
/s
peak
7.36789422453e+12
/s
WASM usage: 'timed_and_validated_integrity::validate'
Metered usage count of the WASM ribosome for this zome function.
total
1252575707
over 888.515401577s
mean rate
7.02831053666e+10
/s
std
1.13339816023e+12
/s
p5
7.751851609e+07
/s
p95
2.26764668993e+11
/s
peak
8.4549870133e+13
/s
Zome call duration: 'timed_and_validated::create_timed_entry'
Duration of this zome call as measured by Holochain internally.
mean
0.008315
s
SD
0.003623
s
p50
0.008637
s
<
p95
0.015171
s
<
p99
0.015599
s
Zome call duration: 'timed_and_validated::get_timed_entries_local'
Duration of this zome call as measured by Holochain internally.
mean
0.082718
s
SD
0.208837
s
p50
0.073963
s
<
p95
0.664319
s
<
p99
0.943531
s
WASM call duration: 'timed_and_validated::create_timed_entry'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.007254
s
SD
0.003131
s
p50
0.007499
s
<
p95
0.013174
s
<
p99
0.013561
s
WASM call duration: 'timed_and_validated::get_timed_entries_local'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.081269
s
SD
0.208628
s
p50
0.072175
s
<
p95
0.662632
s
<
p99
0.941731
s
WASM call duration: 'timed_and_validated_integrity::entry_defs'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.001433
s
SD
0.000832
s
p50
0.001467
s
<
p95
0.002471
s
<
p99
0.005733
s
WASM call duration: 'timed_and_validated_integrity::validate'
Duration of inner WASM calls for this function, excluding Holochain overhead.
mean
0.006327
s
SD
0.003685
s
p50
0.006018
s
<
p95
0.010723
s
<
p99
0.021143
s
Host function call duration: '__hc__create_1'
Duration of this host function call invoked from within WASM.
mean
0.000459
s
SD
0.000212
s
p50
0.000475
s
<
p95
0.000859
s
<
p99
0.000929
s
Host function call duration: '__hc__create_link_1'
Duration of this host function call invoked from within WASM.
mean
0.000444
s
SD
0.000213
s
p50
0.000438
s
<
p95
0.000865
s
<
p99
0.000901
s
Host function call duration: '__hc__get_1'
Duration of this host function call invoked from within WASM.
mean
0.002403
s
SD
0.001818
s
p50
0.003602
s
<
p95
0.007393
s
<
p99
0.008031
s
Host function call duration: '__hc__get_links_1'
Duration of this host function call invoked from within WASM.
mean
0.004109
s
SD
0.007689
s
p50
0.004652
s
<
p95
0.023679
s
<
p99
0.030793
s
Host function call duration: '__hc__must_get_valid_record_1'
Duration of this host function call invoked from within WASM.
mean
0.010056
s
SD
0.004758
s
p50
0.009681
s
<
p95
0.018383
s
<
p99
0.020537
s
Host function call duration: '__hc__zome_info_1'
Duration of this host function call invoked from within WASM.
mean
0.00239
s
SD
0.002402
s
p50
0.002321
s
<
p95
0.00457
s
<
p99
0.015067
s
Post-commit duration
Time spent executing post-commit workflows.
mean
0.000563
s
SD
0.000801
s
p50
0.000646
s
<
p95
0.001424
s
<
p99
0.005499
s
Conductor uptime
Conductor uptime gauge. A drop in the trend indicates a restart during the run.
p5
42.83
s
<
mean
422.36
s
SD
244.48
s
<
p95
803.19
s
Integrated ops
Count of DHT ops integrated since the conductor started. Resets on restart.
total
0
over 888.515309173s
Not enough time data to show a trend.
Integration delay
Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.
mean
1.200096
s
SD
12.646944
s
p50
0
s
<
p95
0.144973
s
<
p99
88.151928
s
Validation attempts per op
Number of validation attempts required per op. Values consistently above 1 indicate retries.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
App validation workflow duration
Time spent running the app validation workflow.
mean
2.669476
s
SD
1.522288
s
p50
0.00206
s
<
p95
0.11518
s
<
p99
10.443836
s
Countersigning workflow duration
Time spent running the countersigning workflow.
mean
0
s
SD
0
s
p50
0
s
≤
p95
0
s
≤
p99
0
s
Integrate DHT ops workflow duration
Time spent running the integration workflow.
mean
0.009577
s
SD
0.242602
s
p50
0.002176
s
<
p95
0.871783
s
<
p99
1.205342
s
Publish DHT ops workflow duration
Time spent running the publish workflow.
mean
0.125182
s
SD
0.058536
s
p50
0.111329
s
<
p95
0.214328
s
<
p99
0.240593
s
System validation workflow duration
Time spent running the sys validation workflow.
mean
5.066482
s
SD
8.661264
s
p50
0.002972
s
<
p95
3.907002
s
<
p99
30.094502
s
Validation receipt workflow duration
Time spent running the validation receipt workflow.
mean
2.231036
s
SD
8.694276
s
p50
0.001076
s
<
p95
0.013054
s
<
p99
12.396602
s
DHT DB connection use time
Time spent holding DHT database connections.
mean
0.005612
s
SD
0.003184
s
p50
0.006233
s
<
p95
0.011331
s
<
p99
0.013047
s
Write transaction duration
Duration of exclusive write transactions across all databases.
mean
0.006164
s
SD
0.003968
s
p50
0.004087
s
<
p95
0.0126
s
<
p99
0.015753
s
Lair keystore: signing
Duration of Ed25519 signing requests to the Lair keystore.
mean
0.000429
s
SD
0.005843
s
p50
0.000375
s
<
p95
0.001577
s
<
p99
0.022426
s
P2P request roundtrip: 'get'
Time spent sending a get request and awaiting its response.
mean
20.344554
s
SD
28.479594
s
p50
4.942527
s
<
p95
60.042087
s
<
p99
60.07023
s
P2P request roundtrip: 'send_validation_receipts'
Time spent sending a send_validation_receipts request and awaiting its response.
mean
0.248449
s
SD
18.7373
s
p50
0.393393
s
<
p95
60.003318
s
<
p99
60.139638
s
Host MetricsClick to toggle visibility.
User CPU usage
CPU usage by user space
49.34
%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5
13.45
KiB/s
<
mean
180.87
KiB/s
SD
302.40
KiB/s
<
p95
693.12
KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5
855
B/s
<
mean
40.80
KiB/s
SD
55.12
KiB/s
<
p95
175.69
KiB/s
Total bytes received
Total bytes received on primary network interface
count
22.18
GiB
mean
25.52
MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.00
GiB
mean
5.76
MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 92.9%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (74.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 37% of hosts overloaded (load5/ncpus > 1.0)

CPU usage
Total CPU usage and kernel CPU usage
Total
p5
14.73
%
<
mean
67.04
%
SD
20.18
%
<
p95
87.58
%
System
17.70
%
CPU percentiles
CPU usage percentiles
p50
74.11
%
p95
87.58
%
p99
92.91
%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
126
hosts
mean time
247.95
s
Memory used percentage
Percentage of memory used
p5
6.9
%
<
mean
12.86
%
SD
5.85
%
<
p95
26.29
%
Memory available percentage
Percentage of available memory
p5
73.71
%
<
mean
87.14
%
SD
5.85
%
<
p95
93.1
%
Max host memory used
Maximum memory usage percentage across all hosts
max
46.46
%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max
74.69
%
Memory growth rate
Rate of memory growth over time
growth
0.37
MiB/s
Disk read throughput
Disk read throughput in MiB/s
0.03
MiB/s
Disk write throughput
Disk write throughput in MiB/s
97.71
MiB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/121
hosts
Mount Point /boot
0/1
hosts
Mount Point /efi-boot
0/2
hosts
Mount Point /etc/hostname
0/13
hosts
Mount Point /etc/hosts
0/13
hosts
Mount Point /etc/nomad.d/client-meta.json
0/14
hosts
Mount Point /etc/resolv.conf
0/13
hosts
Mount Point /nix/store
0/3
hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.18
5 min
2.08
15 min
2.09
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
37.41
%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5
1.4
%
<
mean
23.5785
%
SD
14.5831
%
<
p95
46.89
%
60 second average
22.8395
%
300 second average
22.0069
%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5
0
%
≤
mean
0
%
SD
0
%
≤
p95
0
%
60 second average
0.0000
%
300 second average
0.0000
%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
2.7432
%
SD
3.8048
%
<
p95
9.32
%
60 second average
2.6555
%
300 second average
2.5489
%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5
0
%
<
mean
1.6498
%
SD
2.9437
%
<
p95
3.83
%
60 second average
1.5807
%
300 second average
1.4520
%
Holochain process CPU usage
CPU usage by Holochain process
p5
28.34
%
<
mean
65.02
%
SD
16.28
%
<
p95
82.74
%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5
164.89
KiB
<
mean
377.48
KiB
SD
102.15
KiB
<
p95
502.80
KiB
Holochain process threads
Number of threads in Holochain process
p5
26
threads
<
mean
33.39
threads
SD
6.81
threads
<
p95
38
threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5
65
file descriptors
<
mean
78.91
file descriptors
SD
12.39
file descriptors
<
p95
88
file descriptors