App Install

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Sat, 14 Mar 2026 19:35:10 UTC
Peer count
1
Peer count at end
1
Behaviours
  • minimal (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
app_install_minimal_23094094490
First install time
The time from conductor startup to completion of the first app installation attempt.
1.633489796s
Install app timing
The time taken to install an app.
mean 0.80969s (SD = 0.460758s) p50 0.639183s p95 1.581685s p99 1.801318s
Host Metrics
User CPU usage
CPU usage by user space
6.20%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 18.03 KiB/s < mean 1.45 MiB/s (SD = 1.67 MiB/s) < p956.19 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 5.03 KiB/s < mean 44.95 KiB/s (SD = 45.97 KiB/s) < p95174.03 KiB/s
Total bytes received
Total bytes received on primary network interface
count
448.58 MiB
mean
1.45 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.61 MiB
mean
44.95 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 747.06 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 4.71% < mean 7.71% (SD = 3.56%) < p95 9.5%
System
1.51%
CPU percentiles
CPU usage percentiles
p50
7.17%
p95
9.50%
p99
25.92%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 8.32% < mean 12.01% (SD = 1.89%) < p95 15.32%
Memory available percentage
Percentage of available memory
p5 84.68% < mean 87.99% (SD = 1.89%) < p95 91.68%
Max host memory used
Maximum memory usage percentage across all hosts
max 12.01%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 747.06MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
5.02MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.22
5 min
1.32
15 min
1.10
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0088% (SD = 0.0349%) < p95 0.07%
60 second average
0.0072%
300 second average
0.0003%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 24.75% < mean 36.6603% (SD = 8.491%) < p95 47.73%
60 second average
32.4103%
300 second average
25.6956%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 23.49% < mean 35.4928% (SD = 8.0855%) < p95 46.1%
60 second average
31.4009%
300 second average
25.1506%
Holochain process CPU usage
CPU usage by Holochain process
p5 3.61% < mean 5.52% (SD = 3.76%) < p95 7.58%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 37.59 KiB < mean 668.80 KiB (SD = 285.87 KiB) < p951.08 MiB
Holochain process threads
Number of threads in Holochain process
p5 12 threads < mean 20.29 threads (SD = 3.02 threads) < p95 22 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 37 file descriptors < mean 445.19 file descriptors (SD = 184.49 file descriptors) < p95 665 file descriptors

App Install

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Sat, 14 Mar 2026 19:32:59 UTC
Peer count
1
Peer count at end
1
Behaviours
  • large (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
app_install_large_23094094490
First install time
The time from conductor startup to completion of the first app installation attempt.
3.725768115s
Install app timing
The time taken to install an app.
mean 0.942379s (SD = 0.464189s) p50 0.940512s p95 1.835347s p99 2.004365s
Host Metrics
User CPU usage
CPU usage by user space
25.20%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 2.05 KiB/s < mean 783.99 KiB/s (SD = 2.40 MiB/s) < p958.36 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1.09 KiB/s < mean 16.29 KiB/s (SD = 35.33 KiB/s) < p95127.91 KiB/s
Total bytes received
Total bytes received on primary network interface
count
84.22 MiB
mean
783.99 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.75 MiB
mean
16.29 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 597.55 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 24.91% < mean 34.83% (SD = 15.89%) < p95 83.14%
System
9.63%
CPU percentiles
CPU usage percentiles
p50
30.50%
p95
83.14%
p99
83.14%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.09s
Memory used percentage
Percentage of memory used
p5 23.97% < mean 30.16% (SD = 2.54%) < p95 32.94%
Memory available percentage
Percentage of available memory
p5 67.06% < mean 69.84% (SD = 2.54%) < p95 76.03%
Max host memory used
Maximum memory usage percentage across all hosts
max 30.16%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 597.55MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
9.02MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.11
5 min
2.12
15 min
1.47
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 1.74% < mean 4.6558% (SD = 4.6955%) < p95 18.65%
60 second average
3.5583%
300 second average
1.5225%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 25.46% < mean 34.4442% (SD = 8.1067%) < p95 48.83%
60 second average
34.6300%
300 second average
33.6175%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 13.35% < mean 25.1717% (SD = 6.5976%) < p95 36.91%
60 second average
26.7775%
300 second average
28.5750%
Holochain process CPU usage
CPU usage by Holochain process
p5 4.44% < mean 12.72% (SD = 15.7%) < p95 60.12%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 23.42 KiB < mean 407.76 KiB (SD = 207.63 KiB) < p95697.18 KiB
Holochain process threads
Number of threads in Holochain process
p5 9 threads < mean 15.92 threads (SD = 3.17 threads) < p95 19 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 13 file descriptors < mean 231.77 file descriptors (SD = 159.13 file descriptors) < p95 452 file descriptors

DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Sat, 14 Mar 2026 19:42:10 UTC
Peer count
65
Peer count at end
65
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_23094094490
Create rate
The average number of created records per agent.
Mean
mean 40.5/s
Max
Highest per-partition mean rate
max 100.32/s
Min
Lowest per-partition mean rate
min 6.51/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean 5.2201984844325425e+29s (SD = 4.411872959645519e+30s)
Max
Highest per-partition mean latency
max mean 4.6981786359892883e+30s
Min
Lowest per-partition mean latency
min mean 53.867625s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean 57.08/s
Max
Highest per-partition mean rate
max 245.5/s
Min
Lowest per-partition mean rate
min 0/s
Error count
The number of errors encountered during the scenario.
83
Host Metrics
User CPU usage
CPU usage by user space
34.50%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 942 B/s < mean 2.66 MiB/s (SD = 138.89 MiB/s) < p954.68 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 647 B/s < mean 554.97 KiB/s (SD = 1.60 MiB/s) < p952.08 MiB/s
Total bytes received
Total bytes received on primary network interface
count
296.33 GiB
mean
158.87 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
56.70 GiB
mean
30.40 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.1%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2767.88 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 33% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 4.95% < mean 41.2% (SD = 26.09%) < p95 90.01%
System
6.71%
CPU percentiles
CPU usage percentiles
p50
35.94%
p95
90.01%
p99
97.10%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
25 hosts
mean time
0.27s
Memory used percentage
Percentage of memory used
p5 8.7% < mean 23.69% (SD = 12.49%) < p95 50.74%
Memory available percentage
Percentage of available memory
p5 49.26% < mean 76.31% (SD = 12.49%) < p95 91.3%
Max host memory used
Maximum memory usage percentage across all hosts
max 57.80%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.58%
Memory growth rate
Rate of memory growth over time
growth 2767.88MB/s
Disk read throughput
Disk read throughput in MB/s
2.50MB/s
Disk write throughput
Disk write throughput in MB/s
968.00MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/27hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/31hosts
Mount Point /etc/hosts
0/31hosts
Mount Point /etc/resolv.conf
0/31hosts
Mount Point /nix/store
0/27hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
4.19
5 min
3.81
15 min
2.86
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
32.79%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 9.7871% (SD = 15.9566%) < p95 44.67%
60 second average
9.6862%
300 second average
9.0430%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0026% (SD = 0.1196%) ≤ p95 0%
60 second average
0.0010%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0015% (SD = 0.081%) ≤ p95 0%
60 second average
0.0002%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 27.7779% (SD = 25.1654%) < p95 70.09%
60 second average
27.3263%
300 second average
24.8439%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 20.5742% (SD = 21.1746%) < p95 64.57%
60 second average
20.1789%
300 second average
18.2361%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 30.88% (SD = 23.69%) < p95 76.06%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 239.95 KiB < mean 451.54 KiB (SD = 154.65 KiB) < p95682.86 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 33.81 threads (SD = 14.93 threads) < p95 52 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 80.41 file descriptors (SD = 28.44 file descriptors) < p95 122 file descriptors

First Call

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started
Sat, 14 Mar 2026 18:49:17 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
first_call_23094094490
Zome call timing
The time elapsed between start of install and calling init.
mean 0.015398s (SD = 0.001447s) p50 0.014573s p95 0.017232s p99 0.017232s
Not enough time data to show a trend.
Host Metrics
User CPU usage
CPU usage by user space
12.42%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 85 B/s < mean 2.52 MiB/s (SD = 1.43 MiB/s) < p954.24 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 175 B/s < mean 66.00 KiB/s (SD = 39.26 KiB/s) < p95110.62 KiB/s
Total bytes received
Total bytes received on primary network interface
count
781.05 MiB
mean
2.52 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
19.98 MiB
mean
66.00 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 240.51 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 8.3% < mean 16.25% (SD = 8.33%) < p95 25.41%
System
3.83%
CPU percentiles
CPU usage percentiles
p50
15.81%
p95
25.41%
p99
56.31%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 19.7% < mean 23.14% (SD = 1.36%) < p95 24.15%
Memory available percentage
Percentage of available memory
p5 75.85% < mean 76.86% (SD = 1.36%) < p95 80.3%
Max host memory used
Maximum memory usage percentage across all hosts
max 23.14%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 240.51MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
14.81MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.75
5 min
1.57
15 min
1.02
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0.97% < mean 1.6858% (SD = 0.5%) < p95 2.3%
60 second average
1.4355%
300 second average
0.7110%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 27.93% < mean 46.8565% (SD = 6.9344%) < p95 53.62%
60 second average
42.9432%
300 second average
27.5265%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 24.65% < mean 41.6135% (SD = 6.6718%) < p95 50.4%
60 second average
38.5623%
300 second average
25.2087%
Holochain process CPU usage
CPU usage by Holochain process
p5 3.75% < mean 5.76% (SD = 8.22%) < p95 5.8%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 282.71 KiB ≤ mean 277.09 KiB (SD = 31.38 KiB) < p95283.06 KiB
Holochain process threads
Number of threads in Holochain process
p5 17 threads ≤ mean 16.48 threads (SD = 2.83 threads) < p95 17 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 55 file descriptors < mean 62.52 file descriptors (SD = 20.01 file descriptors) < p95 100 file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sat, 14 Mar 2026 23:55:09 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_23094094490
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean 33.83/s
Max
Highest per-partition mean rate
max 74.02/s
Min
Lowest per-partition mean rate
min 9.2/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean 221.8135s (SD = 110.055593s)
Max
Highest per-partition mean latency
max mean 1025.407856s
Min
Lowest per-partition mean latency
min mean 15.267011s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean 111.76/s
Max
Highest per-partition mean rate
max 994.67/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 13 < mean 40.73 (SD = 14.47) < p95 59
zero-arc
p5 0 < mean 16.09 (SD = 13.74) < p95 38
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
682
Agents affected
59 / 61
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
11
Max
Highest count in any single partition
22
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
683

Local Signals

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started
Sat, 14 Mar 2026 18:51:46 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
local_signals_23094094490
Send timing
The time taken to emit a batch of 10,000 signals.
mean 1.700416s (SD = 0.562675s) p50 2.263091s p95 2.263091s p99 2.263091s
Receive timing
The time taken to receive the completed batch, to the nearest 250ms.
mean 30.001031s (SD = 0.000152s) p50 30.001183s p95 30.001183s p99 30.001183s
Success ratio
The proportion of signals successfully received in a batch.
min 0.8522999882698059 < mean 0.9015 (SD = 0.0492) < max 0.9506999850273132
Host Metrics
User CPU usage
CPU usage by user space
35.42%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 1.49 KiB/s < mean 3.47 MiB/s (SD = 3.15 MiB/s) < p9510.18 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 872 B/s < mean 30.94 KiB/s (SD = 27.13 KiB/s) < p9584.87 KiB/s
Total bytes received
Total bytes received on primary network interface
count
312.23 MiB
mean
3.47 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.72 MiB
mean
30.94 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.7%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 105.25 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 11.04% < mean 44.05% (SD = 26.08%) < p95 98.65%
System
8.64%
CPU percentiles
CPU usage percentiles
p50
41.71%
p95
98.65%
p99
98.65%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.11s
Memory used percentage
Percentage of memory used
p5 46.35% < mean 48.33% (SD = 0.83%) < p95 49.54%
Memory available percentage
Percentage of available memory
p5 50.46% < mean 51.67% (SD = 0.83%) < p95 53.65%
Max host memory used
Maximum memory usage percentage across all hosts
max 48.33%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.97%
Memory growth rate
Rate of memory growth over time
growth 105.25MB/s
Disk read throughput
Disk read throughput in MB/s
0.13MB/s
Disk write throughput
Disk write throughput in MB/s
34.52MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.90
5 min
1.10
15 min
0.91
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 103.69 KiB < mean 218.81 KiB (SD = 50.50 KiB) < p95278.86 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads ≤ mean 1 threads (SD = 0 threads) ≤ p95 1 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors ≤ mean 0 file descriptors (SD = 0 file descriptors) ≤ p95 0 file descriptors

Mixed Arc get_agent_activity

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Sun, 15 Mar 2026 00:11:01 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
mixed_arc_get_agent_activity_23094094490
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
308644
Agents affected
44 / 44
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
7015
Max
Highest count in any single partition
17149
Min
Lowest count in any single partition
1465
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
5259.13
Max
Highest chain head value observed for any single write agent.
9597
Write agents observed
8
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean 0.002568s (SD = 0.045089s)
Max
Highest per-partition mean latency
max mean 0.022464s
Min
Lowest per-partition mean latency
min mean 6.7e-05s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean 41.59/s
Max
Highest per-partition mean rate
max 65.62/s
Min
Lowest per-partition mean rate
min 7.77/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean 0.162949s (SD = 0.168992s)
Max
Highest per-partition mean latency
max mean 0.236924s
Min
Lowest per-partition mean latency
min mean 0.047295s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
6
Agents affected
4 / 18
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
0
Max
Highest count in any single partition
3
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5 40 < mean 54.13 (SD = 8.71) < p95 61
  • behaviour: zero_read
p5 17 < mean 37.84 (SD = 10.22) < p95 50
  • behaviour: zero_write
p5 9 < mean 42.14 (SD = 12.86) < p95 55
Final error count
The total number of errors accumulated across all agents over the run.
34
Host Metrics
User CPU usage
CPU usage by user space
29.63%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 13.93 KiB/s < mean 1.40 MiB/s (SD = 1.58 MiB/s) < p954.68 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 11.75 KiB/s < mean 559.30 KiB/s (SD = 673.00 KiB/s) < p951.84 MiB/s
Total bytes received
Total bytes received on primary network interface
count
226.10 GiB
mean
212.41 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
30.59 GiB
mean
28.74 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1282.31 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.6% < mean 36.34% (SD = 31.56%) < p95 94.35%
System
6.71%
CPU percentiles
CPU usage percentiles
p50
29.44%
p95
94.35%
p99
96.59%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
18 hosts
mean time
0.51s
Memory used percentage
Percentage of memory used
p5 8.37% < mean 22.33% (SD = 13.21%) < p95 49.41%
Memory available percentage
Percentage of available memory
p5 50.59% < mean 77.67% (SD = 13.21%) < p95 91.63%
Max host memory used
Maximum memory usage percentage across all hosts
max 78.02%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.57%
Memory growth rate
Rate of memory growth over time
growth 1282.31MB/s
Disk read throughput
Disk read throughput in MB/s
4.39MB/s
Disk write throughput
Disk write throughput in MB/s
945.76MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/29hosts
Mount Point /boot
0/2hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/32hosts
Mount Point /etc/hosts
0/32hosts
Mount Point /etc/resolv.conf
0/32hosts
Mount Point /nix/store
0/29hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.89
5 min
3.16
15 min
2.26
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
20.63%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 10.9396% (SD = 16.89%) < p95 45.38%
60 second average
10.3307%
300 second average
7.6437%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0007% (SD = 0.0418%) ≤ p95 0%
60 second average
0.0027%
300 second average
0.0029%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0005% (SD = 0.0296%) ≤ p95 0%
60 second average
0.0002%
300 second average
0.0023%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 19.1049% (SD = 22.4528%) < p95 62.91%
60 second average
19.0494%
300 second average
18.7549%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 14.624% (SD = 18.6951%) < p95 57.55%
60 second average
14.7319%
300 second average
15.1379%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 29.23% (SD = 30.41%) < p95 89.18%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 220.83 KiB < mean 343.74 KiB (SD = 115.69 KiB) < p95569.52 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 28.46 threads (SD = 14.23 threads) < p95 49 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 72.3 file descriptors (SD = 27.73 file descriptors) < p95 114 file descriptors

Mixed Arc must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Sun, 15 Mar 2026 00:28:13 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
mixed_arc_must_get_agent_activity_23094094490
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
629.56
Max
Highest chain head value observed for any single write agent.
951
Write agents observed
9
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 137.849445s (SD = 114.971037s)
Max
Highest per-partition mean latency
max mean 286.062935s
Min
Lowest per-partition mean latency
min mean 1.782s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 1.8/s
Max
Highest per-partition mean rate
max 2.36/s
Min
Lowest per-partition mean rate
min 0/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.521946s (SD = 0.476828s)
Max
Highest per-partition mean latency
max mean 0.977082s
Min
Lowest per-partition mean latency
min mean 0.107637s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
4844
Agents affected
17 / 17
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
285
Max
Highest count in any single partition
433
Min
Lowest count in any single partition
155
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5 0 < mean 43.44 (SD = 17.56) < p95 60
  • behaviour: zero_must_get_agent_activity
p5 0 < mean 27.02 (SD = 16.31) < p95 47
  • behaviour: zero_write
p5 0 < mean 31.77 (SD = 14.39) < p95 46
Final error count
The total number of all types of error accumulated over the run by all agents.
8976
Host Metrics
User CPU usage
CPU usage by user space
10.77%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 4.57 KiB/s < mean 1.97 MiB/s (SD = 43.65 MiB/s) < p955.28 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 3.33 KiB/s < mean 161.57 KiB/s (SD = 2.82 MiB/s) < p95443.51 KiB/s
Total bytes received
Total bytes received on primary network interface
count
257.02 GiB
mean
268.56 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
7.35 GiB
mean
7.68 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1202.81 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 3% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.78% < mean 14.72% (SD = 17.65%) < p95 56.89%
System
3.95%
CPU percentiles
CPU usage percentiles
p50
8.10%
p95
56.89%
p99
88.07%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
8 hosts
mean time
0.11s
Memory used percentage
Percentage of memory used
p5 10.36% < mean 24.59% (SD = 13.05%) < p95 50.78%
Memory available percentage
Percentage of available memory
p5 49.22% < mean 75.41% (SD = 13.05%) < p95 89.64%
Max host memory used
Maximum memory usage percentage across all hosts
max 79.11%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.57%
Memory growth rate
Rate of memory growth over time
growth 1202.81MB/s
Disk read throughput
Disk read throughput in MB/s
2.99MB/s
Disk write throughput
Disk write throughput in MB/s
803.31MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/26hosts
Mount Point /efi-boot
0/20hosts
Mount Point /etc/hostname
0/33hosts
Mount Point /etc/hosts
0/33hosts
Mount Point /etc/resolv.conf
0/33hosts
Mount Point /nix/store
0/26hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.76
5 min
2.14
15 min
2.39
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
3.23%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 3.0123% (SD = 9.4331%) < p95 17.35%
60 second average
2.9516%
300 second average
4.3244%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0083% (SD = 0.143%) ≤ p95 0%
60 second average
0.0096%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0034% (SD = 0.0953%) ≤ p95 0%
60 second average
0.0011%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 19.2559% (SD = 21.6232%) < p95 64.28%
60 second average
19.1151%
300 second average
18.5665%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 17.1004% (SD = 19.3824%) < p95 56.85%
60 second average
17.0124%
300 second average
16.1683%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 8.28% (SD = 14.67%) < p95 41.78%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 428.39 KiB < mean 525.64 KiB (SD = 79.35 KiB) < p95662.52 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 27.29 threads (SD = 11.83 threads) < p95 44 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 80.52 file descriptors (SD = 28.42 file descriptors) < p95 124 file descriptors

Remote Call Rate

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Sat, 14 Mar 2026 18:57:21 UTC
Peer count
10
Peer count at end
10
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
remote_call_rate_23094094490
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean -99.892575s (SD = 251.023312s)
Max
Highest per-partition mean latency
max mean 0s
Min
Lowest per-partition mean latency
min mean -151.293879s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean 0.1004s (SD = 0.130167s)
Max
Highest per-partition mean latency
max mean 0.114053s
Min
Lowest per-partition mean latency
min mean 0.084005s
Host Metrics
User CPU usage
CPU usage by user space
11.64%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 20.01 KiB/s < mean 1.60 MiB/s (SD = 1.35 MiB/s) < p954.08 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 19.11 KiB/s < mean 71.83 KiB/s (SD = 46.28 KiB/s) < p95179.46 KiB/s
Total bytes received
Total bytes received on primary network interface
count
7.02 GiB
mean
7.99 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
315.53 MiB
mean
359.00 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 862.05 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 7.49% < mean 16.43% (SD = 15.15%) < p95 65.74%
System
4.79%
CPU percentiles
CPU usage percentiles
p50
10.80%
p95
65.74%
p99
76.96%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.01s
Memory used percentage
Percentage of memory used
p5 9.69% < mean 20.07% (SD = 8.76%) < p95 35.5%
Memory available percentage
Percentage of available memory
p5 64.5% < mean 79.93% (SD = 8.76%) < p95 90.31%
Max host memory used
Maximum memory usage percentage across all hosts
max 31.19%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 862.05MB/s
Disk read throughput
Disk read throughput in MB/s
2.13MB/s
Disk write throughput
Disk write throughput in MB/s
29.34MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/3hosts
Mount Point /efi-boot
0/1hosts
Mount Point /etc/hostname
0/2hosts
Mount Point /etc/hosts
0/2hosts
Mount Point /etc/resolv.conf
0/2hosts
Mount Point /nix/store
0/3hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.76
5 min
1.51
15 min
1.29
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 2.9766% (SD = 9.6113%) < p95 32.52%
60 second average
2.4872%
300 second average
1.4827%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0558% (SD = 0.3444%) < p95 0.09%
60 second average
0.0554%
300 second average
0.0289%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0241% (SD = 0.1459%) < p95 0.04%
60 second average
0.0252%
300 second average
0.0146%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 1.79% < mean 15.9873% (SD = 10.6557%) < p95 35.23%
60 second average
16.7418%
300 second average
19.8781%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 1.69% < mean 13.2029% (SD = 8.368%) < p95 27.99%
60 second average
14.1717%
300 second average
17.6677%
Holochain process CPU usage
CPU usage by Holochain process
p5 1.7% < mean 4.54% (SD = 5.53%) < p95 12.95%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 27.33 KiB < mean 210.74 KiB (SD = 119.51 KiB) < p95539.78 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 20.56 threads (SD = 6.39 threads) < p95 33 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 8 file descriptors < mean 54.7 file descriptors (SD = 20.09 file descriptors) < p95 86 file descriptors

Single-write, many-read

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started
Sat, 14 Mar 2026 19:12:52 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
single_write_many_read_23094094490
read call timing
The time it takes to complete the zome call that retrieves the record.
mean 0.004784s (SD = 0.039681s) p50 0.003168s p95 0.003778s p99 0.007523s
read call rate
The number of read zome calls completed per second.
mean 2075.33/s
Error count
The number of errors collected.
0
Host Metrics
User CPU usage
CPU usage by user space
17.11%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 169 B/s < mean 1.90 MiB/s (SD = 739.54 KiB/s) < p952.75 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 325 B/s < mean 71.23 KiB/s (SD = 66.74 KiB/s) < p95218.54 KiB/s
Total bytes received
Total bytes received on primary network interface
count
589.06 MiB
mean
1.90 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
21.56 MiB
mean
71.23 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 13.83% < mean 23.8% (SD = 6.4%) < p95 31.64%
System
6.69%
CPU percentiles
CPU usage percentiles
p50
23.13%
p95
31.64%
p99
50.81%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 16.74% < mean 17.77% (SD = 0.78%) < p95 19.24%
Memory available percentage
Percentage of available memory
p5 80.76% < mean 82.23% (SD = 0.78%) < p95 83.26%
Max host memory used
Maximum memory usage percentage across all hosts
max 17.77%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth -43.25MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
9.72MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /efi-boot
0/1hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.39
5 min
2.44
15 min
1.84
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0.08% < mean 0.3691% (SD = 0.3954%) < p95 1.1%
60 second average
0.5913%
300 second average
0.8869%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 21.74% < mean 31.9% (SD = 4.8336%) < p95 39.98%
60 second average
33.6334%
300 second average
34.5250%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 19.24% < mean 26.7809% (SD = 4.1023%) < p95 33.18%
60 second average
27.2156%
300 second average
27.7969%
Holochain process CPU usage
CPU usage by Holochain process
p5 2.2% < mean 15% (SD = 7.53%) < p95 23.94%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 26.47 KiB < mean 192.83 KiB (SD = 87.47 KiB) < p95243.76 KiB
Holochain process threads
Number of threads in Holochain process
p5 9 threads < mean 13 threads (SD = 2.06 threads) < p95 14 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 8 file descriptors < mean 45 file descriptors (SD = 20.27 file descriptors) < p95 62 file descriptors

Validation Receipts

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Sat, 14 Mar 2026 20:13:44 UTC
Peer count
20
Peer count at end
20
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
validation_receipts_23094094490
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean 6.461545s (SD = 3.668834s)
Max
Highest per-partition mean latency
max mean 10.845387s
Min
Lowest per-partition mean latency
min mean 5.700342s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean 1.64/s
Max
Highest per-partition mean rate
max 1.74/s
Min
Lowest per-partition mean rate
min 1.36/s
Host Metrics
User CPU usage
CPU usage by user space
5.37%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 242 B/s < mean 255.83 KiB/s (SD = 1.01 MiB/s) < p951.69 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 226 B/s < mean 26.90 KiB/s (SD = 30.46 KiB/s) < p9569.51 KiB/s
Total bytes received
Total bytes received on primary network interface
count
4.38 GiB
mean
4.73 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
471.84 MiB
mean
508.59 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 171.79 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Warning Moderate swap usage (16.6% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 0.9% < mean 7.63% (SD = 8.77%) < p95 21.23%
System
2.26%
CPU percentiles
CPU usage percentiles
p50
5.17%
p95
21.23%
p99
49.39%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
2 hosts
mean time
0.01s
Memory used percentage
Percentage of memory used
p5 8.89% < mean 25.46% (SD = 16.43%) < p95 77.4%
Memory available percentage
Percentage of available memory
p5 22.6% < mean 74.54% (SD = 16.43%) < p95 91.11%
Max host memory used
Maximum memory usage percentage across all hosts
max 79.81%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 16.58%
Memory growth rate
Rate of memory growth over time
growth 171.79MB/s
Disk read throughput
Disk read throughput in MB/s
0.27MB/s
Disk write throughput
Disk write throughput in MB/s
75.34MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/7hosts
Mount Point /efi-boot
0/5hosts
Mount Point /etc/hostname
0/12hosts
Mount Point /etc/hosts
0/12hosts
Mount Point /etc/resolv.conf
0/12hosts
Mount Point /nix/store
0/7hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.02
5 min
1.58
15 min
2.48
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.7148% (SD = 2.0422%) < p95 3.43%
60 second average
0.8226%
300 second average
2.1656%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.018% (SD = 0.375%) ≤ p95 0%
60 second average
0.0162%
300 second average
0.0058%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0139% (SD = 0.3024%) ≤ p95 0%
60 second average
0.0110%
300 second average
0.0006%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 10.9596% (SD = 16.3268%) < p95 52.28%
60 second average
11.5323%
300 second average
16.1305%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 10.3726% (SD = 15.3692%) < p95 48.62%
60 second average
10.9248%
300 second average
14.7857%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 3.57% (SD = 5.55%) < p95 7.75%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 212.08 KiB < mean 246.97 KiB (SD = 28.54 KiB) < p95284.48 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 22.95 threads (SD = 8.18 threads) < p95 34 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 72.38 file descriptors (SD = 23.23 file descriptors) < p95 90 file descriptors

Write/get_agent_activity

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Sat, 14 Mar 2026 20:34:49 UTC
Peer count
65
Peer count at end
65
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_get_agent_activity_23094094490
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
4248.08
Max
Highest chain head value observed for any single write agent.
9145
Write agents observed
13
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean 0.262294s (SD = 0.413558s)
Max
Highest per-partition mean latency
max mean 2.558081s
Min
Lowest per-partition mean latency
min mean 0.103188s
Error count
The number of errors accumulated.
112
Host Metrics
User CPU usage
CPU usage by user space
34.67%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 8.97 KiB/s < mean 1.74 MiB/s (SD = 1.72 MiB/s) < p955.44 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 4.18 KiB/s < mean 480.23 KiB/s (SD = 463.68 KiB/s) < p951.33 MiB/s
Total bytes received
Total bytes received on primary network interface
count
98.55 GiB
mean
105.12 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
27.14 GiB
mean
28.95 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1761.97 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 20% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 3.61% < mean 41.88% (SD = 27.86%) < p95 95.33%
System
7.21%
CPU percentiles
CPU usage percentiles
p50
34.52%
p95
95.33%
p99
98.00%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
24 hosts
mean time
0.46s
Memory used percentage
Percentage of memory used
p5 8.67% < mean 23.61% (SD = 13.46%) < p95 50.19%
Memory available percentage
Percentage of available memory
p5 49.81% < mean 76.39% (SD = 13.46%) < p95 91.33%
Max host memory used
Maximum memory usage percentage across all hosts
max 80.59%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.58%
Memory growth rate
Rate of memory growth over time
growth 1761.97MB/s
Disk read throughput
Disk read throughput in MB/s
3.97MB/s
Disk write throughput
Disk write throughput in MB/s
826.40MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/28hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/35hosts
Mount Point /etc/hosts
0/35hosts
Mount Point /etc/resolv.conf
0/35hosts
Mount Point /nix/store
0/28hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
4.40
5 min
3.37
15 min
2.39
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
19.70%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 10.6183% (SD = 19.508%) < p95 54.54%
60 second average
10.0907%
300 second average
7.6009%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0107% (SD = 0.1758%) ≤ p95 0%
60 second average
0.0080%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0078% (SD = 0.1424%) ≤ p95 0%
60 second average
0.0044%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 26.0781% (SD = 23.2708%) < p95 59.87%
60 second average
25.0486%
300 second average
20.2136%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 19.2918% (SD = 17.9358%) < p95 47.42%
60 second average
18.5732%
300 second average
15.3039%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.15% < mean 30.51% (SD = 26.35%) < p95 88.19%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 110.32 KiB < mean 343.54 KiB (SD = 121.16 KiB) < p95531.92 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 31.77 threads (SD = 12.36 threads) < p95 51 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 44 file descriptors < mean 80.88 file descriptors (SD = 25.37 file descriptors) < p95 124 file descriptors

Write/get_agent_activity with volatile conductors

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Sat, 14 Mar 2026 20:54:02 UTC
Peer count
65
Peer count at end
65
Behaviours
  • get_agent_activity_volatile (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_get_agent_activity_volatile_23094094490
Highest observed action_seq
The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.
Mean
mean 65.4/s
Max
Highest per-partition mean rate
max 1096.12/s
Min
Lowest per-partition mean rate
min 1/s
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean 0.651412s (SD = 1.835897s)
Max
Highest per-partition mean latency
max mean 9.916622s
Min
Lowest per-partition mean latency
min mean 0.006224s
Running volatile conductors count
The number of conductors running by get_agent_activity_volatile peers.
p5 27 conductors < mean 34.48 conductors (SD = 4.55 conductors) < p95 40 conductors
Volatile conductor total running duration
The total running duration of a get_agent_activity_volatile conductor.
Mean
mean 753.354691s (SD = 497.799651s)
Max
Highest per-partition mean latency
max mean 1215.722222s
Min
Lowest per-partition mean latency
min mean 145.428571s
Volatile conductor running duration
The duration that a get_agent_activity_volatile conductor was running before being stopped.
Mean
mean 61.728383s (SD = 47.111382s)
Max
Highest per-partition mean latency
max mean 120.958003s
Min
Lowest per-partition mean latency
min mean 35.510796s
Volatile conductor stopped duration
The duration that a get_agent_activity_volatile conductor was stopped before being started again.
Mean
mean 54.613434s (SD = 53.007771s)
Max
Highest per-partition mean latency
max mean 106.886076s
Min
Lowest per-partition mean latency
min mean 23.057295s
Reached target arc
Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.
  • get_agent_activity_volatile_agent: uhCAk0E-_nIgzM37SElvh__Mb3EvsPNLHawXRhXH-BI-B3yF3ivLL
p5 0 < mean 0.24 (SD = 0.43) < p95 1
  • get_agent_activity_volatile_agent: uhCAk0Uf1AAKRdxGjfq9bK3uBoePczQffRgqjiH3Jn3IBo23B7ZRt
p5 0 < mean 0.15 (SD = 0.36) < p95 1
  • get_agent_activity_volatile_agent: uhCAk0lvMKLseDN5zazneLhOBmIktB13SR_lrXdOf52xrQho2jDe8
p5 0 < mean 0.26 (SD = 0.44) < p95 1
  • get_agent_activity_volatile_agent: uhCAk6G3bXK7kd9cMgm3DvjwuyDIw5NyVjgrAVa8g-4JBeyyqQIhU
p5 0 < mean 0.33 (SD = 0.47) < p95 1
  • get_agent_activity_volatile_agent: uhCAk8iEPzq0SV6bLMUgkSvyWiX0jibe8UH2BQ50KD561GJWrVxkt
p5 0 < mean 0.09 (SD = 0.28) < p95 1
  • get_agent_activity_volatile_agent: uhCAkADOIkw45RHzDgag7E_7bzrrV_BG2byYexXsFS5B_ERc4UuCu
p5 0 < mean 0.13 (SD = 0.34) < p95 1
  • get_agent_activity_volatile_agent: uhCAkE9fo2SBwuQA8pKegr6mB8dG_8gMnxqCXVY4nRYnX-13eYBAE
p5 0 < mean 0.21 (SD = 0.41) < p95 1
  • get_agent_activity_volatile_agent: uhCAkFhbGvClUmhrczpxkHHS4Uz23DOI1Qb7zqLWZdJ4XzRvC7hjg
p5 0 < mean 0.22 (SD = 0.41) < p95 1
  • get_agent_activity_volatile_agent: uhCAkH1or2Ng3QBZbFvyIgoC_SxNVx-JLWUTZvkmdpjcog6ulZnsi
p5 0 < mean 0.08 (SD = 0.28) < p95 1
  • get_agent_activity_volatile_agent: uhCAkHXeGItooWkxXQGgy3g1BEaKs3XrhhWw2TH2IIkbMU5qSrVAk
p5 0 < mean 0.11 (SD = 0.31) < p95 1
  • get_agent_activity_volatile_agent: uhCAkLl3Hm3T66LrsQN2_j4S87zguafEIsei09aRscWpp8cLtKiWR
p5 0 < mean 0.22 (SD = 0.41) < p95 1
  • get_agent_activity_volatile_agent: uhCAkM4U6JeCzuaHVjPc8ypd_h2Jwmpv-9fK9_uNJ3CCay1oCDKHv
p5 0 < mean 0.19 (SD = 0.39) < p95 1
  • get_agent_activity_volatile_agent: uhCAkNJ79S2YpsRrvxEpp0fSiGw0BZsR4X9TZkF2HOaNd4AtX4mNu
p5 0 < mean 0.18 (SD = 0.38) < p95 1
  • get_agent_activity_volatile_agent: uhCAkNwWicUFLERFgSm1rfuM1YUcMbXrmOYjPnRP6TTd6iBBanDiK
p5 0 < mean 0.25 (SD = 0.43) < p95 1
  • get_agent_activity_volatile_agent: uhCAkP8zDjZq6oGGO5NEAX5azUWuCvoUZOk5RcTPKLp4FOn_9xh8G
p5 0 < mean 0.24 (SD = 0.42) < p95 1
  • get_agent_activity_volatile_agent: uhCAkPutyKJfmb8VtBhAY89RaSemDxZLJx_LBLGX44ZTyT_lhecEW
p5 0 < mean 0.57 (SD = 0.49) < p95 1
  • get_agent_activity_volatile_agent: uhCAkV8RcGZ0O7_vgKl2GJm1blz7EB1iXr3Hdn_jpw_Of0sKCzKV9
p5 0 < mean 0.04 (SD = 0.2) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAkXqJxYh_92Iw8APMivvOIfRAFkzrqCw6Rwb8AnDmabwf0vurD
p5 0 < mean 0.28 (SD = 0.45) < p95 1
  • get_agent_activity_volatile_agent: uhCAkYZgl50lC9SetLsQ3Dxz1nv6MdOKljpDYpRN5Gqy0oY6D_8aM
p5 0 ≤ mean 0 (SD = 0) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAk_RLRNab_4SzVO9b84a04_rA3KMjLB5DSHzWiLnvZ4yI2k5ke
p5 0 < mean 0.35 (SD = 0.48) < p95 1
  • get_agent_activity_volatile_agent: uhCAk_TlfA_gnVoILanNtRSaYQCpr0qPYnczQyxLIubsQJSl2tPO9
p5 0 < mean 0.15 (SD = 0.36) < p95 1
  • get_agent_activity_volatile_agent: uhCAkdAJpdT9Q8wJYbTO9YvNHehfcdrkGXtAPVk8DsrGQamUc7EGV
p5 0 < mean 0.23 (SD = 0.42) < p95 1
  • get_agent_activity_volatile_agent: uhCAkdM3iZNY2jiNVqs4lZXTEy-PtVnDCzMlK3w5In4GnOb32PFDr
p5 0 < mean 0.21 (SD = 0.41) < p95 1
  • get_agent_activity_volatile_agent: uhCAkf-QxJAosxOLSlOvgHm__-Z0I4po-gcyYzV0nekk4Hf5t02xy
p5 0 < mean 0.32 (SD = 0.47) < p95 1
  • get_agent_activity_volatile_agent: uhCAkfY6VGghAJD1jYKryyAcy_nys4IkayV2LNiXExyZFkjRD-mYG
p5 0 < mean 0.1 (SD = 0.3) < p95 1
  • get_agent_activity_volatile_agent: uhCAkfrg0dNFTFaIET_u-QRl4LR0GqSgJkwigWvAnf4K8gotuQrop
p5 0 < mean 0.24 (SD = 0.43) < p95 1
  • get_agent_activity_volatile_agent: uhCAkgRz9OLmeI_JKirGrnYAKlLzsGsuGINh_kvGRkMIRawltJRh8
p5 0 < mean 0.11 (SD = 0.31) < p95 1
  • get_agent_activity_volatile_agent: uhCAkgV95hL8B6rLMD5Ph_ehiMBz88QjFZkypXCqk4gMuzerkP6jj
p5 0 < mean 0.22 (SD = 0.41) < p95 1
  • get_agent_activity_volatile_agent: uhCAkhnN1irsT7zr1OaHxJ2DQO3iCqHbPejUDOZC0HWW8kScidVXn
p5 0 < mean 0.4 (SD = 0.49) < p95 1
  • get_agent_activity_volatile_agent: uhCAkiWtmUIsAbfRan9N2gPIvau5p9GKsUR96U1hOba1WWYvIQTGJ
p5 0 < mean 0.15 (SD = 0.36) < p95 1
  • get_agent_activity_volatile_agent: uhCAkik_kYTpf8NzbfDae3lcCMPMrenh_aSI46CcnpnIHzh9MSOsD
p5 0 < mean 0.15 (SD = 0.36) < p95 1
  • get_agent_activity_volatile_agent: uhCAkjmvcPSCi1tEKBAbfpe7hDAzt1kCtrXzvZ-L-6DUvhj0BM0yq
p5 0 < mean 0.28 (SD = 0.45) < p95 1
  • get_agent_activity_volatile_agent: uhCAklGcOZgWqjrjdHly0E5HpdrAVXqEpE1-TqIFAm80GxGxp7Ytm
p5 0 < mean 0.88 (SD = 0.33) < p95 1
  • get_agent_activity_volatile_agent: uhCAkmLMMRo2GJZwpMtTywAtIfNsIgKAKUvq36Z82sZFHkHuVYDP7
p5 0 < mean 0.24 (SD = 0.42) < p95 1
  • get_agent_activity_volatile_agent: uhCAko8A5ilZyLBiWViroHgDAaR70hqTVS_zI28v19BlaX7eClWwO
p5 0 < mean 0.63 (SD = 0.48) < p95 1
  • get_agent_activity_volatile_agent: uhCAkodEnFJLcwOD77JDYsIePDkBG7YJe8N2JKwY28n0i1m3KjpJM
p5 0 ≤ mean 0 (SD = 0) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAkpKxQ0jsAMqq6LMTytbcn9gOiVfZFltbLBQM3c4BEzDppg0uO
p5 0 < mean 0.13 (SD = 0.33) < p95 1
  • get_agent_activity_volatile_agent: uhCAkrW9rG3TzatEpmDzbkgOFiJS3nZUp7HIQD0A7VDJeVi0XoiSh
p5 0 ≤ mean 0 (SD = 0) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAks9h8HfkSo9sUPUPabA-WrJOrE3zqI5wKjQEO8L2ERxJRAp4F
p5 0 < mean 0.13 (SD = 0.34) < p95 1
  • get_agent_activity_volatile_agent: uhCAkvEu-AKflGqnAY8QJhc0rpYqTV4SzLMMYEmQHwLADlsSoCP8D
p5 0 ≤ mean 0 (SD = 0) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAkw3DY5GuLKFE-VZcbfl6EnLzmUQX7QQsR-_D538uhcmpp-Bth
p5 0 < mean 0.09 (SD = 0.28) < p95 1
  • get_agent_activity_volatile_agent: uhCAkwl2miwmUH4HaWeST8Cz2bSzowEPdqKiUCy1lBpPQIDHFs5Ec
p5 0 < mean 0.07 (SD = 0.25) < p95 1
  • get_agent_activity_volatile_agent: uhCAkxOx92xQTfyWPTdwNN_z7lLjJU3BoqrxvDI_3DO6inBEZ0__w
p5 0 < mean 0.28 (SD = 0.45) < p95 1
  • get_agent_activity_volatile_agent: uhCAkxjSLLh8qfr8ULstvtljRra2hd-knCtczPuAF6WjFYtT64pdW
p5 0 < mean 0.22 (SD = 0.42) < p95 1
  • get_agent_activity_volatile_agent: uhCAkyJm_1oskWTpmr35tRB-hYsMIoaPoOmpyaTszpKlDl7JLyIza
p5 0 < mean 0.03 (SD = 0.18) ≤ p95 0
  • get_agent_activity_volatile_agent: uhCAkzKvl38buF8TrW22a-jM9Q-k6N12r2_eRCI_jtdSIGKbZeo_c
p5 0 < mean 0.26 (SD = 0.44) < p95 1
Error count
The number of errors accumulated.
745

Write/Query

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started
Sat, 14 Mar 2026 19:17:55 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_query_23094094490
Write timing
The time it takes to complete the zome function call that writes the entry.
mean 0.025813s (SD = 0.051727s) p50 0.018712s p95 0.041604s p99 0.197868s
Write rate
The number of writes completed per second.
mean 101.96/s
Query timing
The time it takes to complete the chain_query_count_len zome call that queries the source chain.
mean 0.068941s (SD = 0.041581s) p50 0.059797s p95 0.136877s p99 0.169046s
Query rate
The number of chain queries completed per second.
mean 101.96/s
Error count
The number of errors accumulated during the run.
0
Host Metrics
User CPU usage
CPU usage by user space
27.74%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 952 B/s < mean 2.34 MiB/s (SD = 1.39 MiB/s) < p955.62 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1.92 KiB/s < mean 76.70 KiB/s (SD = 56.67 KiB/s) < p95233.59 KiB/s
Total bytes received
Total bytes received on primary network interface
count
726.69 MiB
mean
2.34 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
23.22 MiB
mean
76.70 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 81.25 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 34.87% < mean 43.66% (SD = 6.99%) < p95 64.02%
System
15.92%
CPU percentiles
CPU usage percentiles
p50
42.82%
p95
64.02%
p99
65.92%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 14.79% < mean 17.15% (SD = 0.8%) < p95 18.24%
Memory available percentage
Percentage of available memory
p5 81.76% < mean 82.85% (SD = 0.8%) < p95 85.21%
Max host memory used
Maximum memory usage percentage across all hosts
max 17.15%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.05%
Memory growth rate
Rate of memory growth over time
growth 81.25MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
6.19MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.31
5 min
1.51
15 min
0.90
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 5.23% < mean 11.5747% (SD = 4.1285%) < p95 20.94%
60 second average
10.6378%
300 second average
7.4175%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.22% < mean 3.0559% (SD = 3.5085%) < p95 10.31%
60 second average
2.4525%
300 second average
1.5681%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.12% < mean 2.0875% (SD = 2.3344%) < p95 7.48%
60 second average
1.6884%
300 second average
1.1341%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 29.43% (SD = 8.7%) < p95 35.33%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 194.62 KiB < mean 245.21 KiB (SD = 22.24 KiB) < p95264.16 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 14.94 threads (SD = 3.93 threads) < p95 16 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 56.88 file descriptors (SD = 17.26 file descriptors) < p95 66 file descriptors

Write/Read

Creates an entry and reads it back, then repeats.

Started
Sat, 14 Mar 2026 19:25:55 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_read_23094094490
Write timing
The time taken to write the entry.
mean 0.021018s (SD = 0.010788s) p50 0.01904s p95 0.04029s p99 0.052977s
Write rate
The number of writes completed per second.
mean 195.62/s
Read timing
The time taken to read the entry.
mean 0.029114s (SD = 0.018869s) p50 0.025003s p95 0.062188s p99 0.072215s
Read rate
The number of reads completed per second.
mean 195.59/s
Error count
The number of errors accumulated over the run.
0
Host Metrics
User CPU usage
CPU usage by user space
23.89%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 261 B/s < mean 2.60 MiB/s (SD = 2.19 MiB/s) < p955.51 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 30 B/s < mean 35.53 KiB/s (SD = 28.52 KiB/s) < p9574.10 KiB/s
Total bytes received
Total bytes received on primary network interface
count
754.80 MiB
mean
2.60 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.06 MiB
mean
35.53 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 43.67 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 23.97% < mean 26.66% (SD = 1.99%) < p95 28.97%
System
2.78%
CPU percentiles
CPU usage percentiles
p50
26.51%
p95
28.97%
p99
33.63%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 8.17% < mean 8.47% (SD = 0.38%) < p95 8.77%
Memory available percentage
Percentage of available memory
p5 91.23% < mean 91.53% (SD = 0.38%) < p95 91.83%
Max host memory used
Maximum memory usage percentage across all hosts
max 8.47%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 43.67MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
22.59MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.67
5 min
0.83
15 min
0.43
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.302% (SD = 0.2479%) < p95 0.93%
60 second average
0.3330%
300 second average
0.3493%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.2977% (SD = 0.2482%) < p95 0.93%
60 second average
0.3270%
300 second average
0.3457%
Holochain process CPU usage
CPU usage by Holochain process
p5 18.54% < mean 23.62% (SD = 4.75%) < p95 26%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 255.78 KiB < mean 260.62 KiB (SD = 3.19 KiB) < p95264.06 KiB
Holochain process threads
Number of threads in Holochain process
p5 16 threads < mean 17.2 threads (SD = 3.06 threads) < p95 18 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 61 file descriptors ≤ mean 60.53 file descriptors (SD = 11.61 file descriptors) < p95 68 file descriptors

Write Validated must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Sat, 14 Mar 2026 21:27:57 UTC
Peer count
65
Peer count at end
65
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_validated_must_get_agent_activity_23094094490
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
709.92
Max
Highest chain head value observed for any single write agent.
1875
Write agents observed
12
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 218.763313s (SD = 68.508644s)
Max
Highest per-partition mean latency
max mean 573.394s
Min
Lowest per-partition mean latency
min mean -12.015104s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 3.08/s
Max
Highest per-partition mean rate
max 5.81/s
Min
Lowest per-partition mean rate
min 0/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.899223s (SD = 0.602497s)
Max
Highest per-partition mean latency
max mean 5.043867s
Min
Lowest per-partition mean latency
min mean 0.111623s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
10111
Agents affected
33 / 33
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
306
Max
Highest count in any single partition
787
Min
Lowest count in any single partition
5
Final error count
The total number of all types of error accumulated over the run by all agents.
103517
Host Metrics
User CPU usage
CPU usage by user space
29.03%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 49.10 KiB/s < mean 1.00 MiB/s (SD = 1.62 MiB/s) < p954.84 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 28.20 KiB/s < mean 164.26 KiB/s (SD = 140.93 KiB/s) < p95461.78 KiB/s
Total bytes received
Total bytes received on primary network interface
count
56.85 GiB
mean
60.64 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.08 GiB
mean
9.68 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1480.04 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 20% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 6.66% < mean 42.53% (SD = 29.96%) < p95 90.85%
System
13.50%
CPU percentiles
CPU usage percentiles
p50
30.53%
p95
90.85%
p99
97.34%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
32 hosts
mean time
0.34s
Memory used percentage
Percentage of memory used
p5 10.67% < mean 21.82% (SD = 12.14%) < p95 49.03%
Memory available percentage
Percentage of available memory
p5 50.97% < mean 78.18% (SD = 12.14%) < p95 89.33%
Max host memory used
Maximum memory usage percentage across all hosts
max 60.24%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.58%
Memory growth rate
Rate of memory growth over time
growth 1480.04MB/s
Disk read throughput
Disk read throughput in MB/s
0.15MB/s
Disk write throughput
Disk write throughput in MB/s
575.08MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/41hosts
Mount Point /efi-boot
0/16hosts
Mount Point /etc/hostname
0/22hosts
Mount Point /etc/hosts
0/22hosts
Mount Point /etc/resolv.conf
0/22hosts
Mount Point /nix/store
0/19hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.89
5 min
2.28
15 min
1.74
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
20.31%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 18.1776% (SD = 23.4807%) < p95 65.31%
60 second average
16.9743%
300 second average
12.2780%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0027% (SD = 0.0685%) ≤ p95 0%
60 second average
0.0032%
300 second average
0.0027%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0024% (SD = 0.0633%) ≤ p95 0%
60 second average
0.0028%
300 second average
0.0024%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.05% < mean 22.2605% (SD = 23.2756%) < p95 63.31%
60 second average
21.9816%
300 second average
19.5156%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 18.4085% (SD = 20.7%) < p95 55.86%
60 second average
18.3524%
300 second average
16.7913%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.43% < mean 35.99% (SD = 28.57%) < p95 83.36%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 290.51 KiB < mean 523.64 KiB (SD = 108.85 KiB) < p95674.38 KiB
Holochain process threads
Number of threads in Holochain process
p5 13 threads < mean 36.7 threads (SD = 13.38 threads) < p95 62 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 50 file descriptors < mean 86.44 file descriptors (SD = 22.54 file descriptors) < p95 121 file descriptors

Zero-Arc Create and Read

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Sat, 14 Mar 2026 23:01:04 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_and_read_23094094490
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean 85.53/s
Max
Highest per-partition mean rate
max 191.7/s
Min
Lowest per-partition mean rate
min 16.32/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean 5.40130741144347e+30s (SD = 9.662152431488801e+30s)
Max
Highest per-partition mean latency
max mean 8.101961117165204e+31s
Min
Lowest per-partition mean latency
min mean 14.588927s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean 62.07/s
Max
Highest per-partition mean rate
max 470.67/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 19.05 (SD = 18.71) < p95 58
zero-arc
p5 1 < mean 20.51 (SD = 15.51) < p95 50
Error count
The number of errors accumulated across all nodes.
310

Zero-Arc Create Data

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sat, 14 Mar 2026 22:03:29 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data--high_zero_23094094490
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 64.91/s
Max
Highest per-partition mean rate
max 192.19/s
Min
Lowest per-partition mean rate
min 15.68/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 355.195169s (SD = 155.012578s)
Max
Highest per-partition mean latency
max mean 545.421795s
Min
Lowest per-partition mean latency
min mean 141.665108s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 9.81/s
Max
Highest per-partition mean rate
max 27.42/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 29 < mean 43.75 (SD = 9.14) < p95 57
zero-arc
p5 0 < mean 34.54 (SD = 13.79) < p95 48
Error count
The number of errors accumulated across all nodes.
14
Host Metrics
User CPU usage
CPU usage by user space
38.60%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 5.04 KiB/s < mean 1.08 MiB/s (SD = 1.82 MiB/s) < p955.46 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 4.82 KiB/s < mean 198.17 KiB/s (SD = 270.01 KiB/s) < p95650.80 KiB/s
Total bytes received
Total bytes received on primary network interface
count
198.79 GiB
mean
181.75 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.77 GiB
mean
9.85 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.1%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 3067.72 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 49% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 4.17% < mean 49.16% (SD = 33.1%) < p95 90.04%
System
10.56%
CPU percentiles
CPU usage percentiles
p50
50.60%
p95
90.04%
p99
98.10%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
32 hosts
mean time
0.61s
Memory used percentage
Percentage of memory used
p5 8.55% < mean 18.66% (SD = 10.75%) < p95 37.34%
Memory available percentage
Percentage of available memory
p5 62.66% < mean 81.34% (SD = 10.75%) < p95 91.45%
Max host memory used
Maximum memory usage percentage across all hosts
max 48.42%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.58%
Memory growth rate
Rate of memory growth over time
growth 3067.72MB/s
Disk read throughput
Disk read throughput in MB/s
2.43MB/s
Disk write throughput
Disk write throughput in MB/s
836.17MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/42hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/16hosts
Mount Point /etc/hostname
0/20hosts
Mount Point /etc/hosts
0/20hosts
Mount Point /etc/resolv.conf
0/20hosts
Mount Point /nix/store
0/20hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.14
5 min
2.90
15 min
2.58
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
49.21%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 18.4682% (SD = 20.6495%) < p95 46.55%
60 second average
17.6645%
300 second average
16.3874%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0014% (SD = 0.0838%) ≤ p95 0%
60 second average
0.0018%
300 second average
0.0023%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0014% (SD = 0.0838%) ≤ p95 0%
60 second average
0.0014%
300 second average
0.0009%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.03% < mean 21.963% (SD = 26.3028%) < p95 71.51%
60 second average
21.6083%
300 second average
20.7263%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 18.6535% (SD = 24.8673%) < p95 67.9%
60 second average
18.4202%
300 second average
17.7057%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.4% < mean 43.37% (SD = 32.62%) < p95 86.06%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 225.20 KiB < mean 304.31 KiB (SD = 68.19 KiB) < p95417.53 KiB
Holochain process threads
Number of threads in Holochain process
p5 12 threads < mean 35.79 threads (SD = 18.75 threads) < p95 78 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 50 file descriptors < mean 76.04 file descriptors (SD = 16.77 file descriptors) < p95 105 file descriptors

Zero-arc Create Data (Validated)

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sat, 14 Mar 2026 22:40:09 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_validated--high_zero_23094094490
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 52.02/s
Max
Highest per-partition mean rate
max 113.36/s
Min
Lowest per-partition mean rate
min 14.65/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 378.279886s (SD = 159.258911s)
Max
Highest per-partition mean latency
max mean 412.597739s
Min
Lowest per-partition mean latency
min mean 338.265201s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 36.48/s
Max
Highest per-partition mean rate
max 54.63/s
Min
Lowest per-partition mean rate
min 19.15/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 20 < mean 37.09 (SD = 8.52) < p95 56
zero-arc
p5 0 < mean 26.85 (SD = 14.21) < p95 42
Error count
The number of errors accumulated across all nodes.
8
Host Metrics
User CPU usage
CPU usage by user space
39.13%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 4.06 KiB/s < mean 3.93 MiB/s (SD = 213.72 MiB/s) < p955.42 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 3.07 KiB/s < mean 175.57 KiB/s (SD = 1.08 MiB/s) < p95549.31 KiB/s
Total bytes received
Total bytes received on primary network interface
count
217.58 GiB
mean
184.13 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.50 GiB
mean
8.04 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 3119.36 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 49% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 6.46% < mean 53.32% (SD = 31.71%) < p95 91.85%
System
14.19%
CPU percentiles
CPU usage percentiles
p50
54.47%
p95
91.85%
p99
96.02%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
30 hosts
mean time
0.75s
Memory used percentage
Percentage of memory used
p5 10.77% < mean 22.51% (SD = 13.68%) < p95 51.05%
Memory available percentage
Percentage of available memory
p5 48.95% < mean 77.49% (SD = 13.68%) < p95 89.23%
Max host memory used
Maximum memory usage percentage across all hosts
max 79.31%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.57%
Memory growth rate
Rate of memory growth over time
growth 3119.36MB/s
Disk read throughput
Disk read throughput in MB/s
1.20MB/s
Disk write throughput
Disk write throughput in MB/s
874.48MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/41hosts
Mount Point /efi-boot
0/16hosts
Mount Point /etc/hostname
0/20hosts
Mount Point /etc/hosts
0/20hosts
Mount Point /etc/resolv.conf
0/20hosts
Mount Point /nix/store
0/18hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.52
5 min
3.06
15 min
2.77
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
49.21%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0.02% < mean 24.1558% (SD = 23.8043%) < p95 60.42%
60 second average
22.7937%
300 second average
19.4669%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0019% (SD = 0.0638%) ≤ p95 0%
60 second average
0.0021%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0017% (SD = 0.0589%) ≤ p95 0%
60 second average
0.0019%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 20.8917% (SD = 25.0076%) < p95 68.56%
60 second average
20.3629%
300 second average
19.3223%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 17.0577% (SD = 23.1223%) < p95 64.71%
60 second average
16.7108%
300 second average
16.2280%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.23% < mean 46.64% (SD = 32.22%) < p95 88.19%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 290.36 KiB < mean 492.50 KiB (SD = 112.63 KiB) < p95649.96 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 30.83 threads (SD = 16.03 threads) < p95 53 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 48 file descriptors < mean 77.81 file descriptors (SD = 19.27 file descriptors) < p95 107 file descriptors

Zome Call, Single Value

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started
Sat, 14 Mar 2026 19:30:32 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zome_call_single_value_23094094490
Call timing
The time taken for the call to complete.
mean 0.005266s (SD = 0.00535s) p50 0.004294s p95 0.009763s p99 0.018679s
Call rate
The number of zome calls completed per second.
mean 1877.59/s
Error count
The number of errors accumulated during the run.
0
Host Metrics
User CPU usage
CPU usage by user space
20.85%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 251.02 KiB/s < mean 1.80 MiB/s (SD = 1.75 MiB/s) < p955.38 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 20.07 KiB/s < mean 62.90 KiB/s (SD = 44.61 KiB/s) < p95134.71 KiB/s
Total bytes received
Total bytes received on primary network interface
count
540.25 MiB
mean
1.80 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
18.43 MiB
mean
62.90 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 92.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 344.29 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 33.92% < mean 42.77% (SD = 13.66%) < p95 78.67%
System
21.92%
CPU percentiles
CPU usage percentiles
p50
36.34%
p95
78.67%
p99
92.64%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.03s
Memory used percentage
Percentage of memory used
p5 14.23% < mean 20.76% (SD = 2.11%) < p95 22.03%
Memory available percentage
Percentage of available memory
p5 77.97% < mean 79.24% (SD = 2.11%) < p95 85.77%
Max host memory used
Maximum memory usage percentage across all hosts
max 20.76%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.05%
Memory growth rate
Rate of memory growth over time
growth 344.29MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
3.37MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.15
5 min
1.56
15 min
1.22
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 5.41% < mean 12.3223% (SD = 9.2191%) < p95 31.68%
60 second average
11.6368%
300 second average
9.1884%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.7132% (SD = 1.2097%) < p95 3.49%
60 second average
0.6306%
300 second average
0.8810%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.5348% (SD = 1.0271%) < p95 2.28%
60 second average
0.4606%
300 second average
0.6287%
Holochain process CPU usage
CPU usage by Holochain process
p5 1.67% < mean 6.91% (SD = 12.83%) < p95 40.81%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 110.67 KiB < mean 215.20 KiB (SD = 40.34 KiB) < p95241.22 KiB
Holochain process threads
Number of threads in Holochain process
p5 7 threads < mean 13.58 threads (SD = 2.77 threads) < p95 17 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 11 file descriptors < mean 51.35 file descriptors (SD = 13.98 file descriptors) < p95 64 file descriptors