App Install

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Fri, 13 Mar 2026 20:49:43 UTC
Peer count
1
Peer count at end
1
Behaviours
  • large (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
app_install_large_23066388637
First install time
The time from conductor startup to completion of the first app installation attempt.
0.58353015s
Install app timing
The time taken to install an app.
mean 0.21251s (SD = 0.009397s) p50 0.20999s p95 0.229759s p99 0.258609s
Host Metrics
User CPU usage
CPU usage by user space
44.19%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 815 B/s < mean 416.51 KiB/s (SD = 1.37 MiB/s) < p955.15 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 504 B/s < mean 7.69 KiB/s (SD = 12.89 KiB/s) < p9550.78 KiB/s
Total bytes received
Total bytes received on primary network interface
count
52.88 MiB
mean
416.51 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
999.77 KiB
mean
7.69 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 99.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1562.09 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 36.48% < mean 55.16% (SD = 19.02%) < p95 99.85%
System
10.98%
CPU percentiles
CPU usage percentiles
p50
44.85%
p95
99.85%
p99
99.85%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.15s
Memory used percentage
Percentage of memory used
p5 43.44% < mean 52.38% (SD = 5.91%) < p95 64.26%
Memory available percentage
Percentage of available memory
p5 35.74% < mean 47.62% (SD = 5.91%) < p95 56.56%
Max host memory used
Maximum memory usage percentage across all hosts
max 52.38%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.96%
Memory growth rate
Rate of memory growth over time
growth 1562.09MB/s
Disk read throughput
Disk read throughput in MB/s
0.38MB/s
Disk write throughput
Disk write throughput in MB/s
23.05MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.07
5 min
1.08
15 min
0.75
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 179.33 KiB < mean 723.01 KiB (SD = 442.20 KiB) < p951.58 MiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads ≤ mean 1 threads (SD = 0 threads) ≤ p95 1 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors ≤ mean 0 file descriptors (SD = 0 file descriptors) ≤ p95 0 file descriptors

App Install

Measure app installation speed. This scenario has two roles:

  • minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
  • large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started
Fri, 13 Mar 2026 20:52:02 UTC
Peer count
1
Peer count at end
1
Behaviours
  • minimal (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
app_install_minimal_23066388637
First install time
The time from conductor startup to completion of the first app installation attempt.
1.391337913s
Install app timing
The time taken to install an app.
mean 0.172648s (SD = 0.008088s) p50 0.170965s p95 0.190926s p99 0.204876s
Host Metrics
User CPU usage
CPU usage by user space
40.53%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 780 B/s < mean 630.26 KiB/s (SD = 1.52 MiB/s) < p956.10 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 574 B/s < mean 16.56 KiB/s (SD = 25.97 KiB/s) < p9597.10 KiB/s
Total bytes received
Total bytes received on primary network interface
count
110.79 MiB
mean
630.26 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.91 MiB
mean
16.56 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1814.92 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 36.8% < mean 51.25% (SD = 15.56%) < p95 94.97%
System
10.72%
CPU percentiles
CPU usage percentiles
p50
44.67%
p95
94.97%
p99
94.97%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.11s
Memory used percentage
Percentage of memory used
p5 46.63% < mean 54.63% (SD = 7.02%) < p95 69.23%
Memory available percentage
Percentage of available memory
p5 30.77% < mean 45.37% (SD = 7.02%) < p95 53.37%
Max host memory used
Maximum memory usage percentage across all hosts
max 54.63%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.96%
Memory growth rate
Rate of memory growth over time
growth 1814.92MB/s
Disk read throughput
Disk read throughput in MB/s
0.51MB/s
Disk write throughput
Disk write throughput in MB/s
28.32MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.76
5 min
1.86
15 min
1.11
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 144.68 KiB < mean 797.27 KiB (SD = 576.27 KiB) < p951.94 MiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 1.45 threads (SD = 1.96 threads) < p95 10 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 0.35 file descriptors (SD = 1.53 file descriptors) < p95 7 file descriptors

DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Fri, 13 Mar 2026 21:03:11 UTC
Peer count
65
Peer count at end
65
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_23066388637
Create rate
The average number of created records per agent.
Mean
mean 120.37/s
Max
Highest per-partition mean rate
max 578.33/s
Min
Lowest per-partition mean rate
min 0/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean 8.507059173023461e+31s (SD = 2.432676s)
Max
Highest per-partition mean latency
max mean 3.4028236692093845e+32s
Min
Lowest per-partition mean latency
min mean 3.455707s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean 79.38/s
Max
Highest per-partition mean rate
max 317.5/s
Min
Lowest per-partition mean rate
min 0/s
Error count
The number of errors encountered during the scenario.
0
Cascade duration
The time taken to execute a cascade query.
mean 0.002912s (SD = 0.005454s) p50 0.00054s p95 0.016234s p99 0.023794s
WASM usage
The metered usage of a wasm ribosome.
Total
total 76970over 36401.878882386s mean rate 3.9998174967e+08/s std 1.632300881932e+10/s p5 0/s p95 4.6062513839e+08/s peak 3.0407873100983e+12/s
zome=timed,fn=created_timed_entry
total 0over 36397.850051814s
Not enough time data to show a trend.
zome=timed,fn=get_timed_entries_local
total 0over 32075.895181257s
Not enough time data to show a trend.
zome=timed_integrity,fn=entry_defs
total 0over 36401.878312709s
Not enough time data to show a trend.
post_commit duration
The time spent executing a post commit.
mean 0.00034s (SD = 0.000621s) p50 0.00016s p95 0.001075s p99 0.00277s
Publish DHT ops workflow duration
The time spent running the publish workflow.
mean 0.025706s (SD = 0.057931s) p50 0.015928s p95 0.061146s p99 0.084302s
Integrate DHT ops workflow duration
The time spent running the integration workflow.
mean 0.007769s (SD = 0.059799s) p50 0.002066s p95 0.016007s p99 0.049121s
Countersigning workflow duration
Duration
mean 0.001497s (SD = 0.001725s) p50 0.00088s p95 0.006691s p99 0.008458s
App validation workflow duration
The time spent running the app validation workflow.
mean 2.746951s (SD = 5.674401s) p50 0.008724s p95 17.653157s p99 21.290289s
System validation workflow duration
The time spent running the sys validation workflow.
mean 0.538042s (SD = 2.403282s) p50 0.004913s p95 0.224088s p99 11.527899s
Validation receipt workflow duration
The time spent running the validation receipt workflow.
mean 0.050084s (SD = 0.732774s) p50 0.004494s p95 0.011237s p99 0.480802s
Authored DB
The utilization of connections in the authored database connection pool.
Utilisation
p5 0% < mean 0.05% (SD = 0.06%) < p95 0.11%
Use time
mean 0.001516s (SD = 0.004237s) p50 0.000194s p95 0.009385s p99 0.022971s
Conductor DB
The utilization of connections in the conductor database connection pool.
Utilisation
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
Use time
mean 0.000177s (SD = 0.000397s) p50 9.3e-05s p95 0.000487s p99 0.001663s
DHT DB
The utilization of connections in the DHT database connection pool.
Utilisation
p5 0% < mean 0.17% (SD = 0.1%) < p95 0.33%
Use time
mean 0.000331s (SD = 0.001031s) p50 0.000172s p95 0.00093s p99 0.002759s
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.341756s (SD = 3.109041s) p50 0.081474s p95 0.327596s p99 4.075643s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.114988s (SD = 0.529522s) p50 0.041272s p95 0.32423s p99 1.438371s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1.6e-05s (SD = 0.000114s) p50 1e-05s p95 3.3e-05s p99 7.2e-05s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.001105s (SD = 0.004736s) p50 0.000451s p95 0.003084s p99 0.009592s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.058459s (SD = 0.213898s) p50 0.016458s p95 0.176612s p99 1.065099s
Host Metrics
User CPU usage
CPU usage by user space
39.28%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 51.81 KiB/s < mean 1.73 MiB/s (SD = 1.44 MiB/s) < p954.86 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 16.93 KiB/s < mean 808.11 KiB/s (SD = 903.16 KiB/s) < p952.88 MiB/s
Total bytes received
Total bytes received on primary network interface
count
66.00 GiB
mean
614.42 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.50 GiB
mean
13.95 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.9%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 748.80 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 6.59% < mean 48.38% (SD = 25.14%) < p95 89.4%
System
9.10%
CPU percentiles
CPU usage percentiles
p50
53.88%
p95
89.40%
p99
96.90%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
6 hosts
mean time
0.68s
Memory used percentage
Percentage of memory used
p5 7.89% < mean 19.24% (SD = 9.47%) < p95 40.44%
Memory available percentage
Percentage of available memory
p5 59.56% < mean 80.76% (SD = 9.47%) < p95 92.11%
Max host memory used
Maximum memory usage percentage across all hosts
max 47.00%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.68%
Memory growth rate
Rate of memory growth over time
growth 748.80MB/s
Disk read throughput
Disk read throughput in MB/s
49.35MB/s
Disk write throughput
Disk write throughput in MB/s
1649.14MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/19hosts
Mount Point /efi-boot
0/16hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/19hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.85
5 min
0.75
15 min
0.40
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0.02% < mean 12.7403% (SD = 18.7923%) < p95 54.13%
60 second average
5.6276%
300 second average
1.6442%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0041% (SD = 0.0638%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0041% (SD = 0.0638%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.26% < mean 15.8262% (SD = 14.2686%) < p95 39.13%
60 second average
10.0554%
300 second average
5.9716%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.02% < mean 12.131% (SD = 11.9347%) < p95 34.06%
60 second average
8.2364%
300 second average
5.3584%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 38.69% (SD = 27.03%) < p95 82.97%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 26.37 KiB < mean 234.71 KiB (SD = 78.38 KiB) < p95323.89 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 20.83 threads (SD = 12.46 threads) < p95 38 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 65.2 file descriptors (SD = 35.59 file descriptors) < p95 98 file descriptors

Local Signals

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started
Fri, 13 Mar 2026 19:26:13 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
local_signals_23066388637
Send timing
The time taken to emit a batch of 10,000 signals.
mean 1.374727s (SD = 0.717556s) p50 1.299068s p95 2.509897s p99 2.509897s
Receive timing
The time taken to receive the completed batch, to the nearest 250ms.
mean 24.59211s (SD = 11.474553s) p50 30.001281s p95 30.002102s p99 30.002102s
Success ratio
The proportion of signals successfully received in a batch.
min 0.7925000190734863 < mean 0.9636 (SD = 0.0544) < max 1
Host Metrics
User CPU usage
CPU usage by user space
3.43%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 35 B/s < mean 414.55 KiB/s (SD = 2.18 MiB/s) ≤ p951.14 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 240 B/s < mean 7.35 KiB/s (SD = 35.90 KiB/s) ≤ p951.44 KiB/s
Total bytes received
Total bytes received on primary network interface
count
121.45 MiB
mean
414.55 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.15 MiB
mean
7.35 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 0.93% < mean 6.65% (SD = 10.03%) < p95 33.1%
System
3.22%
CPU percentiles
CPU usage percentiles
p50
1.83%
p95
33.10%
p99
43.43%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 16.45% < mean 17.24% (SD = 1.52%) < p95 20.85%
Memory available percentage
Percentage of available memory
p5 79.15% < mean 82.76% (SD = 1.52%) < p95 83.55%
Max host memory used
Maximum memory usage percentage across all hosts
max 17.24%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth -168.35MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
2.07MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /efi-boot
0/1hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.53
5 min
0.77
15 min
0.34
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0.2% < mean 1.7181% (SD = 2.2393%) < p95 6.25%
60 second average
1.2387%
300 second average
0.4929%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.83% < mean 12.5006% (SD = 18.3489%) < p95 52.29%
60 second average
13.1690%
300 second average
8.7565%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.69% < mean 11.811% (SD = 17.4992%) < p95 50.86%
60 second average
12.4871%
300 second average
8.3274%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.12% < mean 1.64% (SD = 7.05%) ≤ p95 1.35%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 56.33 KiB < mean 230.51 KiB (SD = 50.10 KiB) < p95245.38 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads ≤ mean 9.94 threads (SD = 0.35 threads) < p95 10 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 44 file descriptors < mean 50.03 file descriptors (SD = 10.74 file descriptors) < p95 70 file descriptors

Mixed Arc get_agent_activity

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Sat, 14 Mar 2026 01:15:12 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
mixed_arc_get_agent_activity_23066388637
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
349424
Agents affected
45 / 45
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
7765
Max
Highest count in any single partition
20235
Min
Lowest count in any single partition
2131
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
4875.6
Max
Highest chain head value observed for any single write agent.
8947
Write agents observed
10
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean 0.002833s (SD = 0.048624s)
Max
Highest per-partition mean latency
max mean 0.020462s
Min
Lowest per-partition mean latency
min mean -0.003323s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean 53.81/s
Max
Highest per-partition mean rate
max 112.14/s
Min
Lowest per-partition mean rate
min 17.05/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean 0.163441s (SD = 0.134286s)
Max
Highest per-partition mean latency
max mean 0.448619s
Min
Lowest per-partition mean latency
min mean 0.075441s
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5 0 < mean 49.8 (SD = 14.66) < p95 60
  • behaviour: zero_read
p5 0 < mean 38.04 (SD = 13.76) < p95 54
  • behaviour: zero_write
p5 10 < mean 39.53 (SD = 13.44) < p95 53
Final error count
The total number of errors accumulated across all agents over the run.
23
Host Metrics
User CPU usage
CPU usage by user space
29.61%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 3.16 KiB/s < mean 762.63 KiB/s (SD = 944.99 KiB/s) < p952.37 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1.56 KiB/s < mean 544.58 KiB/s (SD = 646.00 KiB/s) < p951.70 MiB/s
Total bytes received
Total bytes received on primary network interface
count
115.52 GiB
mean
117.12 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
30.22 GiB
mean
30.64 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 551.49 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 25% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.45% < mean 36.2% (SD = 29.49%) < p95 93.9%
System
6.59%
CPU percentiles
CPU usage percentiles
p50
32.15%
p95
93.90%
p99
97.60%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
15 hosts
mean time
0.52s
Memory used percentage
Percentage of memory used
p5 8.01% < mean 20.2% (SD = 12.07%) < p95 42.79%
Memory available percentage
Percentage of available memory
p5 57.21% < mean 79.8% (SD = 12.07%) < p95 91.99%
Max host memory used
Maximum memory usage percentage across all hosts
max 76.68%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.63%
Memory growth rate
Rate of memory growth over time
growth 551.49MB/s
Disk read throughput
Disk read throughput in MB/s
8.86MB/s
Disk write throughput
Disk write throughput in MB/s
638.34MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/27hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/34hosts
Mount Point /etc/hosts
0/34hosts
Mount Point /etc/resolv.conf
0/34hosts
Mount Point /nix/store
0/27hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.29
5 min
2.59
15 min
1.94
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
25.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 10.9701% (SD = 19.4847%) < p95 51.24%
60 second average
10.3982%
300 second average
7.6912%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0085% (SD = 0.1315%) ≤ p95 0%
60 second average
0.0098%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0039% (SD = 0.0989%) ≤ p95 0%
60 second average
0.0028%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 14.3851% (SD = 17.5929%) < p95 50.5%
60 second average
14.2910%
300 second average
14.1079%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 10.0877% (SD = 14.3574%) < p95 43.61%
60 second average
10.2143%
300 second average
10.8965%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.18% < mean 30.08% (SD = 28.58%) < p95 87.97%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 214.30 KiB < mean 320.93 KiB (SD = 100.85 KiB) < p95503.64 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads < mean 25.37 threads (SD = 11.67 threads) < p95 42 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 23 file descriptors < mean 67.72 file descriptors (SD = 20.17 file descriptors) < p95 87 file descriptors

Mixed Arc must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Sat, 14 Mar 2026 01:39:47 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
mixed_arc_must_get_agent_activity_23066388637
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
790.89
Max
Highest chain head value observed for any single write agent.
1138
Write agents observed
9
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 93.775258s (SD = 110.660505s)
Max
Highest per-partition mean latency
max mean 215.349319s
Min
Lowest per-partition mean latency
min mean -46.179612s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 1.55/s
Max
Highest per-partition mean rate
max 1.85/s
Min
Lowest per-partition mean rate
min 1.34/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.227406s (SD = 0.272679s)
Max
Highest per-partition mean latency
max mean 0.634616s
Min
Lowest per-partition mean latency
min mean 0.105699s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
87164
Agents affected
17 / 17
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
5127
Max
Highest count in any single partition
11835
Min
Lowest count in any single partition
483
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5 0 < mean 46.26 (SD = 15.97) < p95 59
  • behaviour: zero_must_get_agent_activity
p5 12 < mean 32.8 (SD = 11.1) < p95 44
  • behaviour: zero_write
p5 9 < mean 32.47 (SD = 12.4) < p95 52
Final error count
The total number of all types of error accumulated over the run by all agents.
94900
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.400238s (SD = 1.562999s) p50 0.106744s p95 2.066305s p99 5.348173s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.2428s (SD = 0.993448s) p50 0.085996s p95 0.665109s p99 3.944235s
Request roundtrip duration: 'must_get_agent_activity'
The time spent sending a must_get_agent_activity request and awaiting its response
mean 0.083777s (SD = 0.141911s) p50 0.064263s p95 0.191483s p99 0.41837s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.650999s (SD = 2.262153s) p50 0.140933s p95 2.763735s p99 6.810891s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 3.3e-05s (SD = 7.2e-05s) p50 1.8e-05s p95 0.000138s p99 0.000327s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.317476s (SD = 1.187205s) p50 0.000654s p95 2.2738s p99 5.548755s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.14504s (SD = 0.74931s) p50 0.004859s p95 0.48435s p99 3.952274s
Handle incoming request duration: 'must_get_agent_activity'
The time spent handling an incoming must_get_agent_activity request
mean 0.443941s (SD = 1.359062s) p50 0.024873s p95 2.923176s p99 7.876052s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.496029s (SD = 1.353608s) p50 0.032476s p95 2.619975s p99 6.379866s
Host Metrics
User CPU usage
CPU usage by user space
15.89%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 4.81 KiB/s < mean 2.46 MiB/s (SD = 113.66 MiB/s) < p953.20 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 2.35 KiB/s < mean 280.91 KiB/s (SD = 655.66 KiB/s) < p951.36 MiB/s
Total bytes received
Total bytes received on primary network interface
count
136.43 GiB
mean
145.52 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.94 GiB
mean
14.86 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 3012.27 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 2% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.73% < mean 20.72% (SD = 23.09%) < p95 76.54%
System
4.83%
CPU percentiles
CPU usage percentiles
p50
12.35%
p95
76.54%
p99
98.02%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
7 hosts
mean time
0.38s
Memory used percentage
Percentage of memory used
p5 9.96% < mean 22.9% (SD = 12.62%) < p95 50.03%
Memory available percentage
Percentage of available memory
p5 49.97% < mean 77.1% (SD = 12.62%) < p95 90.04%
Max host memory used
Maximum memory usage percentage across all hosts
max 81.06%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.63%
Memory growth rate
Rate of memory growth over time
growth 3012.27MB/s
Disk read throughput
Disk read throughput in MB/s
2.21MB/s
Disk write throughput
Disk write throughput in MB/s
582.78MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/27hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/31hosts
Mount Point /etc/hosts
0/31hosts
Mount Point /etc/resolv.conf
0/31hosts
Mount Point /nix/store
0/27hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.01
5 min
1.72
15 min
1.77
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
1.64%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 5.0514% (SD = 13.7329%) < p95 38.98%
60 second average
4.7506%
300 second average
3.9264%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0166% (SD = 0.1841%) ≤ p95 0%
60 second average
0.0209%
300 second average
0.0173%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0141% (SD = 0.1653%) ≤ p95 0%
60 second average
0.0178%
300 second average
0.0148%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 13.7304% (SD = 17.6474%) < p95 51.73%
60 second average
13.7823%
300 second average
13.3016%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 11.3527% (SD = 14.8073%) < p95 42%
60 second average
11.5238%
300 second average
11.4338%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.18% < mean 16.37% (SD = 22.11%) < p95 73.86%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 431.24 KiB < mean 526.98 KiB (SD = 77.65 KiB) < p95644.42 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads < mean 26.22 threads (SD = 9.73 threads) < p95 39 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 45 file descriptors < mean 76.13 file descriptors (SD = 18.99 file descriptors) < p95 91 file descriptors

Remote Call Rate

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started
Fri, 13 Mar 2026 19:36:48 UTC
Peer count
10
Peer count at end
10
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
remote_call_rate_23066388637
Dispatch timing
The time between sending a remote call and the remote handler being invoked.
Mean
mean 0.031022s (SD = 0.08099s)
Max
Highest per-partition mean latency
max mean 0.052853s
Min
Lowest per-partition mean latency
min mean 0.015962s
Round-trip timing
The total elapsed time to get a response to the client.
Mean
mean 0.060181s (SD = 0.089936s)
Max
Highest per-partition mean latency
max mean 0.10221s
Min
Lowest per-partition mean latency
min mean 0.038075s
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.031576s (SD = 0.032179s) p50 0.02274s p95 0.103341s p99 0.10534s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.055437s (SD = 0.070042s) p50 0.043435s p95 0.106094s p99 0.225059s
Request roundtrip duration: 'call_remote'
The time spent sending a call_remote request and awaiting its response
mean 0.051903s (SD = 0.103589s) p50 0.036398s p95 0.109244s p99 0.15437s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 2.8e-05s (SD = 6.4e-05s) p50 2.4e-05s p95 4.1e-05s p99 6.3e-05s
Handle incoming request duration: 'call_remote'
The time spent handling an incoming call_remote request
mean 0.010133s (SD = 0.097235s) p50 0.005444s p95 0.009644s p99 0.075553s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.000805s (SD = 0.000733s) p50 0.000585s p95 0.002193s p99 0.004012s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.008364s (SD = 0.060995s) p50 0.001523s p95 0.009305s p99 0.171718s
Host Metrics
User CPU usage
CPU usage by user space
13.32%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 26.76 KiB/s < mean 192.95 KiB/s (SD = 918.62 KiB/s) < p95668.56 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 24.55 KiB/s < mean 36.36 KiB/s (SD = 14.97 KiB/s) < p9547.75 KiB/s
Total bytes received
Total bytes received on primary network interface
count
847.94 MiB
mean
954.17 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
159.77 MiB
mean
179.78 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 96.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 635.41 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 10.78% < mean 20.72% (SD = 12.12%) < p95 32.89%
System
7.40%
CPU percentiles
CPU usage percentiles
p50
20.35%
p95
32.89%
p99
96.34%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
2 hosts
mean time
0.03s
Memory used percentage
Percentage of memory used
p5 17.54% < mean 18.93% (SD = 1.86%) < p95 25.32%
Memory available percentage
Percentage of available memory
p5 74.68% < mean 81.07% (SD = 1.86%) < p95 82.46%
Max host memory used
Maximum memory usage percentage across all hosts
max 21.33%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.01%
Memory growth rate
Rate of memory growth over time
growth 635.41MB/s
Disk read throughput
Disk read throughput in MB/s
0.21MB/s
Disk write throughput
Disk write throughput in MB/s
12.66MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/4hosts
Mount Point /efi-boot
0/4hosts
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
Mount Point /nix/store
0/4hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.00
5 min
0.77
15 min
0.42
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 4.2389% (SD = 9.677%) < p95 14.13%
60 second average
4.1213%
300 second average
3.1576%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 2.8562% (SD = 5.3777%) < p95 12.79%
60 second average
3.0677%
300 second average
3.0816%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 2.3726% (SD = 4.3293%) < p95 9.79%
60 second average
2.5810%
300 second average
2.6631%
Holochain process CPU usage
CPU usage by Holochain process
p5 1.55% < mean 7.74% (SD = 5.04%) < p95 10.01%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 185.30 KiB < mean 191.78 KiB (SD = 20.77 KiB) < p95214.89 KiB
Holochain process threads
Number of threads in Holochain process
p5 15 threads < mean 20.64 threads (SD = 5.46 threads) < p95 30 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 50 file descriptors < mean 59.08 file descriptors (SD = 10.68 file descriptors) < p95 82 file descriptors

Remote Signals

This scenario tests the throughput of remote_signals operations.

Started
Fri, 13 Mar 2026 19:46:59 UTC
Peer count
10
Peer count at end
10
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
remote_signals_23066388637
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean 0.223504s (SD = 0.889394s) p50 0.048116s p95 0.508671s p99 0.577855s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
map[count:20 mean_rate_per_second:8798.35 measurement_duration:map[nanos:55310126 secs:911] p5_rate_per_second:0 p95_rate_per_second:30206.92 peak_rate_per_second:277700.64 rate_trend:map[trend:[0 0.44 1.09 17356.91 3022.48] window_duration:10s] std_rate_per_second:46389.88]
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.073581s (SD = 0.13234s) p50 0.023096s p95 0.385313s p99 0.478801s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.183704s (SD = 0.202261s) p50 0.033266s p95 0.505488s p99 0.561549s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 2.5e-05s (SD = 8.4e-05s) p50 1.5e-05s p95 4.7e-05s p99 0.000127s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.000672s (SD = 0.000735s) p50 0.000522s p95 0.001507s p99 0.003809s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.004172s (SD = 0.015068s) p50 0.00162s p95 0.011189s p99 0.04399s
Handle incoming request duration: 'remote_signal'
The time spent handling an incoming remote_signal request
mean 0.031947s (SD = 0.618579s) p50 0.005148s p95 0.009529s p99 0.011773s
Host Metrics
User CPU usage
CPU usage by user space
4.63%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 6.90 KiB/s < mean 75.62 KiB/s (SD = 477.74 KiB/s) ≤ p9554.92 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 8.92 KiB/s < mean 24.56 KiB/s (SD = 29.23 KiB/s) < p9556.32 KiB/s
Total bytes received
Total bytes received on primary network interface
count
330.11 MiB
mean
375.59 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
107.23 MiB
mean
122.00 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 67.33 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.77% < mean 7.61% (SD = 11.4%) < p95 31.39%
System
2.98%
CPU percentiles
CPU usage percentiles
p50
3.69%
p95
31.39%
p99
58.63%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
0.03s
Memory used percentage
Percentage of memory used
p5 9.46% < mean 24.84% (SD = 10.16%) < p95 36.85%
Memory available percentage
Percentage of available memory
p5 63.15% < mean 75.16% (SD = 10.16%) < p95 90.54%
Max host memory used
Maximum memory usage percentage across all hosts
max 36.52%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.69%
Memory growth rate
Rate of memory growth over time
growth 67.33MB/s
Disk read throughput
Disk read throughput in MB/s
0.36MB/s
Disk write throughput
Disk write throughput in MB/s
7.07MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /efi-boot
0/1hosts
Mount Point /etc/hostname
0/4hosts
Mount Point /etc/hosts
0/4hosts
Mount Point /etc/resolv.conf
0/4hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.03
5 min
0.93
15 min
0.85
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 2.3179% (SD = 7.9495%) < p95 13.89%
60 second average
2.4330%
300 second average
2.6245%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.1236% (SD = 0.784%) < p95 0.52%
60 second average
0.1312%
300 second average
0.1247%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.0969% (SD = 0.7124%) < p95 0.32%
60 second average
0.1022%
300 second average
0.0944%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.17% < mean 1.13% (SD = 2.87%) < p95 1.81%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 178.81 KiB < mean 205.84 KiB (SD = 25.05 KiB) < p95238.80 KiB
Holochain process threads
Number of threads in Holochain process
p5 17 threads < mean 21.2 threads (SD = 6.6 threads) < p95 34 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 47 file descriptors < mean 56.42 file descriptors (SD = 12.86 file descriptors) < p95 82 file descriptors

Single-write, many-read

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started
Fri, 13 Mar 2026 19:57:28 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
single_write_many_read_23066388637
read call timing
The time it takes to complete the zome call that retrieves the record.
mean 0.003359s (SD = 0.001138s) p50 0.003054s p95 0.005276s p99 0.0082s
read call rate
The number of read zome calls completed per second.
mean 2921.75/s
Error count
The number of errors collected.
0
Host Metrics
User CPU usage
CPU usage by user space
46.87%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 54 B/s < mean 498 B/s (SD = 550 B/s) < p951.71 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 48 B/s < mean 280 B/s (SD = 175 B/s) < p95605 B/s
Total bytes received
Total bytes received on primary network interface
count
141.13 KiB
mean
498 B/s
Total bytes sent
Total bytes sent on primary network interface
count
79.30 KiB
mean
280 B/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 99.4%

Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 99% < mean 99.1% (SD = 0.72%) < p95 99.4%
System
52.22%
CPU percentiles
CPU usage percentiles
p50
99.20%
p95
99.40%
p99
99.40%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
1 hosts
mean time
1.00s
Memory used percentage
Percentage of memory used
p5 38.27% < mean 38.73% (SD = 0.83%) < p95 39.39%
Memory available percentage
Percentage of available memory
p5 60.61% < mean 61.27% (SD = 0.83%) < p95 61.73%
Max host memory used
Maximum memory usage percentage across all hosts
max 38.73%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth -19.14MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
6.24MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.55
5 min
0.75
15 min
0.30
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 32.47% < mean 52.1707% (SD = 6.5234%) < p95 56.01%
60 second average
43.8577%
300 second average
21.3283%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.0307% (SD = 0.1136%) < p95 0.22%
60 second average
0.0257%
300 second average
0.0050%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.0013% (SD = 0.0056%) < p95 0.01%
60 second average
0.0017%
300 second average
0.0000%
Holochain process CPU usage
CPU usage by Holochain process
p5 83.9% < mean 85.59% (SD = 1.29%) < p95 86.64%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 235.18 KiB ≤ mean 232.82 KiB (SD = 22.85 KiB) < p95238.00 KiB
Holochain process threads
Number of threads in Holochain process
p5 9 threads < mean 11.53 threads (SD = 1.06 threads) < p95 12 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 55 file descriptors < mean 56.93 file descriptors (SD = 3.23 file descriptors) < p95 64 file descriptors

Validation Receipts

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Fri, 13 Mar 2026 21:13:23 UTC
Peer count
20
Peer count at end
20
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
validation_receipts_23066388637
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean 6.730329s (SD = 5.933051s)
Max
Highest per-partition mean latency
max mean 7.154519s
Min
Lowest per-partition mean latency
min mean 6.299635s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean 1.7/s
Max
Highest per-partition mean rate
max 1.76/s
Min
Lowest per-partition mean rate
min 1.54/s
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.264341s (SD = 2.296058s) p50 0.175304s p95 0.376962s p99 0.500062s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.344787s (SD = 1.337134s) p50 0.119745s p95 0.926866s p99 2.940676s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 2.4e-05s (SD = 6e-05s) p50 2e-05s p95 4.3e-05s p99 8.2e-05s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.00104s (SD = 0.001825s) p50 0.000801s p95 0.002189s p99 0.005425s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.078501s (SD = 0.328083s) p50 0.007127s p95 0.433156s p99 1.650213s
Host Metrics
User CPU usage
CPU usage by user space
12.38%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 8.67 KiB/s < mean 720.66 KiB/s (SD = 1.51 MiB/s) < p953.19 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 11.27 KiB/s < mean 270.59 KiB/s (SD = 527.35 KiB/s) < p951.26 MiB/s
Total bytes received
Total bytes received on primary network interface
count
10.63 GiB
mean
11.58 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.99 GiB
mean
4.35 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1055.97 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 6% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.97% < mean 15.8% (SD = 17.09%) < p95 47.77%
System
3.43%
CPU percentiles
CPU usage percentiles
p50
7.17%
p95
47.77%
p99
93.31%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
3 hosts
mean time
0.09s
Memory used percentage
Percentage of memory used
p5 6.96% < mean 23.7% (SD = 16.62%) < p95 78.69%
Memory available percentage
Percentage of available memory
p5 21.31% < mean 76.3% (SD = 16.62%) < p95 93.04%
Max host memory used
Maximum memory usage percentage across all hosts
max 79.85%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.68%
Memory growth rate
Rate of memory growth over time
growth 1055.97MB/s
Disk read throughput
Disk read throughput in MB/s
0.03MB/s
Disk write throughput
Disk write throughput in MB/s
100.04MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/9hosts
Mount Point /efi-boot
0/7hosts
Mount Point /etc/hostname
0/8hosts
Mount Point /etc/hosts
0/8hosts
Mount Point /etc/resolv.conf
0/8hosts
Mount Point /nix/store
0/9hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.84
5 min
1.41
15 min
0.91
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
5.88%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 2.9115% (SD = 7.017%) < p95 15.62%
60 second average
2.5574%
300 second average
1.6746%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 17.9022% (SD = 21.427%) < p95 60.76%
60 second average
17.0797%
300 second average
14.1232%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 14.6244% (SD = 17.4173%) < p95 49.48%
60 second average
14.1057%
300 second average
11.9459%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 8.86% (SD = 14.68%) < p95 40.07%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 212.40 KiB < mean 250.02 KiB (SD = 53.96 KiB) < p95332.73 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 23.09 threads (SD = 9.28 threads) < p95 36 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 69.37 file descriptors (SD = 23.34 file descriptors) < p95 92 file descriptors

Write/get_agent_activity

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Fri, 13 Mar 2026 21:23:34 UTC
Peer count
65
Peer count at end
65
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
write_get_agent_activity_23066388637
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
2554.9
Max
Highest chain head value observed for any single write agent.
6905
Write agents observed
10
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean 0.145053s (SD = 0.147092s)
Max
Highest per-partition mean latency
max mean 0.45534s
Min
Lowest per-partition mean latency
min mean 0.073864s
Error count
The number of errors accumulated.
31
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.691372s (SD = 4.478511s) p50 0.100357s p95 1.282414s p99 9.896249s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.241356s (SD = 0.144607s) p50 0.223058s p95 0.475535s p99 0.475535s
Request roundtrip duration: 'get_agent_activity'
The time spent sending a get_agent_activity request and awaiting its response
mean 0.111099s (SD = 0.228368s) p50 0.079413s p95 0.301055s p99 0.531579s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 1.883281s (SD = 5.308989s) p50 0.359784s p95 6.758053s p99 25.727539s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 6.1e-05s (SD = 0.000513s) p50 2.2e-05s p95 0.00018s p99 0.000522s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.299553s (SD = 1.308067s) p50 0.000664s p95 1.093951s p99 9.079278s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.004192s (SD = 0.011025s) p50 0.000339s p95 0.042191s p99 0.042191s
Handle incoming request duration: 'get_agent_activity'
The time spent handling an incoming get_agent_activity request
mean 0.367766s (SD = 1.250737s) p50 0.069161s p95 1.266262s p99 8.870463s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 2.021058s (SD = 9.06611s) p50 0.237345s p95 6.723828s p99 26.736414s
Host Metrics
User CPU usage
CPU usage by user space
37.51%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 12.92 KiB/s < mean 1.44 MiB/s (SD = 1.49 MiB/s) < p954.11 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 7.61 KiB/s < mean 680.70 KiB/s (SD = 490.07 KiB/s) < p951.52 MiB/s
Total bytes received
Total bytes received on primary network interface
count
63.74 GiB
mean
67.99 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
29.46 GiB
mean
31.42 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.7%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2550.38 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 6% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 8.89% < mean 44.27% (SD = 25.66%) < p95 95.14%
System
6.76%
CPU percentiles
CPU usage percentiles
p50
38.31%
p95
95.14%
p99
97.69%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
17 hosts
mean time
0.45s
Memory used percentage
Percentage of memory used
p5 8.45% < mean 22.47% (SD = 12.95%) < p95 48.3%
Memory available percentage
Percentage of available memory
p5 51.7% < mean 77.53% (SD = 12.95%) < p95 91.55%
Max host memory used
Maximum memory usage percentage across all hosts
max 81.63%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.68%
Memory growth rate
Rate of memory growth over time
growth 2550.38MB/s
Disk read throughput
Disk read throughput in MB/s
0.37MB/s
Disk write throughput
Disk write throughput in MB/s
475.42MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/23hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/18hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/23hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
4.23
5 min
3.13
15 min
1.80
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
6.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 9.5312% (SD = 15.1392%) < p95 41.97%
60 second average
8.9664%
300 second average
6.5349%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 28.6146% (SD = 22.7873%) < p95 60.53%
60 second average
27.4586%
300 second average
21.3074%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 20.3187% (SD = 17.7805%) < p95 48.86%
60 second average
19.6253%
300 second average
15.6093%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.22% < mean 35.04% (SD = 24.76%) < p95 90.58%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 231.38 KiB < mean 329.19 KiB (SD = 64.67 KiB) < p95411.82 KiB
Holochain process threads
Number of threads in Holochain process
p5 12 threads < mean 32.6 threads (SD = 11.45 threads) < p95 47 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 45 file descriptors < mean 75.98 file descriptors (SD = 16.61 file descriptors) < p95 90 file descriptors

Write/Query

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started
Fri, 13 Mar 2026 20:07:47 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
write_query_23066388637
Write timing
The time it takes to complete the zome function call that writes the entry.
mean 0.101289s (SD = 0.322065s) p50 0.010325s p95 0.829397s p99 1.685341s
Write rate
The number of writes completed per second.
mean 72.07/s
Query timing
The time it takes to complete the chain_query_count_len zome call that queries the source chain.
mean 0.035513s (SD = 0.021715s) p50 0.034295s p95 0.063401s p99 0.080803s
Query rate
The number of chain queries completed per second.
mean 72.07/s
Error count
The number of errors accumulated during the run.
0
Host Metrics
User CPU usage
CPU usage by user space
12.83%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 118 B/s < mean 273.89 KiB/s (SD = 1.44 MiB/s) ≤ p953.45 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 224 B/s < mean 4.99 KiB/s (SD = 22.96 KiB/s) ≤ p953.65 KiB/s
Total bytes received
Total bytes received on primary network interface
count
80.24 MiB
mean
273.89 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.46 MiB
mean
4.99 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 9.65% < mean 15.35% (SD = 7.75%) < p95 20.72%
System
2.52%
CPU percentiles
CPU usage percentiles
p50
13.81%
p95
20.72%
p99
54.16%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 15.57% < mean 16.6% (SD = 0.85%) < p95 18.26%
Memory available percentage
Percentage of available memory
p5 81.74% < mean 83.4% (SD = 0.85%) < p95 84.43%
Max host memory used
Maximum memory usage percentage across all hosts
max 16.60%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth -13.02MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
6.89MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /efi-boot
0/1hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.77
5 min
0.87
15 min
0.35
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.1061% (SD = 0.2959%) < p95 0.57%
60 second average
0.0897%
300 second average
0.0452%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 22.05% < mean 32.7803% (SD = 8.6057%) < p95 49.48%
60 second average
29.5294%
300 second average
15.7158%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 21.54% < mean 31.3061% (SD = 8.0525%) < p95 45.93%
60 second average
28.1606%
300 second average
14.9355%
Holochain process CPU usage
CPU usage by Holochain process
p5 8.95% < mean 14.16% (SD = 7.64%) < p95 19.26%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 226.50 KiB ≤ mean 223.60 KiB (SD = 37.12 KiB) < p95235.13 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 12.37 threads (SD = 0.98 threads) < p95 13 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 60 file descriptors < mean 61.9 file descriptors (SD = 7.35 file descriptors) < p95 70 file descriptors

Write/Read

Creates an entry and reads it back, then repeats.

Started
Fri, 13 Mar 2026 20:19:07 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
write_read_23066388637
Write timing
The time taken to write the entry.
mean 0.098859s (SD = 0.375252s) p50 0.014789s p95 0.417851s p99 2.379246s
Write rate
The number of writes completed per second.
mean 83.07/s
Read timing
The time taken to read the entry.
mean 0.017312s (SD = 0.014145s) p50 0.015269s p95 0.03341s p99 0.050054s
Read rate
The number of reads completed per second.
mean 83.07/s
Error count
The number of errors accumulated over the run.
0
Host Metrics
User CPU usage
CPU usage by user space
13.99%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 63 B/s < mean 355.48 KiB/s (SD = 1.18 MiB/s) < p953.88 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 187 B/s < mean 10.23 KiB/s (SD = 32.28 KiB/s) < p95111.97 KiB/s
Total bytes received
Total bytes received on primary network interface
count
107.62 MiB
mean
355.48 KiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.10 MiB
mean
10.23 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 11.30 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 8.03% < mean 16.31% (SD = 9.17%) < p95 25.99%
System
2.32%
CPU percentiles
CPU usage percentiles
p50
14.92%
p95
25.99%
p99
58.30%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 15.5% < mean 16.06% (SD = 0.56%) < p95 16.48%
Memory available percentage
Percentage of available memory
p5 83.52% < mean 83.94% (SD = 0.56%) < p95 84.5%
Max host memory used
Maximum memory usage percentage across all hosts
max 16.06%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 11.30MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
7.46MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/1hosts
Mount Point /efi-boot
0/1hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.98
5 min
1.00
15 min
0.41
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.1762% (SD = 0.3316%) < p95 0.64%
60 second average
0.1725%
300 second average
0.0963%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 18.87% < mean 38.5922% (SD = 10.4235%) < p95 51.6%
60 second average
34.1700%
300 second average
17.8734%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 17.63% < mean 37.3206% (SD = 10.3561%) < p95 50.67%
60 second average
32.9797%
300 second average
17.2159%
Holochain process CPU usage
CPU usage by Holochain process
p5 7% < mean 14.75% (SD = 8.97%) < p95 24.38%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 209.55 KiB < mean 217.36 KiB (SD = 13.57 KiB) < p95227.21 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 14.35 threads (SD = 1.88 threads) < p95 16 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 59 file descriptors < mean 59.19 file descriptors (SD = 9.01 file descriptors) < p95 66 file descriptors

Write validated

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started
Fri, 13 Mar 2026 20:29:45 UTC
Peer count
1
Peer count at end
1
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
write_validated_23066388637
Write timing
The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.
mean 0.050996s (SD = 0.022951s) p50 0.045465s p95 0.097575s p99 0.122741s
Write rate
The number of write zome function calls that complete per second.
mean 74.46/s
Read timing
The time it takes to complete the zome function call that reads the data.
mean 0.082023s (SD = 0.022734s) p50 0.078375s p95 0.125007s p99 0.153356s
Read rate
The number of read zome function calls that complete per second.
mean 74.46/s
Error count
The number of errors accumulated over the run.
0
Host Metrics
User CPU usage
CPU usage by user space
36.17%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 691 B/s < mean 833 B/s (SD = 190 B/s) < p951.18 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 14.95 KiB/s < mean 15.15 KiB/s (SD = 166 B/s) < p9515.55 KiB/s
Total bytes received
Total bytes received on primary network interface
count
235.88 KiB
mean
833 B/s
Total bytes sent
Total bytes sent on primary network interface
count
4.29 MiB
mean
15.15 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
NotDetected
Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 36.57% < mean 50.78% (SD = 6.1%) < p95 54.94%
System
14.61%
CPU percentiles
CPU usage percentiles
p50
53.50%
p95
54.94%
p99
60.34%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
0 hosts
mean time
0.00s
Memory used percentage
Percentage of memory used
p5 20.14% < mean 20.62% (SD = 0.35%) < p95 21.13%
Memory available percentage
Percentage of available memory
p5 78.87% < mean 79.38% (SD = 0.35%) < p95 79.86%
Max host memory used
Maximum memory usage percentage across all hosts
max 20.62%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth -20.56MB/s
Disk read throughput
Disk read throughput in MB/s
0.00MB/s
Disk write throughput
Disk write throughput in MB/s
19.27MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /etc/hostname
0/1hosts
Mount Point /etc/hosts
0/1hosts
Mount Point /etc/resolv.conf
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.96
5 min
0.89
15 min
0.34
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 3.45% < mean 4.1673% (SD = 0.4845%) < p95 4.85%
60 second average
3.5507%
300 second average
1.7137%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.42% < mean 0.7717% (SD = 0.2389%) < p95 1.09%
60 second average
0.6827%
300 second average
0.3483%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.2% < mean 0.587% (SD = 0.2817%) < p95 0.99%
60 second average
0.5310%
300 second average
0.2737%
Holochain process CPU usage
CPU usage by Holochain process
p5 33.64% < mean 47.54% (SD = 6.01%) < p95 51.68%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 450.45 KiB ≤ mean 450.16 KiB (SD = 39.51 KiB) < p95458.82 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 13.6 threads (SD = 1.08 threads) < p95 14 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 62 file descriptors < mean 64.1 file descriptors (SD = 3.37 file descriptors) < p95 70 file descriptors

Write Validated must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Fri, 13 Mar 2026 22:23:39 UTC
Peer count
65
Peer count at end
65
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
write_validated_must_get_agent_activity_23066388637
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
658.23
Max
Highest chain head value observed for any single write agent.
1864
Write agents observed
13
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 395.996873s (SD = 59.739867s)
Max
Highest per-partition mean latency
max mean 1138.4294s
Min
Lowest per-partition mean latency
min mean 32.737618s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 2.15/s
Max
Highest per-partition mean rate
max 7.25/s
Min
Lowest per-partition mean rate
min 0/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.687403s (SD = 0.359864s)
Max
Highest per-partition mean latency
max mean 3.357558s
Min
Lowest per-partition mean latency
min mean 0.031182s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
10690
Agents affected
30 / 30
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
356
Max
Highest count in any single partition
809
Min
Lowest count in any single partition
1
Final error count
The total number of all types of error accumulated over the run by all agents.
109233
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.276931s (SD = 2.669052s) p50 0.084265s p95 0.37067s p99 1.121697s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.221094s (SD = 0.157028s) p50 0.174175s p95 0.532312s p99 0.756642s
Request roundtrip duration: 'must_get_agent_activity'
The time spent sending a must_get_agent_activity request and awaiting its response
mean 0.183876s (SD = 0.156964s) p50 0.148426s p95 0.400301s p99 0.810872s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.563985s (SD = 2.378968s) p50 0.222647s p95 1.726348s p99 6.203179s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1.8e-05s (SD = 9.3e-05s) p50 1.2e-05s p95 3.4e-05s p99 9.9e-05s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.001617s (SD = 0.121745s) p50 0.000601s p95 0.002349s p99 0.006741s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.01446s (SD = 0.02368s) p50 0.006025s p95 0.062978s p99 0.133661s
Handle incoming request duration: 'must_get_agent_activity'
The time spent handling an incoming must_get_agent_activity request
mean 0.059737s (SD = 0.103427s) p50 0.03371s p95 0.183172s p99 0.47437s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.512462s (SD = 1.500609s) p50 0.035584s p95 2.227107s p99 6.905106s
Host Metrics
User CPU usage
CPU usage by user space
19.31%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 1.51 KiB/s < mean 2.11 MiB/s (SD = 89.66 MiB/s) < p953.02 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 3.49 KiB/s < mean 178.98 KiB/s (SD = 804.71 KiB/s) < p95575.77 KiB/s
Total bytes received
Total bytes received on primary network interface
count
57.83 GiB
mean
32.90 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
10.98 GiB
mean
6.25 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.9%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 1202.85 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 5% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.98% < mean 24.91% (SD = 21.03%) < p95 72.19%
System
5.60%
CPU percentiles
CPU usage percentiles
p50
20.22%
p95
72.19%
p99
95.93%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
11 hosts
mean time
0.22s
Memory used percentage
Percentage of memory used
p5 10.35% < mean 23.16% (SD = 12.61%) < p95 50.64%
Memory available percentage
Percentage of available memory
p5 49.36% < mean 76.84% (SD = 12.61%) < p95 89.65%
Max host memory used
Maximum memory usage percentage across all hosts
max 81.42%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.67%
Memory growth rate
Rate of memory growth over time
growth 1202.85MB/s
Disk read throughput
Disk read throughput in MB/s
26.74MB/s
Disk write throughput
Disk write throughput in MB/s
845.13MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/28hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/21hosts
Mount Point /etc/hostname
0/32hosts
Mount Point /etc/hosts
0/32hosts
Mount Point /etc/resolv.conf
0/32hosts
Mount Point /nix/store
0/28hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.72
5 min
2.16
15 min
1.77
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
4.76%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 4.4557% (SD = 10.4147%) < p95 20.73%
60 second average
4.2045%
300 second average
3.2023%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0171% (SD = 0.2925%) ≤ p95 0%
60 second average
0.0148%
300 second average
0.0049%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0141% (SD = 0.2564%) ≤ p95 0%
60 second average
0.0116%
300 second average
0.0041%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 23.9623% (SD = 22.992%) < p95 62.36%
60 second average
22.9393%
300 second average
18.6872%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 20.4079% (SD = 19.7984%) < p95 54.63%
60 second average
19.5659%
300 second average
16.0864%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.23% < mean 16.63% (SD = 18.19%) < p95 53.62%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 27.43 KiB < mean 476.28 KiB (SD = 201.80 KiB) < p95658.08 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads < mean 29.42 threads (SD = 13.5 threads) < p95 52 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 8 file descriptors < mean 69.89 file descriptors (SD = 30.67 file descriptors) < p95 96 file descriptors

Zero-Arc Create and Read

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Sat, 14 Mar 2026 00:21:34 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_and_read_23066388637
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean 70.1/s
Max
Highest per-partition mean rate
max 191.39/s
Min
Lowest per-partition mean rate
min 15.6/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean 196.710782s (SD = 19.880273s)
Max
Highest per-partition mean latency
max mean 1423.215784s
Min
Lowest per-partition mean latency
min mean 121.018688s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean 0.2/s
Max
Highest per-partition mean rate
max 2.67/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 30.05 (SD = 22.47) < p95 59
zero-arc
p5 0 < mean 31.17 (SD = 17.92) < p95 51
Error count
The number of errors accumulated across all nodes.
327
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.347643s (SD = 3.393924s) p50 0.120192s p95 0.334577s p99 0.643476s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.248174s (SD = 2.460637s) p50 0.127682s p95 0.331689s p99 0.45896s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.207576s (SD = 0.760561s) p50 0.152111s p95 0.379262s p99 1.099546s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 6.6e-05s (SD = 0.000301s) p50 2.3e-05s p95 0.000171s p99 0.001068s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.002436s (SD = 0.104459s) p50 0.000341s p95 0.004008s p99 0.036435s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.010003s (SD = 0.726739s) p50 0.000324s p95 0.0038s p99 0.043064s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.082359s (SD = 0.542764s) p50 0.023968s p95 0.20182s p99 0.939414s

Zero-Arc Create Data

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Fri, 13 Mar 2026 22:59:58 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_23066388637
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 67.95/s
Max
Highest per-partition mean rate
max 182.94/s
Min
Lowest per-partition mean rate
min 18.31/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 399.237102s (SD = 176.679327s)
Max
Highest per-partition mean latency
max mean 486.985198s
Min
Lowest per-partition mean latency
min mean 342.600883s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 11.1/s
Max
Highest per-partition mean rate
max 19.45/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 30.47 (SD = 22.06) < p95 59
zero-arc
p5 0 < mean 31.32 (SD = 16.55) < p95 49
Error count
The number of errors accumulated across all nodes.
23
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.0587s (SD = 0.617163s) p50 0.032604s p95 0.111587s p99 0.212209s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.124888s (SD = 0.573506s) p50 0.042421s p95 0.296198s p99 1.191711s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1.1e-05s (SD = 7.8e-05s) p50 8e-06s p95 1.8e-05s p99 3.1e-05s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.000836s (SD = 0.008593s) p50 0.000317s p95 0.001649s p99 0.004963s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.054442s (SD = 0.438089s) p50 0.01013s p95 0.096589s p99 0.996956s
Host Metrics
User CPU usage
CPU usage by user space
26.59%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 1.17 KiB/s < mean 733.41 KiB/s (SD = 1.11 MiB/s) < p952.94 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 843 B/s < mean 320.18 KiB/s (SD = 716.19 KiB/s) < p95936.19 KiB/s
Total bytes received
Total bytes received on primary network interface
count
111.29 GiB
mean
113.96 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
16.58 GiB
mean
16.98 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 631.02 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 8% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 3.96% < mean 32.49% (SD = 24.56%) < p95 81.61%
System
5.90%
CPU percentiles
CPU usage percentiles
p50
28.10%
p95
81.61%
p99
98.30%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
11 hosts
mean time
0.32s
Memory used percentage
Percentage of memory used
p5 7.9% < mean 19.65% (SD = 11.84%) < p95 41.59%
Memory available percentage
Percentage of available memory
p5 58.41% < mean 80.35% (SD = 11.84%) < p95 92.1%
Max host memory used
Maximum memory usage percentage across all hosts
max 74.85%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.67%
Memory growth rate
Rate of memory growth over time
growth 631.02MB/s
Disk read throughput
Disk read throughput in MB/s
12.21MB/s
Disk write throughput
Disk write throughput in MB/s
825.99MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/29hosts
Mount Point /boot
0/2hosts
Mount Point /efi-boot
0/20hosts
Mount Point /etc/hostname
0/30hosts
Mount Point /etc/hosts
0/30hosts
Mount Point /etc/resolv.conf
0/30hosts
Mount Point /nix/store
0/28hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.04
5 min
2.39
15 min
1.80
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
8.06%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 7.9914% (SD = 16.1143%) < p95 35.47%
60 second average
7.5635%
300 second average
5.7016%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0033% (SD = 0.1076%) ≤ p95 0%
60 second average
0.0033%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0019% (SD = 0.0577%) ≤ p95 0%
60 second average
0.0020%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 21.6291% (SD = 21.4184%) < p95 62.62%
60 second average
20.8753%
300 second average
18.0168%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 18.5934% (SD = 19.8822%) < p95 58.01%
60 second average
18.0325%
300 second average
15.7890%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.2% < mean 26.18% (SD = 22.43%) < p95 67.1%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 213.02 KiB < mean 315.30 KiB (SD = 96.88 KiB) < p95532.24 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 29.47 threads (SD = 11.26 threads) < p95 49 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 45 file descriptors < mean 73.06 file descriptors (SD = 17.66 file descriptors) < p95 87 file descriptors

Zero-arc Create Data (Validated)

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sat, 14 Mar 2026 00:00:23 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_validated--high_zero_23066388637
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 53.13/s
Max
Highest per-partition mean rate
max 154.19/s
Min
Lowest per-partition mean rate
min 13.69/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 321.28363s (SD = 228.054296s)
Max
Highest per-partition mean latency
max mean 367.656116s
Min
Lowest per-partition mean latency
min mean 294.253544s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 24.62/s
Max
Highest per-partition mean rate
max 49.47/s
Min
Lowest per-partition mean rate
min 12.53/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 20 < mean 36.02 (SD = 11.41) < p95 55
zero-arc
p5 0 < mean 22.56 (SD = 13.67) < p95 40
App validation workflow duration
The time taken to apply app validation to the data.
mean 0.125337s (SD = 1.83089s) p50 0.014155s p95 0.186574s p99 1.178029s
System validation workflow duration
The time taken to apply system validation to the data.
mean 0.047442s (SD = 0.460517s) p50 0.004927s p95 0.103887s p99 0.815579s
Error count
The number of errors accumulated across all nodes.
9
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 2.947774s (SD = 12.73136s) p50 0.079226s p95 0.701966s p99 60.001154s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.219518s (SD = 2.001705s) p50 0.08357s p95 0.352825s p99 1.800468s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1e-05s (SD = 2.1e-05s) p50 8e-06s p95 1.6e-05s p99 4.4e-05s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.000452s (SD = 0.001247s) p50 0.000294s p95 0.001201s p99 0.002295s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.043415s (SD = 0.270814s) p50 0.004413s p95 0.037581s p99 1.291427s
Host Metrics
User CPU usage
CPU usage by user space
25.94%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 4.32 KiB/s < mean 2.66 MiB/s (SD = 99.61 MiB/s) < p954.91 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 3.26 KiB/s < mean 192.28 KiB/s (SD = 252.93 KiB/s) < p95763.29 KiB/s
Total bytes received
Total bytes received on primary network interface
count
150.67 GiB
mean
60.74 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.95 GiB
mean
4.01 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 2074.89 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (21.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 4.94% < mean 33.43% (SD = 24.6%) < p95 81.55%
System
7.49%
CPU percentiles
CPU usage percentiles
p50
29.78%
p95
81.55%
p99
98.50%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
13 hosts
mean time
0.29s
Memory used percentage
Percentage of memory used
p5 10.66% < mean 23.59% (SD = 12.1%) < p95 47.73%
Memory available percentage
Percentage of available memory
p5 52.27% < mean 76.41% (SD = 12.1%) < p95 89.34%
Max host memory used
Maximum memory usage percentage across all hosts
max 77.86%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 21.67%
Memory growth rate
Rate of memory growth over time
growth 2074.89MB/s
Disk read throughput
Disk read throughput in MB/s
7.23MB/s
Disk write throughput
Disk write throughput in MB/s
921.57MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/27hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/20hosts
Mount Point /etc/hostname
0/33hosts
Mount Point /etc/hosts
0/33hosts
Mount Point /etc/resolv.conf
0/33hosts
Mount Point /nix/store
0/27hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.48
5 min
2.99
15 min
2.72
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
9.68%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 8.6637% (SD = 16.4138%) < p95 36.33%
60 second average
8.2204%
300 second average
7.0048%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0021% (SD = 0.0597%) ≤ p95 0%
60 second average
0.0012%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.002% (SD = 0.0551%) ≤ p95 0%
60 second average
0.0004%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 27.7543% (SD = 25.526%) < p95 69.3%
60 second average
26.9173%
300 second average
24.8029%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 24.1222% (SD = 23.6115%) < p95 65.42%
60 second average
23.4661%
300 second average
21.9020%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.18% < mean 25.52% (SD = 21.74%) < p95 70.2%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 428.06 KiB < mean 529.69 KiB (SD = 98.40 KiB) < p95630.69 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads < mean 28.75 threads (SD = 10.51 threads) < p95 43 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 38 file descriptors < mean 75.49 file descriptors (SD = 19.3 file descriptors) < p95 89 file descriptors