DHT Sync Lag

Measure lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

  • write: A simple job that just creates entries with a timestamp field. Those entries are linked to a known base hash. For each write, the metric ws.custom.dht_sync_sent_count is incremented.
  • record_lag: A job that repeatedly queries for links from the known base hash. It keeps track of records that it has seen and when a new record is found, and calculates the time difference between the timestamp of the new record and the current time. That time difference is then recorded as a custom metric called wt.custom.dht_sync_lag.

After each behaviour loop the metric ws.custom.dht_sync_recv_count is incremented.

Started
Sun, 22 Mar 2026 20:11:34 UTC
Peer count
250
Peer count at end
1735
Behaviours
  • record_lag (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
dht_sync_lag_23410727041
Create rate
The average number of created records per agent.
Mean
mean 72.89/s
Max
Highest per-partition mean rate
max 221.86/s
Min
Lowest per-partition mean rate
min 18.04/s
Sync lag timing
The average time between when a record was created and when it is first seen by an agent.
Mean
mean 1.3842546632342554e+30s (SD = 4.529259756612697e+30s)
Max
Highest per-partition mean latency
max mean 2.9069347927919366e+31s
Min
Lowest per-partition mean latency
min mean 14.99181s
Sync lag rate
The average number of created records discovered per agent.
Mean
mean 26.74/s
Max
Highest per-partition mean rate
max 301.67/s
Min
Lowest per-partition mean rate
min 0/s
Error count
The number of errors encountered during the scenario.
10
Host Metrics
User CPU usage
CPU usage by user space
26.57%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 36.95 KiB/s < mean 638.78 KiB/s (SD = 675.57 KiB/s) < p951.88 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1.62 KiB/s < mean 129.49 KiB/s (SD = 429.13 KiB/s) < p95461.66 KiB/s
Total bytes received
Total bytes received on primary network interface
count
68.28 GiB
mean
67.23 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
13.84 GiB
mean
13.63 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.7%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 428.28 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (79.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 1% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.4% < mean 37.99% (SD = 27.94%) < p95 82.71%
System
11.43%
CPU percentiles
CPU usage percentiles
p50
44.82%
p95
82.71%
p99
94.69%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
198 hosts
mean time
0.11s
Memory used percentage
Percentage of memory used
p5 5.56% < mean 10.1% (SD = 7.73%) < p95 23.38%
Memory available percentage
Percentage of available memory
p5 76.62% < mean 89.9% (SD = 7.73%) < p95 94.44%
Max host memory used
Maximum memory usage percentage across all hosts
max 78.66%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 79.70%
Memory growth rate
Rate of memory growth over time
growth 428.28MB/s
Disk read throughput
Disk read throughput in MB/s
1.43MB/s
Disk write throughput
Disk write throughput in MB/s
789.96MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/219hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/8hosts
Mount Point /etc/hostname
0/25hosts
Mount Point /etc/hosts
0/25hosts
Mount Point /etc/resolv.conf
0/25hosts
Mount Point /nix/store
0/12hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.42
5 min
0.73
15 min
0.33
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.81%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 15.8955% (SD = 17.4004%) < p95 47.25%
60 second average
13.8308%
300 second average
6.9434%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 5.516% (SD = 8.4956%) < p95 24.83%
60 second average
4.7274%
300 second average
2.3118%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 3.9609% (SD = 6.9248%) < p95 19.17%
60 second average
3.3726%
300 second average
1.6598%
Holochain process CPU usage
CPU usage by Holochain process
p5 5.52% < mean 47.22% (SD = 20.1%) < p95 81.52%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 137.16 KiB < mean 264.81 KiB (SD = 65.04 KiB) < p95351.51 KiB
Holochain process threads
Number of threads in Holochain process
p5 7 threads < mean 22.64 threads (SD = 16.54 threads) < p95 41 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 32 file descriptors < mean 69.23 file descriptors (SD = 20.61 file descriptors) < p95 96 file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sun, 22 Mar 2026 22:30:33 UTC
Peer count
250
Peer count at end
1494
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
full_arc_create_validated_zero_arc_read_23410727041
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean 57.02/s
Max
Highest per-partition mean rate
max 138.82/s
Min
Lowest per-partition mean rate
min 16.71/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean 49.822023s (SD = 11.869743s)
Max
Highest per-partition mean latency
max mean 796.142019s
Min
Lowest per-partition mean latency
min mean 1.342813s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean 25.55/s
Max
Highest per-partition mean rate
max 490/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 2 < mean 64.8 (SD = 46.32) < p95 162
zero-arc
p5 0 < mean 11.84 (SD = 12.12) < p95 43
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
808
Agents affected
211 / 213
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
4
Max
Highest count in any single partition
5
Min
Lowest count in any single partition
0
Error count
The number of errors accumulated across all nodes.
808

Mixed Arc get_agent_activity

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

  • zero_write: A zero-arc conductor that creates entries on its source chain.
  • full_write: A full-arc conductor that creates entries on its source chain.
  • zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.
Started
Sun, 22 Mar 2026 22:43:47 UTC
Peer count
250
Peer count at end
1741
Behaviours
  • full_write (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
mixed_arc_get_agent_activity_23410727041
Entry created count
The total number of entries created by write agents over the run, partitioned by agent and behaviour.
Total
542606
Agents affected
172 / 172
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
3155
Max
Highest count in any single partition
15270
Min
Lowest count in any single partition
145
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
1230.83
Max
Highest chain head value observed for any single write agent.
5192
Write agents observed
29
Chain head delay timing
The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.
Mean
mean 0.065555s (SD = 0.293002s)
Max
Highest per-partition mean latency
max mean 0.380246s
Min
Lowest per-partition mean latency
min mean -0.119166s
Chain head delay rate
The rate at which readers detect new chain heads appearing on writers' chains.
Mean
mean 15.64/s
Max
Highest per-partition mean rate
max 31.59/s
Min
Lowest per-partition mean rate
min 3/s
get_agent_activity_full zome call timing
The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.
Mean
mean 0.266343s (SD = 0.378706s)
Max
Highest per-partition mean latency
max mean 0.869805s
Min
Lowest per-partition mean latency
min mean 0.006497s
Retrieval errors
A running accumulation of errors encountered by read agents when calling get_agent_activity.
Total
55
Agents affected
42 / 69
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
1
Max
Highest count in any single partition
2
Min
Lowest count in any single partition
0
Open connections
The number of currently open connections to other conductors, partitioned by agent behaviour.
  • behaviour: full_write
p5 6 < mean 49.73 (SD = 37.03) < p95 138
  • behaviour: zero_read
p5 0 < mean 27.68 (SD = 24.94) < p95 94
  • behaviour: zero_write
p5 2 < mean 25.17 (SD = 20.61) < p95 78
Final error count
The total number of errors accumulated across all agents over the run.
88
Host Metrics
User CPU usage
CPU usage by user space
23.66%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 100 B/s < mean 625.99 KiB/s (SD = 691.95 KiB/s) < p951.89 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 0 B/s < mean 92.80 KiB/s (SD = 245.25 KiB/s) < p95382.57 KiB/s
Total bytes received
Total bytes received on primary network interface
count
63.23 GiB
mean
89.93 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
9.37 GiB
mean
13.33 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 93.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 445.89 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (81.4% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 0% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.31% < mean 33.28% (SD = 34.37%) < p95 88.79%
System
9.62%
CPU percentiles
CPU usage percentiles
p50
9.35%
p95
88.79%
p99
93.84%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
182 hosts
mean time
0.23s
Memory used percentage
Percentage of memory used
p5 6.63% < mean 10.77% (SD = 7.14%) < p95 20.55%
Memory available percentage
Percentage of available memory
p5 79.45% < mean 89.23% (SD = 7.14%) < p95 93.37%
Max host memory used
Maximum memory usage percentage across all hosts
max 75.25%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 81.37%
Memory growth rate
Rate of memory growth over time
growth 445.89MB/s
Disk read throughput
Disk read throughput in MB/s
0.80MB/s
Disk write throughput
Disk write throughput in MB/s
457.21MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/220hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/7hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/10hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.27
5 min
0.65
15 min
0.49
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.41%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 17.6255% (SD = 25.2362%) < p95 68.97%
60 second average
14.1382%
300 second average
6.9055%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0003% (SD = 0.0215%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0001% (SD = 0.0096%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 3.0357% (SD = 6.2866%) < p95 10.81%
60 second average
2.5083%
300 second average
1.3153%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 1.7029% (SD = 5.3909%) < p95 4.92%
60 second average
1.4376%
300 second average
0.8093%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 40.77% (SD = 32.58%) < p95 85.07%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 133.72 KiB < mean 251.36 KiB (SD = 61.52 KiB) < p95340.70 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 20.69 threads (SD = 12.77 threads) < p95 41 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 63.05 file descriptors (SD = 21.6 file descriptors) < p95 87 file descriptors

Mixed Arc must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Sun, 22 Mar 2026 22:57:02 UTC
Peer count
250
Peer count at end
1740
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
mixed_arc_must_get_agent_activity_23410727041
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
202.5
Max
Highest chain head value observed for any single write agent.
434
Write agents observed
22
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 37.305912s (SD = 38.775886s)
Max
Highest per-partition mean latency
max mean 172.119444s
Min
Lowest per-partition mean latency
min mean -734.6915s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 1.72/s
Max
Highest per-partition mean rate
max 5/s
Min
Lowest per-partition mean rate
min 0/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.383614s (SD = 0.458072s)
Max
Highest per-partition mean latency
max mean 2.380887s
Min
Lowest per-partition mean latency
min mean 0.052824s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
7489
Agents affected
44 / 44
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
170
Max
Highest count in any single partition
638
Min
Lowest count in any single partition
2
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5 0 < mean 57.49 (SD = 47.93) < p95 154
  • behaviour: zero_must_get_agent_activity
p5 0 < mean 29.32 (SD = 19.16) < p95 75
  • behaviour: zero_write
p5 0 < mean 20.07 (SD = 19) < p95 58
Final error count
The total number of all types of error accumulated over the run by all agents.
12585
Host Metrics
User CPU usage
CPU usage by user space
13.14%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 14.25 KiB/s < mean 350.14 KiB/s (SD = 446.61 KiB/s) < p951.36 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 779 B/s < mean 37.20 KiB/s (SD = 94.10 KiB/s) < p95174.63 KiB/s
Total bytes received
Total bytes received on primary network interface
count
36.01 GiB
mean
50.51 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
3.83 GiB
mean
5.37 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 725.29 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (82.0% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.16% < mean 18.81% (SD = 27.66%) < p95 82.9%
System
5.67%
CPU percentiles
CPU usage percentiles
p50
5.12%
p95
82.90%
p99
93.96%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
215 hosts
mean time
0.09s
Memory used percentage
Percentage of memory used
p5 6.76% < mean 12.16% (SD = 6.87%) < p95 22.63%
Memory available percentage
Percentage of available memory
p5 77.37% < mean 87.84% (SD = 6.87%) < p95 93.24%
Max host memory used
Maximum memory usage percentage across all hosts
max 61.56%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 82.05%
Memory growth rate
Rate of memory growth over time
growth 725.29MB/s
Disk read throughput
Disk read throughput in MB/s
2.01MB/s
Disk write throughput
Disk write throughput in MB/s
379.31MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/222hosts
Mount Point /boot
0/2hosts
Mount Point /efi-boot
0/5hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/10hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.60
5 min
0.56
15 min
0.56
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 6.3861% (SD = 14.9015%) < p95 47.82%
60 second average
5.5094%
300 second average
5.9336%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0001% (SD = 0.0095%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0001% (SD = 0.0095%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 1.5591% (SD = 4.6084%) < p95 10.35%
60 second average
1.3302%
300 second average
1.1328%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 1.0687% (SD = 3.7551%) < p95 5.9%
60 second average
0.9104%
300 second average
0.6965%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 21.79% (SD = 29.21%) < p95 81.46%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 202.00 KiB < mean 408.83 KiB (SD = 101.13 KiB) < p95546.53 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 20.92 threads (SD = 12.92 threads) < p95 42 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 65.82 file descriptors (SD = 23.86 file descriptors) < p95 92 file descriptors

Remote Signals

This scenario tests the throughput of remote_signals operations.

Started
Sun, 22 Mar 2026 19:58:23 UTC
Peer count
250
Peer count at end
1993
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
remote_signals_23410727041
Round trip time
The time from origin signal dispatch to origin receive of the remote side's response signal.
mean 0.386397s (SD = 2.056417s) p50 0.142098s p95 1.130417s p99 3.641028s
Timeouts
The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).
map[count:270 mean_rate_per_second:15140.05 measurement_duration:map[nanos:280172976 secs:1057] p5_rate_per_second:0 p95_rate_per_second:35779 peak_rate_per_second:1.197318008e+07 rate_trend:map[trend:[0 0 0 25.66 0.15 0.11 1.2 6.58 8.61 13.97 25.17 80.13 159.91 331.13 2005.63 20.78 59.88 29.29 187.72 52.92 571.64 165.79 178.19 42.98 728.47 69.32 829.22 185.78 70.33 1313.28 890.47 1614.62 1479.85 998.74 1826.85 419.46 513.74 1068.03 1287.66 1487.54 2067.33 3895.62 10480.29 6396.87 9177.72 22748.43 19222.36 10219.56 98439.89 8791.34 11154.13 21386.29 54688.29 8813.86 9668.19 27179.27 25799.08 13690.46 34157.74 27827.82 48069.82 16213.77 26893.91 19624.49 14713.89 13623.69 18014.52 76335.51 20700.4 11539.51 1251.9 0.24 0.99 0.99 0.99 1 0.99 0.97 1 1 0.99 0.99 0.99 1 0.99 0.99 1 0.99 1 0.99 1 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99 0.99] window_duration:10s] std_rate_per_second:196224.39]
Host Metrics
User CPU usage
CPU usage by user space
5.55%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 27.85 KiB/s < mean 324.29 KiB/s (SD = 396.29 KiB/s) < p951.07 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 4.37 KiB/s < mean 25.33 KiB/s (SD = 32.74 KiB/s) < p9580.95 KiB/s
Total bytes received
Total bytes received on primary network interface
count
34.95 GiB
mean
11.29 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.70 GiB
mean
892.68 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 792.65 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (79.7% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.21% < mean 7.92% (SD = 14.33%) < p95 29.24%
System
2.37%
CPU percentiles
CPU usage percentiles
p50
4.39%
p95
29.24%
p99
86.47%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
131 hosts
mean time
0.03s
Memory used percentage
Percentage of memory used
p5 5.47% < mean 9.37% (SD = 7.38%) < p95 21.03%
Memory available percentage
Percentage of available memory
p5 78.97% < mean 90.63% (SD = 7.38%) < p95 94.53%
Max host memory used
Maximum memory usage percentage across all hosts
max 78.14%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 79.71%
Memory growth rate
Rate of memory growth over time
growth 792.65MB/s
Disk read throughput
Disk read throughput in MB/s
0.85MB/s
Disk write throughput
Disk write throughput in MB/s
86.38MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/218hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/8hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/11hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.21
5 min
0.14
15 min
0.07
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 1.1957% (SD = 3.8858%) < p95 4.04%
60 second average
1.0727%
300 second average
0.6762%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.3863% (SD = 1.7261%) < p95 1.84%
60 second average
0.3595%
300 second average
0.2797%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.3267% (SD = 1.3984%) < p95 1.7%
60 second average
0.3058%
300 second average
0.2430%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.55% < mean 7.19% (SD = 14.45%) < p95 20.98%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 154.03 KiB < mean 220.74 KiB (SD = 33.53 KiB) < p95247.89 KiB
Holochain process threads
Number of threads in Holochain process
p5 8 threads < mean 21.03 threads (SD = 13.09 threads) < p95 40 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 32 file descriptors < mean 66.81 file descriptors (SD = 22.49 file descriptors) < p95 95 file descriptors

Validation Receipts

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started
Sun, 22 Mar 2026 20:24:41 UTC
Peer count
250
Peer count at end
1742
Behaviours
  • default (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
validation_receipts_23410727041
Receipts complete timing
The amount of time between publishing a record and receiving the required number of validation receipts.
Mean
mean 35.419295s (SD = 13.927113s)
Max
Highest per-partition mean latency
max mean 210.426178s
Min
Lowest per-partition mean latency
min mean 10.015375s
Receipts complete rate
The number of complete validation receipt sets collected per second.
Mean
mean 0.89/s
Max
Highest per-partition mean rate
max 1/s
Min
Lowest per-partition mean rate
min 0/s
Host Metrics
User CPU usage
CPU usage by user space
5.84%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 31.71 KiB/s < mean 294.01 KiB/s (SD = 357.24 KiB/s) < p95962.31 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 1.52 KiB/s < mean 25.95 KiB/s (SD = 37.22 KiB/s) < p9591.87 KiB/s
Total bytes received
Total bytes received on primary network interface
count
31.07 GiB
mean
20.01 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
2.72 GiB
mean
1.75 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 835.88 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (79.7% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.35% < mean 8.21% (SD = 14.72%) < p95 30.06%
System
2.37%
CPU percentiles
CPU usage percentiles
p50
3.39%
p95
30.06%
p99
88.88%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
145 hosts
mean time
0.03s
Memory used percentage
Percentage of memory used
p5 5.64% < mean 9.72% (SD = 7.33%) < p95 21.46%
Memory available percentage
Percentage of available memory
p5 78.54% < mean 90.28% (SD = 7.33%) < p95 94.36%
Max host memory used
Maximum memory usage percentage across all hosts
max 77.34%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 79.68%
Memory growth rate
Rate of memory growth over time
growth 835.88MB/s
Disk read throughput
Disk read throughput in MB/s
0.19MB/s
Disk write throughput
Disk write throughput in MB/s
131.17MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/217hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/5hosts
Mount Point /etc/hostname
0/25hosts
Mount Point /etc/hosts
0/25hosts
Mount Point /etc/resolv.conf
0/25hosts
Mount Point /nix/store
0/9hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
0.25
5 min
0.36
15 min
0.39
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 1.2438% (SD = 3.8358%) < p95 4.77%
60 second average
1.1756%
300 second average
2.7651%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.8307% (SD = 3.7795%) < p95 3.75%
60 second average
0.7663%
300 second average
1.1361%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 0.7541% (SD = 3.5284%) < p95 3.42%
60 second average
0.6976%
300 second average
0.9278%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 7.68% (SD = 15.37%) < p95 28.52%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 178.21 KiB < mean 239.98 KiB (SD = 35.84 KiB) < p95272.31 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 22.19 threads (SD = 14.32 threads) < p95 49 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 66.74 file descriptors (SD = 24.01 file descriptors) < p95 96 file descriptors

Write/get_agent_activity

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started
Sun, 22 Mar 2026 20:37:20 UTC
Peer count
250
Peer count at end
5973
Behaviours
  • get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_get_agent_activity_23410727041
Observed chain advancement
The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
4818.05
Max
Highest chain head value observed for any single write agent.
14592
Write agents observed
41
get_agent_activity_full zome call timing
The time taken to call the zome function that retrieves information on a write peer's source chain.
Mean
mean 0.454657s (SD = 0.566522s)
Max
Highest per-partition mean latency
max mean 1.121748s
Min
Lowest per-partition mean latency
min mean 0.15529s
Error count
The number of errors accumulated.
537
Host Metrics
User CPU usage
CPU usage by user space
35.45%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 51.06 KiB/s < mean 745.53 KiB/s (SD = 609.13 KiB/s) < p951.89 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 4.91 KiB/s < mean 129.07 KiB/s (SD = 306.82 KiB/s) < p95362.84 KiB/s
Total bytes received
Total bytes received on primary network interface
count
176.86 GiB
mean
121.55 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
30.50 GiB
mean
20.96 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.9%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 494.65 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (24.6% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.66% < mean 49.05% (SD = 32.22%) < p95 91.66%
System
13.60%
CPU percentiles
CPU usage percentiles
p50
55.00%
p95
91.66%
p99
95.88%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
203 hosts
mean time
0.27s
Memory used percentage
Percentage of memory used
p5 5.86% < mean 11.47% (SD = 7.74%) < p95 23.46%
Memory available percentage
Percentage of available memory
p5 76.54% < mean 88.53% (SD = 7.74%) < p95 94.14%
Max host memory used
Maximum memory usage percentage across all hosts
max 78.83%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 24.61%
Memory growth rate
Rate of memory growth over time
growth 494.65MB/s
Disk read throughput
Disk read throughput in MB/s
2.13MB/s
Disk write throughput
Disk write throughput in MB/s
454.06MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/209hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/8hosts
Mount Point /etc/hostname
0/26hosts
Mount Point /etc/hosts
0/26hosts
Mount Point /etc/resolv.conf
0/26hosts
Mount Point /nix/store
0/14hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
2.26
5 min
1.66
15 min
1.00
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
29.11%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 30.2758% (SD = 29.5209%) < p95 81.77%
60 second average
28.5902%
300 second average
20.6474%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 5.9708% (SD = 8.2742%) < p95 23.12%
60 second average
5.7324%
300 second average
4.3758%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 3.8018% (SD = 7.2713%) < p95 18.23%
60 second average
3.6596%
300 second average
2.8658%
Holochain process CPU usage
CPU usage by Holochain process
p5 1.6% < mean 51.64% (SD = 28.59%) < p95 88.32%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 219.70 KiB < mean 307.74 KiB (SD = 72.26 KiB) < p95409.44 KiB
Holochain process threads
Number of threads in Holochain process
p5 12 threads < mean 28.58 threads (SD = 15.09 threads) < p95 45 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 53 file descriptors < mean 74.2 file descriptors (SD = 17.53 file descriptors) < p95 98 file descriptors

Write Validated must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started
Sun, 22 Mar 2026 21:25:15 UTC
Peer count
250
Peer count at end
7470
Behaviours
  • must_get_agent_activity (1 agent)
  • write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
write_validated_must_get_agent_activity_23410727041
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
363.15
Max
Highest chain head value observed for any single write agent.
1501
Write agents observed
34
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 407.343721s (SD = 67.453407s)
Max
Highest per-partition mean latency
max mean 866.277s
Min
Lowest per-partition mean latency
min mean -187.551083s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 2.09/s
Max
Highest per-partition mean rate
max 8/s
Min
Lowest per-partition mean rate
min 0/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.764702s (SD = 0.375334s)
Max
Highest per-partition mean latency
max mean 4.117741s
Min
Lowest per-partition mean latency
min mean 0.030027s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
35695
Agents affected
126 / 127
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
281
Max
Highest count in any single partition
847
Min
Lowest count in any single partition
0
Final error count
The total number of all types of error accumulated over the run by all agents.
162855
Host Metrics
User CPU usage
CPU usage by user space
25.61%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 35.42 KiB/s < mean 540.75 KiB/s (SD = 537.04 KiB/s) < p951.61 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 3.34 KiB/s < mean 88.00 KiB/s (SD = 189.91 KiB/s) < p95313.39 KiB/s
Total bytes received
Total bytes received on primary network interface
count
118.55 GiB
mean
67.82 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
19.25 GiB
mean
11.01 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 94.2%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 860.95 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (79.8% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 36% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.67% < mean 39.32% (SD = 32.48%) < p95 88.13%
System
13.71%
CPU percentiles
CPU usage percentiles
p50
39.78%
p95
88.13%
p99
94.19%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
197 hosts
mean time
0.15s
Memory used percentage
Percentage of memory used
p5 6.15% < mean 14.08% (SD = 8.71%) < p95 30.46%
Memory available percentage
Percentage of available memory
p5 69.54% < mean 85.92% (SD = 8.71%) < p95 93.85%
Max host memory used
Maximum memory usage percentage across all hosts
max 77.64%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 79.82%
Memory growth rate
Rate of memory growth over time
growth 860.95MB/s
Disk read throughput
Disk read throughput in MB/s
8.09MB/s
Disk write throughput
Disk write throughput in MB/s
434.49MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/196hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/6hosts
Mount Point /etc/hostname
0/27hosts
Mount Point /etc/hosts
0/27hosts
Mount Point /etc/resolv.conf
0/27hosts
Mount Point /nix/store
0/11hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.86
5 min
1.36
15 min
0.94
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
35.56%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 23.5957% (SD = 27.2359%) < p95 73.42%
60 second average
22.1804%
300 second average
15.9471%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0001% (SD = 0.0147%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0.0066%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 5.8799% (SD = 7.7761%) < p95 19.37%
60 second average
5.6039%
300 second average
4.2245%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 3.915% (SD = 6.2907%) < p95 15.63%
60 second average
3.7432%
300 second average
2.8973%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.7% < mean 39.53% (SD = 30.93%) < p95 84.4%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 229.21 KiB < mean 452.42 KiB (SD = 116.33 KiB) < p95627.14 KiB
Holochain process threads
Number of threads in Holochain process
p5 11 threads < mean 33.12 threads (SD = 20.85 threads) < p95 64 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 45 file descriptors < mean 77.87 file descriptors (SD = 21.73 file descriptors) < p95 100 file descriptors

Zero-Arc Create and Read

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Sun, 22 Mar 2026 22:14:18 UTC
Peer count
250
Peer count at end
4732
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_and_read_23410727041
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean 107.08/s
Max
Highest per-partition mean rate
max 355.89/s
Min
Lowest per-partition mean rate
min 32.89/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean 133.328458s (SD = 4.239905s)
Max
Highest per-partition mean latency
max mean 371.408062s
Min
Lowest per-partition mean latency
min mean 69.717018s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean 1.14/s
Max
Highest per-partition mean rate
max 14/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 20.54 (SD = 27.35) < p95 49
zero-arc
p5 0 < mean 16.04 (SD = 17.27) < p95 44
Error count
The number of errors accumulated across all nodes.
416

Zero-Arc Create Data

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sun, 22 Mar 2026 21:48:08 UTC
Peer count
250
Peer count at end
996
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_23410727041
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 122.97/s
Max
Highest per-partition mean rate
max 320.63/s
Min
Lowest per-partition mean rate
min 33.41/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 202.949102s (SD = 10.442576s)
Max
Highest per-partition mean latency
max mean 301.056323s
Min
Lowest per-partition mean latency
min mean 79.353307s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 8.44/s
Max
Highest per-partition mean rate
max 25.33/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 33 (SD = 25.67) < p95 72
zero-arc
p5 2 < mean 21 (SD = 17.86) < p95 47
Error count
The number of errors accumulated across all nodes.
0
Host Metrics
User CPU usage
CPU usage by user space
23.48%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 14.05 KiB/s < mean 363.11 KiB/s (SD = 473.10 KiB/s) < p951.34 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 953 B/s < mean 62.10 KiB/s (SD = 237.88 KiB/s) < p95272.30 KiB/s
Total bytes received
Total bytes received on primary network interface
count
38.27 GiB
mean
52.96 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
6.54 GiB
mean
9.06 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 91.8%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 587.43 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (80.7% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 0% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.4% < mean 34.48% (SD = 26.66%) < p95 79.33%
System
11.00%
CPU percentiles
CPU usage percentiles
p50
37.79%
p95
79.33%
p99
91.75%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
170 hosts
mean time
0.09s
Memory used percentage
Percentage of memory used
p5 6.17% < mean 10.38% (SD = 7.2%) < p95 21.02%
Memory available percentage
Percentage of available memory
p5 78.98% < mean 89.62% (SD = 7.2%) < p95 93.83%
Max host memory used
Maximum memory usage percentage across all hosts
max 74.02%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 80.72%
Memory growth rate
Rate of memory growth over time
growth 587.43MB/s
Disk read throughput
Disk read throughput in MB/s
0.53MB/s
Disk write throughput
Disk write throughput in MB/s
630.31MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/223hosts
Mount Point /boot
0/1hosts
Mount Point /efi-boot
0/4hosts
Mount Point /etc/hostname
0/23hosts
Mount Point /etc/hosts
0/23hosts
Mount Point /etc/resolv.conf
0/23hosts
Mount Point /nix/store
0/7hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.10
5 min
0.77
15 min
0.85
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.40%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 13.1037% (SD = 16.3792%) < p95 45.91%
60 second average
11.1233%
300 second average
8.1469%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0002% (SD = 0.0116%) ≤ p95 0%
60 second average
0.0001%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0002% (SD = 0.0097%) ≤ p95 0%
60 second average
0.0001%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 2.6825% (SD = 5.9851%) < p95 8.32%
60 second average
2.3180%
300 second average
1.7560%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 1.8248% (SD = 5.4228%) < p95 5.13%
60 second average
1.5851%
300 second average
1.1809%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 41.37% (SD = 20.81%) < p95 78.67%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 132.81 KiB < mean 251.01 KiB (SD = 56.15 KiB) < p95327.20 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 21.81 threads (SD = 12.04 threads) < p95 41 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 0 file descriptors < mean 63.7 file descriptors (SD = 19.6 file descriptors) < p95 84 file descriptors

Zero-arc Create Data (Validated)

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Sun, 22 Mar 2026 22:01:29 UTC
Peer count
250
Peer count at end
1490
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.3
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_validated_23410727041
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 66.71/s
Max
Highest per-partition mean rate
max 197/s
Min
Lowest per-partition mean rate
min 26.85/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 169.765232s (SD = 34.283201s)
Max
Highest per-partition mean latency
max mean 243.891963s
Min
Lowest per-partition mean latency
min mean 30.014396s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 4/s
Max
Highest per-partition mean rate
max 22/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 21.67 (SD = 27.61) < p95 63
zero-arc
p5 0 < mean 13.21 (SD = 15.08) < p95 47
Error count
The number of errors accumulated across all nodes.
0
Host Metrics
User CPU usage
CPU usage by user space
28.96%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 4.78 KiB/s < mean 303.36 KiB/s (SD = 424.47 KiB/s) < p951016.67 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 199 B/s < mean 49.19 KiB/s (SD = 157.72 KiB/s) < p95227.95 KiB/s
Total bytes received
Total bytes received on primary network interface
count
31.07 GiB
mean
44.19 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
5.04 GiB
mean
7.16 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 710.33 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (81.0% swap used)

System overload anomaly
System overload anomaly was detected
NotDetected
Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.4% < mean 44.16% (SD = 33.44%) < p95 86.97%
System
15.20%
CPU percentiles
CPU usage percentiles
p50
49.35%
p95
86.97%
p99
95.71%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
215 hosts
mean time
0.19s
Memory used percentage
Percentage of memory used
p5 6.3% < mean 11.89% (SD = 7.86%) < p95 23.51%
Memory available percentage
Percentage of available memory
p5 76.49% < mean 88.11% (SD = 7.86%) < p95 93.7%
Max host memory used
Maximum memory usage percentage across all hosts
max 75.81%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 80.97%
Memory growth rate
Rate of memory growth over time
growth 710.33MB/s
Disk read throughput
Disk read throughput in MB/s
0.57MB/s
Disk write throughput
Disk write throughput in MB/s
526.98MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/222hosts
Mount Point /efi-boot
0/7hosts
Mount Point /etc/hostname
0/24hosts
Mount Point /etc/hosts
0/24hosts
Mount Point /etc/resolv.conf
0/24hosts
Mount Point /nix/store
0/10hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.53
5 min
0.92
15 min
0.82
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
0.00%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 21.9989% (SD = 24.9082%) < p95 60.55%
60 second average
17.6254%
300 second average
10.1765%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 3.3663% (SD = 6.3448%) < p95 9.96%
60 second average
2.7609%
300 second average
1.7442%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 2.1357% (SD = 5.75%) < p95 6.92%
60 second average
1.7693%
300 second average
1.1588%
Holochain process CPU usage
CPU usage by Holochain process
p5 0% < mean 55.76% (SD = 24.78%) < p95 85.39%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 113.59 KiB < mean 366.37 KiB (SD = 124.37 KiB) < p95538.73 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 23.53 threads (SD = 12.76 threads) < p95 44 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 4 file descriptors < mean 68.02 file descriptors (SD = 21.8 file descriptors) < p95 88 file descriptors