Full-Arc Create (Validated) / Zero-Arc Read

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

  • full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
  • zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Fri, 13 Mar 2026 02:02:37 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
full_arc_create_validated_zero_arc_read--high_zero_23030051020
Create rate
The number of timed entries created by the full-arc node(s) per second.
Mean
mean 71.52/s
Max
Highest per-partition mean rate
max 102.43/s
Min
Lowest per-partition mean rate
min 29.26/s
Fetch lag timing
For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.
Mean
mean 203.115684s (SD = 82.545244s)
Max
Highest per-partition mean latency
max mean 1253.7498s
Min
Lowest per-partition mean latency
min mean 18.410198s
Fetch rate
The number of entries per second fetched by zero-arc nodes.
Mean
mean 158.26/s
Max
Highest per-partition mean rate
max 1717.5/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 23.58 (SD = 23.08) < p95 51
zero-arc
p5 0 < mean 10.86 (SD = 11.57) < p95 33
Retrieval errors
Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.
Total
526
Agents affected
55 / 60
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
9
Max
Highest count in any single partition
20
Min
Lowest count in any single partition
0
App validation workflow duration
The time taken to apply app validation to the data.
mean 3.648446s (SD = 20.179811s) p50 0.016547s p95 5.972979s p99 34.479981s
System validation workflow duration
The time taken to apply system validation to the data.
mean 1.635904s (SD = 3.976303s) p50 0.168479s p95 10.646381s p99 10.973633s
Error count
The number of errors accumulated across all nodes.
526
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.094662s (SD = 0.355561s) p50 0.070955s p95 0.263694s p99 0.34559s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 14.906133s (SD = 25.50392s) p50 0.363615s p95 60.001509s p99 60.001923s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 1.098225s (SD = 4.627606s) p50 0.198706s p95 4.333432s p99 13.518541s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 4e-05s (SD = 0.000177s) p50 1.9e-05s p95 7.7e-05s p99 0.000518s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.001403s (SD = 0.044605s) p50 0.000461s p95 0.001506s p99 0.002924s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.140021s (SD = 0.090447s) p50 0.125763s p95 0.281428s p99 0.517575s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.753359s (SD = 3.284115s) p50 0.057939s p95 3.529023s p99 9.873561s

Mixed Arc must_get_agent_activity

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started
Fri, 13 Mar 2026 02:45:54 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full_write (1 agent)
  • zero_must_get_agent_activity (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
mixed_arc_must_get_agent_activity_23030051020
Highest observed action_seq
The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.
Mean max (across write agents)
Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.
823.71
Max
Highest chain head value observed for any single write agent.
962
Write agents observed
7
Chain batch delay timing
The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.
Mean
mean 106.78385s (SD = 122.799565s)
Max
Highest per-partition mean latency
max mean 161.282895s
Min
Lowest per-partition mean latency
min mean 58.008017s
Chain batch delay rate
The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.
Mean
mean 1.67/s
Max
Highest per-partition mean rate
max 1.92/s
Min
Lowest per-partition mean rate
min 1.46/s
create_validated_sample_entry zome call timing
The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.
Mean
mean 0.473504s (SD = 1.255309s)
Max
Highest per-partition mean latency
max mean 0.851284s
Min
Lowest per-partition mean latency
min mean 0.294837s
Retrieval errors
A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.
Total
41678
Agents affected
17 / 17
Mean per agent
Total count divided by number of partitions, rounded to nearest whole number
2452
Max
Highest count in any single partition
5793
Min
Lowest count in any single partition
193
Open connections
The average number of open connections maintained by an agent over the run, broken down by behaviour.
  • behaviour: full_write
p5 0 < mean 44.77 (SD = 14.39) < p95 54
  • behaviour: zero_must_get_agent_activity
p5 0 < mean 31.86 (SD = 14.43) < p95 50
  • behaviour: zero_write
p5 0 < mean 28.91 (SD = 13.19) < p95 46
Final error count
The total number of all types of error accumulated over the run by all agents.
46603
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 2.10397s (SD = 7.80744s) p50 0.066305s p95 10.25088s p99 48.411238s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.633901s (SD = 3.930474s) p50 0.073994s p95 1.100959s p99 14.797621s
Request roundtrip duration: 'must_get_agent_activity'
The time spent sending a must_get_agent_activity request and awaiting its response
mean 0.074337s (SD = 0.304739s) p50 0.042126s p95 0.156082s p99 0.324205s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.975493s (SD = 4.4424s) p50 0.185568s p95 2.650061s p99 19.52274s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 4.1e-05s (SD = 8.6e-05s) p50 1.9e-05s p95 0.000207s p99 0.000411s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.477961s (SD = 1.7286s) p50 0.000654s p95 3.793138s p99 9.774962s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.31602s (SD = 5.108765s) p50 0.006233s p95 0.357134s p99 6.329084s
Handle incoming request duration: 'must_get_agent_activity'
The time spent handling an incoming must_get_agent_activity request
mean 0.592217s (SD = 5.980638s) p50 0.018587s p95 2.313634s p99 10.000802s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.361908s (SD = 0.982448s) p50 0.015473s p95 1.875284s p99 4.534817s
Host Metrics
User CPU usage
CPU usage by user space
12.52%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 598 B/s < mean 794.06 KiB/s (SD = 32.37 MiB/s) < p951.58 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 390 B/s < mean 289.20 KiB/s (SD = 749.36 KiB/s) < p951.68 MiB/s
Total bytes received
Total bytes received on primary network interface
count
39.04 GiB
mean
40.79 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
14.22 GiB
mean
14.85 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
NotDetected
Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 480.45 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
Detected

Critical Heavy swap usage (20.0% swap used)

System overload anomaly
System overload anomaly was detected
Detected

Warning 2% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 1.39% < mean 16.84% (SD = 18.6%) < p95 57.96%
System
4.32%
CPU percentiles
CPU usage percentiles
p50
9.42%
p95
57.96%
p99
85.57%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
8 hosts
mean time
0.09s
Memory used percentage
Percentage of memory used
p5 9.64% < mean 21.4% (SD = 13.67%) < p95 48.78%
Memory available percentage
Percentage of available memory
p5 51.22% < mean 78.6% (SD = 13.67%) < p95 90.36%
Max host memory used
Maximum memory usage percentage across all hosts
max 94.98%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 20.03%
Memory growth rate
Rate of memory growth over time
growth 480.45MB/s
Disk read throughput
Disk read throughput in MB/s
2.15MB/s
Disk write throughput
Disk write throughput in MB/s
325.53MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/22hosts
Mount Point /efi-boot
0/16hosts
Mount Point /etc/hostname
0/33hosts
Mount Point /etc/hosts
0/33hosts
Mount Point /etc/resolv.conf
0/33hosts
Mount Point /nix/store
0/22hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
1.49
5 min
1.41
15 min
1.69
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
1.72%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 0% < mean 2.5822% (SD = 5.9955%) < p95 10.28%
60 second average
2.4295%
300 second average
2.5256%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0332% (SD = 0.4177%) ≤ p95 0%
60 second average
0.0354%
300 second average
0.0290%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.028% (SD = 0.3658%) ≤ p95 0%
60 second average
0.0294%
300 second average
0.0232%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 8.8214% (SD = 13.1709%) < p95 36.02%
60 second average
8.5449%
300 second average
8.5413%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0% < mean 7.2799% (SD = 10.7455%) < p95 29.85%
60 second average
7.0867%
300 second average
7.1122%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.18% < mean 13.08% (SD = 17.3%) < p95 48.04%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 414.87 KiB < mean 532.68 KiB (SD = 113.21 KiB) < p95673.38 KiB
Holochain process threads
Number of threads in Holochain process
p5 10 threads < mean 26.26 threads (SD = 10.78 threads) < p95 46 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 37 file descriptors < mean 78.64 file descriptors (SD = 53.51 file descriptors) < p95 93 file descriptors

Zero-Arc Create and Read

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

  • zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
  • zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
  • full: A full-arc conductor that is just here to serve entries to zero arc nodes.
Started
Fri, 13 Mar 2026 01:31:51 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero_read (1 agent)
  • zero_write (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_and_read_23030051020
Create rate
The number of timed entries created by the zero-arc write node(s) per second.
Mean
mean 160.15/s
Max
Highest per-partition mean rate
max 591.6/s
Min
Lowest per-partition mean rate
min 54.83/s
Fetch lag timing
For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.
Mean
mean 138.135267s (SD = 0.864254s)
Max
Highest per-partition mean latency
max mean 205.476879s
Min
Lowest per-partition mean latency
min mean 117.789479s
Fetch lag rate
The number of entries per second fetched by zero-arc read nodes.
Mean
mean 0/s
Max
Highest per-partition mean rate
max 0/s
Min
Lowest per-partition mean rate
min 0/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 22 < mean 44.41 (SD = 10.26) < p95 56
zero-arc
p5 12 < mean 31.11 (SD = 11.42) < p95 45
Error count
The number of errors accumulated across all nodes.
1
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.674189s (SD = 3.005995s) p50 0.157353s p95 2.160454s p99 9.962232s
Request roundtrip duration: 'get_links'
The time spent sending a get_links request and awaiting its response
mean 0.177761s (SD = 0.703505s) p50 0.121595s p95 0.368501s p99 1.378732s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.223137s (SD = 0.874041s) p50 0.168046s p95 0.440878s p99 1.050106s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 7.1e-05s (SD = 0.000204s) p50 2.8e-05s p95 0.000207s p99 0.000919s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.001434s (SD = 0.002544s) p50 0.000753s p95 0.005345s p99 0.01103s
Handle incoming request duration: 'get_links'
The time spent handling an incoming get_links request
mean 0.000837s (SD = 0.005196s) p50 0.000437s p95 0.002033s p99 0.006688s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.060777s (SD = 0.170888s) p50 0.029495s p95 0.187178s p99 0.491724s

Zero-Arc Create Data

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:
  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Fri, 13 Mar 2026 01:08:34 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data--high_zero_23030051020
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 79.44/s
Max
Highest per-partition mean rate
max 119.56/s
Min
Lowest per-partition mean rate
min 19.76/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 279.694272s (SD = 81.626533s)
Max
Highest per-partition mean latency
max mean 493.834825s
Min
Lowest per-partition mean latency
min mean 65.553719s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 8.88/s
Max
Highest per-partition mean rate
max 16.75/s
Min
Lowest per-partition mean rate
min 1/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 0 < mean 1.18 (SD = 3.91) < p95 12
zero-arc
p5 2 < mean 24.01 (SD = 13.69) < p95 44
Error count
The number of errors accumulated across all nodes.
17
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.460138s (SD = 0.944985s) p50 0.215407s p95 1.737394s p99 4.605449s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.295159s (SD = 1.861014s) p50 0.150717s p95 0.425026s p99 2.222857s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1.8e-05s (SD = 9.9e-05s) p50 9e-06s p95 4.2e-05s p99 0.000217s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.00684s (SD = 0.018277s) p50 0.002109s p95 0.027867s p99 0.079554s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.023642s (SD = 0.206684s) p50 0.006597s p95 0.063693s p99 0.215752s
Host Metrics
User CPU usage
CPU usage by user space
58.85%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 257 B/s < mean 192.23 KiB/s (SD = 945.19 KiB/s) < p95854.69 KiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 543 B/s < mean 16.68 KiB/s (SD = 47.10 KiB/s) < p9595.83 KiB/s
Total bytes received
Total bytes received on primary network interface
count
9.83 GiB
mean
10.60 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
870.38 MiB
mean
938.18 KiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 99.3%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 902.18 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 63% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 30.01% < mean 78.03% (SD = 20.54%) < p95 96.44%
System
19.19%
CPU percentiles
CPU usage percentiles
p50
85.74%
p95
96.44%
p99
99.30%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
57 hosts
mean time
0.72s
Memory used percentage
Percentage of memory used
p5 8.31% < mean 11.83% (SD = 6.84%) < p95 17.98%
Memory available percentage
Percentage of available memory
p5 82.02% < mean 88.17% (SD = 6.84%) < p95 91.69%
Max host memory used
Maximum memory usage percentage across all hosts
max 48.23%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.96%
Memory growth rate
Rate of memory growth over time
growth 902.18MB/s
Disk read throughput
Disk read throughput in MB/s
0.65MB/s
Disk write throughput
Disk write throughput in MB/s
62.46MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/57hosts
Mount Point /efi-boot
0/1hosts
Mount Point /etc/hostname
0/2hosts
Mount Point /etc/hosts
0/2hosts
Mount Point /etc/resolv.conf
0/2hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.01
5 min
2.24
15 min
1.22
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
62.71%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 3.51% < mean 38.2543% (SD = 23.2652%) < p95 86%
60 second average
35.8879%
300 second average
26.0370%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% ≤ mean 0% (SD = 0%) ≤ p95 0%
60 second average
0.0000%
300 second average
0.0000%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.76% < mean 7.362% (SD = 13.7664%) < p95 33.03%
60 second average
8.9808%
300 second average
10.7926%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.17% < mean 5.1168% (SD = 13.1633%) < p95 28.9%
60 second average
6.8159%
300 second average
9.1596%
Holochain process CPU usage
CPU usage by Holochain process
p5 1.05% < mean 61.19% (SD = 26.4%) < p95 86.13%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 10.48 KiB < mean 285.59 KiB (SD = 100.07 KiB) < p95434.79 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 32.35 threads (SD = 23.27 threads) < p95 81 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 4 file descriptors < mean 63.9 file descriptors (SD = 22.92 file descriptors) < p95 81 file descriptors

Zero-arc Create Data (Validated)

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

  • zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
  • full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
Started
Fri, 13 Mar 2026 01:21:24 UTC
Peer count
65
Peer count at end
65
Behaviours
  • full (1 agent)
  • zero (1 agent)
Holochain version
0.6.1-rc.0
Wind Tunnel version
0.6.0
Run ID
zero_arc_create_data_validated--high_zero_23030051020
Create rate
The number of timed entries created by the zero-arc node(s) per second.
Mean
mean 59.69/s
Max
Highest per-partition mean rate
max 98.7/s
Min
Lowest per-partition mean rate
min 15.12/s
Sync lag timing
For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.
Mean
mean 447.175979s (SD = 230.079328s)
Max
Highest per-partition mean latency
max mean 581.092961s
Min
Lowest per-partition mean latency
min mean 361.648852s
Sync lag rate
The number of entries per second received by full nodes.
Mean
mean 14.46/s
Max
Highest per-partition mean rate
max 27.27/s
Min
Lowest per-partition mean rate
min 7.81/s
Open connections
The number of currently open connections to other conductors.
full-arc
p5 5 < mean 29.14 (SD = 14.08) < p95 57
zero-arc
p5 1 < mean 13.45 (SD = 9.05) < p95 33
App validation workflow duration
The time taken to apply app validation to the data.
mean 0.102339s (SD = 1.422453s) p50 0.023214s p95 0.222525s p99 0.411089s
System validation workflow duration
The time taken to apply system validation to the data.
mean 0.04523s (SD = 0.34368s) p50 0.019885s p95 0.148329s p99 0.231729s
Error count
The number of errors accumulated across all nodes.
8
Request roundtrip duration: 'get'
The time spent sending a get request and awaiting its response
mean 0.879978s (SD = 4.721859s) p50 0.222474s p95 1.660047s p99 13.052456s
Request roundtrip duration: 'send_validation_receipts'
The time spent sending a send_validation_receipts request and awaiting its response
mean 0.209236s (SD = 1.172269s) p50 0.113714s p95 0.39206s p99 1.5715s
Handle incoming response duration
The time spent handling an incoming response message of any type
mean 1.6e-05s (SD = 8.6e-05s) p50 8e-06s p95 3.3e-05s p99 0.00016s
Handle incoming request duration: 'get'
The time spent handling an incoming get request
mean 0.004292s (SD = 0.012553s) p50 0.001397s p95 0.017317s p99 0.05517s
Handle incoming request duration: 'send_validation_receipts'
The time spent handling an incoming send_validation_receipts request
mean 0.029916s (SD = 0.195407s) p50 0.008242s p95 0.067926s p99 0.245707s
Host Metrics
User CPU usage
CPU usage by user space
58.12%
Network receive rate (primary)
Rate of bytes received on primary network interface
p5 185 B/s < mean 228.77 KiB/s (SD = 979.38 KiB/s) < p951.02 MiB/s
Network send rate (primary)
Rate of bytes sent on primary network interface
p5 300 B/s < mean 24.35 KiB/s (SD = 58.74 KiB/s) < p95156.55 KiB/s
Total bytes received
Total bytes received on primary network interface
count
12.04 GiB
mean
13.26 MiB/s
Total bytes sent
Total bytes sent on primary network interface
count
1.28 GiB
mean
1.41 MiB/s
CPU spike anomaly
CPU spike anomaly was detected
Detected

Warning CPU p99 reached 97.6%

Memory leak anomaly
Memory leak anomaly was detected
Detected

Warning Memory growing at 903.60 MB/s

Disk full anomaly
Disk full anomaly was detected
NotDetected
Swap thrashing anomaly
Swap thrashing anomaly was detected
NotDetected
System overload anomaly
System overload anomaly was detected
Detected

Warning 89% of hosts overloaded (load5/ncpus > 1.0)

Additional Host Metrics
Hidden by default. Click to toggle visibility.
CPU usage
Total CPU usage and kernel CPU usage
Total
p5 17.09% < mean 80.32% (SD = 22.07%) < p95 95.78%
System
22.20%
CPU percentiles
CPU usage percentiles
p50
88.60%
p95
95.78%
p99
97.60%
CPU usage above 80%
Number of hosts above 80% CPU and mean time spent above threshold for those hosts
count
60 hosts
mean time
0.84s
Memory used percentage
Percentage of memory used
p5 10.58% < mean 13.2% (SD = 4.02%) < p95 23.99%
Memory available percentage
Percentage of available memory
p5 76.01% < mean 86.8% (SD = 4.02%) < p95 89.42%
Max host memory used
Maximum memory usage percentage across all hosts
max 30.88%
Max host swap used percentage
Maximum swap space usage percentage across all hosts
max 0.00%
Memory growth rate
Rate of memory growth over time
growth 903.60MB/s
Disk read throughput
Disk read throughput in MB/s
0.11MB/s
Disk write throughput
Disk write throughput in MB/s
30.27MB/s
Disk space utilization risk
Number of hosts nearing disk space capacity by mount point
Mount Point /
0/61hosts
Mount Point /efi-boot
0/1hosts
Mount Point /etc/hostname
0/3hosts
Mount Point /etc/hosts
0/3hosts
Mount Point /etc/resolv.conf
0/3hosts
Mount Point /nix/store
0/1hosts
System load average
System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.
1 min
3.64
5 min
3.01
15 min
1.83
CPU overloaded hosts
Percentage of hosts that experienced CPU overload
89.06%
CPU pressure
CPU pressure over 10, 60, and 300 second averages
10 second average
p5 2.29% < mean 48.2728% (SD = 21.1134%) < p95 85.4%
60 second average
46.6258%
300 second average
37.4791%
Memory pressure some
Memory pressure (some tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0179% (SD = 0.3278%) ≤ p95 0%
60 second average
0.0149%
300 second average
0.0084%
Memory pressure full
Memory pressure (all tasks stalled) over 10, 60, and 300 second averages
10 second average
p5 0% < mean 0.0122% (SD = 0.245%) ≤ p95 0%
60 second average
0.0097%
300 second average
0.0041%
I/O pressure some
I/O pressure (some tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 1.75% < mean 8.7507% (SD = 16.3007%) < p95 58.28%
60 second average
10.0483%
300 second average
10.5783%
I/O pressure full
I/O pressure (all tasks stalled) over 10, 60 and 300 second averages
10 second average
p5 0.28% < mean 5.8181% (SD = 14.8059%) < p95 48.02%
60 second average
7.1383%
300 second average
8.1600%
Holochain process CPU usage
CPU usage by Holochain process
p5 0.5% < mean 63.09% (SD = 30.44%) < p95 88.14%
Holochain process memory (PSS)
Proportional Set Size memory of Holochain process
p5 2.85 KiB < mean 360.72 KiB (SD = 155.36 KiB) < p95516.57 KiB
Holochain process threads
Number of threads in Holochain process
p5 1 threads < mean 29.08 threads (SD = 21.23 threads) < p95 77 threads
Holochain process file descriptors
Number of file descriptors used by Holochain process
p5 4 file descriptors < mean 64.32 file descriptors (SD = 27.18 file descriptors) < p95 85 file descriptors