App Install

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 24 Sep 2026 09:30:25 UTC

Duration

300s

Peer count

1

Behaviours

minimal (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_minimal_35955922634_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.329995158

s

Install app timing

The time taken to install an app.

mean

0.04405

s

SD

0.020996

s

p50

0.040753

s

<

p95

0.045427

s

<

p99

0.181014

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000061209s

Not enough time data to show a trend.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

111213

over 10.000061209s

mean rate

11121.23

/s

std

0

/s

p5

11121.23

/s

p95

11121.23

/s

peak

11121.23

/s

Not enough time data to show a trend.

WASM call duration: 'callback::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000701

s

SD

2.4e-05

s

p50

0.000695

s

<

p95

0.000748

s

<

p99

0.000777

s

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000726

s

SD

2e-06

s

p50

0.000727

s

≤

p95

0.000727

s

≤

p99

0.000727

s

Post-commit duration

Time spent executing post-commit workflows.

mean

6.6e-05

s

SD

1.1e-05

s

p50

6.3e-05

s

<

p95

8.6e-05

s

<

p99

0.000102

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.57

s

<

mean

12.57

s

SD

5

s

<

p95

17.57

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000078496s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001208

s

SD

0.000119

s

p50

0.001399

s

≤

p95

0.001399

s

≤

p99

0.001399

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001023

s

SD

6.4e-05

s

p50

0.001048

s

≤

p95

0.001048

s

≤

p99

0.001048

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003404

s

SD

0.000129

s

p50

0.003455

s

≤

p95

0.003455

s

≤

p99

0.003455

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001987

s

SD

5.5e-05

s

p50

0.002075

s

≤

p95

0.002075

s

≤

p99

0.002075

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001288

s

SD

6.5e-05

s

p50

0.001392

s

≤

p95

0.001392

s

≤

p99

0.001392

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

5.5e-05

s

SD

2e-06

s

p50

5.8e-05

s

≤

p95

5.8e-05

s

≤

p99

5.8e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000922

s

SD

0.000265

s

p50

0.001203

s

<

p95

0.001324

s

≤

p99

0.001324

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000266

s

SD

0

s

p50

0.000266

s

≤

p95

0.000266

s

≤

p99

0.000266

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.26

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

122.55

KiB/s

<

mean

745.45

KiB/s

SD

622.90

KiB/s

<

p95

1.34

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

44.50

KiB/s

<

mean

51.84

KiB/s

SD

7.34

KiB/s

<

p95

59.18

KiB/s

Total bytes received

Total bytes received on primary network interface

count

14.56

MiB

mean

745.45

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.01

MiB

mean

51.84

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

61.78

%

<

mean

66.52

%

SD

4.74

%

<

p95

71.25

%

System

17.25

%

CPU percentiles

CPU usage percentiles

p50

71.25

%

p95

71.25

%

p99

71.25

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

8.16

%

<

mean

9.24

%

SD

1.11

%

<

p95

10.77

%

Memory available percentage

Percentage of available memory

p5

89.23

%

<

mean

90.76

%

SD

1.11

%

<

p95

91.84

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.24

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-1.03

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.08

MiB/s

Disk write throughput

Disk write throughput in MiB/s

19.93

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.37

5 min

4.20

15 min

2.04

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

12.88

%

<

mean

17.3067

%

SD

3.1588

%

<

p95

20.04

%

60 second average

25.9367

%

300 second average

23.1333

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1067

%

SD

0.1508

%

<

p95

0.32

%

60 second average

0.0200

%

300 second average

0.0033

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

25.01

%

<

mean

33.38

%

SD

8.37

%

<

p95

41.75

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

147.47

KiB

<

mean

475.86

KiB

SD

414.81

KiB

<

p95

1.04

MiB

Holochain process threads

Number of threads in Holochain process

p5

26

threads

<

mean

77.33

threads

SD

69.79

threads

<

p95

176

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

61

file descriptors

<

mean

169.67

file descriptors

SD

152.27

file descriptors

<

p95

385

file descriptors

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 24 Sep 2026 09:28:28 UTC

Duration

300s

Peer count

1

Behaviours

large (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_large_35955922634_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.153040793

s

Install app timing

The time taken to install an app.

mean

0.065944

s

SD

0.005206

s

p50

0.065524

s

<

p95

0.075908

s

<

p99

0.078856

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'large::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000095694s

Not enough time data to show a trend.

WASM call duration: 'large::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003873

s

SD

0.001292

s

p50

0.003247

s

<

p95

0.00557

s

<

p99

0.010501

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.8e-05

s

SD

4.5e-05

s

p50

8.2e-05

s

<

p95

0.000232

s

<

p99

0.000258

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.09

s

<

mean

17.09

s

SD

8.17

s

<

p95

27.09

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 20.001397117s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002004

s

SD

0.000482

s

p50

0.002111

s

<

p95

0.0029

s

≤

p99

0.0029

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001957

s

SD

0.000964

s

p50

0.002033

s

<

p95

0.003928

s

≤

p99

0.003928

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006421

s

SD

0.001447

s

p50

0.006534

s

<

p95

0.009363

s

≤

p99

0.009363

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.00253

s

SD

0.00167

s

p50

0.002559

s

<

p95

0.005941

s

≤

p99

0.005941

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001467

s

SD

7.4e-05

s

p50

0.001396

s

<

p95

0.001508

s

≤

p99

0.001508

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

8.2e-05

s

SD

3.5e-05

s

p50

8.2e-05

s

<

p95

0.000155

s

≤

p99

0.000155

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001693

s

SD

0.000434

s

p50

0.001879

s

<

p95

0.002428

s

≤

p99

0.002428

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000468

s

SD

6.5e-05

s

p50

0.00043

s

<

p95

0.000488

s

≤

p99

0.000488

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

71.47

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.65

MiB/s

<

mean

2.05

MiB/s

SD

411.03

KiB/s

<

p95

2.45

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

67.11

KiB/s

<

mean

77.60

KiB/s

SD

10.49

KiB/s

<

p95

88.09

KiB/s

Total bytes received

Total bytes received on primary network interface

count

41.00

MiB

mean

2.05

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.52

MiB

mean

77.60

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 5.26 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

85.26

%

<

mean

89.62

%

SD

4.37

%

<

p95

93.99

%

System

18.15

%

CPU percentiles

CPU usage percentiles

p50

93.99

%

p95

93.99

%

p99

93.99

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

10.19

%

<

mean

11.53

%

SD

1.42

%

<

p95

13.5

%

Memory available percentage

Percentage of available memory

p5

86.5

%

<

mean

88.47

%

SD

1.42

%

<

p95

89.81

%

Max host memory used

Maximum memory usage percentage across all hosts

max

11.53

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

5.26

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

7.56

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.55

5 min

3.83

15 min

1.63

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

48.94

%

<

mean

52.87

%

SD

3.1688

%

<

p95

56.7

%

60 second average

45.8867

%

300 second average

21.8233

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

21.87

%

<

mean

34.45

%

SD

12.59

%

<

p95

47.04

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

25.76

KiB

<

mean

488.02

KiB

SD

529.57

KiB

<

p95

1.32

MiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

86.75

threads

SD

90.21

threads

<

p95

237

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

190.75

file descriptors

SD

194.23

file descriptors

<

p95

512

file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 24 Sep 2026 09:33:10 UTC

Duration

1800s

Peer count

30

Peer count at end

29

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_35955922634_1

Create rate

The average number of created records per agent.

Mean

mean

20.51

/s

Max

Highest per-partition mean rate

max

35.66

/s

Min

Lowest per-partition mean rate

min

9.96

/s

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

790.81129

s

SD

243.466485

s

Max

Highest per-partition mean latency

max mean

1286.60025

s

Min

Lowest per-partition mean latency

min mean

33.520771

s

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

1.33

/s

Max

Highest per-partition mean rate

max

2

/s

Min

Lowest per-partition mean rate

min

1

/s

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0002

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

93534

Agents affected

24 / 24

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3897

Max

Highest count in any single partition

6534

Min

Lowest count in any single partition

1813

Received count

The total number of entries received by record_lag agents, by agent.

Total

92

Agents affected

3 / 5

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

18

Max

Highest count in any single partition

84

Min

Lowest count in any single partition

0

Error count

The number of errors encountered during the scenario.

7

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000514

s

SD

0.000339

s

p50

0.000498

s

<

p95

0.001478

s

<

p99

0.001989

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

3

over 745.839762532s

mean rate

0.32

/s

std

0.25

/s

p5

0

/s

p95

0.52

/s

peak

0.52

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

292788960

over 1786.798773031s

mean rate

5.9313487257e+08

/s

std

1.75263133843e+09

/s

p5

1.100982224e+07

/s

p95

2.89697723597e+09

/s

peak

5.47459035676e+10

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

563976527

over 1784.243781178s

mean rate

1.55647200928e+09

/s

std

2.23035892959e+09

/s

p5

3.60150419e+07

/s

p95

6.16321330079e+09

/s

peak

6.67339137064e+09

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

39490830

over 1786.798773031s

mean rate

6.949815949e+07

/s

std

2.1643926526e+08

/s

p5

1.48498433e+06

/s

p95

2.7823494141e+08

/s

peak

7.38402558274e+09

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.01414

s

SD

0.015964

s

p50

0.009442

s

<

p95

0.044509

s

<

p99

0.079733

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.011492

s

SD

0.014187

s

p50

0.007052

s

<

p95

0.044801

s

<

p99

0.046071

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013323

s

SD

0.015798

s

p50

0.008575

s

<

p95

0.04344

s

<

p99

0.0781

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010804

s

SD

0.014232

s

p50

0.006102

s

<

p95

0.044215

s

<

p99

0.045474

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001876

s

SD

0.001383

s

p50

0.001535

s

<

p95

0.004473

s

<

p99

0.007616

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.002569

s

SD

0.004839

s

p50

0.000845

s

<

p95

0.012198

s

<

p99

0.022479

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.002467

s

SD

0.004648

s

p50

0.000824

s

<

p95

0.011667

s

<

p99

0.021287

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000485

s

SD

0.001127

s

p50

0.000654

s

<

p95

0.003685

s

<

p99

0.003733

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000989

s

SD

0.001359

s

p50

0.00057

s

<

p95

0.004367

s

<

p99

0.004601

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.003085

s

SD

0.002533

s

p50

0.002475

s

<

p95

0.007845

s

<

p99

0.013696

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00031

s

SD

0.000445

s

p50

0.00016

s

<

p95

0.000978

s

<

p99

0.00262

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

88

s

<

mean

885.57

s

SD

514.9

s

<

p95

1696.56

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

159

over 1786.798854807s

mean rate

60602.58

/s

std

287514.25

/s

p5

0

/s

p95

235632.85

/s

peak

5.33647543e+06

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

72.958198

s

SD

58.959329

s

p50

0.132821

s

<

p95

78.288158

s

<

p99

210.063127

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.082198

s

SD

0.595706

s

p50

0.264611

s

<

p95

1.71437

s

<

p99

2.566756

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.369008

s

SD

0.513457

s

p50

0.386437

s

<

p95

1.47221

s

<

p99

1.582032

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.485514

s

SD

0.328663

s

p50

0.358445

s

<

p95

0.98006

s

<

p99

1.478761

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.863604

s

SD

62.091088

s

p50

27.338577

s

<

p95

157.100339

s

<

p99

293.274297

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

14.195014

s

SD

47.988993

s

p50

0.0029

s

<

p95

87.35049

s

<

p99

277.149219

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002119

s

SD

0.001087

s

p50

0.00199

s

<

p95

0.004178

s

<

p99

0.005518

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003778

s

SD

0.008771

s

p50

0.001546

s

<

p95

0.019915

s

<

p99

0.056372

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.002149

s

SD

0.004104

s

p50

0.0006

s

<

p95

0.010565

s

<

p99

0.019889

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.005063

s

SD

27.465852

s

p50

2.845174

s

<

p95

60.013449

s

<

p99

72.854644

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.797986

s

SD

18.791187

s

p50

0.875508

s

<

p95

60.001166

s

<

p99

60.001396

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

70.36

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

149.27

KiB/s

<

mean

633.91

KiB/s

SD

732.20

KiB/s

<

p95

1.07

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

60.02

KiB/s

<

mean

505.83

KiB/s

SD

385.75

KiB/s

<

p95

1.20

MiB/s

Total bytes received

Total bytes received on primary network interface

count

29.66

GiB

mean

16.97

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

23.67

GiB

mean

13.54

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (72.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 79% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

41.75

%

<

mean

83.66

%

SD

18.89

%

<

p95

97.95

%

System

13.30

%

CPU percentiles

CPU usage percentiles

p50

92.71

%

p95

97.95

%

p99

98.25

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

23

hosts

mean time

1631.32

s

Memory used percentage

Percentage of memory used

p5

7.12

%

<

mean

23.61

%

SD

15.31

%

<

p95

44.66

%

Memory available percentage

Percentage of available memory

p5

55.34

%

<

mean

76.39

%

SD

15.31

%

<

p95

92.88

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.25

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

72.50

%

Memory growth rate

Rate of memory growth over time

growth

0.23

MiB/s

Disk read throughput

Disk read throughput in MiB/s

4.02

MiB/s

Disk write throughput

Disk write throughput in MiB/s

170.64

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/14

hosts

Mount Point /etc/hosts

0/14

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/14

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.94

5 min

5.20

15 min

3.61

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

79.31

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.73

%

<

mean

43.1798

%

SD

23.3898

%

<

p95

73.72

%

60 second average

42.0978

%

300 second average

36.8912

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0008

%

SD

0.0342

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0006

%

SD

0.0313

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

5.197

%

SD

12.4003

%

<

p95

37.46

%

60 second average

5.0457

%

300 second average

4.3379

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.2978

%

SD

4.3291

%

<

p95

10.57

%

60 second average

1.2504

%

300 second average

1.0664

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

45.19

%

SD

37.54

%

<

p95

95.61

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

243.87

KiB

<

mean

435.30

KiB

SD

161.51

KiB

<

p95

668.37

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

35.19

threads

SD

12.43

threads

<

p95

54

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

78.24

file descriptors

SD

23.44

file descriptors

<

p95

101

file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Thu, 24 Sep 2026 09:13:41 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_35955922634_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.011149

s

SD

0.002483

s

p50

0.010609

s

<

p95

0.015584

s

<

p99

0.021879

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000187839s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000187839s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

242592

over 10.000187839s

mean rate

24258.74

/s

std

0

/s

p5

24258.74

/s

p95

24258.74

/s

peak

24258.74

/s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004746

s

SD

6.6e-05

s

p50

0.004861

s

≤

p95

0.004861

s

≤

p99

0.004861

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004004

s

SD

0.000516

s

p50

0.004079

s

<

p95

0.005171

s

<

p99

0.005275

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001346

s

SD

0.000315

s

p50

0.001257

s

<

p95

0.002481

s

≤

p99

0.002481

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001167

s

SD

4.1e-05

s

p50

0.001239

s

≤

p95

0.001239

s

≤

p99

0.001239

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000333

s

SD

3.3e-05

s

p50

0.000354

s

<

p95

0.00037

s

≤

p99

0.00037

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00207

s

SD

2e-05

s

p50

0.002106

s

≤

p95

0.002106

s

≤

p99

0.002106

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9.6e-05

s

SD

5.6e-05

s

p50

7.7e-05

s

<

p95

0.000259

s

<

p99

0.000347

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.85

s

<

mean

11.85

s

SD

5

s

<

p95

16.85

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.00003243s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00106

s

SD

6.5e-05

s

p50

0.001173

s

≤

p95

0.001173

s

≤

p99

0.001173

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.0013

s

SD

2.2e-05

s

p50

0.001337

s

≤

p95

0.001337

s

≤

p99

0.001337

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004126

s

SD

8.3e-05

s

p50

0.004272

s

≤

p95

0.004272

s

≤

p99

0.004272

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002488

s

SD

0.000137

s

p50

0.002523

s

≤

p95

0.002523

s

≤

p99

0.002523

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001748

s

SD

0.000282

s

p50

0.002241

s

≤

p95

0.002241

s

≤

p99

0.002241

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3e-05

s

SD

1e-06

s

p50

3.1e-05

s

≤

p95

3.1e-05

s

≤

p99

3.1e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001046

s

SD

0.000346

s

p50

0.001589

s

<

p95

0.00162

s

≤

p99

0.00162

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000271

s

SD

3e-06

s

p50

0.000275

s

≤

p95

0.000275

s

≤

p99

0.000275

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

51.27

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

61.13

KiB/s

<

mean

2.55

MiB/s

SD

2.49

MiB/s

<

p95

5.03

MiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

29.75

KiB/s

<

mean

55.84

KiB/s

SD

26.10

KiB/s

<

p95

81.94

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

50.91

MiB

mean

2.55

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.09

MiB

mean

55.84

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 7.47 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

51.09

%

<

mean

60.49

%

SD

9.4

%

<

p95

69.89

%

Not enough time data to show a trend.

System

9.22

%

CPU percentiles

CPU usage percentiles

p50

69.89

%

p95

69.89

%

p99

69.89

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

12.34

%

<

mean

19.49

%

SD

8.48

%

<

p95

31.4

%

Memory available percentage

Percentage of available memory

p5

68.6

%

<

mean

80.51

%

SD

8.48

%

<

p95

87.66

%

Max host memory used

Maximum memory usage percentage across all hosts

max

19.49

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

7.47

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

12.14

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.53

5 min

0.17

15 min

0.05

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.85

%

<

mean

4.32

%

SD

3.3732

%

<

p95

8.89

%

60 second average

1.2367

%

300 second average

0.2800

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.5667

%

SD

0.4215

%

<

p95

1.01

%

60 second average

0.1833

%

300 second average

0.1000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4433

%

SD

0.3194

%

<

p95

0.74

%

60 second average

0.1433

%

300 second average

0.0900

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

24.22

%

SD

24.22

%

<

p95

48.44

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

252.61

KiB

<

mean

897.79

KiB

SD

645.18

KiB

<

p95

1.51

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

108.67

threads

SD

130.96

threads

<

p95

293

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

240.67

file descriptors

SD

281.09

file descriptors

<

p95

635

file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 24 Sep 2026 12:27:52 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

full_arc_create_validated_zero_arc_read--high_zero_35955922634_1

Create rate

The number of timed entries created by the full-arc node(s) per second.

Mean

mean

31.76

/s

Max

Highest per-partition mean rate

max

32.45

/s

Min

Lowest per-partition mean rate

min

31.07

/s

Fetch lag timing

For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.

Mean

mean

176.991097

s

SD

14.616261

s

Max

Highest per-partition mean latency

max mean

1067.644974

s

Min

Lowest per-partition mean latency

min mean

17.162659

s

Fetch rate

The number of entries per second fetched by zero-arc nodes.

Mean

mean

36.43

/s

Max

Highest per-partition mean rate

max

255

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc nodes.

0.0179

Entry created count

Total number of entries created by full-arc nodes, by agent.

Total

5660

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2830

Max

Highest count in any single partition

2939

Min

Lowest count in any single partition

2721

Received count

Total number of entries fetched by zero-arc nodes, by agent.

Total

2743

Agents affected

7 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

102

Max

Highest count in any single partition

720

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

26

<

mean

27.57

SD

2.19

<

p95

28

zero-arc

p5

0

<

mean

8.26

SD

7.61

<

p95

22

Retrieval errors

Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.

Total

365

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

14

Max

Highest count in any single partition

14

Min

Lowest count in any single partition

3

Error count

The number of errors accumulated across all nodes.

366

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.13389

s

SD

0.212788

s

p50

0.137781

s

<

p95

0.540922

s

<

p99

1.211565

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

11

over 812.510973542s

mean rate

235.19

/s

std

1557.36

/s

p5

0

/s

p95

1260.8

/s

peak

51396.05

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

170747151

over 872.024824638s

mean rate

1.60014215e+06

/s

std

325284.9

/s

p5

1.11214708e+06

/s

p95

2.26671132e+06

/s

peak

2.37722935e+06

/s

WASM usage: 'timed_and_validated::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

491046230

over 888.300862533s

mean rate

2.94969477444e+09

/s

std

2.61208171424e+10

/s

p5

1.706698168e+07

/s

p95

6.33263927962e+09

/s

peak

7.59872677771e+11

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

17258740

over 888.300862533s

mean rate

3.1745143404e+08

/s

std

1.59347662785e+09

/s

p5

6.41482962e+06

/s

p95

9.8025647176e+08

/s

peak

3.82633577538e+10

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.300862533s

Not enough time data to show a trend.

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009617

s

SD

0.001415

s

p50

0.009509

s

<

p95

0.01115

s

<

p99

0.01132

s

Zome call duration: 'timed_and_validated::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.033341

s

SD

0.160015

s

p50

0.010639

s

<

p95

0.441433

s

<

p99

0.756734

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008974

s

SD

0.001386

s

p50

0.00886

s

<

p95

0.010463

s

<

p99

0.010632

s

WASM call duration: 'timed_and_validated::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.405366

s

SD

25.495415

s

p50

0.421469

s

<

p95

100.97127

s

<

p99

112.274015

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001436

s

SD

0.000761

s

p50

0.001044

s

<

p95

0.003246

s

<

p99

0.00335

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010529

s

SD

0.003303

s

p50

0.00126

s

<

p95

0.010035

s

<

p99

0.018972

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000789

s

SD

0.000165

s

p50

0.000779

s

<

p95

0.000968

s

<

p99

0.000989

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000716

s

SD

0.000144

s

p50

0.000719

s

<

p95

0.000879

s

<

p99

0.000891

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.192781

s

SD

0.894252

s

p50

0.29232

s

<

p95

2.019437

s

<

p99

4.659521

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.039876

s

SD

1.325883

s

p50

0.018201

s

<

p95

1.233507

s

<

p99

8.13091

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.023821

s

SD

0.011363

s

p50

0.015558

s

<

p95

0.038216

s

<

p99

0.042114

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002393

s

SD

0.001117

s

p50

0.001604

s

<

p95

0.004761

s

<

p99

0.004925

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000204

s

SD

6.2e-05

s

p50

0.000106

s

<

p95

0.000267

s

<

p99

0.000282

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.24

s

<

mean

445.28

s

SD

255.7

s

<

p95

846.98

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 888.300824017s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

143.507067

s

SD

35.128153

s

p50

0

s

<

p95

100.998784

s

<

p99

142.196868

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

5.280585

s

SD

3.023311

s

p50

0.001158

s

<

p95

7.85837

s

<

p99

18.267456

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.050643

s

SD

0.016996

s

p50

0.001368

s

<

p95

0.020588

s

<

p99

0.095195

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.173015

s

SD

0.043173

s

p50

0.011105

s

<

p95

0.126416

s

<

p99

0.246537

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

17.165365

s

SD

14.649687

s

p50

0.002324

s

<

p95

19.24983

s

<

p99

101.603852

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

5.387538

s

SD

5.205993

s

p50

0.002109

s

<

p95

12.634631

s

<

p99

31.719182

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003008

s

SD

0.002374

s

p50

0.000114

s

<

p95

0.006112

s

<

p99

0.012863

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006244

s

SD

0.002841

s

p50

0.001716

s

<

p95

0.00795

s

<

p99

0.016859

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000404

s

SD

0.000137

s

p50

0.000302

s

<

p95

0.000671

s

<

p99

0.000702

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.792684

s

SD

29.912728

s

p50

0.197349

s

<

p95

60.001607

s

<

p99

60.001774

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

11.775997

s

SD

26.06413

s

p50

1.102028

s

<

p95

60.001456

s

<

p99

60.001719

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

2.129277

s

SD

25.205632

s

p50

2.297667

s

<

p95

60.001288

s

<

p99

60.002357

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

7.86

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

685

B/s

<

mean

47.97

KiB/s

SD

169.85

KiB/s

<

p95

197.09

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

495

B/s

<

mean

33.23

KiB/s

SD

184.02

KiB/s

<

p95

69.70

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.10

GiB

mean

1.26

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

779.11

MiB

mean

896.42

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (70.4% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 11% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.69

%

<

mean

14.85

%

SD

26.03

%

<

p95

97.84

%

System

6.99

%

CPU percentiles

CPU usage percentiles

p50

3.01

%

p95

97.84

%

p99

98.75

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

3

hosts

mean time

566.29

s

Memory used percentage

Percentage of memory used

p5

6.9

%

<

mean

20.14

%

SD

14.83

%

<

p95

37.96

%

Memory available percentage

Percentage of available memory

p5

62.04

%

<

mean

79.86

%

SD

14.83

%

<

p95

93.1

%

Max host memory used

Maximum memory usage percentage across all hosts

max

85.46

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

70.43

%

Memory growth rate

Rate of memory growth over time

growth

0.09

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.09

MiB/s

Disk write throughput

Disk write throughput in MiB/s

24.29

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.85

5 min

0.99

15 min

1.35

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

10.71

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

6.1385

%

SD

21.5477

%

<

p95

83.65

%

60 second average

6.0919

%

300 second average

6.3357

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4578

%

SD

1.7771

%

<

p95

2.49

%

60 second average

0.4684

%

300 second average

0.5289

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.3623

%

SD

1.6716

%

<

p95

1.95

%

60 second average

0.3733

%

300 second average

0.3868

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.17

%

<

mean

8.22

%

SD

24.43

%

<

p95

96.09

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

358.16

KiB

<

mean

415.91

KiB

SD

52.23

KiB

<

p95

520.62

KiB

Holochain process threads

Number of threads in Holochain process

p5

21

threads

<

mean

26.22

threads

SD

5.26

threads

<

p95

36

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

52

file descriptors

<

mean

60.8

file descriptors

SD

8.39

file descriptors

<

p95

81

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Thu, 24 Sep 2026 09:15:17 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_35955922634_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.217037

s

SD

0.590443

s

p50

1.312992

s

<

p95

2.292754

s

≤

p99

2.292754

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

27.968677

s

SD

6.097065

s

p50

30.000926

s

<

p95

30.001711

s

≤

p99

30.001711

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.486299991607666

<

mean

0.5882

SD

0.0754

<

max

0.7142000198364258

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

275720238

over 280.004599602s

mean rate

984687.24

/s

std

1.4307174e+06

/s

p5

0

/s

p95

3.06353964e+06

/s

peak

3.0635397e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.198887

s

SD

0.022803

s

p50

0.196028

s

<

p95

0.229746

s

≤

p99

0.229746

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.197957

s

SD

0.022749

s

p50

0.195005

s

<

p95

0.228821

s

≤

p99

0.228821

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

1.4e-05

s

SD

1e-06

s

p50

1.4e-05

s

<

p95

1.6e-05

s

≤

p99

1.6e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

90000

over 280.004610982s

mean rate

321.42

/s

std

463.42

/s

p5

0

/s

p95

999.99

/s

peak

999.99

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

6.8e-05

s

SD

0

s

p50

6.8e-05

s

≤

p95

6.8e-05

s

≤

p99

6.8e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.86

s

<

mean

146.86

s

SD

83.67

s

<

p95

276.86

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

3782

over 280.004614138s

mean rate

13.51

/s

std

21.15

/s

p5

0

/s

p95

58.7

/s

peak

60.2

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004604907s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001146

s

SD

0

s

p50

0.001146

s

≤

p95

0.001146

s

≤

p99

0.001146

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001292

s

SD

0

s

p50

0.001292

s

≤

p95

0.001292

s

≤

p99

0.001292

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.002799

s

SD

0.000188

s

p50

0.002798

s

<

p95

0.003196

s

≤

p99

0.003196

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002558

s

SD

0

s

p50

0.002558

s

≤

p95

0.002558

s

≤

p99

0.002558

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00295

s

SD

0

s

p50

0.00295

s

≤

p95

0.00295

s

≤

p99

0.00295

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000124

s

SD

2.8e-05

s

p50

0.000121

s

<

p95

0.000144

s

≤

p99

0.000144

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000865

s

SD

0.000201

s

p50

0.001142

s

≤

p95

0.001142

s

≤

p99

0.001142

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000256

s

SD

0

s

p50

0.000256

s

≤

p95

0.000256

s

≤

p99

0.000256

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

15.94

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

933

B/s

<

mean

1.07

MiB/s

SD

2.85

MiB/s

<

p95

8.69

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

116

B/s

<

mean

29.76

KiB/s

SD

44.34

KiB/s

<

p95

143.89

KiB/s

Total bytes received

Total bytes received on primary network interface

count

309.57

MiB

mean

1.07

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.43

MiB

mean

29.76

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 5.31 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

2.21

%

<

mean

23.5

%

SD

22.4

%

<

p95

66.48

%

System

7.56

%

CPU percentiles

CPU usage percentiles

p50

20.76

%

p95

66.48

%

p99

98.30

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

10.34

s

Memory used percentage

Percentage of memory used

p5

14.23

%

<

mean

21.71

%

SD

6.63

%

<

p95

32.82

%

Memory available percentage

Percentage of available memory

p5

67.18

%

<

mean

78.29

%

SD

6.63

%

<

p95

85.77

%

Max host memory used

Maximum memory usage percentage across all hosts

max

21.71

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

5.31

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.12

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.00

5 min

0.54

15 min

0.22

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.32

%

<

mean

9.61

%

SD

11.1064

%

<

p95

41.29

%

60 second average

7.7453

%

300 second average

3.3770

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0193

%

300 second average

0.0200

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0170

%

300 second average

0.0167

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.5797

%

SD

0.6

%

<

p95

1.83

%

60 second average

0.7620

%

300 second average

0.5713

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4013

%

SD

0.4861

%

<

p95

1.6

%

60 second average

0.6137

%

300 second average

0.5027

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.4

%

<

mean

6.16

%

SD

8.93

%

<

p95

29.14

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

100.18

KiB

<

mean

190.90

KiB

SD

34.37

KiB

<

p95

205.92

KiB

Holochain process threads

Number of threads in Holochain process

p5

6

threads

<

mean

24

threads

SD

7.54

threads

<

p95

31

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

11

file descriptors

<

mean

65.47

file descriptors

SD

18.56

file descriptors

<

p95

80

file descriptors

Mixed Arc `get_agent_activity`

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

zero_write: A zero-arc conductor that creates entries on its source chain.
full_write: A full-arc conductor that creates entries on its source chain.
zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.

Started

Thu, 24 Sep 2026 12:43:30 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_get_agent_activity_35955922634_1

Entry created count

The total number of entries created by write agents over the run, partitioned by agent and behaviour.

Total

107782

Agents affected

20 / 20

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5389

Max

Highest count in any single partition

10587

Min

Lowest count in any single partition

2252

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2806.5

Max

Highest chain head value observed for any single write agent.

3778

Write agents observed

4

Chain head delay timing

The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.

Mean

mean

0.01424

s

SD

0.194256

s

Max

Highest per-partition mean latency

max mean

0.046678

s

Min

Lowest per-partition mean latency

min mean

-0.004104

s

Chain head delay rate

The rate at which readers detect new chain heads appearing on writers' chains.

Mean

mean

19.3

/s

Max

Highest per-partition mean rate

max

50.63

/s

Min

Lowest per-partition mean rate

min

3.68

/s

get_agent_activity_full zome call timing

The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.

Mean

mean

0.20833

s

SD

0.226646

s

Max

Highest per-partition mean latency

max mean

0.39533

s

Min

Lowest per-partition mean latency

min mean

0.072311

s

Retrieval errors

A running accumulation of errors encountered by read agents when calling get_agent_activity.

Total

40

Agents affected

9 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4

Max

Highest count in any single partition

8

Min

Lowest count in any single partition

2

Open connections

The number of currently open connections to other conductors, partitioned by agent behaviour.

behaviour: full_write

p5

28

≤

mean

27.88

SD

1.15

<

p95

28

behaviour: zero_read

p5

12

<

mean

18.19

SD

4.16

<

p95

25

behaviour: zero_write

p5

0

<

mean

19.61

SD

7.14

<

p95

26

Final error count

The total number of errors accumulated across all agents over the run.

40

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.153973

s

SD

0.181453

s

p50

0.125535

s

<

p95

0.662945

s

≤

p99

0.662945

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 750.063670055s

Not enough time data to show a trend.

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006485

s

SD

0.002925

s

p50

0.005444

s

<

p95

0.01244

s

≤

p99

0.01244

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004791

s

SD

0.003424

s

p50

0.004317

s

<

p95

0.014628

s

<

p99

0.016096

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.171728

s

SD

0.136438

s

p50

0.188192

s

<

p95

0.5144

s

<

p99

0.563196

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.014315

s

SD

0.213254

s

p50

0.029689

s

<

p95

0.667999

s

≤

p99

0.667999

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005597

s

SD

0.002694

s

p50

0.004547

s

<

p95

0.011362

s

≤

p99

0.011362

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004143

s

SD

0.003336

s

p50

0.003656

s

<

p95

0.013817

s

<

p99

0.015266

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.170774

s

SD

0.136254

s

p50

0.187389

s

<

p95

0.512888

s

<

p99

0.561684

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013679

s

SD

0.213104

s

p50

0.028606

s

<

p95

0.66693

s

≤

p99

0.66693

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001327

s

SD

0.00069

s

p50

0.00126

s

<

p95

0.003275

s

<

p99

0.003726

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

4.1e-05

s

SD

2.1e-05

s

p50

3.3e-05

s

<

p95

8.4e-05

s

≤

p99

8.4e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000725

s

SD

0.001539

s

p50

0.000476

s

<

p95

0.005218

s

<

p99

0.006373

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000544

s

SD

0.000333

s

p50

0.000408

s

<

p95

0.001658

s

≤

p99

0.001658

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.168901

s

SD

0.135873

s

p50

0.18604

s

<

p95

0.509968

s

<

p99

0.55878

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.010588

s

SD

0.212659

s

p50

0.025707

s

<

p95

0.662993

s

≤

p99

0.662993

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

7e-06

s

SD

4e-06

s

p50

9e-06

s

<

p95

1.7e-05

s

≤

p99

1.7e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002095

s

SD

0.001283

s

p50

0.002036

s

<

p95

0.005596

s

<

p99

0.006473

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000188

s

SD

0.000196

s

p50

0.000114

s

<

p95

0.000724

s

<

p99

0.000905

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.23

s

<

mean

446.84

s

SD

256.62

s

<

p95

847.25

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 889.154474516s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

9.270406

s

SD

9.764724

s

p50

0

s

<

p95

0.034028

s

<

p99

61.575786

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00768

s

SD

0.061032

s

p50

0.001444

s

<

p95

0.240358

s

<

p99

0.262295

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.08644

s

SD

0.457497

s

p50

0.001678

s

<

p95

1.188156

s

<

p99

1.288213

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.095676

s

SD

0.065506

s

p50

0.055786

s

<

p95

0.198474

s

<

p99

0.234875

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.985417

s

SD

9.870869

s

p50

0.004057

s

<

p95

33.895006

s

<

p99

47.622408

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

4.405862

s

SD

16.220325

s

p50

0.001897

s

<

p95

2.263647

s

<

p99

116.808544

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00236

s

SD

0.001302

s

p50

0.001467

s

<

p95

0.004009

s

<

p99

0.004912

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002405

s

SD

0.002844

s

p50

0.001164

s

<

p95

0.002773

s

<

p99

0.021223

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000687

s

SD

0.001305

s

p50

0.000342

s

<

p95

0.004548

s

<

p99

0.006041

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

8.491527

s

SD

29.868752

s

p50

0.21768

s

<

p95

60.004718

s

<

p99

60.005368

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.072688

s

SD

0.162682

s

p50

0.175719

s

<

p95

0.548662

s

≤

p99

0.548662

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.140404

s

SD

29.386183

s

p50

0.244276

s

<

p95

60.001345

s

<

p99

60.001511

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.251529

s

SD

0.811658

s

p50

0.135485

s

<

p95

2.260214

s

≤

p99

2.260214

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.02

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.82

KiB/s

<

mean

312.62

KiB/s

SD

275.10

KiB/s

<

p95

818.20

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

2.01

KiB/s

<

mean

291.47

KiB/s

SD

226.33

KiB/s

<

p95

644.33

KiB/s

Total bytes received

Total bytes received on primary network interface

count

7.16

GiB

mean

8.24

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.68

GiB

mean

7.69

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.4%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (70.4% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 57% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.85

%

<

mean

59.46

%

SD

39.04

%

<

p95

98.1

%

System

11.44

%

CPU percentiles

CPU usage percentiles

p50

78.30

%

p95

98.10

%

p99

98.40

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

15

hosts

mean time

820.45

s

Memory used percentage

Percentage of memory used

p5

6.05

%

<

mean

19.08

%

SD

14.4

%

<

p95

34.77

%

Memory available percentage

Percentage of available memory

p5

65.23

%

<

mean

80.92

%

SD

14.4

%

<

p95

93.95

%

Max host memory used

Maximum memory usage percentage across all hosts

max

82.74

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

70.39

%

Memory growth rate

Rate of memory growth over time

growth

0.27

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.05

MiB/s

Disk write throughput

Disk write throughput in MiB/s

77.65

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.50

5 min

2.78

15 min

2.03

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

57.14

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

29.9618

%

SD

27.0803

%

<

p95

78.87

%

60 second average

28.4559

%

300 second average

22.4777

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.6415

%

SD

5.4089

%

<

p95

6.47

%

60 second average

1.5593

%

300 second average

1.2441

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.492

%

SD

1.8925

%

<

p95

3.15

%

60 second average

0.4870

%

300 second average

0.4415

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.3

%

<

mean

53.79

%

SD

39.66

%

<

p95

96.28

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

231.38

KiB

<

mean

321.38

KiB

SD

74.36

KiB

<

p95

450.72

KiB

Holochain process threads

Number of threads in Holochain process

p5

24

threads

<

mean

32.89

threads

SD

5.13

threads

<

p95

39

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

57

file descriptors

<

mean

74.65

file descriptors

SD

11.59

file descriptors

<

p95

89

file descriptors

Mixed Arc `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started

Thu, 24 Sep 2026 12:59:12 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full_write (1 agent)
zero_must_get_agent_activity (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_must_get_agent_activity_35955922634_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1668.75

Max

Highest chain head value observed for any single write agent.

1732

Write agents observed

4

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

78.171622

s

SD

63.718707

s

Max

Highest per-partition mean latency

max mean

217.475718

s

Min

Lowest per-partition mean latency

min mean

6.729178

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

1.9

/s

Max

Highest per-partition mean rate

max

2.54

/s

Min

Lowest per-partition mean rate

min

1.28

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

1.616925

s

SD

1.860128

s

Max

Highest per-partition mean latency

max mean

7.582193

s

Min

Lowest per-partition mean latency

min mean

0.341014

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

12341

Agents affected

9 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1371

Max

Highest count in any single partition

5799

Min

Lowest count in any single partition

167

Open connections

The average number of open connections maintained by an agent over the run, broken down by behaviour.

behaviour: full_write

p5

28

≤

mean

27.84

SD

1.07

<

p95

28

behaviour: zero_must_get_agent_activity

p5

10

<

mean

14.92

SD

3.13

<

p95

19

behaviour: zero_write

p5

0

<

mean

15.08

SD

5.89

<

p95

24

Final error count

The total number of all types of error accumulated over the run by all agents.

20516

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.07214

s

SD

0.488998

s

p50

0.194259

s

<

p95

1.237813

s

<

p99

2.615198

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 811.049972866s

Not enough time data to show a trend.

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007112

s

SD

0.002988

s

p50

0.005977

s

<

p95

0.012919

s

≤

p99

0.012919

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.065954

s

SD

0.031439

s

p50

0.048289

s

<

p95

0.123111

s

<

p99

0.146501

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009293

s

SD

0.001933

s

p50

0.006592

s

<

p95

0.011787

s

<

p99

0.011942

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.285737

s

SD

0.346208

s

p50

0.214966

s

<

p95

1.075357

s

<

p99

1.570136

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.020742

s

SD

0.25532

s

p50

0.24656

s

<

p95

0.837957

s

≤

p99

0.837957

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00591

s

SD

0.002575

s

p50

0.005085

s

<

p95

0.01099

s

≤

p99

0.01099

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.065103

s

SD

0.031318

s

p50

0.047466

s

<

p95

0.122004

s

<

p99

0.145743

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007806

s

SD

0.001628

s

p50

0.00557

s

<

p95

0.009859

s

<

p99

0.009995

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.323116

s

SD

0.440841

s

p50

0.33163

s

<

p95

1.675571

s

<

p99

2.134567

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.020134

s

SD

0.255267

s

p50

0.245357

s

<

p95

0.837195

s

≤

p99

0.837195

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001252

s

SD

0.00067

s

p50

0.001297

s

<

p95

0.003022

s

<

p99

0.003404

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.01544

s

SD

0.139508

s

p50

0.004858

s

<

p95

0.180343

s

<

p99

0.871507

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.5e-05

s

SD

2.1e-05

s

p50

2.3e-05

s

<

p95

8e-05

s

<

p99

9e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000476

s

SD

0.000184

s

p50

0.000427

s

<

p95

0.000725

s

<

p99

0.000881

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000441

s

SD

0.000244

s

p50

0.000428

s

<

p95

0.000963

s

<

p99

0.001268

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.261487

s

SD

0.36969

s

p50

0.277757

s

<

p95

1.150029

s

<

p99

1.830672

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.052622

s

SD

0.574135

s

p50

0.055482

s

<

p95

2.021219

s

<

p99

2.987247

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.027597

s

SD

0.020937

s

p50

0.012161

s

<

p95

0.065351

s

<

p99

0.086419

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

6e-06

s

SD

6e-06

s

p50

1.1e-05

s

<

p95

2.3e-05

s

≤

p99

2.3e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001959

s

SD

0.001098

s

p50

0.002356

s

<

p95

0.004895

s

<

p99

0.006229

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000144

s

SD

7e-05

s

p50

0.000125

s

<

p95

0.000278

s

<

p99

0.000349

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.24

s

<

mean

446.3

s

SD

256.32

s

<

p95

847.02

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 889.077790146s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

138.621571

s

SD

72.670805

s

p50

0

s

<

p95

197.577063

s

<

p99

309.767947

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

2.587163

s

SD

1.403738

s

p50

0.002091

s

<

p95

3.819623

s

<

p99

5.899803

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.155138

s

SD

0.154177

s

p50

0.001609

s

<

p95

0.404024

s

<

p99

0.570891

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.116078

s

SD

0.139004

s

p50

0.017097

s

<

p95

0.407123

s

<

p99

0.660087

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

10.931533

s

SD

11.440801

s

p50

0.003645

s

<

p95

31.079182

s

<

p99

49.835262

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

20.693958

s

SD

29.061695

s

p50

0.002255

s

<

p95

62.016182

s

<

p99

209.4085

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002881

s

SD

0.001907

s

p50

0.0003

s

<

p95

0.005826

s

<

p99

0.006871

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002154

s

SD

0.003415

s

p50

0.001569

s

<

p95

0.008498

s

<

p99

0.015707

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00039

s

SD

0.000148

s

p50

0.000375

s

<

p95

0.000632

s

<

p99

0.000775

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.232264

s

SD

29.890571

s

p50

0.27651

s

<

p95

60.001609

s

<

p99

60.001697

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.199709

s

SD

25.411048

s

p50

0.260048

s

<

p95

60.001581

s

<

p99

60.001868

s

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.175598

s

SD

22.599581

s

p50

0.249954

s

<

p95

60.001624

s

<

p99

60.002009

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.168628

s

SD

6.494967

s

p50

0.198101

s

<

p95

15.213099

s

<

p99

15.234853

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

21.40

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.26

KiB/s

<

mean

126.38

KiB/s

SD

444.83

KiB/s

<

p95

206.90

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

1.61

KiB/s

<

mean

73.93

KiB/s

SD

97.33

KiB/s

<

p95

236.57

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.90

GiB

mean

3.33

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.69

GiB

mean

1.95

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (70.3% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 32% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.18

%

<

mean

31.78

%

SD

30.6

%

<

p95

92.18

%

System

10.38

%

CPU percentiles

CPU usage percentiles

p50

19.10

%

p95

92.18

%

p99

97.65

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

9

hosts

mean time

353.93

s

Memory used percentage

Percentage of memory used

p5

7.02

%

<

mean

21.28

%

SD

14.91

%

<

p95

40.39

%

Memory available percentage

Percentage of available memory

p5

59.62

%

<

mean

78.72

%

SD

14.91

%

<

p95

92.98

%

Max host memory used

Maximum memory usage percentage across all hosts

max

86.14

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

70.35

%

Memory growth rate

Rate of memory growth over time

growth

0.52

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1180.18

MiB/s

Disk write throughput

Disk write throughput in MiB/s

81.74

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.75

5 min

2.09

15 min

2.14

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

32.14

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.1896

%

SD

16.5361

%

<

p95

54.2

%

60 second average

9.7106

%

300 second average

13.6699

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.7758

%

SD

2.388

%

<

p95

6

%

60 second average

0.7198

%

300 second average

0.4976

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.648

%

SD

2.1047

%

<

p95

6

%

60 second average

0.5940

%

300 second average

0.4067

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.2951

%

SD

16.8248

%

<

p95

54.97

%

60 second average

6.0811

%

300 second average

5.0227

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.6523

%

SD

13.7997

%

<

p95

47.7

%

60 second average

4.4316

%

300 second average

3.4080

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.7

%

<

mean

23.59

%

SD

29.9

%

<

p95

86.08

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

351.23

KiB

<

mean

434.01

KiB

SD

74.46

KiB

<

p95

565.18

KiB

Holochain process threads

Number of threads in Holochain process

p5

26

threads

<

mean

33.18

threads

SD

6.12

threads

<

p95

42

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

61

file descriptors

<

mean

73.35

file descriptors

SD

10.27

file descriptors

<

p95

83

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 24 Sep 2026 09:16:59 UTC

Duration

900s

Peer count

10

Peer count at end

10

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_35955922634_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.075952

s

SD

0.060467

s

Max

Highest per-partition mean latency

max mean

0.096876

s

Min

Lowest per-partition mean latency

min mean

0.042036

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.152239

s

SD

0.083528

s

Max

Highest per-partition mean latency

max mean

0.186949

s

Min

Lowest per-partition mean latency

min mean

0.079758

s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

185642512

over 885.933476504s

mean rate

1.042429639e+08

/s

std

1.0659661315e+08

/s

p5

195885.22

/s

p95

3.3987901569e+08

/s

peak

4.2037474001e+08

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

39541558

over 885.933476504s

mean rate

8.78894078e+06

/s

std

2.287240396e+07

/s

p5

25908.45

/s

p95

7.308218529e+07

/s

peak

1.0401658769e+08

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 885.933476504s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.149305

s

SD

0.043084

s

p50

0.166946

s

<

p95

0.208969

s

<

p99

0.301777

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.003129

s

SD

0.000762

s

p50

0.002777

s

<

p95

0.004349

s

<

p99

0.004386

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.148248

s

SD

0.043026

s

p50

0.165571

s

<

p95

0.20793

s

<

p99

0.300495

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002104

s

SD

0.000558

s

p50

0.001887

s

<

p95

0.003

s

<

p99

0.003039

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002527

s

SD

0.000738

s

p50

0.002759

s

<

p95

0.003277

s

≤

p99

0.003277

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.145808

s

SD

0.042937

s

p50

0.162775

s

<

p95

0.205007

s

<

p99

0.297718

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00043

s

SD

0.000112

s

p50

0.000447

s

<

p95

0.000559

s

≤

p99

0.000559

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

1e-05

s

SD

3e-06

s

p50

1e-05

s

<

p95

1.5e-05

s

<

p99

1.6e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000291

s

SD

7.5e-05

s

p50

0.000264

s

<

p95

0.000447

s

≤

p99

0.000447

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00014

s

SD

4.9e-05

s

p50

0.000138

s

<

p95

0.000218

s

≤

p99

0.000218

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.9

s

<

mean

444.96

s

SD

255.8

s

<

p95

846.79

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

103

over 885.933596672s

mean rate

18.5

/s

std

87.6

/s

p5

0

/s

p95

85.93

/s

peak

882.91

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.665897

s

SD

1.59692

s

p50

0.877996

s

<

p95

5.122882

s

<

p99

5.681273

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002493

s

SD

0.000913

s

p50

0.002356

s

<

p95

0.004422

s

<

p99

0.005085

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.078855

s

SD

0.032047

s

p50

0.084193

s

<

p95

0.132265

s

<

p99

0.141899

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005818

s

SD

0.001825

s

p50

0.005878

s

<

p95

0.008995

s

<

p99

0.009817

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.194795

s

SD

0.497193

s

p50

0.031981

s

<

p95

1.713398

s

≤

p99

1.713398

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.190269

s

SD

0.132793

s

p50

0.1578

s

<

p95

0.421611

s

<

p99

0.674078

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00014

s

SD

3e-05

s

p50

0.000137

s

<

p95

0.000195

s

<

p99

0.000219

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00102

s

SD

0.000417

s

p50

0.001118

s

<

p95

0.002796

s

≤

p99

0.002796

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000442

s

SD

9.2e-05

s

p50

0.000431

s

<

p95

0.000578

s

<

p99

0.000602

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.105216

s

SD

0.035663

s

p50

0.112714

s

<

p95

0.162702

s

<

p99

0.167248

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.15411

s

SD

0.034856

s

p50

0.165052

s

<

p95

0.194094

s

<

p99

0.198462

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.145242

s

SD

0.042922

s

p50

0.16224

s

<

p95

0.204523

s

<

p99

0.297144

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

12.86

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.07

KiB/s

<

mean

212.96

KiB/s

SD

1.05

MiB/s

<

p95

1009.42

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.53

KiB/s

<

mean

16.46

KiB/s

SD

19.80

KiB/s

<

p95

37.88

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.60

GiB

mean

1.87

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

127.00

MiB

mean

147.79

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.79 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

6.75

%

<

mean

19.32

%

SD

20.82

%

<

p95

73.34

%

System

6.46

%

CPU percentiles

CPU usage percentiles

p50

11.20

%

p95

73.34

%

p99

89.76

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

58.84

s

Memory used percentage

Percentage of memory used

p5

7.36

%

<

mean

18.83

%

SD

9.98

%

<

p95

45.43

%

Memory available percentage

Percentage of available memory

p5

54.57

%

<

mean

81.17

%

SD

9.98

%

<

p95

92.64

%

Max host memory used

Maximum memory usage percentage across all hosts

max

44.57

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

5.50

%

Memory growth rate

Rate of memory growth over time

growth

1.79

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.12

MiB/s

Disk write throughput

Disk write throughput in MiB/s

19.84

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/2

hosts

Mount Point /etc/hosts

0/2

hosts

Mount Point /etc/nomad.d/client-meta.json

0/8

hosts

Mount Point /etc/resolv.conf

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.80

5 min

0.59

15 min

0.31

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.13

%

<

mean

6.4138

%

SD

9.9603

%

<

p95

30.75

%

60 second average

6.2905

%

300 second average

4.7243

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0002

%

SD

0.0043

%

≤

p95

0

%

60 second average

0.0001

%

300 second average

0.0002

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0002

%

SD

0.0043

%

≤

p95

0

%

60 second average

0.0001

%

300 second average

0.0001

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1953

%

SD

0.7492

%

<

p95

1.07

%

60 second average

0.1988

%

300 second average

0.1725

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1209

%

SD

0.4325

%

<

p95

0.74

%

60 second average

0.1232

%

300 second average

0.1095

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.88

%

<

mean

9.91

%

SD

14.06

%

<

p95

47.33

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

192.36

KiB

<

mean

224.88

KiB

SD

59.40

KiB

<

p95

261.87

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

29.03

threads

SD

9.47

threads

<

p95

32

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

61

file descriptors

<

mean

73.33

file descriptors

SD

20.69

file descriptors

<

p95

80

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 24 Sep 2026 09:18:30 UTC

Duration

900s

Peer count

10

Peer count at end

10

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_35955922634_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.151274

s

SD

0.413672

s

p50

0.149724

s

<

p95

0.209145

s

<

p99

0.278341

s

Timeout rate

The proportion of remote signal round-trips that timed out.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 886.03765873s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

31303468

over 881.182963624s

mean rate

1.35288950659e+09

/s

std

2.53745311172e+10

/s

p5

0

/s

p95

7.8111337172e+08

/s

peak

5.446e+11

/s

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

13132024

over 881.182963624s

mean rate

1.342009806e+07

/s

std

4.80856562e+07

/s

p5

0

/s

p95

5.45952699e+07

/s

peak

4.3493474831e+08

/s

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.002778

s

SD

0.000559

s

p50

0.002731

s

<

p95

0.003883

s

<

p99

0.004764

s

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.002732

s

SD

0.000554

s

p50

0.002676

s

<

p95

0.003922

s

<

p99

0.004472

s

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002227

s

SD

0.000795

s

p50

0.002584

s

<

p95

0.003494

s

≤

p99

0.003494

s

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001858

s

SD

0.000429

s

p50

0.001878

s

<

p95

0.002523

s

<

p99

0.003423

s

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001802

s

SD

0.000411

s

p50

0.001823

s

<

p95

0.002518

s

<

p99

0.003121

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000371

s

SD

0.000101

s

p50

0.000417

s

<

p95

0.00048

s

≤

p99

0.00048

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

7.2e-05

s

SD

2.2e-05

s

p50

6.4e-05

s

<

p95

0.000108

s

<

p99

0.000116

s

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

8.6e-05

s

SD

4.2e-05

s

p50

7.1e-05

s

<

p95

0.000157

s

<

p99

0.000179

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

3e-06

s

p50

5e-06

s

<

p95

1e-05

s

<

p99

1.9e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000265

s

SD

0.000135

s

p50

0.000238

s

<

p95

0.000609

s

≤

p99

0.000609

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1735

over 881.182992698s

mean rate

58262.77

/s

std

832651.3

/s

p5

0

/s

p95

14436.37

/s

peak

1.675977654e+07

/s

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1734

over 881.182989156s

mean rate

63898.26

/s

std

1.12565656e+06

/s

p5

0

/s

p95

12674.59

/s

peak

2.4e+07

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000179

s

SD

9.1e-05

s

p50

0.000171

s

<

p95

0.000376

s

≤

p99

0.000376

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.42

s

<

mean

444.09

s

SD

255.82

s

<

p95

843.87

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

117

over 886.037637999s

mean rate

1516.83

/s

std

25422.39

/s

p5

0

/s

p95

94.9

/s

peak

602541.63

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

2.377352

s

SD

1.755825

s

p50

2.614461

s

<

p95

5.600772

s

<

p99

7.954988

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002668

s

SD

0.001036

s

p50

0.002998

s

<

p95

0.004219

s

<

p99

0.006013

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.096469

s

SD

0.034293

s

p50

0.104219

s

<

p95

0.155826

s

≤

p99

0.155826

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.006804

s

SD

0.002559

s

p50

0.007099

s

<

p95

0.012103

s

<

p99

0.016925

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.224125

s

SD

0.324864

s

p50

0.231985

s

<

p95

0.5976

s

<

p99

1.339249

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.139217

s

SD

0.044933

s

p50

0.13338

s

<

p95

0.214081

s

<

p99

0.265698

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000148

s

SD

4.7e-05

s

p50

0.000147

s

<

p95

0.000239

s

<

p99

0.000297

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001274

s

SD

0.000871

s

p50

0.001336

s

<

p95

0.004418

s

≤

p99

0.004418

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000504

s

SD

9.6e-05

s

p50

0.000502

s

<

p95

0.000695

s

<

p99

0.000786

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.075146

s

SD

0.038796

s

p50

0.082684

s

<

p95

0.127012

s

<

p99

0.154472

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.108238

s

SD

0.016634

s

p50

0.111966

s

<

p95

0.131369

s

<

p99

0.142783

s

P2P received remote signals

Count of remote signals received.

total

1735

over 881.183016741s

mean rate

73044.58

/s

std

1.48958215e+06

/s

p5

0

/s

p95

11513.71

/s

peak

3.205128205e+07

/s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

19.80

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

13.29

KiB/s

<

mean

506.08

KiB/s

SD

1.53

MiB/s

<

p95

2.63

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

13.31

KiB/s

<

mean

28.79

KiB/s

SD

53.66

KiB/s

<

p95

96.10

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.71

GiB

mean

1.97

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

99.82

MiB

mean

114.85

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

7.22

%

<

mean

28.75

%

SD

25.98

%

<

p95

81.06

%

System

8.95

%

CPU percentiles

CPU usage percentiles

p50

17.35

%

p95

81.06

%

p99

93.85

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

60.70

s

Memory used percentage

Percentage of memory used

p5

6.16

%

<

mean

23.96

%

SD

12.31

%

<

p95

46.12

%

Memory available percentage

Percentage of available memory

p5

53.88

%

<

mean

76.04

%

SD

12.31

%

<

p95

93.84

%

Max host memory used

Maximum memory usage percentage across all hosts

max

45.13

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

5.50

%

Memory growth rate

Rate of memory growth over time

growth

0.75

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.13

MiB/s

Disk write throughput

Disk write throughput in MiB/s

14.14

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

0/2

hosts

Mount Point /etc/hosts

0/2

hosts

Mount Point /etc/nomad.d/client-meta.json

0/3

hosts

Mount Point /etc/resolv.conf

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.29

5 min

1.01

15 min

0.54

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.12

%

<

mean

10.5007

%

SD

12.8477

%

<

p95

31.78

%

60 second average

10.5265

%

300 second average

8.5550

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.006

%

≤

p95

0

%

60 second average

0.0002

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.006

%

≤

p95

0

%

60 second average

0.0002

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.2637

%

SD

0.9482

%

<

p95

1.31

%

60 second average

0.2694

%

300 second average

0.2545

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.1328

%

SD

0.4581

%

<

p95

0.77

%

60 second average

0.1353

%

300 second average

0.1352

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.82

%

<

mean

12.39

%

SD

19.62

%

<

p95

60.14

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

189.51

KiB

<

mean

220.74

KiB

SD

84.62

KiB

<

p95

272.69

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

28.2

threads

SD

13.2

threads

<

p95

32

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

70.34

file descriptors

SD

29.16

file descriptors

<

p95

79

file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started

Thu, 24 Sep 2026 09:20:06 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

single_write_many_read_35955922634_1

read call timing

The time it takes to complete the zome call that retrieves the record.

mean

0.003681

s

SD

0.003262

s

p50

0.003338

s

<

p95

0.005497

s

<

p99

0.00721

s

read call rate

The number of read zome calls completed per second.

mean

2688.04

/s

Error count

The number of errors collected.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000294

s

SD

1.4e-05

s

p50

0.000295

s

<

p95

0.000308

s

<

p99

0.00037

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 270.00365963s

Not enough time data to show a trend.

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

2422037940

over 270.00365963s

mean rate

8.97038872e+06

/s

std

205456.63

/s

p5

8.60503919e+06

/s

p95

9.18205552e+06

/s

peak

9.19227877e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 270.00365963s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 270.00365963s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009484

s

SD

0

s

p50

0.009484

s

≤

p95

0.009484

s

≤

p99

0.009484

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001867

s

SD

6e-05

s

p50

0.001867

s

<

p95

0.001902

s

<

p99

0.002187

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008255

s

SD

0

s

p50

0.008255

s

≤

p95

0.008255

s

≤

p99

0.008255

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001348

s

SD

4.8e-05

s

p50

0.001348

s

<

p95

0.001382

s

<

p99

0.001607

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002999

s

SD

0

s

p50

0.002999

s

≤

p95

0.002999

s

≤

p99

0.002999

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002046

s

SD

0

s

p50

0.002046

s

≤

p95

0.002046

s

≤

p99

0.002046

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000523

s

SD

4.8e-05

s

p50

0.00057

s

≤

p95

0.00057

s

≤

p99

0.00057

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000326

s

SD

1.5e-05

s

p50

0.000326

s

<

p95

0.00034

s

<

p99

0.000405

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004516

s

SD

0

s

p50

0.004516

s

≤

p95

0.004516

s

≤

p99

0.004516

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000336

s

SD

0

s

p50

0.000336

s

≤

p95

0.000336

s

≤

p99

0.000336

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.81

s

<

mean

141.82

s

SD

80.78

s

<

p95

266.82

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 270.003672394s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002021

s

SD

0

s

p50

0.002021

s

≤

p95

0.002021

s

≤

p99

0.002021

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002012

s

SD

0

s

p50

0.002012

s

≤

p95

0.002012

s

≤

p99

0.002012

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005848

s

SD

0.001016

s

p50

0.006024

s

<

p95

0.007765

s

≤

p99

0.007765

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002603

s

SD

0

s

p50

0.002603

s

≤

p95

0.002603

s

≤

p99

0.002603

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.008527

s

SD

0

s

p50

0.008527

s

≤

p95

0.008527

s

≤

p99

0.008527

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

4.7e-05

s

SD

2e-06

s

p50

4.7e-05

s

<

p95

4.9e-05

s

<

p99

6e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001229

s

SD

6.4e-05

s

p50

0.001263

s

≤

p95

0.001263

s

≤

p99

0.001263

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000408

s

SD

0

s

p50

0.000408

s

≤

p95

0.000408

s

≤

p99

0.000408

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

38.65

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

17.62

KiB/s

<

mean

222.80

KiB/s

SD

833.72

KiB/s

<

p95

1.28

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

14.09

KiB/s

<

mean

18.22

KiB/s

SD

9.65

KiB/s

<

p95

32.49

KiB/s

Total bytes received

Total bytes received on primary network interface

count

60.92

MiB

mean

222.80

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

4.98

MiB

mean

18.22

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

59.97

%

<

mean

61.55

%

SD

1.54

%

<

p95

64.19

%

System

22.90

%

CPU percentiles

CPU usage percentiles

p50

61.46

%

p95

64.19

%

p99

67.87

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

24.5

%

<

mean

25.54

%

SD

0.78

%

<

p95

26.38

%

Memory available percentage

Percentage of available memory

p5

73.62

%

<

mean

74.46

%

SD

0.78

%

<

p95

75.5

%

Max host memory used

Maximum memory usage percentage across all hosts

max

25.54

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.21

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

2.70

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.08

5 min

1.11

15 min

0.46

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

14.71

%

<

mean

29.3438

%

SD

4.6324

%

<

p95

31.36

%

60 second average

26.1879

%

300 second average

14.2807

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0.11

%

<

mean

2.3321

%

SD

2.6317

%

<

p95

8.58

%

60 second average

1.8390

%

300 second average

0.8145

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0.07

%

<

mean

1.1455

%

SD

1.2216

%

<

p95

3.28

%

60 second average

0.9393

%

300 second average

0.4366

%

Holochain process CPU usage

CPU usage by Holochain process

p5

43.74

%

<

mean

45.5

%

SD

8.63

%

<

p95

47.7

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

197.02

KiB

<

mean

235.54

KiB

SD

25.52

KiB

<

p95

242.99

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

23.48

threads

SD

1.79

threads

<

p95

26

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

62

file descriptors

<

mean

64.97

file descriptors

SD

3.38

file descriptors

<

p95

69

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 24 Sep 2026 09:34:58 UTC

Duration

900s

Peer count

20

Peer count at end

20

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_35955922634_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

6.480768

s

SD

2.417906

s

Max

Highest per-partition mean latency

max mean

7.858138

s

Min

Lowest per-partition mean latency

min mean

5.81339

s

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.44

/s

Max

Highest per-partition mean rate

max

1.6

/s

Min

Lowest per-partition mean rate

min

1.26

/s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

4013685

over 882.756928154s

mean rate

626766.4

/s

std

965171.19

/s

p5

0

/s

p95

2.87298364e+06

/s

peak

5.91314379e+06

/s

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

14715841

over 882.756928154s

mean rate

2.03193437e+06

/s

std

2.95213227e+06

/s

p5

46696.38

/s

p95

8.60198389e+06

/s

peak

1.461390498e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 882.756928154s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

502208

over 887.034921255s

mean rate

77083.27

/s

std

119217.96

/s

p5

0

/s

p95

353542.5

/s

peak

733605.6

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005553

s

SD

0.001521

s

p50

0.005111

s

<

p95

0.00862

s

<

p99

0.009699

s

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.018869

s

SD

0.009192

s

p50

0.01299

s

<

p95

0.031719

s

<

p99

0.045014

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004787

s

SD

0.001372

s

p50

0.004428

s

<

p95

0.007596

s

<

p99

0.0086

s

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.017929

s

SD

0.009024

s

p50

0.012148

s

<

p95

0.030312

s

<

p99

0.043702

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003059

s

SD

0.002204

s

p50

0.002343

s

<

p95

0.010028

s

≤

p99

0.010028

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001658

s

SD

0.00057

s

p50

0.001513

s

<

p95

0.002844

s

<

p99

0.003444

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000527

s

SD

0.000384

s

p50

0.000546

s

<

p95

0.001437

s

<

p99

0.001845

s

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.015843

s

SD

0.008595

s

p50

0.010284

s

<

p95

0.026956

s

<

p99

0.040097

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002598

s

SD

0.000777

s

p50

0.002466

s

<

p95

0.004307

s

<

p99

0.00466

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000201

s

SD

7.5e-05

s

p50

0.000187

s

<

p95

0.000343

s

<

p99

0.000391

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.91

s

<

mean

442.3

s

SD

254.84

s

<

p95

837.45

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

6915

over 887.034938283s

mean rate

150.76

/s

std

342.01

/s

p5

2.45

/s

p95

652.46

/s

peak

4085.5

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.363038

s

SD

1.306577

s

p50

1.393584

s

<

p95

4.904182

s

<

p99

5.705609

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002967

s

SD

0.007559

s

p50

0.001865

s

<

p95

0.016658

s

<

p99

0.034483

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.038724

s

SD

0.026274

s

p50

0.044501

s

<

p95

0.08803

s

<

p99

0.169144

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.041583

s

SD

0.021735

s

p50

0.027113

s

<

p95

0.075759

s

<

p99

0.10584

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.042583

s

SD

0.463672

s

p50

0.042645

s

<

p95

1.031331

s

<

p99

2.683546

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

2.927393

s

SD

10.471682

s

p50

4.594807

s

<

p95

31.55816

s

<

p99

47.384139

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000739

s

SD

0.000337

s

p50

0.000529

s

<

p95

0.001316

s

<

p99

0.00164

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003471

s

SD

0.005463

s

p50

0.001817

s

<

p95

0.006253

s

<

p99

0.029955

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000316

s

SD

0.0003

s

p50

0.000276

s

<

p95

0.000551

s

<

p99

0.000618

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.086363

s

SD

0.066898

s

p50

0.099636

s

<

p95

0.192647

s

<

p99

0.196297

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.142877

s

SD

0.04546

s

p50

0.12642

s

<

p95

0.220051

s

<

p99

0.234319

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

75.97

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

0

B/s

<

mean

572.43

KiB/s

SD

406.55

KiB/s

<

p95

944.51

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

0

B/s

<

mean

494.23

KiB/s

SD

354.15

KiB/s

<

p95

1.12

MiB/s

Total bytes received

Total bytes received on primary network interface

count

9.13

GiB

mean

10.50

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.88

GiB

mean

9.07

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Warning Moderate swap usage (15.9% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 90% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

63.71

%

<

mean

89.62

%

SD

11.08

%

<

p95

98

%

System

13.65

%

CPU percentiles

CPU usage percentiles

p50

94.23

%

p95

98.00

%

p99

98.25

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

19

hosts

mean time

807.25

s

Memory used percentage

Percentage of memory used

p5

14.49

%

<

mean

26.02

%

SD

9.12

%

<

p95

43.66

%

Memory available percentage

Percentage of available memory

p5

56.35

%

<

mean

73.98

%

SD

9.12

%

<

p95

93.11

%

Max host memory used

Maximum memory usage percentage across all hosts

max

46.40

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

15.91

%

Memory growth rate

Rate of memory growth over time

growth

0.35

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.28

MiB/s

Disk write throughput

Disk write throughput in MiB/s

119.97

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/9

hosts

Mount Point /etc/hosts

0/9

hosts

Mount Point /etc/nomad.d/client-meta.json

0/9

hosts

Mount Point /etc/resolv.conf

0/9

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

6.62

5 min

5.20

15 min

3.00

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

90.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

19.3

%

<

mean

53.3274

%

SD

18.7563

%

<

p95

76.58

%

60 second average

52.4778

%

300 second average

41.3844

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0017

%

SD

0.0386

%

≤

p95

0

%

60 second average

0.0014

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.9106

%

SD

14.772

%

<

p95

49.71

%

60 second average

6.7084

%

300 second average

5.1179

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.4434

%

SD

4.3656

%

<

p95

13.07

%

60 second average

1.4493

%

300 second average

1.1890

%

Holochain process CPU usage

CPU usage by Holochain process

p5

1.6

%

<

mean

6.39

%

SD

5.97

%

<

p95

12.81

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

238.50

KiB

<

mean

263.13

KiB

SD

36.10

KiB

<

p95

304.09

KiB

Holochain process threads

Number of threads in Holochain process

p5

30

threads

<

mean

32.76

threads

SD

5.32

threads

<

p95

38

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

73

file descriptors

<

mean

77.59

file descriptors

SD

9.15

file descriptors

<

p95

86

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 24 Sep 2026 10:03:46 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_35955922634_1

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2431.2

Max

Highest chain head value observed for any single write agent.

3692

Write agents observed

5

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.266983

s

SD

0.562305

s

Max

Highest per-partition mean latency

max mean

1.110592

s

Min

Lowest per-partition mean latency

min mean

0.049719

s

Error count

The number of errors accumulated.

16

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.052074

s

SD

1.066336

s

p50

0.040078

s

<

p95

4.136176

s

<

p99

4.38034

s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005591

s

SD

0.001911

s

p50

0.00563

s

<

p95

0.00836

s

≤

p99

0.00836

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004669

s

SD

0.002812

s

p50

0.004534

s

<

p95

0.012896

s

<

p99

0.015004

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.070606

s

SD

0.3739

s

p50

0.047029

s

<

p95

0.921023

s

<

p99

1.681051

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.004929

s

SD

1.393263

s

p50

0.004757

s

<

p95

4.384667

s

≤

p99

4.384667

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004722

s

SD

0.001709

s

p50

0.004702

s

<

p95

0.00725

s

≤

p99

0.00725

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003995

s

SD

0.002628

s

p50

0.003862

s

<

p95

0.011664

s

<

p99

0.013717

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.069822

s

SD

0.373894

s

p50

0.046213

s

<

p95

0.919934

s

<

p99

1.679656

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004205

s

SD

1.393115

s

p50

0.003893

s

<

p95

4.383613

s

≤

p99

4.383613

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001252

s

SD

0.000628

s

p50

0.001155

s

<

p95

0.003067

s

<

p99

0.00369

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.9e-05

s

SD

9e-06

s

p50

2.9e-05

s

<

p95

4.4e-05

s

≤

p99

4.4e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000612

s

SD

0.000719

s

p50

0.000579

s

<

p95

0.002613

s

<

p99

0.003593

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.0004

s

SD

0.000147

s

p50

0.000398

s

<

p95

0.000651

s

≤

p99

0.000651

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.067977

s

SD

0.373868

s

p50

0.043789

s

<

p95

0.917937

s

<

p99

1.676654

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001163

s

SD

1.392925

s

p50

0.001109

s

<

p95

4.380384

s

≤

p99

4.380384

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.1e-05

s

SD

5e-06

s

p50

8e-06

s

<

p95

1.9e-05

s

≤

p99

1.9e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001954

s

SD

0.000978

s

p50

0.001723

s

<

p95

0.003734

s

<

p99

0.006226

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000194

s

SD

0.00041

s

p50

0.000117

s

<

p95

0.000865

s

<

p99

0.002225

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.94

s

<

mean

443.32

s

SD

254.72

s

<

p95

838.13

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

1713

over 889.666563575s

mean rate

62614.19

/s

std

90292.57

/s

p5

259.62

/s

p95

274132.62

/s

peak

426691.23

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

179.231405

s

SD

137.471131

s

p50

46.409765

s

<

p95

405.012685

s

<

p99

519.717683

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.016863

s

SD

0.172697

s

p50

0.069749

s

<

p95

0.449408

s

<

p99

0.924299

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.098927

s

SD

0.493895

s

p50

0.647108

s

<

p95

1.270438

s

<

p99

1.740971

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.170419

s

SD

0.235409

s

p50

0.064762

s

<

p95

0.293461

s

<

p99

1.835675

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

2.73844

s

SD

28.674952

s

p50

18.141758

s

<

p95

85.519038

s

<

p99

109.421745

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

24.915157

s

SD

39.47686

s

p50

1.76928

s

<

p95

129.027343

s

<

p99

141.72048

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00291

s

SD

0.001465

s

p50

0.002465

s

<

p95

0.005439

s

<

p99

0.006884

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00258

s

SD

0.003429

s

p50

0.001326

s

<

p95

0.006647

s

<

p99

0.022515

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000524

s

SD

0.00047

s

p50

0.000306

s

<

p95

0.000998

s

<

p99

0.003144

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.949259

s

SD

28.791361

s

p50

7.348063

s

<

p95

60.001732

s

<

p99

60.007007

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

1.245348

s

SD

1.34561

s

p50

0.482793

s

<

p95

4.377266

s

≤

p99

4.377266

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.0576

s

SD

26.461631

s

p50

0.091199

s

<

p95

60.001394

s

<

p99

60.002156

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.724179

s

SD

19.941687

s

p50

1.309668

s

<

p95

60.000918

s

<

p99

60.003627

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

54.93

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

84.92

KiB/s

<

mean

410.24

KiB/s

SD

747.58

KiB/s

<

p95

908.73

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

49.96

KiB/s

<

mean

311.41

KiB/s

SD

228.91

KiB/s

<

p95

729.10

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.35

GiB

mean

10.76

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

7.10

GiB

mean

8.17

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (73.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 64% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

20.7

%

<

mean

66.92

%

SD

25.92

%

<

p95

98.35

%

System

11.99

%

CPU percentiles

CPU usage percentiles

p50

74.78

%

p95

98.35

%

p99

98.50

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

17

hosts

mean time

643.89

s

Memory used percentage

Percentage of memory used

p5

6.47

%

<

mean

20.31

%

SD

14.51

%

<

p95

37.96

%

Memory available percentage

Percentage of available memory

p5

62.04

%

<

mean

79.69

%

SD

14.51

%

<

p95

93.53

%

Max host memory used

Maximum memory usage percentage across all hosts

max

83.52

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

73.49

%

Memory growth rate

Rate of memory growth over time

growth

0.31

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.51

MiB/s

Disk write throughput

Disk write throughput in MiB/s

137.86

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.12

5 min

4.35

15 min

4.42

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

64.29

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

29.0393

%

SD

28.6836

%

<

p95

87.27

%

60 second average

28.9973

%

300 second average

30.9513

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0002

%

300 second average

0.0002

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0001

%

300 second average

0.0001

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.6494

%

SD

8.2681

%

<

p95

27.14

%

60 second average

3.6918

%

300 second average

3.9172

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.7772

%

SD

5.4882

%

<

p95

16.76

%

60 second average

1.8015

%

300 second average

1.6490

%

Holochain process CPU usage

CPU usage by Holochain process

p5

19.26

%

<

mean

61.28

%

SD

25.71

%

<

p95

96.96

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

262.58

KiB

<

mean

355.02

KiB

SD

70.14

KiB

<

p95

491.50

KiB

Holochain process threads

Number of threads in Holochain process

p5

30

threads

<

mean

35.23

threads

SD

8.7

threads

<

p95

52

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

71

file descriptors

<

mean

79.7

file descriptors

SD

12.33

file descriptors

<

p95

97

file descriptors

Write/`get_agent_activity` with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 24 Sep 2026 10:19:26 UTC

Duration

1800s

Peer count

30

Peer count at end

25

Behaviours

get_agent_activity_volatile (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_volatile_35955922634_1

Highest observed action_seq

The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.

Mean

mean

41.98

/s

Max

Highest per-partition mean rate

max

100.56

/s

Min

Lowest per-partition mean rate

min

9.95

/s

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.278023

s

SD

0.281514

s

Max

Highest per-partition mean latency

max mean

0.678918

s

Min

Lowest per-partition mean latency

min mean

0.062989

s

Running volatile conductors count

The number of conductors running by get_agent_activity_volatile peers.

p5

10

conductors

<

mean

14.33

conductors

SD

2.46

conductors

<

p95

18

conductors

Volatile conductor total running duration

The total running duration of a get_agent_activity_volatile conductor.

Mean

mean

699.419799

s

SD

373.086948

s

Max

Highest per-partition mean latency

max mean

826.290323

s

Min

Lowest per-partition mean latency

min mean

563.116279

s

Volatile conductor running duration

The duration that a get_agent_activity_volatile conductor was running before being stopped.

Mean

mean

31.934

s

SD

21.493694

s

Max

Highest per-partition mean latency

max mean

43.799912

s

Min

Lowest per-partition mean latency

min mean

24.446736

s

Volatile conductor stopped duration

The duration that a get_agent_activity_volatile conductor was stopped before being started again.

Mean

mean

26.054055

s

SD

22.384432

s

Max

Highest per-partition mean latency

max mean

38.874046

s

Min

Lowest per-partition mean latency

min mean

15.892089

s

Reached target arc

Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.

get_agent_activity_volatile_agent: uhCAk-Wr-ORzrCknBppFw7o9_dY39o3EB01MSkG0DMQxOVJVxN9Rq

p5

0

<

mean

0.21

SD

0.4

<

p95

1

get_agent_activity_volatile_agent: uhCAk0VgeSRDQ1inEUrPJBGv4ysDWSr_an2WakJilkxIscXkkSwty

p5

0

<

mean

0.3

SD

0.46

<

p95

1

get_agent_activity_volatile_agent: uhCAk2hZLGebgrqAa3Qj9L_k66-fqyCsBmEDwpwcWoS0rIfwvL252

p5

0

<

mean

0.28

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkCdBcPTn40HYDeLtBp1mvwOoz7quRoxhffSxNJrw_0WdSdgbi

p5

0

<

mean

0.29

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkEs4h1ZWeu0c4UPJV_8IxAUvC8KVjgQI7Hm5zUFf8qN1YgTDi

p5

0

<

mean

0.19

SD

0.39

<

p95

1

get_agent_activity_volatile_agent: uhCAkLgRhQnNfLUldi_oxrQGinNM2cj3NmkLjfT6fadLzCrYYpH0b

p5

0

<

mean

0.35

SD

0.48

<

p95

1

get_agent_activity_volatile_agent: uhCAkSF56i0MwMkUzlm9TORJtnCDwqbAi5yZIxx71aCDhLyg1_qtd

p5

0

<

mean

0.27

SD

0.44

<

p95

1

get_agent_activity_volatile_agent: uhCAkVe-YqIjWT-aWSKWYetOC_g49zgEItjnT_N3cqubyP_7OtVmf

p5

0

<

mean

0.25

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkW6bBm4zCsOEClMd2FbtZc3LRdtPVcwlkFhRNxnLni4QrKktT

p5

0

<

mean

0.27

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkWfICwFRCLuzxIeyPkJimTm2BIVItKf4JYVfnqfLuxrAe4qNZ

p5

0

<

mean

0.27

SD

0.44

<

p95

1

get_agent_activity_volatile_agent: uhCAk_9zck3sWUp7j_Ph6W5ndSgeID93d8q84IeTU59VLBM8MXQEk

p5

0

<

mean

0.18

SD

0.38

<

p95

1

get_agent_activity_volatile_agent: uhCAka_0eamiLatT0iybppUuKOoNTHOdl1CJZRVyPooeWL16_Z_oQ

p5

0

<

mean

0.27

SD

0.44

<

p95

1

get_agent_activity_volatile_agent: uhCAkcoDl7VYhOi1HlymgpRoqxKZcS7BBV5l2K9VnNAyydPsU-NIC

p5

0

<

mean

0.25

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkdwsg1N1YkwkGuXyGWmfvzQXmf5c4U68iZuA7yNKHJt82PvuM

p5

0

<

mean

0.24

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkh_gVGo-bDS-MdCqtr55cN3hpR3RrNESlWbKyMTb5XyE4qJPZ

p5

0

<

mean

0.2

SD

0.4

<

p95

1

get_agent_activity_volatile_agent: uhCAkk5euRczxXEFJ-j2dnEr119yntXehy2j64V6TRo0CAyDpgytx

p5

0

<

mean

0.24

SD

0.43

<

p95

1

get_agent_activity_volatile_agent: uhCAkrWLzKdPImk4zXN2OljwO59rONJ3D-ldTes4Mpvpy4dCPndbu

p5

0

<

mean

0.27

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAks_7mVBtdyB5oSE0o5lihKrPS1ZgNznYXbWMLMfzEPUUyn2pI

p5

0

<

mean

0.28

SD

0.45

<

p95

1

get_agent_activity_volatile_agent: uhCAkx39k8JfUJh0RhdQZMZT7OPupR5Kp7QjciQpk3dRRAuxJtHAO

p5

0

<

mean

0.3

SD

0.46

<

p95

1

get_agent_activity_volatile_agent: uhCAkxihP577co21gmgp5MxiJ-fnQdx53awn8Qslo0mq5VYyAYVfr

p5

0

<

mean

0.29

SD

0.45

<

p95

1

Error count

The number of errors accumulated.

210

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.095709

s

SD

1.589434

s

p50

0.353707

s

<

p95

1.135511

s

<

p99

1.886238

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 1634.396228819s

Not enough time data to show a trend.

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005299

s

SD

0.001629

s

p50

0.00545

s

<

p95

0.008257

s

≤

p99

0.008257

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005599

s

SD

0.002103

s

p50

0.005714

s

<

p95

0.010596

s

<

p99

0.012096

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.099793

s

SD

0.399858

s

p50

0.390126

s

<

p95

1.177448

s

<

p99

1.689645

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.005212

s

SD

0.323093

s

p50

0.004568

s

<

p95

1.062945

s

≤

p99

1.062945

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004414

s

SD

0.001462

s

p50

0.00431

s

<

p95

0.007117

s

≤

p99

0.007117

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004861

s

SD

0.001945

s

p50

0.004939

s

<

p95

0.009446

s

<

p99

0.010887

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.101037

s

SD

1.615003

s

p50

0.369103

s

<

p95

1.163584

s

<

p99

1.893986

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004714

s

SD

0.323073

s

p50

0.003996

s

<

p95

1.062151

s

≤

p99

1.062151

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001608

s

SD

0.000653

s

p50

0.001666

s

<

p95

0.003188

s

<

p99

0.003625

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.2e-05

s

SD

1.1e-05

s

p50

3.1e-05

s

<

p95

4.6e-05

s

≤

p99

4.6e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00063

s

SD

0.000311

s

p50

0.000602

s

<

p95

0.001333

s

<

p99

0.001586

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000396

s

SD

0.000135

s

p50

0.000358

s

<

p95

0.0006

s

≤

p99

0.0006

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.098552

s

SD

1.614585

s

p50

0.364647

s

<

p95

1.154549

s

<

p99

1.886304

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.002393

s

SD

0.32335

s

p50

0.000537

s

<

p95

1.059938

s

≤

p99

1.059938

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

4e-06

s

p50

7e-06

s

<

p95

1.3e-05

s

<

p99

3.6e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002621

s

SD

0.001016

s

p50

0.002557

s

<

p95

0.004081

s

<

p99

0.00621

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000185

s

SD

9.6e-05

s

p50

0.000225

s

<

p95

0.000402

s

<

p99

0.000409

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

3.61

s

<

mean

332.3

s

SD

512.1

s

<

p95

1527.86

s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

239.259542

s

SD

245.527643

s

p50

98.795899

s

<

p95

852.387933

s

<

p99

1130.862199

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.013091

s

SD

0.121623

s

p50

0.006456

s

<

p95

0.348746

s

<

p99

0.464986

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.192358

s

SD

0.298684

s

p50

0.017684

s

<

p95

0.665538

s

<

p99

1.413792

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.351339

s

SD

0.625073

s

p50

0.094097

s

<

p95

1.419014

s

<

p99

3.091486

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

1.946071

s

SD

32.520207

s

p50

14.092147

s

<

p95

98.574734

s

<

p99

130.839971

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

61.11237

s

SD

57.987792

s

p50

28.733298

s

<

p95

159.679666

s

<

p99

210.504401

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.009179

s

SD

0.004147

s

p50

0.001613

s

<

p95

0.013493

s

<

p99

0.015149

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.005016

s

SD

0.047005

s

p50

0.001058

s

<

p95

0.008272

s

<

p99

0.123091

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000508

s

SD

0.000213

s

p50

0.000433

s

<

p95

0.000885

s

<

p99

0.001237

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

8.028327

s

SD

26.806968

s

p50

0.593802

s

<

p95

60.001665

s

<

p99

60.001771

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.545514

s

SD

0.302411

s

p50

0.456697

s

<

p95

1.058576

s

≤

p99

1.058576

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.095971

s

SD

4.783769

s

p50

0.359703

s

<

p95

1.15396

s

<

p99

1.913883

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.442737

s

SD

24.602043

s

p50

0.504285

s

<

p95

60.001624

s

<

p99

60.001946

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

27.95

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.55

KiB/s

<

mean

210.83

KiB/s

SD

458.74

KiB/s

<

p95

745.77

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

2.15

KiB/s

<

mean

186.68

KiB/s

SD

295.27

KiB/s

<

p95

836.01

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.83

GiB

mean

5.63

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.71

GiB

mean

4.98

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.9%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (73.8% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 24% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.48

%

<

mean

37.18

%

SD

35.79

%

<

p95

98.78

%

System

9.23

%

CPU percentiles

CPU usage percentiles

p50

22.42

%

p95

98.78

%

p99

98.90

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

9

hosts

mean time

1209.45

s

Memory used percentage

Percentage of memory used

p5

5.54

%

<

mean

19.23

%

SD

13.66

%

<

p95

37.2

%

Memory available percentage

Percentage of available memory

p5

62.8

%

<

mean

80.77

%

SD

13.66

%

<

p95

94.46

%

Max host memory used

Maximum memory usage percentage across all hosts

max

77.55

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

73.83

%

Memory growth rate

Rate of memory growth over time

growth

0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.83

MiB/s

Disk write throughput

Disk write throughput in MiB/s

43.54

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/14

hosts

Mount Point /etc/hosts

0/14

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/14

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.52

5 min

2.69

15 min

3.19

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

24.14

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

19.5267

%

SD

34.2914

%

<

p95

97.45

%

60 second average

19.4849

%

300 second average

20.2884

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.073

%

SD

4.3548

%

<

p95

4.42

%

60 second average

1.1893

%

300 second average

1.6520

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.6936

%

SD

3.8377

%

<

p95

2.15

%

60 second average

0.7656

%

300 second average

0.9605

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

29.14

%

SD

38.4

%

<

p95

97.69

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

56.82

KiB

<

mean

212.87

KiB

SD

133.08

KiB

<

p95

448.42

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

27.09

threads

SD

11.56

threads

<

p95

44

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

68.45

file descriptors

SD

26.06

file descriptors

<

p95

97

file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started

Thu, 24 Sep 2026 09:21:47 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_query_35955922634_1

Write timing

The time it takes to complete the zome function call that writes the entry.

mean

0.031618

s

SD

0.015764

s

p50

0.030973

s

<

p95

0.05416

s

<

p99

0.069492

s

Write rate

The number of writes completed per second.

mean

88

/s

Query timing

The time it takes to complete the chain_query_count_len zome call that queries the source chain.

mean

0.072345

s

SD

0.039408

s

p50

0.071499

s

<

p95

0.133181

s

<

p99

0.153791

s

Query rate

The number of chain queries completed per second.

mean

88.04

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::chain_query_count_len'

Metered usage count of the WASM ribosome for this zome function.

total

83032429082

over 280.006033527s

mean rate

2.9653813965e+08

/s

std

2.997185107e+07

/s

p5

2.4480111424e+08

/s

p95

3.1985712875e+08

/s

peak

3.2160906342e+08

/s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

92486280

over 280.006033527s

mean rate

330299.84

/s

std

212218.06

/s

p5

185245.35

/s

p95

754705.47

/s

peak

1.19378245e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.006033527s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

22948352

over 280.006033527s

mean rate

81956.34

/s

std

52666.26

/s

p5

45538.93

/s

p95

187262.88

/s

peak

296209.77

/s

Zome call duration: 'crud::chain_query_count_len'

Duration of this zome call as measured by Holochain internally.

mean

0.029732

s

SD

0.009557

s

p50

0.027243

s

<

p95

0.038696

s

<

p99

0.03945

s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003308

s

SD

5.4e-05

s

p50

0.003303

s

<

p95

0.003333

s

<

p99

0.003336

s

WASM call duration: 'crud::chain_query_count_len'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.029108

s

SD

0.009526

s

p50

0.026627

s

<

p95

0.038047

s

<

p99

0.0388

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00275

s

SD

4.1e-05

s

p50

0.002746

s

<

p95

0.00277

s

<

p99

0.002772

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002701

s

SD

0

s

p50

0.002701

s

≤

p95

0.002701

s

≤

p99

0.002701

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000966

s

SD

1.6e-05

s

p50

0.000962

s

<

p95

0.000981

s

<

p99

0.000982

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000267

s

SD

0.000107

s

p50

0.000478

s

≤

p95

0.000478

s

≤

p99

0.000478

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.012462

s

SD

0.004214

s

p50

0.011486

s

<

p95

0.016205

s

<

p99

0.016482

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001472

s

SD

5e-05

s

p50

0.001444

s

<

p95

0.001552

s

<

p99

0.001556

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000119

s

SD

8e-06

s

p50

0.000119

s

<

p95

0.000123

s

≤

p99

0.000123

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.67

s

<

mean

146.67

s

SD

83.67

s

<

p95

276.67

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.006021366s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000351

s

SD

0

s

p50

0.000351

s

≤

p95

0.000351

s

≤

p99

0.000351

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001789

s

SD

0.00016

s

p50

0.00178

s

<

p95

0.001853

s

<

p99

0.00186

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.025184

s

SD

0.008065

s

p50

0.023338

s

<

p95

0.032103

s

<

p99

0.032663

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002464

s

SD

0

s

p50

0.002464

s

≤

p95

0.002464

s

≤

p99

0.002464

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000644

s

SD

0

s

p50

0.000644

s

≤

p95

0.000644

s

≤

p99

0.000644

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002366

s

SD

0.000799

s

p50

0.0022

s

<

p95

0.003038

s

<

p99

0.003091

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001064

s

SD

0.000704

s

p50

0.002437

s

≤

p95

0.002437

s

≤

p99

0.002437

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000237

s

SD

4e-06

s

p50

0.000237

s

<

p95

0.000239

s

≤

p99

0.000239

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

55.18

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

14.76

KiB/s

<

mean

353.82

KiB/s

SD

1.74

MiB/s

≤

p95

21.63

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.82

KiB/s

<

mean

23.72

KiB/s

SD

26.13

KiB/s

<

p95

24.86

KiB/s

Total bytes received

Total bytes received on primary network interface

count

100.20

MiB

mean

353.82

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.72

MiB

mean

23.72

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

67.72

%

<

mean

71.95

%

SD

1.74

%

<

p95

74.02

%

System

16.77

%

CPU percentiles

CPU usage percentiles

p50

72.46

%

p95

74.02

%

p99

74.07

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

25.63

%

<

mean

26.02

%

SD

0.61

%

<

p95

26.42

%

Memory available percentage

Percentage of available memory

p5

73.58

%

<

mean

73.98

%

SD

0.61

%

<

p95

74.37

%

Max host memory used

Maximum memory usage percentage across all hosts

max

26.02

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.30

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

2.75

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.16

5 min

1.59

15 min

0.81

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

17.77

%

<

mean

27.1147

%

SD

4.5902

%

<

p95

31.78

%

60 second average

24.4997

%

300 second average

15.7550

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.267

%

SD

0.3214

%

<

p95

0.93

%

60 second average

0.2833

%

300 second average

0.3987

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.093

%

SD

0.1487

%

<

p95

0.34

%

60 second average

0.1160

%

300 second average

0.2523

%

Holochain process CPU usage

CPU usage by Holochain process

p5

54.95

%

<

mean

58.31

%

SD

10.98

%

<

p95

62.84

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

191.42

KiB

<

mean

256.04

KiB

SD

29.46

KiB

<

p95

271.26

KiB

Holochain process threads

Number of threads in Holochain process

p5

22

threads

<

mean

22.5

threads

SD

1.02

threads

<

p95

25

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

61

file descriptors

<

mean

62.2

file descriptors

SD

2.6

file descriptors

<

p95

66

file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started

Thu, 24 Sep 2026 09:23:31 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_read_35955922634_1

Write timing

The time taken to write the entry.

mean

0.040781

s

SD

0.027739

s

p50

0.032094

s

<

p95

0.088588

s

<

p99

0.117526

s

Write rate

The number of writes completed per second.

mean

111.64

/s

Read timing

The time taken to read the entry.

mean

0.044023

s

SD

0.031376

s

p50

0.036227

s

<

p95

0.100038

s

<

p99

0.132511

s

Read rate

The number of reads completed per second.

mean

111.61

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000523

s

SD

8.2e-05

s

p50

0.000486

s

<

p95

0.000637

s

<

p99

0.000651

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

107546175

over 280.007054978s

mean rate

384086.31

/s

std

289534.92

/s

p5

185243.35

/s

p95

1.03943255e+06

/s

peak

1.48882962e+06

/s

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

104930560

over 280.007054978s

mean rate

374745.03

/s

std

282403.85

/s

p5

180682.89

/s

p95

1.01382927e+06

/s

peak

1.4521568e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.007054978s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

13342560

over 280.007054978s

mean rate

47651.11

/s

std

35920.73

/s

p5

22981.95

/s

p95

128955.69

/s

peak

184709.48

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002714

s

SD

0.000155

s

p50

0.002688

s

<

p95

0.002904

s

<

p99

0.003046

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001886

s

SD

0.00016

s

p50

0.001838

s

<

p95

0.002104

s

<

p99

0.002129

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002204

s

SD

0.000129

s

p50

0.002183

s

<

p95

0.002362

s

<

p99

0.0025

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001374

s

SD

0.000135

s

p50

0.00133

s

<

p95

0.001562

s

<

p99

0.001584

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000786

s

SD

0

s

p50

0.000786

s

≤

p95

0.000786

s

≤

p99

0.000786

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000694

s

SD

4.1e-05

s

p50

0.000688

s

<

p95

0.000741

s

<

p99

0.000809

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000273

s

SD

3.6e-05

s

p50

0.000244

s

<

p95

0.000292

s

<

p99

0.000296

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000551

s

SD

8.3e-05

s

p50

0.000514

s

<

p95

0.000667

s

<

p99

0.000681

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001121

s

SD

6.8e-05

s

p50

0.00111

s

<

p95

0.001202

s

<

p99

0.001278

s

Post-commit duration

Time spent executing post-commit workflows.

mean

6.2e-05

s

SD

8e-06

s

p50

6e-05

s

<

p95

7.3e-05

s

<

p99

7.4e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

18.11

s

<

mean

148.11

s

SD

83.67

s

<

p95

278.11

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.007051545s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000428

s

SD

0

s

p50

0.000428

s

≤

p95

0.000428

s

≤

p99

0.000428

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.000909

s

SD

0.000173

s

p50

0.000848

s

<

p95

0.001116

s

<

p99

0.001128

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.027783

s

SD

0.010543

s

p50

0.025294

s

<

p95

0.03934

s

<

p99

0.040392

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002218

s

SD

0

s

p50

0.002218

s

≤

p95

0.002218

s

≤

p99

0.002218

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002212

s

SD

0

s

p50

0.002212

s

≤

p95

0.002212

s

≤

p99

0.002212

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00241

s

SD

0.000883

s

p50

0.00222

s

<

p95

0.003327

s

<

p99

0.003402

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000751

s

SD

0.000101

s

p50

0.00058

s

<

p95

0.000864

s

<

p99

0.000883

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000244

s

SD

1.5e-05

s

p50

0.000239

s

<

p95

0.000264

s

<

p99

0.000265

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

56.16

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.06

KiB/s

<

mean

820.67

KiB/s

SD

2.12

MiB/s

<

p95

8.34

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.51

KiB/s

<

mean

31.53

KiB/s

SD

39.99

KiB/s

<

p95

172.09

KiB/s

Total bytes received

Total bytes received on primary network interface

count

232.42

MiB

mean

820.67

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.93

MiB

mean

31.53

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.8%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 3.61 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

45.61

%

<

mean

71.99

%

SD

17.73

%

<

p95

90.21

%

System

15.83

%

CPU percentiles

CPU usage percentiles

p50

79.46

%

p95

90.21

%

p99

93.85

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

134.48

s

Memory used percentage

Percentage of memory used

p5

7.13

%

<

mean

8.66

%

SD

1.09

%

<

p95

10.11

%

Memory available percentage

Percentage of available memory

p5

89.89

%

<

mean

91.34

%

SD

1.09

%

<

p95

92.87

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.66

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

3.61

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.54

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.35

5 min

1.59

15 min

0.67

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.37

%

<

mean

23.6527

%

SD

17.6076

%

<

p95

48.51

%

60 second average

17.0543

%

300 second average

7.2097

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.38

%

<

mean

33.15

%

SD

10.4

%

<

p95

45.57

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

124.95

KiB

<

mean

287.49

KiB

SD

101.31

KiB

<

p95

434.86

KiB

Holochain process threads

Number of threads in Holochain process

p5

7

threads

<

mean

26.03

threads

SD

6.16

threads

<

p95

31

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

11

file descriptors

<

mean

60.32

file descriptors

SD

14.02

file descriptors

<

p95

70

file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started

Thu, 24 Sep 2026 09:25:15 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_35955922634_1

Write timing

The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.

mean

0.081317

s

SD

0.040756

s

p50

0.079583

s

<

p95

0.142469

s

<

p99

0.195567

s

Write rate

The number of write zome function calls that complete per second.

mean

47.25

/s

Read timing

The time it takes to complete the zome function call that reads the data.

mean

0.121367

s

SD

0.055969

s

p50

0.118316

s

<

p95

0.198096

s

<

p99

0.286663

s

Read rate

The number of read zome function calls that complete per second.

mean

47.21

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.005878

s

SD

0.002121

s

p50

0.005447

s

<

p95

0.008806

s

<

p99

0.009146

s

WASM usage: 'validated::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

41848380

over 270.009441039s

mean rate

154989.44

/s

std

70171.59

/s

p5

92995.64

/s

p95

305557.54

/s

peak

381945.43

/s

WASM usage: 'validated::update_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

49448484

over 270.009441039s

mean rate

183137.19

/s

std

83030.59

/s

p5

106044.34

/s

p95

361336.76

/s

peak

451669.19

/s

WASM usage: 'validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

85605116

over 280.009471979s

mean rate

305724.33

/s

std

157075.05

/s

p5

168613.43

/s

p95

710646.43

/s

peak

775537.4

/s

WASM usage: 'validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

733072463

over 270.009441039s

mean rate

2.7150037e+06

/s

std

1.23204787e+06

/s

p5

1.58455763e+06

/s

p95

5.38809139e+06

/s

peak

6.69157168e+06

/s

Zome call duration: 'validated::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003779

s

SD

0.000307

s

p50

0.00379

s

<

p95

0.003999

s

<

p99

0.004

s

Zome call duration: 'validated::update_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.010814

s

SD

0.00266

s

p50

0.010334

s

<

p95

0.013178

s

<

p99

0.01335

s

WASM call duration: 'validated::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003117

s

SD

0.000266

s

p50

0.003126

s

<

p95

0.003309

s

≤

p99

0.003309

s

WASM call duration: 'validated::update_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010154

s

SD

0.002622

s

p50

0.009671

s

<

p95

0.012496

s

<

p99

0.012666

s

WASM call duration: 'validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000882

s

SD

5.5e-05

s

p50

0.000873

s

<

p95

0.000926

s

<

p99

0.000927

s

WASM call duration: 'validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003519

s

SD

0.000459

s

p50

0.003443

s

<

p95

0.003875

s

<

p99

0.003887

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000382

s

SD

4e-05

s

p50

0.000389

s

<

p95

0.000408

s

<

p99

0.00041

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000341

s

SD

3.2e-05

s

p50

0.000345

s

<

p95

0.000362

s

<

p99

0.000363

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.00544

s

SD

0.001677

s

p50

0.00506

s

<

p95

0.006966

s

<

p99

0.006998

s

Host function call duration: '__hc__update_1'

Duration of this host function call invoked from within WASM.

mean

0.007196

s

SD

0.002397

s

p50

0.006723

s

<

p95

0.009413

s

<

p99

0.009575

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001456

s

SD

0.000137

s

p50

0.001487

s

<

p95

0.001711

s

<

p99

0.001727

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000114

s

SD

1e-05

s

p50

0.000113

s

<

p95

0.000122

s

≤

p99

0.000122

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.5

s

<

mean

147.51

s

SD

83.67

s

<

p95

277.51

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.009452471s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001303

s

SD

0

s

p50

0.001303

s

≤

p95

0.001303

s

≤

p99

0.001303

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001532

s

SD

0.000247

s

p50

0.001539

s

<

p95

0.001614

s

<

p99

0.002724

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.050069

s

SD

0.018206

s

p50

0.04476

s

<

p95

0.066321

s

<

p99

0.067871

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.004214

s

SD

0

s

p50

0.004214

s

≤

p95

0.004214

s

≤

p99

0.004214

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001228

s

SD

0

s

p50

0.001228

s

≤

p95

0.001228

s

≤

p99

0.001228

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003692

s

SD

0.001425

s

p50

0.003317

s

<

p95

0.004904

s

<

p99

0.00509

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001512

s

SD

0.000123

s

p50

0.001578

s

<

p95

0.001608

s

<

p99

0.001613

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.0003

s

SD

2.9e-05

s

p50

0.000297

s

<

p95

0.00032

s

<

p99

0.000321

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

64.80

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.06

KiB/s

<

mean

1013.31

KiB/s

SD

2.17

MiB/s

<

p95

8.34

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.51

KiB/s

<

mean

36.87

KiB/s

SD

41.77

KiB/s

<

p95

170.22

KiB/s

Total bytes received

Total bytes received on primary network interface

count

277.08

MiB

mean

1013.31

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.08

MiB

mean

36.87

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

74.85

%

<

mean

83.62

%

SD

6.35

%

<

p95

93.85

%

System

18.82

%

CPU percentiles

CPU usage percentiles

p50

84.26

%

p95

93.85

%

p99

93.99

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

192.86

s

Memory used percentage

Percentage of memory used

p5

8.84

%

<

mean

9.57

%

SD

0.92

%

<

p95

10.89

%

Memory available percentage

Percentage of available memory

p5

89.11

%

<

mean

90.43

%

SD

0.92

%

<

p95

91.16

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.57

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

0.85

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.25

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.11

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.20

5 min

2.74

15 min

1.19

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

22.9

%

<

mean

35.4869

%

SD

11.7515

%

<

p95

52.97

%

60 second average

30.5100

%

300 second average

14.7486

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.26

%

<

mean

26.31

%

SD

9.49

%

<

p95

46.24

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.65

KiB

<

mean

276.07

KiB

SD

234.45

KiB

<

p95

493.03

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

31.44

threads

SD

37.79

threads

<

p95

76

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

7

file descriptors

<

mean

72.12

file descriptors

SD

81.2

file descriptors

<

p95

173

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 24 Sep 2026 11:56:29 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_35955922634_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

66.94

/s

Max

Highest per-partition mean rate

max

80.44

/s

Min

Lowest per-partition mean rate

min

51.52

/s

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

215.501429

s

SD

54.698541

s

Max

Highest per-partition mean latency

max mean

489.778686

s

Min

Lowest per-partition mean latency

min mean

10.573235

s

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

77.88

/s

Max

Highest per-partition mean rate

max

405

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0.0043

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

74354

Agents affected

12 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

6196

Max

Highest count in any single partition

7645

Min

Lowest count in any single partition

4766

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

3529

Agents affected

10 / 11

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

321

Max

Highest count in any single partition

1275

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

25

<

mean

27.11

SD

3.72

<

p95

28

zero-arc

p5

5

<

mean

9.18

SD

3.94

<

p95

16

Error count

The number of errors accumulated across all nodes.

138

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.015447

s

SD

0.273287

s

p50

0.203564

s

<

p95

1.049488

s

<

p99

1.325794

s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

457608672

over 875.952342328s

mean rate

5.4777611169e+08

/s

std

9.7656779081e+08

/s

p5

3.01887488e+06

/s

p95

3.16689493714e+09

/s

peak

3.65041427459e+09

/s

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

462808056

over 876.793943453s

mean rate

6.95553232733e+12

/s

std

4.97413760639e+13

/s

p5

3.26503335e+06

/s

p95

1.81771356294e+13

/s

peak

7.63116541637e+14

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

302427

over 887.597271166s

mean rate

1.41118437677e+09

/s

std

1.47378740133e+10

/s

p5

0

/s

p95

7.1997978478e+08

/s

peak

3.89388752279e+11

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004846

s

SD

0.001178

s

p50

0.00499

s

<

p95

0.006741

s

<

p99

0.00696

s

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.972064

s

SD

41.282584

s

p50

5.376176

s

<

p95

143.243233

s

<

p99

167.606868

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004285

s

SD

0.001062

s

p50

0.004437

s

<

p95

0.005889

s

<

p99

0.006073

s

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.971343

s

SD

41.282556

s

p50

5.375184

s

<

p95

143.242622

s

<

p99

167.606

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000903

s

SD

0.000478

s

p50

0.001122

s

<

p95

0.002256

s

<

p99

0.002678

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000267

s

SD

7.5e-05

s

p50

0.000264

s

<

p95

0.000404

s

<

p99

0.000417

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000253

s

SD

8.1e-05

s

p50

0.00025

s

<

p95

0.000404

s

<

p99

0.000417

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.015351

s

SD

0.276727

s

p50

0.200595

s

<

p95

1.056612

s

<

p99

1.336546

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.03443

s

SD

0.466034

s

p50

0.299988

s

<

p95

1.826352

s

<

p99

2.027786

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001362

s

SD

0.000714

s

p50

0.001747

s

<

p95

0.003389

s

<

p99

0.003995

s

Post-commit duration

Time spent executing post-commit workflows.

mean

9e-05

s

SD

6e-05

s

p50

0.000109

s

<

p95

0.000269

s

<

p99

0.000284

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.96

s

<

mean

442.15

s

SD

254.08

s

<

p95

837.71

s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

13.397527

s

SD

9.407926

s

p50

0

s

<

p95

6.816505

s

<

p99

68.798603

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.072378

s

SD

0.182876

s

p50

0.001326

s

<

p95

0.183869

s

<

p99

1.676441

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.026922

s

SD

0.358257

s

p50

0.00156

s

<

p95

1.063576

s

<

p99

1.127908

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.092163

s

SD

0.041243

s

p50

0.019961

s

<

p95

0.121357

s

<

p99

0.155761

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.650197

s

SD

15.194358

s

p50

0.002912

s

<

p95

20.008259

s

<

p99

70.252726

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

2.533673

s

SD

5.95034

s

p50

0.002515

s

<

p95

0.805761

s

<

p99

34.898826

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001306

s

SD

0.000887

s

p50

0.001

s

<

p95

0.002387

s

<

p99

0.003501

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001763

s

SD

0.001837

s

p50

0.001266

s

<

p95

0.002973

s

<

p99

0.012403

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000217

s

SD

0.000114

s

p50

0.000273

s

<

p95

0.000506

s

<

p99

0.000652

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.49072

s

SD

29.843943

s

p50

0.396221

s

<

p95

60.001882

s

<

p99

60.003733

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.031025

s

SD

14.31857

s

p50

0.243582

s

<

p95

60.000763

s

≤

p99

60.000763

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.637407

s

SD

0.345183

s

p50

0.578299

s

<

p95

0.881033

s

<

p99

1.816201

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

32.85

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

1.18

KiB/s

<

mean

432.07

KiB/s

SD

1.17

MiB/s

<

p95

1.17

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

776

B/s

<

mean

363.77

KiB/s

SD

1.12

MiB/s

<

p95

570.44

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.75

GiB

mean

11.35

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.21

GiB

mean

9.56

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.9%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.1% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 21% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.88

%

<

mean

43.83

%

SD

32.37

%

<

p95

94.56

%

System

10.97

%

CPU percentiles

CPU usage percentiles

p50

52.35

%

p95

94.56

%

p99

95.94

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

4

hosts

mean time

838.55

s

Memory used percentage

Percentage of memory used

p5

6.52

%

<

mean

19.49

%

SD

14.21

%

<

p95

37.1

%

Memory available percentage

Percentage of available memory

p5

62.9

%

<

mean

80.51

%

SD

14.21

%

<

p95

93.48

%

Max host memory used

Maximum memory usage percentage across all hosts

max

81.16

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.12

%

Memory growth rate

Rate of memory growth over time

growth

0.21

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.20

MiB/s

Disk write throughput

Disk write throughput in MiB/s

89.81

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.49

5 min

2.46

15 min

2.48

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

21.43

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

12.699

%

SD

16.6606

%

<

p95

42.54

%

60 second average

12.4731

%

300 second average

12.2653

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.9813

%

SD

3.1657

%

<

p95

5.01

%

60 second average

1.1058

%

300 second average

1.3835

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.4251

%

SD

2.319

%

<

p95

1.7

%

60 second average

0.5537

%

300 second average

0.8451

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.37

%

<

mean

36.32

%

SD

30.81

%

<

p95

92.1

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

233.20

KiB

<

mean

318.14

KiB

SD

68.59

KiB

<

p95

447.45

KiB

Holochain process threads

Number of threads in Holochain process

p5

23

threads

<

mean

32.23

threads

SD

6.41

threads

<

p95

45

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

56

file descriptors

<

mean

71.55

file descriptors

SD

11.18

file descriptors

<

p95

86

file descriptors

Zero-Arc Create Data

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 24 Sep 2026 11:09:24 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data--high_zero_35955922634_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

69.24

/s

Max

Highest per-partition mean rate

max

83.98

/s

Min

Lowest per-partition mean rate

min

51.1

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

355.421159

s

SD

248.168306

s

Max

Highest per-partition mean latency

max mean

355.421159

s

Min

Lowest per-partition mean latency

min mean

355.421159

s

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

85.83

/s

Max

Highest per-partition mean rate

max

85.83

/s

Min

Lowest per-partition mean rate

min

85.83

/s

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0055

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

162513

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

6019

Max

Highest count in any single partition

7399

Min

Lowest count in any single partition

4461

Received count

Total number of entries received by full-arc nodes, by agent.

Total

1795

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

898

Max

Highest count in any single partition

1795

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

6

<

mean

23.04

SD

7.52

<

p95

28

zero-arc

p5

4

<

mean

11.65

SD

6.1

<

p95

26

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000316

s

SD

6.8e-05

s

p50

0.000252

s

<

p95

0.000345

s

<

p99

0.000423

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

45

over 740.006992266s

mean rate

2.27

/s

std

1.45

/s

p5

0.23

/s

p95

4.5

/s

peak

4.5

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

388717152

over 810.650514953s

mean rate

7.6415527385e+08

/s

std

1.89851305146e+09

/s

p5

1.79381696e+06

/s

p95

3.51615211912e+09

/s

peak

1.53528309076e+10

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

61553484951

over 878.689494377s

mean rate

3.2775816771e+09

/s

std

2.21897943027e+09

/s

p5

5.91345194e+06

/s

p95

6.68572127299e+09

/s

peak

7.03265921346e+09

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

23527944

over 888.521850456s

mean rate

8.942510134e+07

/s

std

2.3611243398e+08

/s

p5

0

/s

p95

3.6194408774e+08

/s

peak

2.07076125886e+09

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004921

s

SD

0.000961

s

p50

0.005048

s

<

p95

0.006451

s

<

p99

0.008131

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.022891

s

SD

0.0444

s

p50

0.004682

s

<

p95

0.126764

s

<

p99

0.13737

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004364

s

SD

0.00087

s

p50

0.004513

s

<

p95

0.005586

s

<

p99

0.00728

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.022174

s

SD

0.044439

s

p50

0.003896

s

<

p95

0.126117

s

<

p99

0.136718

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00094

s

SD

0.00027

s

p50

0.000965

s

<

p95

0.001403

s

<

p99

0.001891

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000268

s

SD

5.6e-05

s

p50

0.000268

s

<

p95

0.000373

s

<

p99

0.000455

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000264

s

SD

5.7e-05

s

p50

0.000273

s

<

p95

0.000386

s

<

p99

0.000446

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000323

s

SD

3.7e-05

s

p50

0.00032

s

<

p95

0.000373

s

<

p99

0.00054

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000955

s

SD

0.001687

s

p50

0.000283

s

<

p95

0.004809

s

<

p99

0.005168

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001408

s

SD

0.00028

s

p50

0.001422

s

<

p95

0.001971

s

<

p99

0.002259

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.0001

s

SD

4.8e-05

s

p50

9.5e-05

s

<

p95

0.00017

s

<

p99

0.000256

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.92

s

<

mean

443.1

s

SD

254.55

s

<

p95

837.98

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

10733

over 888.521814463s

mean rate

11528.26

/s

std

68615.18

/s

p5

0

/s

p95

0

/s

peak

611239.56

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

2.105367

s

SD

6.33355

s

p50

0

s

≤

p95

0

s

<

p99

38.198985

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.051411

s

SD

0.06076

s

p50

0.001355

s

<

p95

0.099055

s

<

p99

0.160628

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.00484

s

SD

0.202948

s

p50

0.000998

s

<

p95

0.505861

s

<

p99

1.087652

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.088234

s

SD

0.038715

s

p50

0.07317

s

<

p95

0.125921

s

<

p99

0.152092

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.863051

s

SD

7.708329

s

p50

0.002617

s

<

p95

18.408799

s

<

p99

29.062237

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

7.270272

s

SD

9.282909

s

p50

0.001339

s

<

p95

0.004384

s

<

p99

52.287817

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00238

s

SD

0.001628

s

p50

0.003786

s

<

p95

0.004993

s

<

p99

0.006162

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003571

s

SD

0.005885

s

p50

0.001327

s

<

p95

0.004389

s

<

p99

0.037529

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000214

s

SD

5.6e-05

s

p50

0.000219

s

<

p95

0.000339

s

<

p99

0.000419

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

11.835904

s

SD

29.894483

s

p50

0.201501

s

<

p95

60.000806

s

<

p99

60.000862

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.123542

s

SD

0.010614

s

p50

0.122842

s

<

p95

0.12914

s

<

p99

0.129297

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.08

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

2.67

KiB/s

<

mean

186.61

KiB/s

SD

672.84

KiB/s

<

p95

895.98

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

894

B/s

<

mean

122.02

KiB/s

SD

135.20

KiB/s

<

p95

242.71

KiB/s

Total bytes received

Total bytes received on primary network interface

count

4.26

GiB

mean

4.90

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.78

GiB

mean

3.20

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (70.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 36% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

3.78

%

<

mean

60.44

%

SD

26.51

%

<

p95

94.81

%

System

12.36

%

CPU percentiles

CPU usage percentiles

p50

59.62

%

p95

94.81

%

p99

95.71

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

10

hosts

mean time

700.43

s

Memory used percentage

Percentage of memory used

p5

6.46

%

<

mean

20.07

%

SD

14.66

%

<

p95

37.12

%

Memory available percentage

Percentage of available memory

p5

62.88

%

<

mean

79.93

%

SD

14.66

%

<

p95

93.54

%

Max host memory used

Maximum memory usage percentage across all hosts

max

84.54

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

70.54

%

Memory growth rate

Rate of memory growth over time

growth

0.35

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.03

MiB/s

Disk write throughput

Disk write throughput in MiB/s

95.86

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.76

5 min

2.74

15 min

2.69

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

35.71

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

13.5862

%

SD

14.189

%

<

p95

38.09

%

60 second average

13.3850

%

300 second average

13.5112

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.979

%

SD

6.1827

%

<

p95

20.74

%

60 second average

2.0019

%

300 second average

2.0202

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.1742

%

SD

4.5141

%

<

p95

8.96

%

60 second average

1.2244

%

300 second average

1.2319

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.8

%

<

mean

54.32

%

SD

26.65

%

<

p95

93.22

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

231.81

KiB

<

mean

327.03

KiB

SD

60.53

KiB

<

p95

440.39

KiB

Holochain process threads

Number of threads in Holochain process

p5

24

threads

<

mean

33.14

threads

SD

5.14

threads

<

p95

40

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

62

file descriptors

<

mean

74.9

file descriptors

SD

9.24

file descriptors

<

p95

81

file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 24 Sep 2026 11:40:50 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data_validated--high_zero_35955922634_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

62.06

/s

Max

Highest per-partition mean rate

max

78.11

/s

Min

Lowest per-partition mean rate

min

46.49

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

317.204496

s

SD

28.186428

s

Max

Highest per-partition mean latency

max mean

634.074421

s

Min

Lowest per-partition mean latency

min mean

0.33457

s

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0069

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

146553

Agents affected

27 / 27

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5428

Max

Highest count in any single partition

6918

Min

Lowest count in any single partition

4051

Received count

Total number of entries received by full-arc nodes, by agent.

Total

2022

Agents affected

2 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1011

Max

Highest count in any single partition

2021

Min

Lowest count in any single partition

1

Open connections

The number of currently open connections to other conductors.

full-arc

p5

5

<

mean

22.53

SD

8.03

<

p95

28

zero-arc

p5

3

<

mean

9.64

SD

5.96

<

p95

22

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.002628

s

SD

0.003782

s

p50

0.007392

s

<

p95

0.012674

s

<

p99

0.015716

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

187

over 704.248085853s

mean rate

8768.86

/s

std

6022.64

/s

p5

90.6

/s

p95

15844.88

/s

peak

16006.18

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

461965524

over 844.353678431s

mean rate

5.280304945e+08

/s

std

4.08576411533e+09

/s

p5

399210.12

/s

p95

1.1582290105e+09

/s

peak

9.63592476695e+10

/s

WASM usage: 'timed_and_validated::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

406915716

over 874.249086267s

mean rate

1.04980812231e+09

/s

std

1.02003619694e+09

/s

p5

7.090270651e+07

/s

p95

3.57382408808e+09

/s

peak

4.23459023507e+09

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

270454608

over 887.702543311s

mean rate

3.0077238151e+08

/s

std

2.17521699533e+09

/s

p5

0

/s

p95

7.0709114641e+08

/s

peak

5.61145678192e+10

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

2138962816

over 887.702543311s

mean rate

2.94529672216e+09

/s

std

1.78446529761e+10

/s

p5

0

/s

p95

5.9679055754e+09

/s

peak

4.43797497249e+11

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.005161

s

SD

0.001424

s

p50

0.005403

s

<

p95

0.008175

s

<

p99

0.011178

s

Zome call duration: 'timed_and_validated::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.008651

s

SD

0.002752

s

p50

0.007685

s

<

p95

0.014154

s

<

p99

0.017736

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004573

s

SD

0.001291

s

p50

0.004837

s

<

p95

0.00733

s

<

p99

0.010026

s

WASM call duration: 'timed_and_validated::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008017

s

SD

0.002735

s

p50

0.007059

s

<

p95

0.013538

s

<

p99

0.01712

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001006

s

SD

0.000636

s

p50

0.000934

s

<

p95

0.002055

s

<

p99

0.004501

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004349

s

SD

0.00238

s

p50

0.003792

s

<

p95

0.006876

s

<

p99

0.017113

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000273

s

SD

7e-05

s

p50

0.00027

s

<

p95

0.000423

s

<

p99

0.000557

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00027

s

SD

7.7e-05

s

p50

0.000268

s

<

p95

0.000423

s

<

p99

0.000603

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.00037

s

SD

0.000954

s

p50

0.000599

s

<

p95

0.002839

s

<

p99

0.003018

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000643

s

SD

0.000244

s

p50

0.000577

s

<

p95

0.001126

s

<

p99

0.001392

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.004914

s

SD

0.003488

s

p50

0.007754

s

<

p95

0.012783

s

<

p99

0.015845

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001576

s

SD

0.001029

s

p50

0.001463

s

<

p95

0.002722

s

<

p99

0.008055

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000106

s

SD

5.2e-05

s

p50

0.000108

s

<

p95

0.000232

s

<

p99

0.000307

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.97

s

<

mean

442.92

s

SD

254.43

s

<

p95

837.93

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 887.702597361s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

16.023113

s

SD

54.524076

s

p50

0

s

<

p95

0.839788

s

<

p99

6.349859

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

5.077204

s

SD

1.543478

s

p50

0.001117

s

<

p95

4.294538

s

<

p99

8.91789

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.009708

s

SD

0.260361

s

p50

0.001117

s

<

p95

0.966384

s

<

p99

1.196018

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.080497

s

SD

0.033868

s

p50

0.067605

s

<

p95

0.115131

s

<

p99

0.132642

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.493019

s

SD

6.900284

s

p50

0.002811

s

<

p95

6.839111

s

<

p99

38.38079

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.127221

s

SD

0.096893

s

p50

0.002069

s

<

p95

0.065282

s

<

p99

0.537163

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003063

s

SD

0.001607

s

p50

0.00364

s

<

p95

0.005389

s

<

p99

0.006199

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003192

s

SD

0.004765

s

p50

0.001471

s

<

p95

0.00554

s

<

p99

0.031512

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000216

s

SD

7.1e-05

s

p50

0.000211

s

<

p95

0.000386

s

<

p99

0.000479

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

31.027357

s

SD

28.358

s

p50

3.46636

s

<

p95

60.011612

s

<

p99

60.011646

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.248324

s

SD

0.059942

s

p50

0.191583

s

<

p95

0.298133

s

<

p99

0.417404

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

43.73

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

7.57

KiB/s

<

mean

125.11

KiB/s

SD

521.84

KiB/s

<

p95

751.79

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

3.64

KiB/s

<

mean

77.10

KiB/s

SD

55.74

KiB/s

<

p95

186.14

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.86

GiB

mean

3.29

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.76

GiB

mean

2.03

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (74.6% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 29% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

24.58

%

<

mean

57.15

%

SD

22.54

%

<

p95

89.61

%

System

13.42

%

CPU percentiles

CPU usage percentiles

p50

52.94

%

p95

89.61

%

p99

97.65

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

8

hosts

mean time

685.68

s

Memory used percentage

Percentage of memory used

p5

7.59

%

<

mean

21.77

%

SD

15.15

%

<

p95

40.83

%

Memory available percentage

Percentage of available memory

p5

59.17

%

<

mean

78.23

%

SD

15.15

%

<

p95

92.41

%

Max host memory used

Maximum memory usage percentage across all hosts

max

87.33

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

74.56

%

Memory growth rate

Rate of memory growth over time

growth

0.17

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.92

MiB/s

Disk write throughput

Disk write throughput in MiB/s

95.06

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/2

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/13

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.56

5 min

2.56

15 min

2.58

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

28.57

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

12.05

%

SD

14.5374

%

<

p95

31.39

%

60 second average

11.9258

%

300 second average

12.4951

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0005

%

SD

0.0125

%

≤

p95

0

%

60 second average

0.0005

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0005

%

SD

0.0125

%

≤

p95

0

%

60 second average

0.0005

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.8075

%

SD

5.2646

%

<

p95

16.07

%

60 second average

1.8602

%

300 second average

1.8785

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.4086

%

SD

4.7137

%

<

p95

14.64

%

60 second average

1.4626

%

300 second average

1.4582

%

Holochain process CPU usage

CPU usage by Holochain process

p5

23.37

%

<

mean

50.83

%

SD

22.84

%

<

p95

87.92

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

377.32

KiB

<

mean

466.14

KiB

SD

83.47

KiB

<

p95

620.27

KiB

Holochain process threads

Number of threads in Holochain process

p5

31

threads

<

mean

34.86

threads

SD

7.32

threads

<

p95

39

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

66

file descriptors

<

mean

76.95

file descriptors

SD

10.13

file descriptors

<

p95

84

file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started

Thu, 24 Sep 2026 09:26:58 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zome_call_single_value_35955922634_1

Call timing

The time taken for the call to complete.

mean

0.003561

s

SD

0.002322

s

p50

0.002896

s

<

p95

0.006796

s

<

p99

0.011222

s

Call rate

The number of zome calls completed per second.

mean

2767.82

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'return_single_value::get_value'

Metered usage count of the WASM ribosome for this zome function.

total

312204429

over 280.003387563s

mean rate

1.11500513e+06

/s

std

400231.41

/s

p5

686049.23

/s

p95

1.75391551e+06

/s

peak

1.75799232e+06

/s

Zome call duration: 'return_single_value::get_value'

Duration of this zome call as measured by Holochain internally.

mean

0.001421

s

SD

0.000125

s

p50

0.00153

s

<

p95

0.001646

s

<

p99

0.001654

s

WASM call duration: 'return_single_value::get_value'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000893

s

SD

8.5e-05

s

p50

0.000975

s

<

p95

0.001048

s

<

p99

0.001054

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000162

s

SD

0

s

p50

0.000162

s

≤

p95

0.000162

s

≤

p99

0.000162

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.2

s

<

mean

147.21

s

SD

83.67

s

<

p95

277.21

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003363463s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001158

s

SD

0

s

p50

0.001158

s

≤

p95

0.001158

s

≤

p99

0.001158

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001412

s

SD

0

s

p50

0.001412

s

≤

p95

0.001412

s

≤

p99

0.001412

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003779

s

SD

0.00062

s

p50

0.004125

s

<

p95

0.004837

s

≤

p99

0.004837

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001849

s

SD

0

s

p50

0.001849

s

≤

p95

0.001849

s

≤

p99

0.001849

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002567

s

SD

0

s

p50

0.002567

s

≤

p95

0.002567

s

≤

p99

0.002567

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

8.1e-05

s

SD

1.3e-05

s

p50

9.3e-05

s

<

p95

0.000108

s

<

p99

0.00011

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001035

s

SD

5.3e-05

s

p50

0.001107

s

≤

p95

0.001107

s

≤

p99

0.001107

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000335

s

SD

0

s

p50

0.000335

s

≤

p95

0.000335

s

≤

p99

0.000335

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

54.28

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

19.74

KiB/s

<

mean

1.22

MiB/s

SD

2.18

MiB/s

<

p95

8.39

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.51

KiB/s

<

mean

44.34

KiB/s

SD

44.52

KiB/s

<

p95

169.99

KiB/s

Total bytes received

Total bytes received on primary network interface

count

340.81

MiB

mean

1.22

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

12.12

MiB

mean

44.34

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

34.46

%

<

mean

72.36

%

SD

20.81

%

<

p95

93.85

%

System

18.09

%

CPU percentiles

CPU usage percentiles

p50

79.65

%

p95

93.85

%

p99

93.99

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

139.29

s

Memory used percentage

Percentage of memory used

p5

7.45

%

<

mean

9.31

%

SD

1.29

%

<

p95

10.89

%

Memory available percentage

Percentage of available memory

p5

89.11

%

<

mean

90.69

%

SD

1.29

%

<

p95

92.55

%

Max host memory used

Maximum memory usage percentage across all hosts

max

9.31

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-3.13

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.33

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.29

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.70

5 min

3.64

15 min

1.67

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

2.72

%

<

mean

31.079

%

SD

17.1125

%

<

p95

52.97

%

60 second average

32.6579

%

300 second average

20.2852

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.0169

%

SD

0.0616

%

<

p95

0.12

%

60 second average

0.0076

%

300 second average

0.0007

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.18

%

<

mean

23.34

%

SD

7.39

%

<

p95

41.72

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

25.76

KiB

<

mean

267.05

KiB

SD

260.36

KiB

<

p95

1.04

MiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

38.39

threads

SD

45.92

threads

<

p95

176

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

87.03

file descriptors

SD

99.05

file descriptors

<

p95

385

file descriptors

App Install

App Install

DHT Sync Lag

First Call

Full-Arc Create (Validated) / Zero-Arc Read

Local Signals

Mixed Arc get_agent_activity

Mixed Arc must_get_agent_activity

Remote Call Rate

Remote Signals

Single-write, many-read

Validation Receipts

Write/get_agent_activity

Write/get_agent_activity with volatile conductors

Write/Query

Write/Read

Write validated

Zero-Arc Create and Read

Zero-Arc Create Data

Zero-arc Create Data (Validated)

Zome Call, Single Value

Mixed Arc `get_agent_activity`

Mixed Arc `must_get_agent_activity`

Write/`get_agent_activity`

Write/`get_agent_activity` with volatile conductors