App Install

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 27 Aug 2026 11:31:54 UTC

Duration

300s

Peer count

1

Behaviours

large (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_large_33058136248_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.130410446

s

Install app timing

The time taken to install an app.

mean

0.067118

s

SD

0.004392

s

p50

0.067116

s

<

p95

0.075869

s

<

p99

0.076882

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'large::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.00021228s

Not enough time data to show a trend.

WASM call duration: 'large::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003909

s

SD

0.000904

s

p50

0.003691

s

<

p95

0.005314

s

<

p99

0.007473

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000103

s

SD

4.6e-05

s

p50

8.2e-05

s

<

p95

0.0002

s

<

p99

0.000248

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.29

s

<

mean

17.29

s

SD

8.17

s

<

p95

27.29

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 20.000908365s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001534

s

SD

0.000245

s

p50

0.001536

s

<

p95

0.002045

s

≤

p99

0.002045

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002017

s

SD

0.000158

s

p50

0.002165

s

<

p95

0.002323

s

≤

p99

0.002323

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.007022

s

SD

0.002336

s

p50

0.007278

s

<

p95

0.011988

s

≤

p99

0.011988

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002842

s

SD

0.000383

s

p50

0.002674

s

<

p95

0.003192

s

≤

p99

0.003192

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001893

s

SD

0.00065

s

p50

0.002131

s

<

p95

0.003263

s

≤

p99

0.003263

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7.8e-05

s

SD

7e-06

s

p50

7.8e-05

s

<

p95

9.3e-05

s

≤

p99

9.3e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00174

s

SD

0.000369

s

p50

0.001673

s

<

p95

0.002396

s

≤

p99

0.002396

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000496

s

SD

8.3e-05

s

p50

0.000497

s

<

p95

0.000499

s

≤

p99

0.000499

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

68.76

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

76.67

KiB/s

<

mean

792.85

KiB/s

SD

1004.20

KiB/s

<

p95

2.16

MiB/s

Not enough time data to show a trend.

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

33.16

KiB/s

<

mean

46.15

KiB/s

SD

17.28

KiB/s

<

p95

70.57

KiB/s

Not enough time data to show a trend.

Total bytes received

Total bytes received on primary network interface

count

23.23

MiB

mean

792.85

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.35

MiB

mean

46.15

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 92.3%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 7.95 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

85.35

%

<

mean

87.94

%

SD

3.13

%

<

p95

92.35

%

Not enough time data to show a trend.

System

19.18

%

CPU percentiles

CPU usage percentiles

p50

86.13

%

p95

92.35

%

p99

92.35

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

300.00

s

Memory used percentage

Percentage of memory used

p5

8.77

%

<

mean

11

%

SD

2.16

%

<

p95

14.26

%

Not enough time data to show a trend.

Memory available percentage

Percentage of available memory

p5

85.74

%

<

mean

89

%

SD

2.16

%

<

p95

91.23

%

Not enough time data to show a trend.

Max host memory used

Maximum memory usage percentage across all hosts

max

11.00

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

7.95

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

4.76

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.84

5 min

3.94

15 min

2.19

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

37.61

%

<

mean

45.55

%

SD

5.4423

%

<

p95

52.78

%

Not enough time data to show a trend.

60 second average

43.7825

%

300 second average

25.2400

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

Not enough time data to show a trend.

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

11

%

<

mean

19.65

%

SD

6.11

%

<

p95

24.02

%

Not enough time data to show a trend.

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

23.40

KiB

<

mean

631.23

KiB

SD

676.50

KiB

<

p95

1.77

MiB

Not enough time data to show a trend.

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

114.4

threads

SD

126.89

threads

<

p95

339

threads

Not enough time data to show a trend.

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

245

file descriptors

SD

270.51

file descriptors

<

p95

721

file descriptors

Not enough time data to show a trend.

App Install

↑ Back to index

Measure app installation speed. This scenario has two roles:

minimal: Installs a simple app which implements initialisation callbacks but otherwise doesn't contain a lot of code.
large: Installs a larger app that contains some dependencies and some generated data as padding to make the bundle larger.

These are intended to catch a compilation slowdown or issues with copying around large WASMs (e.g. accidental cloning or tracing). In each case, the behaviour will uninstall the app it installed so that it can re-install it on the next iteration.

Started

Thu, 27 Aug 2026 11:33:30 UTC

Duration

300s

Peer count

1

Behaviours

minimal (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

app_install_minimal_33058136248_1

First install time

The time from conductor startup to completion of the first app installation attempt.

0.358185079

s

Install app timing

The time taken to install an app.

mean

0.041716

s

SD

0.010996

s

p50

0.039883

s

<

p95

0.044451

s

<

p99

0.11159

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'callback::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 9.999935696s

Not enough time data to show a trend.

WASM usage: 'callback_integrity::genesis_self_check_2'

Metered usage count of the WASM ribosome for this zome function.

total

110942

over 9.999935696s

mean rate

11094.27

/s

std

0

/s

p5

11094.27

/s

p95

11094.27

/s

peak

11094.27

/s

Not enough time data to show a trend.

WASM call duration: 'callback::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000724

s

SD

4.3e-05

s

p50

0.000713

s

<

p95

0.000765

s

<

p99

0.000939

s

WASM call duration: 'callback_integrity::genesis_self_check_2'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000976

s

SD

0.000162

s

p50

0.001046

s

≤

p95

0.001046

s

≤

p99

0.001046

s

Post-commit duration

Time spent executing post-commit workflows.

mean

6.9e-05

s

SD

2.3e-05

s

p50

6.3e-05

s

<

p95

8.5e-05

s

<

p99

0.000199

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

7.51

s

<

mean

12.51

s

SD

5

s

<

p95

17.51

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 9.999855356s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001206

s

SD

5.8e-05

s

p50

0.001299

s

≤

p95

0.001299

s

≤

p99

0.001299

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.00109

s

SD

7e-06

s

p50

0.001093

s

≤

p95

0.001093

s

≤

p99

0.001093

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003572

s

SD

9.3e-05

s

p50

0.003719

s

≤

p95

0.003719

s

≤

p99

0.003719

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002293

s

SD

0.000197

s

p50

0.002605

s

≤

p95

0.002605

s

≤

p99

0.002605

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.001678

s

SD

0.000331

s

p50

0.002204

s

≤

p95

0.002204

s

≤

p99

0.002204

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

5.6e-05

s

SD

1e-06

s

p50

5.7e-05

s

≤

p95

5.7e-05

s

≤

p99

5.7e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000951

s

SD

0.000273

s

p50

0.001322

s

<

p95

0.00135

s

≤

p99

0.00135

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000333

s

SD

3.3e-05

s

p50

0.000347

s

≤

p95

0.000347

s

≤

p99

0.000347

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.83

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

131.44

KiB/s

<

mean

395.56

KiB/s

SD

264.12

KiB/s

<

p95

659.68

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

44.80

KiB/s

<

mean

49.03

KiB/s

SD

4.23

KiB/s

<

p95

53.26

KiB/s

Total bytes received

Total bytes received on primary network interface

count

7.73

MiB

mean

395.56

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

980.67

KiB

mean

49.03

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 7.45 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

63.13

%

<

mean

67.48

%

SD

4.34

%

<

p95

71.82

%

System

17.65

%

CPU percentiles

CPU usage percentiles

p50

71.82

%

p95

71.82

%

p99

71.82

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

8.42

%

<

mean

10.36

%

SD

1.99

%

<

p95

13.1

%

Memory available percentage

Percentage of available memory

p5

86.9

%

<

mean

89.64

%

SD

1.99

%

<

p95

91.58

%

Max host memory used

Maximum memory usage percentage across all hosts

max

10.36

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

7.45

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.01

MiB/s

Disk write throughput

Disk write throughput in MiB/s

6.44

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.86

5 min

4.54

15 min

2.59

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

17.49

%

<

mean

19.4033

%

SD

1.4687

%

<

p95

21.06

%

60 second average

27.8767

%

300 second average

25.4467

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

28.83

%

<

mean

35.95

%

SD

7.12

%

<

p95

43.07

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

139.40

KiB

<

mean

999.77

KiB

SD

763.23

KiB

<

p95

1.95

MiB

Holochain process threads

Number of threads in Holochain process

p5

29

threads

<

mean

191.33

threads

SD

144.01

threads

<

p95

379

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

61

file descriptors

<

mean

413.33

file descriptors

SD

309.79

file descriptors

<

p95

815

file descriptors

DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 27 Aug 2026 11:35:09 UTC

Duration

1800s

Peer count

30

Peer count at end

27

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_33058136248_1

Create rate

The average number of created records per agent.

Mean

mean

17

/s

Max

Highest per-partition mean rate

max

25.7

/s

Min

Lowest per-partition mean rate

min

7.53

/s

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

460.33807

s

SD

349.881087

s

Max

Highest per-partition mean latency

max mean

722.225809

s

Min

Lowest per-partition mean latency

min mean

54.633563

s

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

469.93

/s

Max

Highest per-partition mean rate

max

1656

/s

Min

Lowest per-partition mean rate

min

2.5

/s

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0331

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

72470

Agents affected

23 / 23

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3151

Max

Highest count in any single partition

4683

Min

Lowest count in any single partition

1384

Received count

The total number of entries received by record_lag agents, by agent.

Total

11983

Agents affected

4 / 5

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2397

Max

Highest count in any single partition

4704

Min

Lowest count in any single partition

0

Error count

The number of errors encountered during the scenario.

12

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000543

s

SD

0.001384

s

p50

0.000584

s

<

p95

0.00454

s

<

p99

0.004771

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

7

over 1700.075407867s

mean rate

1.9

/s

std

1.22

/s

p5

0

/s

p95

3.57

/s

peak

3.57

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

244339282

over 1788.039872543s

mean rate

3.5632558732e+08

/s

std

3.68330256206e+09

/s

p5

4.35304928e+06

/s

p95

6.8388567666e+08

/s

peak

1.59942803818e+11

/s

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

554726111

over 1784.61067166s

mean rate

2.84586040607e+10

/s

std

3.2079548884e+10

/s

p5

1.75386971187e+09

/s

p95

1.03483137365e+11

/s

peak

1.22000782324e+11

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

31155168

over 1788.039872543s

mean rate

4.758922837e+07

/s

std

4.3460099693e+08

/s

p5

565885.25

/s

p95

9.379009457e+07

/s

peak

2.03939574617e+10

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.023393

s

SD

0.026309

s

p50

0.016441

s

<

p95

0.064591

s

<

p99

0.136471

s

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.128882

s

SD

0.265438

s

p50

0.05348

s

<

p95

0.796378

s

<

p99

0.990628

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.022227

s

SD

0.02598

s

p50

0.015263

s

<

p95

0.061525

s

<

p99

0.13422

s

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.127797

s

SD

0.265741

s

p50

0.049437

s

<

p95

0.795527

s

<

p99

0.989732

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002821

s

SD

0.002362

s

p50

0.002419

s

<

p95

0.010162

s

<

p99

0.011477

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.004648

s

SD

0.008736

s

p50

0.001862

s

<

p95

0.01965

s

<

p99

0.043037

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.004394

s

SD

0.008371

s

p50

0.001769

s

<

p95

0.018756

s

<

p99

0.041108

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000538

s

SD

0.001445

s

p50

0.000587

s

<

p95

0.004707

s

<

p99

0.004858

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.005482

s

SD

0.006754

s

p50

0.005072

s

<

p95

0.021783

s

<

p99

0.026518

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00467

s

SD

0.004181

s

p50

0.003842

s

<

p95

0.016197

s

<

p99

0.019768

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00059

s

SD

0.001049

s

p50

0.000322

s

<

p95

0.001869

s

<

p99

0.006391

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

87.85

s

<

mean

890.81

s

SD

513.41

s

<

p95

1688.05

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

18893

over 1788.039805021s

mean rate

103724.01

/s

std

608973.87

/s

p5

29.95

/s

p95

416882.04

/s

peak

1.957827927e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

113.792875

s

SD

104.775315

s

p50

1.912472

s

<

p95

162.544836

s

<

p99

711.289152

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.128337

s

SD

0.697654

s

p50

0.195653

s

<

p95

2.053413

s

<

p99

3.365558

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

2e-06

s

p50

0

s

≤

p95

0

s

<

p99

1e-05

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.459996

s

SD

0.622282

s

p50

0.839661

s

<

p95

1.810922

s

<

p99

2.276884

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.417058

s

SD

0.317679

s

p50

0.334411

s

<

p95

1.031477

s

<

p99

1.5045

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.253341

s

SD

42.31496

s

p50

19.103074

s

<

p95

112.913965

s

<

p99

195.069623

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

48.503338

s

SD

55.746515

s

p50

0.10432

s

<

p95

159.914246

s

<

p99

199.18549

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00176

s

SD

0.001896

s

p50

0.001948

s

<

p95

0.006233

s

<

p99

0.009469

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.009469

s

SD

0.015694

s

p50

0.002102

s

<

p95

0.047688

s

<

p99

0.067027

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.003433

s

SD

0.007441

s

p50

0.001587

s

<

p95

0.017684

s

<

p99

0.038851

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.704901

s

SD

28.685307

s

p50

0.661928

s

<

p95

60.011711

s

<

p99

60.05647

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.263912

s

SD

5.644504

s

p50

0.229205

s

<

p95

15.180432

s

<

p99

15.861572

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

73.11

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

271.84

KiB/s

<

mean

811.03

KiB/s

SD

739.63

KiB/s

<

p95

1.39

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

85.69

KiB/s

<

mean

668.82

KiB/s

SD

437.85

KiB/s

<

p95

1.37

MiB/s

Total bytes received

Total bytes received on primary network interface

count

38.53

GiB

mean

22.04

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

31.77

GiB

mean

18.18

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.6%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 93% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

58.03

%

<

mean

88.35

%

SD

17.58

%

<

p95

98.3

%

System

15.24

%

CPU percentiles

CPU usage percentiles

p50

95.71

%

p95

98.30

%

p99

98.62

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

27

hosts

mean time

1555.46

s

Memory used percentage

Percentage of memory used

p5

12

%

<

mean

24.61

%

SD

9.86

%

<

p95

43.61

%

Memory available percentage

Percentage of available memory

p5

56.39

%

<

mean

75.39

%

SD

9.86

%

<

p95

88

%

Max host memory used

Maximum memory usage percentage across all hosts

max

47.70

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.47

%

Memory growth rate

Rate of memory growth over time

growth

0.45

MiB/s

Disk read throughput

Disk read throughput in MiB/s

171.82

MiB/s

Disk write throughput

Disk write throughput in MiB/s

178.62

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/9

hosts

Mount Point /efi-boot

0/8

hosts

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/9

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

Mount Point /nix/store

0/9

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

10.67

5 min

9.50

15 min

6.66

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

92.86

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

5.52

%

<

mean

62.9679

%

SD

25.7608

%

<

p95

96.95

%

60 second average

61.4781

%

300 second average

54.2444

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.107

%

SD

0.8901

%

≤

p95

0

%

60 second average

0.1028

%

300 second average

0.0978

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0241

%

SD

0.1554

%

≤

p95

0

%

60 second average

0.0272

%

300 second average

0.0400

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.7859

%

SD

19.814

%

<

p95

56.37

%

60 second average

15.5206

%

300 second average

14.1064

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.9551

%

SD

4.3218

%

<

p95

11.7

%

60 second average

1.9935

%

300 second average

2.0923

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

28.16

%

SD

27.1

%

<

p95

88.91

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.41

KiB

<

mean

297.21

KiB

SD

148.04

KiB

<

p95

532.43

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

27.15

threads

SD

19.32

threads

<

p95

40

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

61.51

file descriptors

SD

37.23

file descriptors

<

p95

89

file descriptors

First Call

↑ Back to index

This scenario installs a simple app which implements the init callback, and measures the time elapsed between start of install and calling init. It will uninstall the app it installed so that it can re-install it on the next iteration. This is required to re-run the initialisation callback.

Started

Thu, 27 Aug 2026 11:16:37 UTC

Duration

300s

Peer count

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

first_call_33058136248_1

Zome call timing

The time elapsed between start of install and calling init.

mean

0.010943

s

SD

0.003435

s

p50

0.010239

s

<

p95

0.015139

s

<

p99

0.026744

s

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000247631s

Not enough time data to show a trend.

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 10.000247631s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

246512

over 10.000247631s

mean rate

24650.59

/s

std

0

/s

p5

24650.59

/s

p95

24650.59

/s

peak

24650.59

/s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.004423

s

SD

0.00034

s

p50

0.005044

s

≤

p95

0.005044

s

≤

p99

0.005044

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003728

s

SD

0.000772

s

p50

0.003634

s

<

p95

0.005153

s

<

p99

0.005856

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001439

s

SD

0.000429

s

p50

0.001268

s

<

p95

0.002673

s

≤

p99

0.002673

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001103

s

SD

8.5e-05

s

p50

0.001257

s

≤

p95

0.001257

s

≤

p99

0.001257

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000338

s

SD

3.5e-05

s

p50

0.000348

s

<

p95

0.000418

s

≤

p99

0.000418

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001954

s

SD

0.000117

s

p50

0.002168

s

≤

p95

0.002168

s

≤

p99

0.002168

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000121

s

SD

0.000105

s

p50

8.6e-05

s

<

p95

0.000433

s

<

p99

0.000545

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

6.92

s

<

mean

11.92

s

SD

5

s

<

p95

16.92

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 10.000068108s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001008

s

SD

0.00017

s

p50

0.001319

s

≤

p95

0.001319

s

≤

p99

0.001319

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001255

s

SD

0.000124

s

p50

0.001482

s

≤

p95

0.001482

s

≤

p99

0.001482

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.004115

s

SD

0.000536

s

p50

0.005092

s

≤

p95

0.005092

s

≤

p99

0.005092

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002732

s

SD

0.000509

s

p50

0.003663

s

≤

p95

0.003663

s

≤

p99

0.003663

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00148

s

SD

0.000724

s

p50

0.002804

s

≤

p95

0.002804

s

≤

p99

0.002804

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

3.2e-05

s

SD

5e-06

s

p50

4.1e-05

s

≤

p95

4.1e-05

s

≤

p99

4.1e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001073

s

SD

0.000358

s

p50

0.001622

s

<

p95

0.001789

s

≤

p99

0.001789

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000275

s

SD

1.3e-05

s

p50

0.0003

s

≤

p95

0.0003

s

≤

p99

0.0003

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

47.61

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

55.78

KiB/s

<

mean

593.75

KiB/s

SD

537.96

KiB/s

<

p95

1.11

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

26.45

KiB/s

<

mean

28.08

KiB/s

SD

1.63

KiB/s

<

p95

29.71

KiB/s

Total bytes received

Total bytes received on primary network interface

count

11.60

MiB

mean

593.75

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

561.63

KiB

mean

28.08

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 6.93 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

52.23

%

<

mean

61.32

%

SD

9.09

%

<

p95

70.41

%

System

13.71

%

CPU percentiles

CPU usage percentiles

p50

70.41

%

p95

70.41

%

p99

70.41

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

15.04

%

<

mean

21.93

%

SD

7.73

%

<

p95

32.72

%

Memory available percentage

Percentage of available memory

p5

67.28

%

<

mean

78.07

%

SD

7.73

%

<

p95

84.96

%

Max host memory used

Maximum memory usage percentage across all hosts

max

21.93

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

6.93

MiB/s

Disk read throughput

Disk read throughput in MiB/s

265.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.06

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.23

5 min

2.08

15 min

3.37

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.77

%

<

mean

7.5533

%

SD

3.4536

%

<

p95

12.12

%

60 second average

3.6467

%

300 second average

3.4100

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

2.52

%

<

mean

3.0767

%

SD

0.4573

%

<

p95

3.64

%

60 second average

3.5867

%

300 second average

5.0500

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.9

%

<

mean

2.5767

%

SD

0.6995

%

<

p95

3.54

%

60 second average

3.4167

%

300 second average

4.9533

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

67.26

%

<

mean

73.8633

%

SD

7.7691

%

<

p95

84.77

%

60 second average

84.2500

%

300 second average

83.9233

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

38.41

%

<

mean

53.0233

%

SD

18.4094

%

<

p95

78.99

%

60 second average

77.4500

%

300 second average

80.5400

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

22.89

%

SD

22.89

%

<

p95

45.77

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

242.96

KiB

<

mean

800.75

KiB

SD

557.80

KiB

<

p95

1.33

MiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

99

threads

SD

123.36

threads

<

p95

273

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

221

file descriptors

SD

267.9

file descriptors

<

p95

598

file descriptors

Full-Arc Create (Validated) / Zero-Arc Read

↑ Back to index

A full-arc/zero-arc mixed scenario where full-arc nodes create data that gets validated and zero-arc nodes read the data. The scenario has two roles:

full: A full-arc conductor that creates entries with a timestamp field. Those entries get validated and then retrieved by zero-arc nodes.
zero: A zero-arc conductor that reads the entries created by the full-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 27 Aug 2026 12:29:43 UTC

Duration

900s

Peer count

30

Peer count at end

27

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

full_arc_create_validated_zero_arc_read_33058136248_1

Create rate

The number of timed entries created by the full-arc node(s) per second.

Mean

mean

37.55

/s

Max

Highest per-partition mean rate

max

44.79

/s

Min

Lowest per-partition mean rate

min

27.02

/s

Fetch lag timing

For each entry, the time lag between when it was created by a full-arc node and when a zero-arc node fetched it from the network.

Mean

mean

104.393192

s

SD

7.393088

s

Max

Highest per-partition mean latency

max mean

1258.627157

s

Min

Lowest per-partition mean latency

min mean

3.027401

s

Fetch rate

The number of entries per second fetched by zero-arc nodes.

Mean

mean

122.56

/s

Max

Highest per-partition mean rate

max

267

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc nodes.

0.0344

Entry created count

Total number of entries created by full-arc nodes, by agent.

Total

13573

Agents affected

4 / 4

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3393

Max

Highest count in any single partition

3937

Min

Lowest count in any single partition

2276

Received count

Total number of entries fetched by zero-arc nodes, by agent.

Total

11192

Agents affected

13 / 24

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

466

Max

Highest count in any single partition

1899

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

2

<

mean

15.92

SD

10.75

<

p95

26

zero-arc

p5

0

<

mean

4.59

SD

5.62

<

p95

15

Retrieval errors

Errors encountered by zero-arc nodes when attempting to retrieve entries created by full-arc nodes.

Total

308

Agents affected

23 / 24

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

13

Max

Highest count in any single partition

14

Min

Lowest count in any single partition

0

Error count

The number of errors accumulated across all nodes.

311

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.029101

s

SD

0.124849

s

p50

0.095617

s

<

p95

0.328719

s

<

p99

0.580809

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

4

over 809.248170087s

mean rate

936.02

/s

std

2697.03

/s

p5

0

/s

p95

4366.93

/s

peak

14226.68

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.006992

s

SD

0.00363

s

p50

0.005532

s

<

p95

0.015084

s

<

p99

0.015801

s

Zome call duration: 'timed_and_validated::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.321316

s

SD

110.825468

s

p50

0.514908

s

<

p95

326.46623

s

<

p99

399.073666

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006299

s

SD

0.003347

s

p50

0.004882

s

<

p95

0.01373

s

<

p99

0.014457

s

WASM call duration: 'timed_and_validated::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

1.835959

s

SD

82.771774

s

p50

3.539909

s

<

p95

212.692651

s

<

p99

248.594914

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001257

s

SD

0.000508

s

p50

0.001573

s

<

p95

0.002242

s

<

p99

0.0025

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008195

s

SD

0.003362

s

p50

0.001264

s

<

p95

0.010305

s

<

p99

0.015779

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000448

s

SD

0.000253

s

p50

0.000362

s

<

p95

0.001009

s

<

p99

0.001049

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000428

s

SD

0.000201

s

p50

0.000351

s

<

p95

0.000827

s

<

p99

0.000859

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.029281

s

SD

0.172462

s

p50

0.140389

s

<

p95

0.419633

s

<

p99

0.628664

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.08431

s

SD

4.301758

s

p50

0.162524

s

<

p95

12.689215

s

<

p99

17.506075

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.012664

s

SD

0.007641

s

p50

0.012275

s

<

p95

0.028018

s

<

p99

0.02976

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002052

s

SD

0.000831

s

p50

0.002404

s

<

p95

0.003799

s

<

p99

0.004283

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000125

s

SD

5.4e-05

s

p50

0.000106

s

<

p95

0.000197

s

<

p99

0.00031

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.75

s

<

mean

445.56

s

SD

255.9

s

<

p95

846.33

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 889.053920087s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

179.947416

s

SD

60.299068

s

p50

0

s

<

p95

94.726127

s

<

p99

370.097071

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

5.272404

s

SD

2.922881

s

p50

0.001351

s

<

p95

6.478684

s

<

p99

17.036965

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

<

p99

1e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.074989

s

SD

0.066181

s

p50

0.001567

s

<

p95

0.077271

s

<

p99

0.40163

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.15561

s

SD

0.051324

s

p50

0.019651

s

<

p95

0.174745

s

<

p99

0.240652

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

20.750818

s

SD

17.460099

s

p50

0.002584

s

<

p95

52.119881

s

<

p99

83.946864

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

1.03823

s

SD

12.080284

s

p50

0.001952

s

<

p95

0.044024

s

<

p99

116.181623

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001832

s

SD

0.001892

s

p50

0.000145

s

<

p95

0.005541

s

<

p99

0.009258

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.009392

s

SD

0.00569

s

p50

0.001859

s

<

p95

0.013326

s

<

p99

0.036312

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000317

s

SD

0.000122

s

p50

0.000296

s

<

p95

0.000588

s

<

p99

0.000654

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

2.83243

s

SD

29.81549

s

p50

0.95639

s

<

p95

60.001427

s

<

p99

60.002103

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.84536

s

SD

16.032405

s

p50

0.868609

s

<

p95

60.001073

s

<

p99

60.001399

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.246822

s

SD

4.603373

s

p50

0.218658

s

<

p95

16.428881

s

≤

p99

16.428881

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

17.92

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

710

B/s

<

mean

135.54

KiB/s

SD

295.69

KiB/s

<

p95

673.92

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

631

B/s

<

mean

107.25

KiB/s

SD

326.62

KiB/s

<

p95

553.57

KiB/s

Total bytes received

Total bytes received on primary network interface

count

2.99

GiB

mean

3.44

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.36

GiB

mean

2.72

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/6 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 15% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

0.8

%

<

mean

24.51

%

SD

31.65

%

<

p95

88.78

%

System

6.59

%

CPU percentiles

CPU usage percentiles

p50

3.65

%

p95

88.78

%

p99

95.66

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

7

hosts

mean time

446.59

s

Memory used percentage

Percentage of memory used

p5

6.74

%

<

mean

15.6

%

SD

11.09

%

<

p95

42.61

%

Memory available percentage

Percentage of available memory

p5

57.39

%

<

mean

84.4

%

SD

11.09

%

<

p95

93.26

%

Max host memory used

Maximum memory usage percentage across all hosts

max

44.69

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.23

%

Memory growth rate

Rate of memory growth over time

growth

0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.31

MiB/s

Disk write throughput

Disk write throughput in MiB/s

57.60

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/9

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/8

hosts

Mount Point /etc/hostname

1/6

hosts

Mount Point /etc/hosts

1/6

hosts

Mount Point /etc/nomad.d/client-meta.json

1/12

hosts

Mount Point /etc/resolv.conf

1/6

hosts

Mount Point /nix/store

0/9

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.14

5 min

2.10

15 min

2.50

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

15.38

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

6.9534

%

SD

15.9447

%

<

p95

49.53

%

60 second average

6.8986

%

300 second average

6.6624

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0004

%

SD

0.0207

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

4.0162

%

SD

11.894

%

<

p95

29.26

%

60 second average

3.9298

%

300 second average

4.0205

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.8361

%

SD

4.8804

%

<

p95

12.12

%

60 second average

1.8283

%

300 second average

2.0483

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.12

%

<

mean

11.61

%

SD

25.17

%

<

p95

83.09

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

367.54

KiB

<

mean

447.92

KiB

SD

108.13

KiB

<

p95

685.81

KiB

Holochain process threads

Number of threads in Holochain process

p5

23

threads

<

mean

32.22

threads

SD

12.59

threads

<

p95

56

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

53

file descriptors

<

mean

64.41

file descriptors

SD

12.28

file descriptors

<

p95

81

file descriptors

Local Signals

↑ Back to index

This scenario is for testing the speed and reliability of local signals. It uses a zome with a function that will emit 10,000 signals. The scenario measures how long it takes to send the signals and how many have been received by the time the zome call ends.

Started

Thu, 27 Aug 2026 11:18:15 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

local_signals_33058136248_1

Send timing

The time taken to emit a batch of 10,000 signals.

mean

1.496577

s

SD

0.790625

s

p50

1.760506

s

<

p95

2.632498

s

≤

p99

2.632498

s

Receive timing

The time taken to receive the completed batch, to the nearest 250ms.

mean

27.649435

s

SD

7.055531

s

p50

30.001216

s

<

p95

30.002225

s

≤

p99

30.002225

s

Success ratio

The proportion of signals successfully received in a batch.

min

0.33739998936653137

<

mean

0.5882

SD

0.1011

<

max

0.7121999859809875

Holochain MetricsClick to toggle visibility.

WASM usage: 'signal::emit_10k_signals'

Metered usage count of the WASM ribosome for this zome function.

total

246763944

over 280.004473162s

mean rate

881308.3

/s

std

1.39347078e+06

/s

p5

0

/s

p95

3.08453479e+06

/s

peak

3.08498916e+06

/s

Zome call duration: 'signal::emit_10k_signals'

Duration of this zome call as measured by Holochain internally.

mean

0.212265

s

SD

0.031081

s

p50

0.216497

s

<

p95

0.236401

s

≤

p99

0.236401

s

WASM call duration: 'signal::emit_10k_signals'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.211494

s

SD

0.031105

s

p50

0.215715

s

<

p95

0.235699

s

≤

p99

0.235699

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

1.5e-05

s

SD

2e-06

s

p50

1.5e-05

s

<

p95

1.7e-05

s

≤

p99

1.7e-05

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

80000

over 280.004482599s

mean rate

285.71

/s

std

431.76

/s

p5

0

/s

p95

999.99

/s

peak

999.99

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000127

s

SD

0

s

p50

0.000127

s

≤

p95

0.000127

s

≤

p99

0.000127

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.88

s

<

mean

146.89

s

SD

83.67

s

<

p95

276.89

s

Dropped WebSocket signals

Count of signals dropped from the app WebSocket due to channel overload.

total

3016

over 280.004425192s

mean rate

10.77

/s

std

21.24

/s

p5

0

/s

p95

70.3

/s

peak

78.4

/s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.004431627s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001678

s

SD

0

s

p50

0.001678

s

≤

p95

0.001678

s

≤

p99

0.001678

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001174

s

SD

0

s

p50

0.001174

s

≤

p95

0.001174

s

≤

p99

0.001174

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003277

s

SD

0.000311

s

p50

0.003216

s

<

p95

0.003541

s

≤

p99

0.003541

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.001142

s

SD

0

s

p50

0.001142

s

≤

p95

0.001142

s

≤

p99

0.001142

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000432

s

SD

0

s

p50

0.000432

s

≤

p95

0.000432

s

≤

p99

0.000432

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000139

s

SD

2.1e-05

s

p50

0.000141

s

<

p95

0.000152

s

≤

p99

0.000152

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000853

s

SD

0.000206

s

p50

0.001136

s

≤

p95

0.001136

s

≤

p99

0.001136

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000247

s

SD

0

s

p50

0.000247

s

≤

p95

0.000247

s

≤

p99

0.000247

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

10.18

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

480

B/s

<

mean

260.56

KiB/s

SD

1.26

MiB/s

≤

p95

227.03

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

21

B/s

<

mean

13.87

KiB/s

SD

20.23

KiB/s

<

p95

41.03

KiB/s

Total bytes received

Total bytes received on primary network interface

count

73.79

MiB

mean

260.56

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

3.93

MiB

mean

13.87

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.84 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

9.88

%

<

mean

24.24

%

SD

12.67

%

<

p95

55.56

%

System

14.06

%

CPU percentiles

CPU usage percentiles

p50

22.90

%

p95

55.56

%

p99

62.14

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

17.76

%

<

mean

23.21

%

SD

3.81

%

<

p95

29.23

%

Memory available percentage

Percentage of available memory

p5

70.77

%

<

mean

76.79

%

SD

3.81

%

<

p95

82.24

%

Max host memory used

Maximum memory usage percentage across all hosts

max

23.21

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

2.84

MiB/s

Disk read throughput

Disk read throughput in MiB/s

271.13

MiB/s

Disk write throughput

Disk write throughput in MiB/s

2.06

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.44

5 min

2.79

15 min

3.36

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.13

%

<

mean

9.614

%

SD

4.3216

%

<

p95

16.2

%

60 second average

8.4220

%

300 second average

5.6777

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

3.05

%

<

mean

3.963

%

SD

0.6013

%

<

p95

5.03

%

60 second average

4.0713

%

300 second average

4.6497

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

2.58

%

<

mean

3.652

%

SD

0.5947

%

<

p95

4.73

%

60 second average

3.7610

%

300 second average

4.4353

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

72.87

%

<

mean

78.5873

%

SD

3.8513

%

<

p95

83.91

%

60 second average

79.6547

%

300 second average

81.9160

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

49.35

%

<

mean

63.6833

%

SD

9.0061

%

<

p95

78.89

%

60 second average

66.4030

%

300 second average

73.5733

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.5

%

<

mean

8.9

%

SD

11.22

%

<

p95

40.76

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.75

KiB

<

mean

177.68

KiB

SD

82.79

KiB

<

p95

248.94

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

21.74

threads

SD

9.57

threads

<

p95

29

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

59.17

file descriptors

SD

24.18

file descriptors

<

p95

76

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 27 Aug 2026 11:19:52 UTC

Duration

900s

Peer count

10

Peer count at end

9

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_33058136248_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

0.056166

s

SD

0.128066

s

Max

Highest per-partition mean latency

max mean

0.100415

s

Min

Lowest per-partition mean latency

min mean

-0.223261

s

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.127303

s

SD

0.105376

s

Max

Highest per-partition mean latency

max mean

0.160881

s

Min

Lowest per-partition mean latency

min mean

0.061415

s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

148954131

over 888.189389438s

mean rate

1.7532599961e+08

/s

std

3.8078216907e+08

/s

p5

612390.04

/s

p95

1.22433991004e+09

/s

peak

1.64246024765e+09

/s

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

47921968

over 888.189389438s

mean rate

1.23692603e+06

/s

std

1.50209244e+06

/s

p5

11136.33

/s

p95

4.44947094e+06

/s

peak

5.84149312e+06

/s

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 888.189389438s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.127241

s

SD

0.050728

s

p50

0.164429

s

<

p95

0.205273

s

<

p99

0.268636

s

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.002652

s

SD

0.001118

s

p50

0.002655

s

<

p95

0.00451

s

<

p99

0.00452

s

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.126362

s

SD

0.050614

s

p50

0.163502

s

<

p95

0.204268

s

<

p99

0.267175

s

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001727

s

SD

0.000723

s

p50

0.001824

s

<

p95

0.003086

s

<

p99

0.003107

s

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002401

s

SD

0.000979

s

p50

0.002581

s

<

p95

0.004271

s

≤

p99

0.004271

s

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.124489

s

SD

0.050391

s

p50

0.16172

s

<

p95

0.202416

s

<

p99

0.26432

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000405

s

SD

0.000187

s

p50

0.000401

s

<

p95

0.000825

s

≤

p99

0.000825

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

s

SD

6e-06

s

p50

8e-06

s

<

p95

1.9e-05

s

<

p99

2.6e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000231

s

SD

0.0001

s

p50

0.00023

s

<

p95

0.000436

s

<

p99

0.000492

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000153

s

SD

0.000133

s

p50

0.000127

s

<

p95

0.000524

s

≤

p99

0.000524

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.96

s

<

mean

446.03

s

SD

256.37

s

<

p95

846.97

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

95

over 888.189247847s

mean rate

3.69

/s

std

17.31

/s

p5

0

/s

p95

12.32

/s

peak

110.04

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.277919

s

SD

0.629824

s

p50

1.19593

s

<

p95

2.067603

s

<

p99

2.546628

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002472

s

SD

0.002009

s

p50

0.002484

s

<

p95

0.007943

s

<

p99

0.008695

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.06165

s

SD

0.044633

s

p50

0.070483

s

<

p95

0.192479

s

≤

p99

0.192479

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.005305

s

SD

0.004309

s

p50

0.004933

s

<

p95

0.015912

s

<

p99

0.020356

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.25716

s

SD

0.212576

s

p50

0.228804

s

<

p95

0.600851

s

<

p99

0.872577

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.142345

s

SD

0.066714

s

p50

0.153662

s

<

p95

0.220663

s

<

p99

0.279472

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00012

s

SD

5.1e-05

s

p50

0.00013

s

<

p95

0.00021

s

<

p99

0.000256

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001027

s

SD

0.000771

s

p50

0.001025

s

<

p95

0.002755

s

<

p99

0.00379

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000344

s

SD

0.000159

s

p50

0.00038

s

<

p95

0.000662

s

<

p99

0.00067

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.085314

s

SD

0.036097

s

p50

0.078402

s

<

p95

0.1668

s

<

p99

0.176076

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.146713

s

SD

0.060187

s

p50

0.166454

s

<

p95

0.224625

s

≤

p99

0.224625

s

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.124043

s

SD

0.050332

s

p50

0.161455

s

<

p95

0.201943

s

<

p99

0.263565

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

11.53

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.49

KiB/s

<

mean

184.01

KiB/s

SD

939.75

KiB/s

<

p95

339.78

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

8.76

KiB/s

<

mean

19.05

KiB/s

SD

24.23

KiB/s

<

p95

53.58

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.41

GiB

mean

1.62

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

149.04

MiB

mean

171.48

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.93 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/4 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

5.3

%

<

mean

19.25

%

SD

19.59

%

<

p95

71.83

%

System

7.72

%

CPU percentiles

CPU usage percentiles

p50

12.35

%

p95

71.83

%

p99

89.46

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

3

hosts

mean time

70.79

s

Memory used percentage

Percentage of memory used

p5

6.87

%

<

mean

23.75

%

SD

13.3

%

<

p95

43.67

%

Memory available percentage

Percentage of available memory

p5

56.34

%

<

mean

76.25

%

SD

13.3

%

<

p95

93.17

%

Max host memory used

Maximum memory usage percentage across all hosts

max

42.63

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.58

%

Memory growth rate

Rate of memory growth over time

growth

1.93

MiB/s

Disk read throughput

Disk read throughput in MiB/s

3053.24

MiB/s

Disk write throughput

Disk write throughput in MiB/s

24.95

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

1/4

hosts

Mount Point /etc/hosts

1/4

hosts

Mount Point /etc/nomad.d/client-meta.json

1/7

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/4

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.34

5 min

1.16

15 min

1.36

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

10.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

6.412

%

SD

9.3307

%

<

p95

27.73

%

60 second average

6.3040

%

300 second average

4.9674

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.8626

%

SD

1.6508

%

<

p95

4.4

%

60 second average

0.8846

%

300 second average

0.9689

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.7833

%

SD

1.5347

%

<

p95

4.25

%

60 second average

0.8051

%

300 second average

0.9041

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

9.4045

%

SD

23.7193

%

<

p95

76.69

%

60 second average

9.2850

%

300 second average

9.3204

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.0117

%

SD

16.4593

%

<

p95

60.43

%

60 second average

6.0307

%

300 second average

6.7493

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

7.09

%

SD

12.54

%

<

p95

36.88

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

207.99

KiB

<

mean

248.45

KiB

SD

104.76

KiB

<

p95

395.88

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

32.95

threads

SD

27.57

threads

<

p95

89

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

64.27

file descriptors

SD

44.68

file descriptors

<

p95

79

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 27 Aug 2026 11:21:35 UTC

Duration

900s

Peer count

10

Peer count at end

8

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_33058136248_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.157463

s

SD

0.470817

s

p50

0.149891

s

<

p95

0.241483

s

<

p99

0.35418

s

Timeout rate

The proportion of remote signal round-trips that timed out.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 883.705043286s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

31787845

over 883.705019832s

mean rate

1.9404786058e+08

/s

std

8.8877626107e+08

/s

p5

0

/s

p95

9.6703860459e+08

/s

peak

1.3835047619e+10

/s

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

13606920

over 883.705043286s

mean rate

7.816990122e+07

/s

std

4.5904044688e+08

/s

p5

0

/s

p95

7.810529078e+07

/s

peak

6.10650887574e+09

/s

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.001948

s

SD

0.000821

s

p50

0.001947

s

<

p95

0.00317

s

<

p99

0.003956

s

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.001896

s

SD

0.000649

s

p50

0.001968

s

<

p95

0.002637

s

<

p99

0.003793

s

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002596

s

SD

0.002242

s

p50

0.00106

s

<

p95

0.007253

s

≤

p99

0.007253

s

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001247

s

SD

0.000641

s

p50

0.001236

s

<

p95

0.002257

s

<

p99

0.003118

s

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001194

s

SD

0.000437

s

p50

0.001237

s

<

p95

0.001801

s

<

p99

0.002389

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00057

s

SD

0.00065

s

p50

0.000209

s

<

p95

0.002119

s

≤

p99

0.002119

s

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

4.6e-05

s

SD

2.4e-05

s

p50

4.8e-05

s

<

p95

8.8e-05

s

<

p99

0.000101

s

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

4.9e-05

s

SD

3.8e-05

s

p50

3.8e-05

s

<

p95

0.000121

s

<

p99

0.000142

s

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

3e-06

s

SD

2e-06

s

p50

4e-06

s

<

p95

7e-06

s

≤

p99

7e-06

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000335

s

SD

0.000501

s

p50

0.000144

s

<

p95

0.00181

s

≤

p99

0.00181

s

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1753

over 883.70504613s

mean rate

4743.15

/s

std

15989.95

/s

p5

0

/s

p95

29434.27

/s

peak

223289.05

/s

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1751

over 883.705144005s

mean rate

5464.27

/s

std

16599.41

/s

p5

0

/s

p95

34019.01

/s

peak

186497.58

/s

Post-commit duration

Time spent executing post-commit workflows.

mean

7.8e-05

s

SD

4.8e-05

s

p50

4.6e-05

s

<

p95

0.000195

s

≤

p99

0.000195

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

47.44

s

<

mean

446.91

s

SD

256.91

s

<

p95

847.49

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

76

over 883.705104784s

mean rate

39654.69

/s

std

1.00979674e+06

/s

p5

0

/s

p95

5014.17

/s

peak

2.839931153e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.169607

s

SD

1.040156

s

p50

0.81157

s

<

p95

3.931983

s

<

p99

4.288898

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.002313

s

SD

0.002056

s

p50

0.002525

s

<

p95

0.006639

s

<

p99

0.008906

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.066036

s

SD

0.034709

s

p50

0.07216

s

<

p95

0.122898

s

<

p99

0.235172

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.00531

s

SD

0.003126

s

p50

0.005885

s

<

p95

0.011002

s

<

p99

0.013597

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.168279

s

SD

0.467894

s

p50

0.023855

s

<

p95

1.678739

s

≤

p99

1.678739

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.205394

s

SD

0.128444

s

p50

0.239925

s

<

p95

0.379017

s

<

p99

0.694794

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000113

s

SD

6.6e-05

s

p50

0.000124

s

<

p95

0.000246

s

<

p99

0.000305

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001054

s

SD

0.000798

s

p50

0.000947

s

<

p95

0.002976

s

<

p99

0.003437

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.00039

s

SD

0.000179

s

p50

0.000427

s

<

p95

0.000643

s

<

p99

0.000977

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.07459

s

SD

0.037605

s

p50

0.074625

s

<

p95

0.171926

s

≤

p99

0.171926

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.143159

s

SD

0.047632

s

p50

0.160887

s

<

p95

0.193734

s

<

p99

0.203378

s

P2P received remote signals

Count of remote signals received.

total

1753

over 883.705004899s

mean rate

10404.01

/s

std

43043.74

/s

p5

0

/s

p95

49857.9

/s

peak

546448.09

/s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

14.61

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

16.05

KiB/s

<

mean

362.62

KiB/s

SD

1.10

MiB/s

<

p95

1.26

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

13.75

KiB/s

<

mean

68.87

KiB/s

SD

207.68

KiB/s

<

p95

295.62

KiB/s

Total bytes received

Total bytes received on primary network interface

count

1.23

GiB

mean

1.42

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

239.45

MiB

mean

275.50

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.6%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.24 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/3 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

5.01

%

<

mean

23.58

%

SD

25.19

%

<

p95

87.45

%

System

8.97

%

CPU percentiles

CPU usage percentiles

p50

14.09

%

p95

87.45

%

p99

95.59

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

2

hosts

mean time

166.85

s

Memory used percentage

Percentage of memory used

p5

6.41

%

<

mean

28.45

%

SD

13.35

%

<

p95

43.77

%

Memory available percentage

Percentage of available memory

p5

56.23

%

<

mean

71.55

%

SD

13.35

%

<

p95

93.59

%

Max host memory used

Maximum memory usage percentage across all hosts

max

43.05

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.58

%

Memory growth rate

Rate of memory growth over time

growth

1.24

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2738.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

23.04

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

1/3

hosts

Mount Point /etc/hosts

1/3

hosts

Mount Point /etc/nomad.d/client-meta.json

1/3

hosts

Mount Point /etc/resolv.conf

1/3

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.76

5 min

1.48

15 min

1.46

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

9.0354

%

SD

14.6455

%

<

p95

47.2

%

60 second average

8.0644

%

300 second average

6.0951

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.796

%

SD

1.6133

%

<

p95

4.34

%

60 second average

0.8099

%

300 second average

0.9025

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.784

%

SD

1.5929

%

<

p95

4.32

%

60 second average

0.7981

%

300 second average

0.8933

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.7617

%

SD

5.6913

%

<

p95

12.28

%

60 second average

1.7414

%

300 second average

1.3906

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.8781

%

SD

2.5775

%

<

p95

8.18

%

60 second average

0.9159

%

300 second average

0.9041

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.04

%

<

mean

7.51

%

SD

16.85

%

<

p95

37.91

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

203.97

KiB

<

mean

270.07

KiB

SD

148.53

KiB

<

p95

396.28

KiB

Holochain process threads

Number of threads in Holochain process

p5

25

threads

<

mean

45.86

threads

SD

34.58

threads

<

p95

89

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

76.78

file descriptors

SD

53.64

file descriptors

<

p95

82

file descriptors

Single-write, many-read

↑ Back to index

Creates an entry in the agent setup, then read one's own record back. This tests the maximum read performance for reading back local data when the chain is short.

Started

Thu, 27 Aug 2026 11:23:24 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

single_write_many_read_33058136248_1

read call timing

The time it takes to complete the zome call that retrieves the record.

mean

0.003009

s

SD

0.001296

s

p50

0.002744

s

<

p95

0.004538

s

<

p99

0.005954

s

read call rate

The number of read zome calls completed per second.

mean

3311.72

/s

Error count

The number of errors collected.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000219

s

SD

5.4e-05

s

p50

0.000212

s

<

p95

0.000234

s

<

p99

0.000504

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001603783s

Not enough time data to show a trend.

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

3112146576

over 280.001603783s

mean rate

1.111474614e+07

/s

std

2.09801381e+06

/s

p5

8.41574525e+06

/s

p95

1.484901572e+07

/s

peak

1.503523804e+07

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001603783s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.001603783s

Not enough time data to show a trend.

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002873

s

SD

0

s

p50

0.002873

s

≤

p95

0.002873

s

≤

p99

0.002873

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.001334

s

SD

0.000202

s

p50

0.001303

s

<

p95

0.001411

s

<

p99

0.002391

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00236

s

SD

0

s

p50

0.00236

s

≤

p95

0.00236

s

≤

p99

0.00236

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000935

s

SD

0.000152

s

p50

0.000913

s

<

p95

0.000989

s

<

p99

0.001733

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000853

s

SD

0

s

p50

0.000853

s

≤

p95

0.000853

s

≤

p99

0.000853

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001297

s

SD

0

s

p50

0.001297

s

≤

p95

0.001297

s

≤

p99

0.001297

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000255

s

SD

2.3e-05

s

p50

0.000279

s

≤

p95

0.000279

s

≤

p99

0.000279

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000241

s

SD

5.6e-05

s

p50

0.000234

s

<

p95

0.000257

s

<

p99

0.00054

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001196

s

SD

0

s

p50

0.001196

s

≤

p95

0.001196

s

≤

p99

0.001196

s

Post-commit duration

Time spent executing post-commit workflows.

mean

4.8e-05

s

SD

0

s

p50

4.8e-05

s

≤

p95

4.8e-05

s

≤

p99

4.8e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

18.07

s

<

mean

148.07

s

SD

83.67

s

<

p95

278.07

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.001608071s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001003

s

SD

0

s

p50

0.001003

s

≤

p95

0.001003

s

≤

p99

0.001003

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.000955

s

SD

0

s

p50

0.000955

s

≤

p95

0.000955

s

≤

p99

0.000955

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.001944

s

SD

0.000301

s

p50

0.001993

s

<

p95

0.002532

s

≤

p99

0.002532

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.002578

s

SD

0

s

p50

0.002578

s

≤

p95

0.002578

s

≤

p99

0.002578

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00243

s

SD

0

s

p50

0.00243

s

≤

p95

0.00243

s

≤

p99

0.00243

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

4.2e-05

s

SD

7e-06

s

p50

4e-05

s

<

p95

4.5e-05

s

<

p99

7.6e-05

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000798

s

SD

0.000163

s

p50

0.001037

s

≤

p95

0.001037

s

≤

p99

0.001037

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000364

s

SD

0

s

p50

0.000364

s

≤

p95

0.000364

s

≤

p99

0.000364

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

31.21

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

18.69

KiB/s

<

mean

309.20

KiB/s

SD

999.24

KiB/s

<

p95

2.86

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

14.35

KiB/s

<

mean

21.99

KiB/s

SD

17.84

KiB/s

<

p95

76.77

KiB/s

Total bytes received

Total bytes received on primary network interface

count

87.57

MiB

mean

309.20

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.23

MiB

mean

21.99

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.55 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

33.89

%

<

mean

45.96

%

SD

16.56

%

<

p95

71.45

%

System

14.75

%

CPU percentiles

CPU usage percentiles

p50

34.26

%

p95

71.45

%

p99

71.51

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

6.36

%

<

mean

6.77

%

SD

0.53

%

<

p95

7.59

%

Memory available percentage

Percentage of available memory

p5

92.41

%

<

mean

93.23

%

SD

0.53

%

<

p95

93.64

%

Max host memory used

Maximum memory usage percentage across all hosts

max

6.77

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.55

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

3.44

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.91

5 min

0.96

15 min

0.87

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

2.12

%

<

mean

8.1817

%

SD

8.8694

%

<

p95

22.61

%

60 second average

5.6387

%

300 second average

2.7560

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

18.17

%

<

mean

28.57

%

SD

7.36

%

<

p95

38.59

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

221.19

KiB

<

mean

227.24

KiB

SD

5.16

KiB

<

p95

234.21

KiB

Holochain process threads

Number of threads in Holochain process

p5

25

threads

<

mean

25.97

threads

SD

1.22

threads

<

p95

27

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

59

file descriptors

<

mean

61.33

file descriptors

SD

2.81

file descriptors

<

p95

64

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 27 Aug 2026 11:36:48 UTC

Duration

900s

Peer count

20

Peer count at end

19

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_33058136248_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

7.453881

s

SD

3.043158

s

Max

Highest per-partition mean latency

max mean

10.542363

s

Min

Lowest per-partition mean latency

min mean

6.309006

s

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.31

/s

Max

Highest per-partition mean rate

max

1.47

/s

Min

Lowest per-partition mean rate

min

1.09

/s

Error count

The total number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

4146420

over 884.118570037s

mean rate

2.17850097e+06

/s

std

6.06541591e+06

/s

p5

0

/s

p95

1.060825164e+07

/s

peak

9.097271709e+07

/s

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

14731210

over 884.118570037s

mean rate

6.13688686e+06

/s

std

2.079695376e+07

/s

p5

55483.37

/s

p95

3.154351311e+07

/s

peak

3.0903891202e+08

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 884.118570037s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

515034

over 886.666756623s

mean rate

267833.42

/s

std

745146.63

/s

p5

0

/s

p95

1.30640715e+06

/s

peak

1.120329838e+07

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009571

s

SD

0.007261

s

p50

0.006653

s

<

p95

0.028301

s

<

p99

0.033129

s

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.028013

s

SD

0.026638

s

p50

0.015174

s

<

p95

0.07103

s

<

p99

0.157524

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008487

s

SD

0.006505

s

p50

0.00593

s

<

p95

0.024225

s

<

p99

0.030171

s

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.026675

s

SD

0.025638

s

p50

0.014087

s

<

p95

0.06635

s

<

p99

0.152187

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002608

s

SD

0.001743

s

p50

0.00196

s

<

p95

0.006698

s

<

p99

0.006954

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002781

s

SD

0.001954

s

p50

0.002034

s

<

p95

0.006902

s

<

p99

0.009926

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001045

s

SD

0.000949

s

p50

0.000536

s

<

p95

0.003951

s

<

p99

0.004548

s

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.023316

s

SD

0.024018

s

p50

0.011266

s

<

p95

0.057652

s

<

p99

0.144571

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004618

s

SD

0.003608

s

p50

0.003234

s

<

p95

0.013074

s

<

p99

0.017149

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000353

s

SD

0.000399

s

p50

0.000205

s

<

p95

0.001587

s

<

p99

0.001803

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

46.5

s

<

mean

443.29

s

SD

255.46

s

<

p95

845.02

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

5731

over 886.666754676s

mean rate

289.96

/s

std

620.84

/s

p5

2.56

/s

p95

1434.63

/s

peak

10642.97

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

1.244649

s

SD

1.218454

s

p50

1.248664

s

<

p95

4.762149

s

<

p99

5.319178

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.00452

s

SD

0.005708

s

p50

0.002564

s

<

p95

0.020255

s

<

p99

0.02213

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.047872

s

SD

0.04705

s

p50

0.052009

s

<

p95

0.175666

s

<

p99

0.203987

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.060675

s

SD

0.056104

s

p50

0.031072

s

<

p95

0.145029

s

<

p99

0.333499

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.037292

s

SD

0.831198

s

p50

0.037777

s

<

p95

0.99818

s

<

p99

5.084405

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

2.127615

s

SD

14.741978

s

p50

7.387682

s

<

p95

41.540176

s

<

p99

56.044564

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001139

s

SD

0.001122

s

p50

0.000651

s

<

p95

0.003376

s

<

p99

0.006434

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.002856

s

SD

0.002822

s

p50

0.001615

s

<

p95

0.010014

s

<

p99

0.014016

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000472

s

SD

0.000444

s

p50

0.000333

s

<

p95

0.001662

s

<

p99

0.002038

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.06513

s

SD

0.05879

s

p50

0.061367

s

<

p95

0.15649

s

<

p99

0.236931

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.170582

s

SD

0.055917

s

p50

0.160584

s

<

p95

0.247855

s

<

p99

0.310168

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

70.78

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

0

B/s

<

mean

700.14

KiB/s

SD

867.11

KiB/s

<

p95

1.08

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

0

B/s

<

mean

518.83

KiB/s

SD

380.77

KiB/s

<

p95

1.26

MiB/s

Total bytes received

Total bytes received on primary network interface

count

11.20

GiB

mean

12.89

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

8.30

GiB

mean

9.55

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 85% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

25.74

%

<

mean

87.95

%

SD

20.65

%

<

p95

98.4

%

System

17.17

%

CPU percentiles

CPU usage percentiles

p50

97.23

%

p95

98.40

%

p99

98.72

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

17

hosts

mean time

885.80

s

Memory used percentage

Percentage of memory used

p5

12.91

%

<

mean

29.21

%

SD

10.36

%

<

p95

46.19

%

Memory available percentage

Percentage of available memory

p5

53.81

%

<

mean

70.79

%

SD

10.36

%

<

p95

87.09

%

Max host memory used

Maximum memory usage percentage across all hosts

max

51.32

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.58

%

Memory growth rate

Rate of memory growth over time

growth

0.82

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2354.45

MiB/s

Disk write throughput

Disk write throughput in MiB/s

164.46

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/10

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

10.41

5 min

7.92

15 min

4.66

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

85.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0.75

%

<

mean

69.9531

%

SD

29.5514

%

<

p95

97.67

%

60 second average

68.0037

%

300 second average

52.5121

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2162

%

SD

0.7353

%

<

p95

1.66

%

60 second average

0.2279

%

300 second average

0.2825

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.123

%

SD

0.4242

%

<

p95

1

%

60 second average

0.1348

%

300 second average

0.1948

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

6.2316

%

SD

12.2385

%

<

p95

45.47

%

60 second average

6.2179

%

300 second average

5.8915

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

0.334

%

SD

0.9788

%

<

p95

2.26

%

60 second average

0.4387

%

300 second average

1.1009

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

28.6

%

SD

17.49

%

<

p95

50.01

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

92.91

KiB

<

mean

278.68

KiB

SD

104.47

KiB

<

p95

410.21

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

33.91

threads

SD

20.48

threads

<

p95

53

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

75.11

file descriptors

SD

31.71

file descriptors

<

p95

99

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 27 Aug 2026 11:38:25 UTC

Duration

900s

Peer count

30

Peer count at end

28

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_33058136248_1

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2111.75

Max

Highest chain head value observed for any single write agent.

2524

Write agents observed

4

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.350739

s

SD

0.338473

s

Max

Highest per-partition mean latency

max mean

1.106427

s

Min

Lowest per-partition mean latency

min mean

0.128853

s

Error count

The number of errors accumulated.

26

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.022941

s

SD

0.072031

s

p50

0.001138

s

<

p95

0.215831

s

<

p99

0.258467

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 270.195581467s

Not enough time data to show a trend.

WASM usage: 'agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 886.35827993s

Not enough time data to show a trend.

WASM usage: 'agent_activity::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

30497568

over 886.35827993s

mean rate

4.616543853e+07

/s

std

5.228852652e+07

/s

p5

1.384158e+06

/s

p95

1.6811596851e+08

/s

peak

1.9715144677e+08

/s

WASM usage: 'agent_activity::get_agent_activity_full'

Metered usage count of the WASM ribosome for this zome function.

total

553837375

over 850.215659019s

mean rate

5.6061445271e+08

/s

std

7.6341425258e+08

/s

p5

2.056548051e+07

/s

p95

2.8470560391e+09

/s

peak

3.90310710979e+09

/s

WASM usage: 'agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

525081947

over 886.324336598s

mean rate

6.0442308816e+08

/s

std

9.6018707154e+08

/s

p5

1.210901865e+07

/s

p95

2.99303105351e+09

/s

peak

4.57314241983e+09

/s

WASM usage: 'agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

39607314

over 888.788816853s

mean rate

5.673437699e+07

/s

std

8.632913174e+07

/s

p5

1.15113593e+06

/s

p95

2.2229489176e+08

/s

peak

6.1305918983e+08

/s

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.014291

s

SD

0.026033

s

p50

0.006581

s

<

p95

0.092597

s

≤

p99

0.092597

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00955

s

SD

0.011726

s

p50

0.013617

s

<

p95

0.046854

s

<

p99

0.049628

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.131416

s

SD

0.07449

s

p50

0.120074

s

<

p95

0.257559

s

<

p99

0.288175

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.008427

s

SD

0.008402

s

p50

0.007557

s

<

p95

0.034771

s

<

p99

0.037253

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.01313

s

SD

0.025764

s

p50

0.004603

s

<

p95

0.090762

s

≤

p99

0.090762

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008443

s

SD

0.010645

s

p50

0.01184

s

<

p95

0.042181

s

<

p99

0.044731

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.130013

s

SD

0.074399

s

p50

0.118939

s

<

p95

0.256321

s

<

p99

0.287196

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007271

s

SD

0.007419

s

p50

0.006637

s

<

p95

0.030503

s

<

p99

0.032872

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00249

s

SD

0.002713

s

p50

0.002727

s

<

p95

0.009695

s

<

p99

0.013929

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

2.5e-05

s

SD

1.4e-05

s

p50

2.4e-05

s

<

p95

6.3e-05

s

≤

p99

6.3e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001555

s

SD

0.002492

s

p50

0.001689

s

<

p95

0.007793

s

<

p99

0.008413

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000511

s

SD

0.000367

s

p50

0.00032

s

<

p95

0.001324

s

≤

p99

0.001324

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.126334

s

SD

0.074632

s

p50

0.115149

s

<

p95

0.252414

s

<

p99

0.282776

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000813

s

SD

0.001113

s

p50

0.000765

s

<

p95

0.003513

s

<

p99

0.005306

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

2.1e-05

s

SD

1.7e-05

s

p50

2.3e-05

s

<

p95

6.1e-05

s

<

p99

6.2e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004069

s

SD

0.00652

s

p50

0.003595

s

<

p95

0.019123

s

<

p99

0.036483

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000392

s

SD

0.002588

s

p50

0.000161

s

<

p95

0.002244

s

<

p99

0.014513

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.4

s

<

mean

432.98

s

SD

251.34

s

<

p95

835.5

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

12062

over 888.788706949s

mean rate

42966.27

/s

std

107568.79

/s

p5

38.93

/s

p95

198174.32

/s

peak

1.70095696e+06

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

55.942244

s

SD

50.557742

s

p50

8.117444

s

<

p95

140.55409

s

<

p99

263.330544

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.05925

s

SD

0.228797

s

p50

0.143369

s

<

p95

0.653214

s

<

p99

1.242354

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.607497

s

SD

0.480303

s

p50

1.042347

s

<

p95

1.491892

s

<

p99

1.875293

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.13437

s

SD

0.091609

s

p50

0.048709

s

<

p95

0.27954

s

<

p99

0.388232

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

4.015485

s

SD

18.756267

s

p50

11.392571

s

<

p95

54.962439

s

<

p99

89.279096

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

22.816865

s

SD

31.907337

s

p50

0.114092

s

<

p95

87.214489

s

<

p99

126.307242

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003165

s

SD

0.001988

s

p50

0.003389

s

<

p95

0.006852

s

<

p99

0.009334

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006259

s

SD

0.010182

s

p50

0.002277

s

<

p95

0.031922

s

<

p99

0.044013

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.001208

s

SD

0.001555

s

p50

0.000712

s

<

p95

0.005539

s

<

p99

0.006774

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.376582

s

SD

26.673025

s

p50

0.218636

s

<

p95

60.003886

s

<

p99

60.113194

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.087188

s

SD

27.951813

s

p50

0.06469

s

<

p95

60.003024

s

<

p99

60.009231

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.205024

s

SD

0.189899

s

p50

0.162335

s

<

p95

0.68684

s

<

p99

0.831745

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

72.26

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

295.82

KiB/s

<

mean

787.30

KiB/s

SD

537.97

KiB/s

<

p95

1.41

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

99.45

KiB/s

<

mean

700.05

KiB/s

SD

436.85

KiB/s

<

p95

1.38

MiB/s

Total bytes received

Total bytes received on primary network interface

count

18.57

GiB

mean

21.36

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

16.51

GiB

mean

19.00

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.7%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.18 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/12 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 90% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

54.05

%

<

mean

88.69

%

SD

16.77

%

<

p95

98.4

%

System

16.43

%

CPU percentiles

CPU usage percentiles

p50

96.78

%

p95

98.40

%

p99

98.72

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

26

hosts

mean time

834.39

s

Memory used percentage

Percentage of memory used

p5

15.52

%

<

mean

28.2

%

SD

9.87

%

<

p95

46.71

%

Memory available percentage

Percentage of available memory

p5

53.29

%

<

mean

71.8

%

SD

9.87

%

<

p95

84.48

%

Max host memory used

Maximum memory usage percentage across all hosts

max

51.65

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.59

%

Memory growth rate

Rate of memory growth over time

growth

1.18

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2117.29

MiB/s

Disk write throughput

Disk write throughput in MiB/s

208.71

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/9

hosts

Mount Point /efi-boot

0/8

hosts

Mount Point /etc/hostname

1/12

hosts

Mount Point /etc/hosts

1/12

hosts

Mount Point /etc/nomad.d/client-meta.json

1/9

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/12

hosts

Mount Point /nix/store

0/9

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

12.71

5 min

9.85

15 min

5.67

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

89.66

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

12.7

%

<

mean

68.3581

%

SD

28.1116

%

<

p95

97.56

%

60 second average

66.6574

%

300 second average

53.1874

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0494

%

SD

0.2572

%

≤

p95

0

%

60 second average

0.0535

%

300 second average

0.0739

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0479

%

SD

0.2538

%

≤

p95

0

%

60 second average

0.0518

%

300 second average

0.0721

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.5526

%

SD

19.4645

%

<

p95

54.98

%

60 second average

15.1962

%

300 second average

12.2508

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

1.9383

%

SD

4.4966

%

<

p95

12.26

%

60 second average

1.9624

%

300 second average

1.8392

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.27

%

<

mean

29.12

%

SD

16.64

%

<

p95

53.32

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

1.70

KiB

<

mean

260.44

KiB

SD

119.45

KiB

<

p95

365.57

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

32.01

threads

SD

17.88

threads

<

p95

52

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

72.03

file descriptors

SD

30.59

file descriptors

<

p95

96

file descriptors

Write/`get_agent_activity` with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 27 Aug 2026 11:40:05 UTC

Duration

1800s

Peer count

30

Peer count at end

22

Behaviours

get_agent_activity_volatile (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_volatile_33058136248_1

Highest observed action_seq

The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.

Mean

mean

32.18

/s

Max

Highest per-partition mean rate

max

59.31

/s

Min

Lowest per-partition mean rate

min

10.09

/s

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.271727

s

SD

0.484379

s

Max

Highest per-partition mean latency

max mean

0.747421

s

Min

Lowest per-partition mean latency

min mean

0.108006

s

Running volatile conductors count

The number of conductors running by get_agent_activity_volatile peers.

p5

9

conductors

<

mean

12.33

conductors

SD

2.17

conductors

<

p95

16

conductors

Volatile conductor total running duration

The total running duration of a get_agent_activity_volatile conductor.

Mean

mean

681.180028

s

SD

358.849967

s

Max

Highest per-partition mean latency

max mean

904.833333

s

Min

Lowest per-partition mean latency

min mean

587.954545

s

Volatile conductor running duration

The duration that a get_agent_activity_volatile conductor was running before being stopped.

Mean

mean

36.69734

s

SD

18.988379

s

Max

Highest per-partition mean latency

max mean

139.984678

s

Min

Lowest per-partition mean latency

min mean

24.227282

s

Volatile conductor stopped duration

The duration that a get_agent_activity_volatile conductor was stopped before being started again.

Mean

mean

22.736074

s

SD

18.32598

s

Max

Highest per-partition mean latency

max mean

35.396098

s

Min

Lowest per-partition mean latency

min mean

7.973688

s

Reached target arc

Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.

get_agent_activity_volatile_agent: uhCAk1os_eaqH4QbCBJjRZr8fTg3UQA4_gWADfq6jtjjhZ4RI2cd8

p5

0

<

mean

0.09

SD

0.29

<

p95

1

get_agent_activity_volatile_agent: uhCAk3LhASPN3fYlR3jNkG9JTXv0-Ynn9Hm-DNdUuUc0k7aeTp6Vc

p5

0

<

mean

0.12

SD

0.33

<

p95

1

get_agent_activity_volatile_agent: uhCAk81tDOw5UeEVaGcHdD2M4v3hruC7K064El42ZXkOxiA9XI30h

p5

0

<

mean

0.05

SD

0.21

≤

p95

0

get_agent_activity_volatile_agent: uhCAkCpCnKBFwOi-Vebin_jeUtHnvuCs1rph-CtwRJPHUMaGKxWQt

p5

0

<

mean

0.06

SD

0.25

<

p95

1

get_agent_activity_volatile_agent: uhCAkG1gnUjOrRR0SQfWyq8wFSkZbvoJn_q-iciRCOWh1SrKVRdQD

p5

0

<

mean

0.15

SD

0.36

<

p95

1

get_agent_activity_volatile_agent: uhCAkG4XRhZTE7rc-OtM4LmLkuQy01e3BbCLibXjgVno1mTXMqmux

p5

0

<

mean

0.17

SD

0.38

<

p95

1

get_agent_activity_volatile_agent: uhCAkRtvTdHWfo5gdPLupTmXj-gjDYyY7X4FAfKM71088WHTnPvh0

p5

0

≤

mean

0

SD

0

≤

p95

0

get_agent_activity_volatile_agent: uhCAkRwBHRJwF4DvNch6Plrg5J4CN0t6l7xMJwiIkIW8O2AjqEGB6

p5

0

≤

mean

0

SD

0

≤

p95

0

get_agent_activity_volatile_agent: uhCAkW50h2Pdj4KBNErLLa0BeJtf2z_a9kznRs8GNVwHC98vxvseE

p5

0

<

mean

0.1

SD

0.3

<

p95

1

get_agent_activity_volatile_agent: uhCAkXHK_IrZan_hZWuHN9fTS9HAxggfu56EAkzkDNBYS-4iyqOSP

p5

0

<

mean

0.12

SD

0.32

<

p95

1

get_agent_activity_volatile_agent: uhCAklz9Hkt4vJK6pie-oO-4PMTkA7rQYx_Gq-CxquxJRINdRXg0t

p5

0

<

mean

0.07

SD

0.25

<

p95

1

get_agent_activity_volatile_agent: uhCAknLWQ2iwUpWzJG4NtCfjgT3QoI9zSS6P0sK-Zocx5muxATsuH

p5

0

<

mean

0.13

SD

0.33

<

p95

1

get_agent_activity_volatile_agent: uhCAknto1iGHkSKNoDwrnL6KsNekJr7YzLBxHRvN5R_fVbtqrumGO

p5

0

<

mean

0.03

SD

0.17

≤

p95

0

get_agent_activity_volatile_agent: uhCAkqDukojcEZjeKbtcz8jOwOSKUDI9JYYqdeuONfnBpukszLT1a

p5

0

<

mean

0.13

SD

0.33

<

p95

1

get_agent_activity_volatile_agent: uhCAkr-BAuAWtLxb3tlQL0LxpFT7ciWZblzr5Bn33Cu-MdMqJCii3

p5

0

<

mean

0.11

SD

0.32

<

p95

1

get_agent_activity_volatile_agent: uhCAkscdfbP7s8JiNvcHsqqBN-il1dIT8V7hXxIF7lBlV6qvYx8KO

p5

0

<

mean

0.16

SD

0.37

<

p95

1

get_agent_activity_volatile_agent: uhCAkvE6ltG-fqfqpkZiLQwh6kXkOoT4-y8p3p8h9STwB0NorTEkj

p5

0

<

mean

0.16

SD

0.37

<

p95

1

get_agent_activity_volatile_agent: uhCAkvwucp6UYdE7bC7UzC_aUPKSYaf2n8y62T1WC5SYPamBzrijQ

p5

0

<

mean

0.14

SD

0.35

<

p95

1

get_agent_activity_volatile_agent: uhCAkwGSKqh45sdw7plIXIQTEzUToaLcrLe6t7db8uGBKEfX-oPTo

p5

0

<

mean

0.12

SD

0.33

<

p95

1

Error count

The number of errors accumulated.

147

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.096012

s

SD

2.466928

s

p50

0.346359

s

<

p95

1.040753

s

<

p99

1.687212

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 1670.855487543s

Not enough time data to show a trend.

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007322

s

SD

0.003844

s

p50

0.005769

s

<

p95

0.01638

s

≤

p99

0.01638

s

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.011641

s

SD

0.007386

s

p50

0.01136

s

<

p95

0.027194

s

<

p99

0.029419

s

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.160189

s

SD

2.530077

s

p50

0.376714

s

<

p95

1.135886

s

<

p99

1.891261

s

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.003521

s

SD

0.345585

s

p50

0.00759

s

<

p95

0.985445

s

≤

p99

0.985445

s

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006357

s

SD

0.003597

s

p50

0.004859

s

<

p95

0.014929

s

≤

p99

0.014929

s

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010355

s

SD

0.00697

s

p50

0.009819

s

<

p95

0.024969

s

<

p99

0.027117

s

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.155949

s

SD

2.515552

s

p50

0.357119

s

<

p95

1.04803

s

<

p99

1.705664

s

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002941

s

SD

0.294125

s

p50

0.006756

s

<

p95

0.782447

s

<

p99

0.812864

s

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003452

s

SD

0.00242

s

p50

0.002872

s

<

p95

0.008313

s

<

p99

0.008727

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.2e-05

s

SD

1.3e-05

s

p50

2.9e-05

s

<

p95

6.5e-05

s

≤

p99

6.5e-05

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001308

s

SD

0.000925

s

p50

0.001206

s

<

p95

0.003397

s

<

p99

0.004032

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000805

s

SD

0.000726

s

p50

0.000486

s

<

p95

0.002613

s

≤

p99

0.002613

s

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.152196

s

SD

2.515861

s

p50

0.350724

s

<

p95

1.040811

s

<

p99

1.700052

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000474

s

SD

0.294318

s

p50

0.000953

s

<

p95

0.77977

s

<

p99

0.8051

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

5e-06

s

SD

5e-06

s

p50

7e-06

s

<

p95

1.6e-05

s

<

p99

2.8e-05

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.005415

s

SD

0.003272

s

p50

0.003292

s

<

p95

0.011558

s

<

p99

0.013263

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.00029

s

SD

0.000153

s

p50

0.000313

s

<

p95

0.000559

s

<

p99

0.000639

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

3.26

s

<

mean

333.11

s

SD

504.36

s

<

p95

1476.35

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

20656

over 1787.593914045s

mean rate

100518.33

/s

std

1.66033647e+06

/s

p5

0

/s

p95

116493.2

/s

peak

5.688384943e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

105.667275

s

SD

129.34174

s

p50

45.928025

s

<

p95

279.727833

s

<

p99

785.815708

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.032282

s

SD

0.116888

s

p50

0.013017

s

<

p95

0.319762

s

<

p99

0.530945

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

<

p99

2e-06

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.071207

s

SD

0.19028

s

p50

0.026646

s

<

p95

0.426599

s

<

p99

0.814449

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.335393

s

SD

0.535461

s

p50

0.092481

s

<

p95

1.140994

s

<

p99

2.940639

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

3.566126

s

SD

25.535899

s

p50

0.840798

s

<

p95

74.826067

s

<

p99

98.387303

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

46.294475

s

SD

41.643434

s

p50

2.997868

s

<

p95

130.305662

s

<

p99

166.041865

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.010245

s

SD

0.004967

s

p50

0.001316

s

<

p95

0.015398

s

<

p99

0.019164

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.037162

s

SD

0.229928

s

p50

0.001644

s

<

p95

0.059552

s

<

p99

0.304595

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000945

s

SD

0.001316

s

p50

0.000564

s

<

p95

0.002291

s

<

p99

0.004019

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

9.021261

s

SD

28.322609

s

p50

0.39924

s

<

p95

60.012589

s

<

p99

60.0193

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.451222

s

SD

0.332481

s

p50

0.685505

s

<

p95

0.81093

s

≤

p99

0.81093

s

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.14547

s

SD

0.310124

s

p50

0.349112

s

<

p95

1.015477

s

<

p99

1.4306

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.580657

s

SD

27.272779

s

p50

0.715613

s

<

p95

60.002088

s

<

p99

60.033838

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

45.39

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

4.02

KiB/s

<

mean

576.14

KiB/s

SD

687.78

KiB/s

<

p95

1.40

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

3.22

KiB/s

<

mean

509.81

KiB/s

SD

497.58

KiB/s

<

p95

1.31

MiB/s

Total bytes received

Total bytes received on primary network interface

count

25.38

GiB

mean

14.52

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

22.46

GiB

mean

12.85

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/6 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 58% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

1.53

%

<

mean

56

%

SD

38.72

%

<

p95

97.92

%

System

10.60

%

CPU percentiles

CPU usage percentiles

p50

66.95

%

p95

97.92

%

p99

98.35

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

16

hosts

mean time

1287.67

s

Memory used percentage

Percentage of memory used

p5

7.5

%

<

mean

18.58

%

SD

9.62

%

<

p95

41.31

%

Memory available percentage

Percentage of available memory

p5

58.69

%

<

mean

81.42

%

SD

9.62

%

<

p95

92.5

%

Max host memory used

Maximum memory usage percentage across all hosts

max

43.28

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.23

%

Memory growth rate

Rate of memory growth over time

growth

0.39

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2.09

MiB/s

Disk write throughput

Disk write throughput in MiB/s

90.23

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/18

hosts

Mount Point /boot

0/1

hosts

Mount Point /efi-boot

0/16

hosts

Mount Point /etc/hostname

1/6

hosts

Mount Point /etc/hosts

1/6

hosts

Mount Point /etc/nomad.d/client-meta.json

1/3

hosts

Mount Point /etc/resolv.conf

1/6

hosts

Mount Point /nix/store

0/18

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.65

5 min

7.14

15 min

5.26

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

57.69

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

33.6546

%

SD

34.637

%

<

p95

86.55

%

60 second average

33.4596

%

300 second average

30.9345

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.3671

%

SD

18.8919

%

<

p95

54.3

%

60 second average

15.2579

%

300 second average

14.2997

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

2.8416

%

SD

5.5967

%

<

p95

14.47

%

60 second average

2.8370

%

300 second average

2.7040

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

23

%

SD

28.11

%

<

p95

95.22

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

45.00

KiB

<

mean

206.85

KiB

SD

139.94

KiB

<

p95

416.29

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

28.74

threads

SD

15.39

threads

<

p95

46

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

7

file descriptors

<

mean

67.5

file descriptors

SD

26.73

file descriptors

<

p95

87

file descriptors

Write/Query

↑ Back to index

Creates an entry, then queries the source chain and performs a simple operation on the entries, then repeats.

Started

Thu, 27 Aug 2026 11:25:03 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_query_33058136248_1

Write timing

The time it takes to complete the zome function call that writes the entry.

mean

0.033702

s

SD

0.025678

s

p50

0.031805

s

<

p95

0.05506

s

<

p99

0.123887

s

Write rate

The number of writes completed per second.

mean

90.29

/s

Query timing

The time it takes to complete the chain_query_count_len zome call that queries the source chain.

mean

0.073349

s

SD

0.044946

s

p50

0.071431

s

<

p95

0.135911

s

<

p99

0.166574

s

Query rate

The number of chain queries completed per second.

mean

90.32

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'crud::chain_query_count_len'

Metered usage count of the WASM ribosome for this zome function.

total

76408152976

over 270.009091641s

mean rate

2.8298369747e+08

/s

std

2.497553692e+07

/s

p5

2.1974597649e+08

/s

p95

3.070350213e+08

/s

peak

3.0846807979e+08

/s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

82427970

over 270.009091641s

mean rate

305278.52

/s

std

132527.71

/s

p5

193050.28

/s

p95

557613.53

/s

peak

772086.79

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 270.009091641s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

23365816

over 280.009562162s

mean rate

83446.4

/s

std

53534.71

/s

p5

47974.28

/s

p95

190165.12

/s

peak

306364.78

/s

Zome call duration: 'crud::chain_query_count_len'

Duration of this zome call as measured by Holochain internally.

mean

0.029322

s

SD

0.008722

s

p50

0.027585

s

<

p95

0.038203

s

<

p99

0.038997

s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003385

s

SD

1.7e-05

s

p50

0.003386

s

<

p95

0.00339

s

<

p99

0.003391

s

WASM call duration: 'crud::chain_query_count_len'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.028693

s

SD

0.008694

s

p50

0.026958

s

<

p95

0.037549

s

<

p99

0.038341

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002825

s

SD

1.2e-05

s

p50

0.002827

s

<

p95

0.002829

s

<

p99

0.00283

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001798

s

SD

0

s

p50

0.001798

s

≤

p95

0.001798

s

≤

p99

0.001798

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000987

s

SD

0.000173

s

p50

0.000983

s

<

p95

0.000996

s

<

p99

0.00193

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000276

s

SD

3.7e-05

s

p50

0.000348

s

≤

p95

0.000348

s

≤

p99

0.000348

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.011941

s

SD

0.003694

s

p50

0.011291

s

<

p95

0.015581

s

<

p99

0.01589

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001504

s

SD

3.5e-05

s

p50

0.001479

s

<

p95

0.001563

s

<

p99

0.001567

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000117

s

SD

8e-06

s

p50

0.000119

s

<

p95

0.000121

s

<

p99

0.000122

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

16.77

s

<

mean

146.78

s

SD

83.67

s

<

p95

276.78

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.009506507s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000711

s

SD

0

s

p50

0.000711

s

≤

p95

0.000711

s

≤

p99

0.000711

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.00172

s

SD

0.000153

s

p50

0.001722

s

<

p95

0.001746

s

<

p99

0.00245

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.024926

s

SD

0.007806

s

p50

0.022912

s

<

p95

0.031946

s

<

p99

0.032525

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003541

s

SD

0

s

p50

0.003541

s

≤

p95

0.003541

s

≤

p99

0.003541

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000337

s

SD

0

s

p50

0.000337

s

≤

p95

0.000337

s

≤

p99

0.000337

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002359

s

SD

0.000799

s

p50

0.002185

s

<

p95

0.003026

s

<

p99

0.00308

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001596

s

SD

0.000258

s

p50

0.001083

s

<

p95

0.001631

s

<

p99

0.001646

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000246

s

SD

3.2e-05

s

p50

0.000247

s

<

p95

0.000249

s

<

p99

0.000418

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

51.53

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

10.10

KiB/s

<

mean

12.09

KiB/s

SD

2.29

KiB/s

<

p95

16.04

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

12.19

KiB/s

<

mean

13.95

KiB/s

SD

1.62

KiB/s

<

p95

17.48

KiB/s

Total bytes received

Total bytes received on primary network interface

count

3.42

MiB

mean

12.09

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

3.95

MiB

mean

13.95

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 100% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

64.22

%

<

mean

70.25

%

SD

2.74

%

<

p95

73.02

%

System

18.72

%

CPU percentiles

CPU usage percentiles

p50

71.25

%

p95

73.02

%

p99

74.14

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

0

hosts

mean time

0.00

s

Memory used percentage

Percentage of memory used

p5

21.76

%

<

mean

22.06

%

SD

0.69

%

<

p95

22.63

%

Memory available percentage

Percentage of available memory

p5

77.37

%

<

mean

77.94

%

SD

0.69

%

<

p95

78.24

%

Max host memory used

Maximum memory usage percentage across all hosts

max

22.06

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-0.22

MiB/s

Disk read throughput

Disk read throughput in MiB/s

256.78

MiB/s

Disk write throughput

Disk write throughput in MiB/s

1.92

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.04

5 min

3.48

15 min

3.46

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

100.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

21.19

%

<

mean

27.7537

%

SD

3.2732

%

<

p95

30.56

%

60 second average

24.8183

%

300 second average

16.1657

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.84

%

<

mean

3.165

%

SD

0.8364

%

<

p95

4.78

%

60 second average

3.3580

%

300 second average

3.8250

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

1.24

%

<

mean

1.8843

%

SD

0.4606

%

<

p95

2.66

%

60 second average

2.2417

%

300 second average

3.1720

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

67.61

%

<

mean

71.539

%

SD

3.0193

%

<

p95

76.96

%

60 second average

72.5870

%

300 second average

76.3320

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

20.92

%

<

mean

24.2597

%

SD

6.0093

%

<

p95

33.99

%

60 second average

31.0643

%

300 second average

50.2763

%

Holochain process CPU usage

CPU usage by Holochain process

p5

51.38

%

<

mean

57.96

%

SD

2.98

%

<

p95

60.98

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.19

KiB

<

mean

213.07

KiB

SD

53.35

KiB

<

p95

238.46

KiB

Holochain process threads

Number of threads in Holochain process

p5

8

threads

<

mean

22.97

threads

SD

4.97

threads

<

p95

26

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

27

file descriptors

<

mean

62.55

file descriptors

SD

12.32

file descriptors

<

p95

70

file descriptors

Write/Read

↑ Back to index

Creates an entry and reads it back, then repeats.

Started

Thu, 27 Aug 2026 11:26:46 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_read_33058136248_1

Write timing

The time taken to write the entry.

mean

0.043746

s

SD

0.026849

s

p50

0.03638

s

<

p95

0.090066

s

<

p99

0.104336

s

Write rate

The number of writes completed per second.

mean

108.62

/s

Read timing

The time taken to read the entry.

mean

0.046642

s

SD

0.029569

s

p50

0.039964

s

<

p95

0.098088

s

<

p99

0.115217

s

Read rate

The number of reads completed per second.

mean

108.62

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.000522

s

SD

8e-05

s

p50

0.000501

s

<

p95

0.000616

s

<

p99

0.000624

s

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

106627335

over 280.005170817s

mean rate

380805.99

/s

std

251449.87

/s

p5

196561.01

/s

p95

890045.08

/s

peak

1.31538612e+06

/s

WASM usage: 'crud::get_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

99517585

over 280.005170817s

mean rate

355414.42

/s

std

234404.88

/s

p5

183434.94

/s

p95

830609.14

/s

peak

1.22527787e+06

/s

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 280.005170817s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

13131166

over 280.005170817s

mean rate

46896.29

/s

std

30966.07

/s

p5

24206.51

/s

p95

109609.13

/s

peak

161989.92

/s

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003036

s

SD

0.000123

s

p50

0.002985

s

<

p95

0.003191

s

<

p99

0.003201

s

Zome call duration: 'crud::get_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.002065

s

SD

0.000139

s

p50

0.002017

s

<

p95

0.00224

s

<

p99

0.002258

s

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00247

s

SD

0.000106

s

p50

0.002427

s

<

p95

0.002605

s

<

p99

0.002614

s

WASM call duration: 'crud::get_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.0015

s

SD

0.00012

s

p50

0.00146

s

<

p95

0.001648

s

<

p99

0.001663

s

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001128

s

SD

0

s

p50

0.001128

s

≤

p95

0.001128

s

≤

p99

0.001128

s

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000785

s

SD

3.1e-05

s

p50

0.00077

s

<

p95

0.000826

s

<

p99

0.000828

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000285

s

SD

2.1e-05

s

p50

0.000248

s

<

p95

0.000309

s

<

p99

0.000313

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.000555

s

SD

8.1e-05

s

p50

0.000533

s

<

p95

0.000649

s

<

p99

0.000658

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001259

s

SD

5.4e-05

s

p50

0.001234

s

<

p95

0.00133

s

<

p99

0.001334

s

Post-commit duration

Time spent executing post-commit workflows.

mean

8.3e-05

s

SD

5e-06

s

p50

8.2e-05

s

<

p95

9e-05

s

<

p99

9.2e-05

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.56

s

<

mean

147.56

s

SD

83.67

s

<

p95

277.57

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.005161355s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.000775

s

SD

0

s

p50

0.000775

s

≤

p95

0.000775

s

≤

p99

0.000775

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.001136

s

SD

0.000134

s

p50

0.001088

s

<

p95

0.001283

s

<

p99

0.001295

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.030202

s

SD

0.010637

s

p50

0.027591

s

<

p95

0.041118

s

<

p99

0.042065

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003725

s

SD

0

s

p50

0.003725

s

≤

p95

0.003725

s

≤

p99

0.003725

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.00094

s

SD

0

s

p50

0.00094

s

≤

p95

0.00094

s

≤

p99

0.00094

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002607

s

SD

0.000919

s

p50

0.002387

s

<

p95

0.003503

s

<

p99

0.003575

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.000941

s

SD

0.000253

s

p50

0.001392

s

≤

p95

0.001392

s

≤

p99

0.001392

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000252

s

SD

2.2e-05

s

p50

0.000245

s

<

p95

0.000277

s

<

p99

0.00028

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

61.01

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

19.43

KiB/s

<

mean

766.48

KiB/s

SD

1.87

MiB/s

<

p95

4.65

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

14.59

KiB/s

<

mean

34.49

KiB/s

SD

37.87

KiB/s

<

p95

108.10

KiB/s

Total bytes received

Total bytes received on primary network interface

count

217.07

MiB

mean

766.48

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

9.77

MiB

mean

34.49

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.48 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

69.17

%

<

mean

79.49

%

SD

8.07

%

<

p95

90

%

System

18.49

%

CPU percentiles

CPU usage percentiles

p50

79.22

%

p95

90.00

%

p99

94.04

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

134.48

s

Memory used percentage

Percentage of memory used

p5

7.41

%

<

mean

8.29

%

SD

0.73

%

<

p95

9.33

%

Memory available percentage

Percentage of available memory

p5

90.67

%

<

mean

91.71

%

SD

0.73

%

<

p95

92.59

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.29

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

2.48

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.00

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.54

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.29

5 min

2.39

15 min

1.44

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

17.73

%

<

mean

30.3363

%

SD

11.8723

%

<

p95

48.06

%

60 second average

23.7497

%

300 second average

11.3347

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.57

%

<

mean

30.7

%

SD

9.01

%

<

p95

41.41

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

19.08

KiB

<

mean

260.10

KiB

SD

126.29

KiB

<

p95

443.89

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

25.03

threads

SD

7.8

threads

<

p95

32

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

7

file descriptors

<

mean

56.76

file descriptors

SD

18.98

file descriptors

<

p95

70

file descriptors

Write validated

↑ Back to index

Creates an entry, updates it and links the create to the update, then repeats. Each of the three actions is validated by the included hApp.

Started

Thu, 27 Aug 2026 11:28:26 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_33058136248_1

Write timing

The time it takes to complete the zome function call that writes the entry, updates it, and links to the update.

mean

0.081987

s

SD

0.040242

s

p50

0.087484

s

<

p95

0.13416

s

<

p99

0.191584

s

Write rate

The number of write zome function calls that complete per second.

mean

46.61

/s

Read timing

The time it takes to complete the zome function call that reads the data.

mean

0.121913

s

SD

0.053932

s

p50

0.132232

s

<

p95

0.193426

s

<

p99

0.264658

s

Read rate

The number of read zome function calls that complete per second.

mean

46.61

/s

Error count

The number of errors accumulated over the run.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.005835

s

SD

0.002129

s

p50

0.005461

s

<

p95

0.008905

s

<

p99

0.009196

s

WASM usage: 'validated::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

43096700

over 270.00304346s

mean rate

159615.54

/s

std

71296.09

/s

p5

103019.11

/s

p95

312492.09

/s

peak

391472.27

/s

WASM usage: 'validated::update_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

50664108

over 270.00304346s

mean rate

187642.65

/s

std

83578.42

/s

p5

125245.12

/s

p95

367655.95

/s

peak

456538.25

/s

WASM usage: 'validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

83738728

over 280.003132s

mean rate

299063.46

/s

std

150195.29

/s

p5

186876.39

/s

p95

691442.01

/s

peak

739248.05

/s

WASM usage: 'validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

758138391

over 270.00304346s

mean rate

2.80788713e+06

/s

std

1.25435559e+06

/s

p5

1.85889822e+06

/s

p95

5.51199932e+06

/s

peak

6.86326064e+06

/s

Zome call duration: 'validated::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.003846

s

SD

0.000305

s

p50

0.00383

s

<

p95

0.004082

s

<

p99

0.004091

s

Zome call duration: 'validated::update_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.010933

s

SD

0.002668

s

p50

0.010447

s

<

p95

0.013331

s

<

p99

0.013459

s

WASM call duration: 'validated::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003179

s

SD

0.000272

s

p50

0.003164

s

<

p95

0.00339

s

<

p99

0.003398

s

WASM call duration: 'validated::update_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.010277

s

SD

0.002633

s

p50

0.009795

s

<

p95

0.01265

s

<

p99

0.01278

s

WASM call duration: 'validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.0009

s

SD

0.000133

s

p50

0.000901

s

<

p95

0.000944

s

<

p99

0.001544

s

WASM call duration: 'validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003563

s

SD

0.000455

s

p50

0.003501

s

<

p95

0.003933

s

<

p99

0.003944

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000396

s

SD

4.9e-05

s

p50

0.00039

s

<

p95

0.000436

s

<

p99

0.000438

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000353

s

SD

3.6e-05

s

p50

0.000346

s

<

p95

0.000381

s

<

p99

0.000382

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.005362

s

SD

0.001648

s

p50

0.005004

s

<

p95

0.006911

s

<

p99

0.007021

s

Host function call duration: '__hc__update_1'

Duration of this host function call invoked from within WASM.

mean

0.007222

s

SD

0.002405

s

p50

0.006735

s

<

p95

0.009469

s

<

p99

0.009609

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.001491

s

SD

0.000137

s

p50

0.001522

s

<

p95

0.001758

s

<

p99

0.001778

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000111

s

SD

7e-06

s

p50

0.000111

s

<

p95

0.000116

s

≤

p99

0.000116

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.55

s

<

mean

147.55

s

SD

83.67

s

<

p95

277.55

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.003131514s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001127

s

SD

0

s

p50

0.001127

s

≤

p95

0.001127

s

≤

p99

0.001127

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.00149

s

SD

0.000196

s

p50

0.0015

s

<

p95

0.001552

s

<

p99

0.002317

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.050082

s

SD

0.018995

s

p50

0.044791

s

<

p95

0.066553

s

<

p99

0.067663

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003798

s

SD

0

s

p50

0.003798

s

≤

p95

0.003798

s

≤

p99

0.003798

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.000778

s

SD

0

s

p50

0.000778

s

≤

p95

0.000778

s

≤

p99

0.000778

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.003699

s

SD

0.001417

s

p50

0.003348

s

<

p95

0.004884

s

<

p99

0.004969

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001536

s

SD

0.00042

s

p50

0.002269

s

≤

p95

0.002269

s

≤

p99

0.002269

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000306

s

SD

5e-05

s

p50

0.000304

s

<

p95

0.000331

s

<

p99

0.000508

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

64.81

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

19.44

KiB/s

<

mean

817.18

KiB/s

SD

1.92

MiB/s

<

p95

6.19

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

14.96

KiB/s

<

mean

37.74

KiB/s

SD

39.32

KiB/s

<

p95

137.35

KiB/s

Total bytes received

Total bytes received on primary network interface

count

231.43

MiB

mean

817.18

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.69

MiB

mean

37.74

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.61 MiB/s

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

74.87

%

<

mean

83.21

%

SD

6.11

%

<

p95

92.35

%

System

18.40

%

CPU percentiles

CPU usage percentiles

p50

83.16

%

p95

92.35

%

p99

94.04

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

175.86

s

Memory used percentage

Percentage of memory used

p5

8.18

%

<

mean

8.97

%

SD

1.21

%

<

p95

11.64

%

Memory available percentage

Percentage of available memory

p5

88.36

%

<

mean

91.03

%

SD

1.21

%

<

p95

91.82

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.97

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

1.61

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.11

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.85

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.30

5 min

3.27

15 min

1.86

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

23.06

%

<

mean

35.801

%

SD

10.6599

%

<

p95

49.29

%

60 second average

33.2293

%

300 second average

18.2510

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

16.49

%

<

mean

25.72

%

SD

8.64

%

<

p95

41.2

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

22.00

KiB

<

mean

315.76

KiB

SD

317.87

KiB

<

p95

974.38

KiB

Holochain process threads

Number of threads in Holochain process

p5

9

threads

<

mean

38.97

threads

SD

58.15

threads

<

p95

172

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

8

file descriptors

<

mean

85.41

file descriptors

SD

124.13

file descriptors

<

p95

372

file descriptors

Write Validated `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started

Thu, 27 Aug 2026 12:05:30 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

must_get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_must_get_agent_activity_33058136248_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1468

Max

Highest chain head value observed for any single write agent.

1809

Write agents observed

4

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

200.797925

s

SD

84.457455

s

Max

Highest per-partition mean latency

max mean

629.801538

s

Min

Lowest per-partition mean latency

min mean

5.72422

s

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

3.42

/s

Max

Highest per-partition mean rate

max

5.71

/s

Min

Lowest per-partition mean rate

min

0

/s

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.350882

s

SD

0.468354

s

Max

Highest per-partition mean latency

max mean

0.954204

s

Min

Lowest per-partition mean latency

min mean

0.039852

s

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

4921

Agents affected

16 / 16

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

308

Max

Highest count in any single partition

743

Min

Lowest count in any single partition

6

Final error count

The total number of all types of error accumulated over the run by all agents.

52432

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.024726

s

SD

0.027656

s

p50

0.001297

s

<

p95

0.07302

s

<

p99

0.120396

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

0

over 811.028087251s

Not enough time data to show a trend.

WASM usage: 'validated_must_get_agent_activity::announce_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

0

over 885.706508356s

Not enough time data to show a trend.

WASM usage: 'validated_must_get_agent_activity::create_sample_entries_batch'

Metered usage count of the WASM ribosome for this zome function.

total

2517656453

over 885.706508356s

mean rate

6.5150030932e+08

/s

std

9.2654028432e+08

/s

p5

4983.32

/s

p95

2.75274073118e+09

/s

peak

4.48311645793e+09

/s

WASM usage: 'validated_must_get_agent_activity::create_validated_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

8817715

over 883.8281979s

mean rate

4.726752706e+07

/s

std

1.579909286e+08

/s

p5

394287.84

/s

p95

2.2514836799e+08

/s

peak

4.35501915111e+09

/s

WASM usage: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Metered usage count of the WASM ribosome for this zome function.

total

475303227

over 883.8281979s

mean rate

1.21064303529e+09

/s

std

2.51755044562e+09

/s

p5

7.69676505e+06

/s

p95

7.24855855822e+09

/s

peak

1.70852488545e+10

/s

WASM usage: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Metered usage count of the WASM ribosome for this zome function.

total

28617

over 883.888141435s

mean rate

2.25990472e+06

/s

std

3.33014627e+06

/s

p5

0

/s

p95

1.053522858e+07

/s

peak

1.592526947e+07

/s

WASM usage: 'validated_must_get_agent_activity_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

240975636

over 886.998768533s

mean rate

1.56856458813e+09

/s

std

2.27205332977e+10

/s

p5

5.3876431e+06

/s

p95

2.43996478336e+09

/s

peak

6.45389555523e+11

/s

WASM usage: 'validated_must_get_agent_activity_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

7907755080

over 885.707285046s

mean rate

1.85894689373e+11

/s

std

3.27917621064e+12

/s

p5

2.964051857e+07

/s

p95

1.85992137994e+11

/s

peak

1.03530464223e+14

/s

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.004179

s

SD

0.002184

s

p50

0.003909

s

<

p95

0.009465

s

≤

p99

0.009465

s

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.116694

s

SD

0.143219

s

p50

0.066395

s

<

p95

0.474361

s

<

p99

0.754129

s

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008729

s

SD

0.004089

s

p50

0.006624

s

<

p95

0.014411

s

<

p99

0.028202

s

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.010457

s

SD

0.005981

s

p50

0.006847

s

<

p95

0.022177

s

<

p99

0.032172

s

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.009625

s

SD

0.023117

s

p50

0.005824

s

<

p95

0.008938

s

<

p99

0.112232

s

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.003392

s

SD

0.001704

s

p50

0.003223

s

<

p95

0.007458

s

≤

p99

0.007458

s

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.115365

s

SD

0.141926

s

p50

0.065254

s

<

p95

0.468947

s

<

p99

0.74726

s

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007666

s

SD

0.003816

s

p50

0.005737

s

<

p95

0.013115

s

<

p99

0.025917

s

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008

s

SD

0.005521

s

p50

0.005931

s

<

p95

0.016528

s

<

p99

0.037326

s

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008663

s

SD

0.023175

s

p50

0.004853

s

<

p95

0.007685

s

<

p99

0.111715

s

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001672

s

SD

0.000914

s

p50

0.001627

s

<

p95

0.003494

s

<

p99

0.005464

s

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.012986

s

SD

0.014942

s

p50

0.011071

s

<

p95

0.033213

s

<

p99

0.074046

s

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

3.4e-05

s

SD

3.8e-05

s

p50

2.2e-05

s

<

p95

0.000153

s

<

p99

0.000159

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000943

s

SD

0.000871

s

p50

0.000592

s

<

p95

0.001669

s

<

p99

0.004603

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.001099

s

SD

0.001384

s

p50

0.000411

s

<

p95

0.001728

s

<

p99

0.008316

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001229

s

SD

0.016954

s

p50

0.000705

s

<

p95

0.003462

s

<

p99

0.109783

s

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.028112

s

SD

0.031999

s

p50

0.021577

s

<

p95

0.096447

s

<

p99

0.179455

s

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.049068

s

SD

0.077406

s

p50

0.019005

s

<

p95

0.227191

s

<

p99

0.401755

s

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

3e-05

s

SD

8.9e-05

s

p50

7e-06

s

<

p95

2e-05

s

<

p99

0.000422

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002776

s

SD

0.002031

s

p50

0.002653

s

<

p95

0.005874

s

<

p99

0.012747

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000268

s

SD

0.000333

s

p50

0.000204

s

<

p95

0.001108

s

<

p99

0.001622

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.58

s

<

mean

436.52

s

SD

252.12

s

<

p95

828.33

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

28432

over 886.998658456s

mean rate

302303.16

/s

std

3.97710034e+06

/s

p5

347.99

/s

p95

366128.2

/s

peak

9.361351063e+07

/s

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

53.163149

s

SD

39.279799

s

p50

41.273996

s

<

p95

142.505462

s

<

p99

187.701546

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

1.206456

s

SD

2.321061

s

p50

2.267959

s

<

p95

7.207958

s

<

p99

12.739868

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

1e-06

s

SD

3e-06

s

p50

0

s

≤

p95

0

s

<

p99

1.7e-05

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.177495

s

SD

0.323137

s

p50

0.273736

s

<

p95

1.105701

s

<

p99

1.344037

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.208546

s

SD

0.198864

s

p50

0.113682

s

<

p95

0.492523

s

<

p99

1.113387

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

1.004416

s

SD

13.347041

s

p50

6.512451

s

<

p95

41.933574

s

<

p99

69.236248

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

20.899915

s

SD

29.053638

s

p50

10.873623

s

<

p95

92.468593

s

<

p99

130.864479

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004181

s

SD

0.003083

s

p50

0.003091

s

<

p95

0.010142

s

<

p99

0.012774

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.012168

s

SD

0.008675

s

p50

0.002351

s

<

p95

0.027016

s

<

p99

0.034378

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000681

s

SD

0.000751

s

p50

0.000455

s

<

p95

0.00187

s

<

p99

0.004658

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.622076

s

SD

29.132333

s

p50

0.185676

s

<

p95

60.00201

s

<

p99

60.002347

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.327965

s

SD

0

s

p50

0.327965

s

≤

p95

0.327965

s

≤

p99

0.327965

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.218009

s

SD

4.304206

s

p50

0.228248

s

<

p95

15.471981

s

<

p99

16.577918

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

62.58

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

54.80

KiB/s

<

mean

407.27

KiB/s

SD

862.08

KiB/s

<

p95

1.31

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

25.01

KiB/s

<

mean

260.94

KiB/s

SD

229.08

KiB/s

<

p95

606.87

KiB/s

Total bytes received

Total bytes received on primary network interface

count

10.26

GiB

mean

11.80

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.57

GiB

mean

7.56

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 2.05 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 90% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

19.63

%

<

mean

77.61

%

SD

24.59

%

<

p95

98.55

%

System

15.03

%

CPU percentiles

CPU usage percentiles

p50

86.66

%

p95

98.55

%

p99

98.77

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

27

hosts

mean time

606.49

s

Memory used percentage

Percentage of memory used

p5

14.71

%

<

mean

26.09

%

SD

9.91

%

<

p95

44.44

%

Memory available percentage

Percentage of available memory

p5

55.56

%

<

mean

73.91

%

SD

9.91

%

<

p95

85.29

%

Max host memory used

Maximum memory usage percentage across all hosts

max

50.06

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.48

%

Memory growth rate

Rate of memory growth over time

growth

2.05

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2224.36

MiB/s

Disk write throughput

Disk write throughput in MiB/s

184.58

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/13

hosts

Mount Point /efi-boot

0/12

hosts

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/7

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

Mount Point /nix/store

0/13

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

8.90

5 min

8.64

15 min

8.45

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

90.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.27

%

<

mean

44.7713

%

SD

31.6799

%

<

p95

96.87

%

60 second average

44.6193

%

300 second average

45.7383

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.2116

%

SD

0.988

%

<

p95

1

%

60 second average

0.2255

%

300 second average

0.2533

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1066

%

SD

0.5607

%

<

p95

1

%

60 second average

0.1189

%

300 second average

0.1350

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

17.7831

%

SD

18.4769

%

<

p95

51.56

%

60 second average

17.5988

%

300 second average

16.6441

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

3.6492

%

SD

5.9764

%

<

p95

14.93

%

60 second average

3.6515

%

300 second average

3.2258

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.35

%

<

mean

25.16

%

SD

14.12

%

<

p95

44.15

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

50.56

KiB

<

mean

280.75

KiB

SD

115.39

KiB

<

p95

471.07

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

32.86

threads

SD

13.85

threads

<

p95

49

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

71.15

file descriptors

SD

23.05

file descriptors

<

p95

89

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 27 Aug 2026 12:27:57 UTC

Duration

900s

Peer count

30

Peer count at end

27

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_33058136248_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

47.22

/s

Max

Highest per-partition mean rate

max

80.51

/s

Min

Lowest per-partition mean rate

min

33.72

/s

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

386.967978

s

SD

0.697338

s

Max

Highest per-partition mean latency

max mean

540.801157

s

Min

Lowest per-partition mean latency

min mean

213.668056

s

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0.0083

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

50954

Agents affected

12 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4246

Max

Highest count in any single partition

7448

Min

Lowest count in any single partition

2963

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

5088

Agents affected

12 / 12

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

424

Max

Highest count in any single partition

424

Min

Lowest count in any single partition

424

Open connections

The number of currently open connections to other conductors.

full-arc

p5

0

<

mean

10.98

SD

9.68

<

p95

27

zero-arc

p5

3

<

mean

11.21

SD

5.28

<

p95

20

Error count

The number of errors accumulated across all nodes.

82

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.019227

s

SD

0.348439

s

p50

0.101197

s

<

p95

0.558587

s

<

p99

2.418126

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

1

over 671.775836823s

mean rate

0.72

/s

std

0.47

/s

p5

0

/s

p95

1.27

/s

peak

1.28

/s

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

235549609

over 872.30499952s

mean rate

2.2704000197e+08

/s

std

2.5393993881e+08

/s

p5

427143.81

/s

p95

7.848927469e+08

/s

peak

8.2845558715e+08

/s

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

1345278594

over 871.534077895s

mean rate

3.28790804364e+09

/s

std

7.92791777841e+09

/s

p5

880634.49

/s

p95

2.26099091107e+10

/s

peak

8.72438971622e+10

/s

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

30121296

over 887.42160859s

mean rate

4.5787135436e+08

/s

std

1.34873405398e+09

/s

p5

112766.78

/s

p95

2.34871624598e+09

/s

peak

2.74544879863e+10

/s

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.010061

s

SD

0.005756

s

p50

0.009541

s

<

p95

0.026333

s

<

p99

0.027026

s

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

0.96089

s

SD

2.190257

s

p50

0.775582

s

<

p95

4.772787

s

<

p99

11.340082

s

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.009228

s

SD

0.005453

s

p50

0.008617

s

<

p95

0.024536

s

<

p99

0.025145

s

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.960207

s

SD

2.18985

s

p50

0.774927

s

<

p95

4.771872

s

<

p99

11.337453

s

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001848

s

SD

0.001122

s

p50

0.001638

s

<

p95

0.00438

s

<

p99

0.004902

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000655

s

SD

0.000454

s

p50

0.000634

s

<

p95

0.001945

s

<

p99

0.00209

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000628

s

SD

0.000462

s

p50

0.000655

s

<

p95

0.002012

s

<

p99

0.002113

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.012037

s

SD

0.146233

s

p50

0.032464

s

<

p95

0.310081

s

<

p99

0.609643

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.399789

s

SD

0.754206

s

p50

0.3905

s

<

p95

2.832625

s

<

p99

3.598758

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002873

s

SD

0.001837

s

p50

0.002539

s

<

p95

0.007831

s

<

p99

0.008278

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000202

s

SD

0.000295

s

p50

0.000146

s

<

p95

0.001113

s

<

p99

0.001339

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

44.06

s

<

mean

432.97

s

SD

250.88

s

<

p95

827.21

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 887.421667935s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

13.39279

s

SD

32.155517

s

p50

0

s

<

p95

36.749894

s

<

p99

192.622831

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.169962

s

SD

0.14385

s

p50

0.002008

s

<

p95

0.224914

s

<

p99

0.926628

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.034119

s

SD

0.370373

s

p50

0.002094

s

<

p95

1.171684

s

<

p99

1.605227

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.137788

s

SD

0.068321

s

p50

0.023233

s

<

p95

0.198321

s

<

p99

0.219595

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.444707

s

SD

6.933538

s

p50

0.002899

s

<

p95

10.555544

s

<

p99

37.040803

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

1.991626

s

SD

4.185983

s

p50

0.001479

s

<

p95

0.131021

s

<

p99

28.528669

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002392

s

SD

0.001931

s

p50

0.001137

s

<

p95

0.00557

s

<

p99

0.006453

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.010626

s

SD

0.012294

s

p50

0.003721

s

<

p95

0.03742

s

<

p99

0.056154

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000498

s

SD

0.000568

s

p50

0.000534

s

<

p95

0.002318

s

<

p99

0.002367

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.441894

s

SD

29.721038

s

p50

0.304198

s

<

p95

60.002732

s

<

p99

60.006369

s

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.218698

s

SD

29.052026

s

p50

0.261155

s

<

p95

60.003659

s

<

p99

60.004641

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.076543

s

SD

0.117248

s

p50

0.229534

s

<

p95

0.381921

s

≤

p99

0.381921

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

58.23

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.32

KiB/s

<

mean

310.49

KiB/s

SD

595.48

KiB/s

<

p95

1.06

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

6.19

KiB/s

<

mean

214.36

KiB/s

SD

180.81

KiB/s

<

p95

463.61

KiB/s

Total bytes received

Total bytes received on primary network interface

count

7.27

GiB

mean

8.36

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.02

GiB

mean

5.77

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 89% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

16.29

%

<

mean

73.49

%

SD

27.21

%

<

p95

98.65

%

System

15.26

%

CPU percentiles

CPU usage percentiles

p50

83.49

%

p95

98.65

%

p99

98.75

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

24

hosts

mean time

594.66

s

Memory used percentage

Percentage of memory used

p5

12.25

%

<

mean

28.51

%

SD

10.11

%

<

p95

45.55

%

Memory available percentage

Percentage of available memory

p5

54.45

%

<

mean

71.49

%

SD

10.11

%

<

p95

87.75

%

Max host memory used

Maximum memory usage percentage across all hosts

max

52.40

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.49

%

Memory growth rate

Rate of memory growth over time

growth

0.68

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.56

MiB/s

Disk write throughput

Disk write throughput in MiB/s

145.12

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/10

hosts

Mount Point /efi-boot

0/9

hosts

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/8

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

Mount Point /nix/store

0/10

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.41

5 min

7.18

15 min

7.31

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

89.29

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

42.5695

%

SD

32.9738

%

<

p95

92.6

%

60 second average

43.5711

%

300 second average

42.2540

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

12.8372

%

SD

19.0873

%

<

p95

56.16

%

60 second average

12.8606

%

300 second average

12.3205

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

5.9718

%

SD

11.3749

%

<

p95

34.69

%

60 second average

5.9228

%

300 second average

5.8491

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.02

%

<

mean

16.08

%

SD

17.08

%

<

p95

43.61

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.12

KiB

<

mean

285.20

KiB

SD

132.33

KiB

<

p95

487.63

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

30.51

threads

SD

13.7

threads

<

p95

45

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

66.6

file descriptors

SD

22.71

file descriptors

<

p95

87

file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 27 Aug 2026 12:26:09 UTC

Duration

900s

Peer count

30

Peer count at end

29

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data_validated--high_zero_33058136248_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

43.13

/s

Max

Highest per-partition mean rate

max

91.19

/s

Min

Lowest per-partition mean rate

min

22.64

/s

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

119.851145

s

SD

1.749548

s

Max

Highest per-partition mean latency

max mean

119.851145

s

Min

Lowest per-partition mean latency

min mean

119.851145

s

Not enough time data to show a trend.

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

0

/s

Max

Highest per-partition mean rate

max

0

/s

Min

Lowest per-partition mean rate

min

0

/s

Not enough time data to show a trend.

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0032

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

107381

Agents affected

28 / 28

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3835

Max

Highest count in any single partition

8186

Min

Lowest count in any single partition

1941

Received count

Total number of entries received by full-arc nodes, by agent.

Total

678

Agents affected

1 / 2

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

339

Max

Highest count in any single partition

678

Min

Lowest count in any single partition

0

Open connections

The number of currently open connections to other conductors.

full-arc

p5

9

<

mean

21.89

SD

7.33

<

p95

29

zero-arc

p5

2

<

mean

13.08

SD

6.08

<

p95

22

Error count

The number of errors accumulated across all nodes.

0

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.005578

s

SD

0.005251

s

p50

0.008069

s

<

p95

0.018052

s

<

p99

0.022617

s

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

573

over 770.426436555s

mean rate

485.72

/s

std

651.65

/s

p5

0

/s

p95

1849.76

/s

peak

2163.17

/s

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

185817758

over 880.758495344s

mean rate

4.0999488133e+08

/s

std

7.8562310343e+08

/s

p5

1.584430407e+07

/s

p95

2.08092233382e+09

/s

peak

4.98703839304e+09

/s

WASM usage: 'timed_and_validated::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

173853515

over 880.42915168s

mean rate

3.6293043818e+08

/s

std

2.3960592381e+08

/s

p5

3.92032488e+06

/s

p95

7.0639609749e+08

/s

peak

7.5305107854e+08

/s

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

103026276

over 887.195693534s

mean rate

2.5005726668e+08

/s

std

4.7038168189e+08

/s

p5

3.56833742e+06

/s

p95

1.19716034057e+09

/s

peak

4.21729855866e+09

/s

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

862008886

over 885.395119144s

mean rate

2.73854849867e+09

/s

std

8.50079113794e+09

/s

p5

5.345853373e+07

/s

p95

1.07968519708e+10

/s

peak

1.24335532276e+11

/s

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008529

s

SD

0.004823

s

p50

0.008226

s

<

p95

0.023424

s

<

p99

0.024937

s

Zome call duration: 'timed_and_validated::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.05693

s

SD

0.058502

s

p50

0.016041

s

<

p95

0.177465

s

<

p99

0.193468

s

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007742

s

SD

0.004436

s

p50

0.007397

s

<

p95

0.020734

s

<

p99

0.023106

s

WASM call duration: 'timed_and_validated::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.055698

s

SD

0.058666

s

p50

0.014308

s

<

p95

0.176503

s

<

p99

0.192498

s

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001611

s

SD

0.000982

s

p50

0.001395

s

<

p95

0.003853

s

<

p99

0.005379

s

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006618

s

SD

0.003686

s

p50

0.005397

s

<

p95

0.014551

s

<

p99

0.017556

s

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000514

s

SD

0.000358

s

p50

0.000468

s

<

p95

0.00156

s

<

p99

0.001883

s

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000503

s

SD

0.000385

s

p50

0.000476

s

<

p95

0.001751

s

<

p99

0.001968

s

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.002751

s

SD

0.000259

s

p50

0.00263

s

<

p95

0.003017

s

<

p99

0.003019

s

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.00202

s

SD

0.001545

s

p50

0.001103

s

<

p95

0.005256

s

<

p99

0.005719

s

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.00935

s

SD

0.005066

s

p50

0.008629

s

<

p95

0.018337

s

<

p99

0.023236

s

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002611

s

SD

0.001633

s

p50

0.002377

s

<

p95

0.006586

s

<

p99

0.007937

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000189

s

SD

0.000173

s

p50

0.000183

s

<

p95

0.000513

s

<

p99

0.000998

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

45.58

s

<

mean

435.93

s

SD

251.89

s

<

p95

834.91

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 887.195688651s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

6.030895

s

SD

19.400232

s

p50

0

s

<

p95

0.302554

s

<

p99

114.978777

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

5.762398

s

SD

2.350032

s

p50

0.001619

s

<

p95

3.854766

s

<

p99

14.236839

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.010198

s

SD

0.269402

s

p50

0.001797

s

<

p95

0.942057

s

<

p99

1.324004

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.112931

s

SD

0.05351

s

p50

0.102106

s

<

p95

0.190262

s

<

p99

0.240034

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

7.443437

s

SD

3.83557

s

p50

0.003125

s

<

p95

2.301059

s

<

p99

22.277753

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

4.011314

s

SD

5.312998

s

p50

0.001468

s

<

p95

0.012558

s

<

p99

38.445664

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004338

s

SD

0.002457

s

p50

0.004216

s

<

p95

0.008544

s

<

p99

0.010686

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.011004

s

SD

0.016855

s

p50

0.002605

s

<

p95

0.049069

s

<

p99

0.085203

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000394

s

SD

0.00029

s

p50

0.000357

s

<

p95

0.001194

s

<

p99

0.001314

s

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.710488

s

SD

29.891024

s

p50

0.042603

s

<

p95

60.011738

s

<

p99

60.014727

s

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.094654

s

SD

0.039502

s

p50

0.014767

s

<

p95

0.095006

s

<

p99

0.096388

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

61.76

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

25.38

KiB/s

<

mean

378.24

KiB/s

SD

753.24

KiB/s

<

p95

1.26

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

42.48

KiB/s

<

mean

234.54

KiB/s

SD

168.12

KiB/s

<

p95

499.46

KiB/s

Total bytes received

Total bytes received on primary network interface

count

9.14

GiB

mean

10.64

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.67

GiB

mean

6.60

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.8%

Memory leak anomaly

Memory leak anomaly was detected

Detected

Warning Memory growing at 1.60 MiB/s

Disk full anomaly

Disk full anomaly was detected

Detected

Critical Disk /etc/resolv.conf: 1/11 hosts above 90% capacity

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

Detected

Warning 93% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

28.75

%

<

mean

78.63

%

SD

22.19

%

<

p95

98.6

%

System

16.88

%

CPU percentiles

CPU usage percentiles

p50

85.11

%

p95

98.60

%

p99

98.75

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

26

hosts

mean time

650.24

s

Memory used percentage

Percentage of memory used

p5

17.63

%

<

mean

29.59

%

SD

9.56

%

<

p95

47.04

%

Memory available percentage

Percentage of available memory

p5

52.96

%

<

mean

70.41

%

SD

9.56

%

<

p95

82.37

%

Max host memory used

Maximum memory usage percentage across all hosts

max

53.16

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

1.67

%

Memory growth rate

Rate of memory growth over time

growth

1.60

MiB/s

Disk read throughput

Disk read throughput in MiB/s

2166.49

MiB/s

Disk write throughput

Disk write throughput in MiB/s

168.58

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/10

hosts

Mount Point /efi-boot

0/9

hosts

Mount Point /etc/hostname

1/11

hosts

Mount Point /etc/hosts

1/11

hosts

Mount Point /etc/nomad.d/client-meta.json

1/10

hosts

Mount Point /etc/nomad.d/meta.json

0/1

hosts

Mount Point /etc/resolv.conf

1/11

hosts

Mount Point /nix/store

0/10

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

7.78

5 min

7.32

15 min

7.33

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

93.33

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

3.12

%

<

mean

46.8891

%

SD

30.5777

%

<

p95

92.47

%

60 second average

46.1594

%

300 second average

42.9847

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.1766

%

SD

0.8098

%

<

p95

1

%

60 second average

0.1796

%

300 second average

0.1821

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

<

mean

0.0789

%

SD

0.3086

%

<

p95

1

%

60 second average

0.0826

%

300 second average

0.0889

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

15.9404

%

SD

21.9678

%

<

p95

62.81

%

60 second average

15.6978

%

300 second average

15.0931

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

<

mean

5.8664

%

SD

11.0031

%

<

p95

33.47

%

60 second average

5.9305

%

300 second average

6.1376

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0.03

%

<

mean

19.22

%

SD

15.67

%

<

p95

44.37

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

24.51

KiB

<

mean

306.67

KiB

SD

139.68

KiB

<

p95

564.31

KiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

31.2

threads

SD

13.46

threads

<

p95

44

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

6

file descriptors

<

mean

68.08

file descriptors

SD

22.18

file descriptors

<

p95

86

file descriptors

Zome Call, Single Value

↑ Back to index

Calls a zome function that returns a fixed value. This tests the maximum performance of zome calls when the zome function is not calling into Holochain other otherwise doing any work.

Started

Thu, 27 Aug 2026 11:30:10 UTC

Duration

300s

Peer count

1

Peer count at end

1

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zome_call_single_value_33058136248_1

Call timing

The time taken for the call to complete.

mean

0.003511

s

SD

0.002226

s

p50

0.002859

s

<

p95

0.006677

s

<

p99

0.010721

s

Call rate

The number of zome calls completed per second.

mean

2821.5

/s

Error count

The number of errors accumulated during the run.

0

Holochain MetricsClick to toggle visibility.

WASM usage: 'return_single_value::get_value'

Metered usage count of the WASM ribosome for this zome function.

total

312291776

over 280.001974964s

mean rate

1.11532012e+06

/s

std

392695.1

/s

p5

718438.43

/s

p95

1.75111435e+06

/s

peak

1.77710119e+06

/s

Zome call duration: 'return_single_value::get_value'

Duration of this zome call as measured by Holochain internally.

mean

0.001427

s

SD

0.000129

s

p50

0.00156

s

<

p95

0.001654

s

<

p99

0.001677

s

WASM call duration: 'return_single_value::get_value'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.000893

s

SD

8.9e-05

s

p50

0.000983

s

<

p95

0.001058

s

<

p99

0.001072

s

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000113

s

SD

0

s

p50

0.000113

s

≤

p95

0.000113

s

≤

p99

0.000113

s

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

p5

17.22

s

<

mean

147.22

s

SD

83.67

s

<

p95

277.22

s

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

0

over 280.001979186s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

App validation workflow duration

Time spent running the app validation workflow.

mean

0.001988

s

SD

0

s

p50

0.001988

s

≤

p95

0.001988

s

≤

p99

0.001988

s

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

0

s

SD

0

s

p50

0

s

≤

p95

0

s

≤

p99

0

s

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.002153

s

SD

0

s

p50

0.002153

s

≤

p95

0.002153

s

≤

p99

0.002153

s

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.003763

s

SD

0.00065

s

p50

0.004034

s

<

p95

0.004968

s

≤

p99

0.004968

s

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.003967

s

SD

0

s

p50

0.003967

s

≤

p95

0.003967

s

≤

p99

0.003967

s

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.002061

s

SD

0

s

p50

0.002061

s

≤

p95

0.002061

s

≤

p99

0.002061

s

DHT DB connection use time

Time spent holding DHT database connections.

mean

7.8e-05

s

SD

1.2e-05

s

p50

9.2e-05

s

<

p95

9.9e-05

s

<

p99

0.0001

s

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.001695

s

SD

0.000454

s

p50

0.002319

s

≤

p95

0.002319

s

≤

p99

0.002319

s

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000376

s

SD

0

s

p50

0.000376

s

≤

p95

0.000376

s

≤

p99

0.000376

s

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

52.61

%

Network receive rate (primary)

Rate of bytes received on primary network interface

p5

20.67

KiB/s

<

mean

759.27

KiB/s

SD

1.77

MiB/s

<

p95

6.15

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

p5

15.22

KiB/s

<

mean

35.24

KiB/s

SD

36.78

KiB/s

<

p95

135.68

KiB/s

Total bytes received

Total bytes received on primary network interface

count

215.03

MiB

mean

759.27

KiB/s

Total bytes sent

Total bytes sent on primary network interface

count

9.98

MiB

mean

35.24

KiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 94.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

NotDetected

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

p5

34.01

%

<

mean

70.45

%

SD

21.81

%

<

p95

92.36

%

System

17.84

%

CPU percentiles

CPU usage percentiles

p50

75.46

%

p95

92.36

%

p99

94.01

%

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

1

hosts

mean time

124.14

s

Memory used percentage

Percentage of memory used

p5

6.04

%

<

mean

8.78

%

SD

1.81

%

<

p95

13.1

%

Memory available percentage

Percentage of available memory

p5

86.9

%

<

mean

91.22

%

SD

1.81

%

<

p95

93.96

%

Max host memory used

Maximum memory usage percentage across all hosts

max

8.78

%

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

0.00

%

Memory growth rate

Rate of memory growth over time

growth

-3.12

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.11

MiB/s

Disk write throughput

Disk write throughput in MiB/s

5.26

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /etc/nomad.d/client-meta.json

0/1

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

5.37

5 min

3.95

15 min

2.27

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

%

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

p5

1.87

%

<

mean

29.5867

%

SD

17.6844

%

<

p95

49.29

%

60 second average

32.2123

%

300 second average

22.8537

%

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

p5

0

%

≤

mean

0

%

SD

0

%

≤

p95

0

%

60 second average

0.0000

%

300 second average

0.0000

%

Holochain process CPU usage

CPU usage by Holochain process

p5

0

%

<

mean

16.6

%

SD

11.06

%

<

p95

35.08

%

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

p5

23.40

KiB

<

mean

338.85

KiB

SD

442.34

KiB

<

p95

1.75

MiB

Holochain process threads

Number of threads in Holochain process

p5

1

threads

<

mean

49.38

threads

SD

88.44

threads

<

p95

339

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

p5

0

file descriptors

<

mean

108.16

file descriptors

SD

189.67

file descriptors

<

p95

721

file descriptors

App Install

App Install

DHT Sync Lag

First Call

Full-Arc Create (Validated) / Zero-Arc Read

Local Signals

Remote Call Rate

Remote Signals

Single-write, many-read

Validation Receipts

Write/get_agent_activity

Write/get_agent_activity with volatile conductors

Write/Query

Write/Read

Write validated

Write Validated must_get_agent_activity

Zero-Arc Create and Read

Zero-arc Create Data (Validated)

Zome Call, Single Value

Write/`get_agent_activity`

Write/`get_agent_activity` with volatile conductors

Write Validated `must_get_agent_activity`