DHT Sync Lag

↑ Back to index

Measures the lag time between an agent publishing data and other peers being able to see it. This scenario has two roles:

write: Creates entries with a timestamp field, linked to a known base hash so that record_lag agents can discover them.
record_lag: Repeatedly queries for links from the known base hash. When a new entry is discovered, it calculates the time difference between the entry's timestamp and the current time, giving the sync lag for that entry.

Started

Thu, 24 Sep 2026 03:02:26 UTC

Duration

1800s

Peer count

150

Behaviours

record_lag (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

dht_sync_lag_35946754846_1

Create rate

The average number of created records per agent.

Mean

mean

11.66

Max

Highest per-partition mean rate

max

Min

Lowest per-partition mean rate

min

4.82

Sync lag timing

The average time between when a record was created and when it is first seen by an agent.

Mean

mean

2.7374132551499996e+28

7.188511838219999e+29

Max

Highest per-partition mean latency

max mean

4.9273438592699995e+29

Min

Lowest per-partition mean latency

min mean

28.643265

Sync lag rate

The average number of created records discovered per agent.

Mean

mean

112.03

Max

Highest per-partition mean rate

max

1381

Min

Lowest per-partition mean rate

min

Delivery ratio

The proportion of created entries successfully received by all record_lag agents.

0.0025

Sent count

The total number of entries sent (created) by write agents, by agent.

Total

274204

Agents affected

126 / 126

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

2176

Max

Highest count in any single partition

7355

Min

Lowest count in any single partition

933

Received count

The total number of entries received by record_lag agents, by agent.

Total

15557

Agents affected

18 / 23

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

676

Max

Highest count in any single partition

6906

Min

Lowest count in any single partition

Error count

The number of errors encountered during the scenario.

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.002107

0.002527

p50

0.0022

p95

0.008165

p99

0.009651

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 1650.076035288s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

65966880

over 1790.316740159s

mean rate

6.03882765094e+09

std

1.17753058813e+11

1.654735048e+07

p95

9.10153018975e+09

peak

8.65855391544e+12

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

4201908240

over 1788.326621486s

mean rate

8.36042295931e+10

std

5.75428742008e+11

2.0246266533e+08

p95

2.08563164084e+11

peak

9.86490525409e+12

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

8897490

over 1790.316740159s

mean rate

1.26485659138e+09

std

1.79616451961e+10

4.03769164e+06

p95

3.30001838509e+09

peak

1.16784963723e+12

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.038588

0.039902

p50

0.039316

p95

0.129949

p99

0.155805

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.105291

0.19287

p50

0.058818

p95

0.555879

p99

0.842991

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.036695

0.038696

p50

0.037154

p95

0.125449

p99

0.151022

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.10391

0.19287

p50

0.058045

p95

0.554035

p99

0.842069

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002377

0.001339

p50

0.002466

p95

0.005358

p99

0.006174

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.010701

0.013423

p50

0.010623

p95

0.041657

p99

0.051133

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.010471

0.013627

p50

0.009965

p95

0.041595

p99

0.052548

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.002093

0.003341

p50

0.003116

p95

0.009966

p99

0.014928

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.004456

0.006634

p50

0.003332

p95

0.01783

p99

0.031013

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.00612

0.005274

p50

0.00623

p95

0.018263

p99

0.021609

Post-commit duration

Time spent executing post-commit workflows.

mean

0.003185

0.003768

p50

0.002462

p95

0.011378

p99

0.015906

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

85.44

mean

879.1

508.71

p95

1673.64

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

124.62359

130.630642

p50

6.885329

p95

273.030428

p99

722.291076

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.383531

1.182151

p50

0.362581

p95

3.288084

p99

5.443462

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.342728

0.484416

p50

0.255112

p95

1.446568

p99

1.641475

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

3.167628

4.379747

p50

2.606522

p95

12.432362

p99

16.927226

System validation workflow duration

Time spent running the sys validation workflow.

mean

27.022513

57.051265

p50

31.19999

p95

162.090644

p99

294.163777

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

23.279879

42.722519

p50

0.392189

p95

101.743044

p99

233.267866

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004686

0.003517

p50

0.006016

p95

0.011902

p99

0.012937

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.009561

0.008534

p50

0.004402

p95

0.025512

p99

0.031012

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.010645

0.013156

p50

0.008441

p95

0.040088

p99

0.050806

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

3.062541

28.037301

p50

3.162423

p95

60.060492

p99

60.114983

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

5.804908

27.271225

p50

6.903511

p95

60.060951

p99

61.939146

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

55.14

Network receive rate (primary)

Rate of bytes received on primary network interface

128.97

KiB/s

mean

1.22

MiB/s

772.02

KiB/s

p95

2.53

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

13.43

KiB/s

mean

147.08

KiB/s

165.48

KiB/s

p95

401.52

KiB/s

Total bytes received

Total bytes received on primary network interface

count

313.57

GiB

mean

179.39

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

36.99

GiB

mean

21.16

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (20.8% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 95% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

43.96

mean

75.98

17.22

p95

95.89

System

20.85

CPU percentiles

CPU usage percentiles

p50

81.43

p95

95.89

p99

98.35

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

129

hosts

mean time

1118.34

Memory used percentage

Percentage of memory used

8.68

mean

13.19

7.49

p95

25.01

Memory available percentage

Percentage of available memory

74.99

mean

86.81

7.49

p95

91.32

Max host memory used

Maximum memory usage percentage across all hosts

max

81.01

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

20.82

Memory growth rate

Rate of memory growth over time

growth

0.36

MiB/s

Disk read throughput

Disk read throughput in MiB/s

4.12

MiB/s

Disk write throughput

Disk write throughput in MiB/s

143.11

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/14

hosts

Mount Point /etc/hosts

0/14

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/14

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.95

5 min

3.46

15 min

2.46

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

94.63

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

3.61

mean

48.5231

18.5952

p95

70.41

60 second average

47.1251

300 second average

41.2460

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

0.04

mean

11.3108

7.6293

p95

22.6

60 second average

11.0707

300 second average

9.6473

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

3.3564

3.7339

p95

8.64

60 second average

3.2968

300 second average

2.8972

Holochain process CPU usage

CPU usage by Holochain process

47.64

mean

72.45

14.43

p95

93.59

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

250.98

KiB

mean

445.16

KiB

133.33

KiB

p95

651.61

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

35.2

threads

5.14

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

85.77

file descriptors

10.68

file descriptors

p95

file descriptors

Mixed Arc `get_agent_activity`

↑ Back to index

A mixed-arc scenario that measures the DHT's ability to propagate agent activity and make it available via get_agent_activity calls. The scenario has three roles:

zero_write: A zero-arc conductor that creates entries on its source chain.
full_write: A full-arc conductor that creates entries on its source chain.
zero_read: A zero-arc conductor that picks a random writer and repeatedly calls get_agent_activity to observe new entries appearing on the writer's chain, measuring the lag between creation and discovery.

Started

Thu, 24 Sep 2026 06:44:32 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

full_write (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_get_agent_activity_35946754846_1

Entry created count

The total number of entries created by write agents over the run, partitioned by agent and behaviour.

Total

535040

Agents affected

106 / 106

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

5048

Max

Highest count in any single partition

10759

Min

Lowest count in any single partition

1588

Observed chain advancement

The maximum highest-observed action sequence per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any reader successfully tracking that point counts as propagation). Captures how far readers tracked writers’ chains during the run.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2791

Max

Highest chain head value observed for any single write agent.

4659

Write agents observed

Chain head delay timing

The time lag between when a writer creates a new entry and when a reader discovers it via get_agent_activity. This is measured per new action in the chain.

Mean

mean

0.125203

0.736496

Max

Highest per-partition mean latency

max mean

1.872235

Min

Lowest per-partition mean latency

min mean

-0.0651

Chain head delay rate

The rate at which readers detect new chain heads appearing on writers' chains.

Mean

mean

11.67

Max

Highest per-partition mean rate

max

61.28

Min

Lowest per-partition mean rate

min

1.33

get_agent_activity_full zome call timing

The time taken to complete the get_agent_activity_full zome call that queries a writer's chain.

Mean

mean

0.5586

1.086279

Max

Highest per-partition mean latency

max mean

0.951766

Min

Lowest per-partition mean latency

min mean

0.061477

Retrieval errors

A running accumulation of errors encountered by read agents when calling get_agent_activity.

Total

299

Agents affected

42 / 43

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

Max

Highest count in any single partition

Min

Lowest count in any single partition

Open connections

The number of currently open connections to other conductors, partitioned by agent behaviour.

behaviour: full_write

mean

37.33

41.11

p95

146

behaviour: zero_read

mean

25.39

15.97

p95

behaviour: zero_write

mean

23.63

15.09

p95

Final error count

The total number of errors accumulated across all agents over the run.

319

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.407084

6.671438

p50

1.616096

p95

22.255479

p99

33.503034

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 800.973818757s

mean rate

117.47

std

366.75

p95

537.6

peak

11880.32

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007938

0.003076

p50

0.007656

p95

0.011242

p99

0.020881

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00609

0.003145

p50

0.006212

p95

0.011126

p99

0.021003

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.232615

1.763118

p50

0.541875

p95

2.879475

p99

8.7871

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.068151

1.830912

p50

1.193998

p95

5.748661

p99

6.879163

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006585

0.002752

p50

0.006236

p95

0.009662

p99

0.018082

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005037

0.002947

p50

0.00503

p95

0.010061

p99

0.019515

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.414123

5.027948

p50

1.120913

p95

13.454939

p99

26.969813

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.2642

7.814842

p50

2.599384

p95

22.37714

p99

33.508959

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001562

0.000446

p50

0.001635

p95

0.002306

p99

0.00301

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

6.4e-05

4.4e-05

p50

5e-05

p95

0.000158

p99

0.000212

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00084

0.001932

p50

0.000535

p95

0.00418

p99

0.011008

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000655

0.000317

p50

0.000606

p95

0.001061

p99

0.001354

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.411978

5.027694

p50

1.118026

p95

13.449789

p99

26.967171

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.262322

7.814591

p50

2.594131

p95

22.371934

p99

33.50418

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

4e-06

p50

1.5e-05

p95

2.1e-05

p99

2.8e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002488

0.00155

p50

0.002776

p95

0.004427

p99

0.0069

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000596

0.000671

p50

0.000513

p95

0.001947

p99

0.003148

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.51

mean

430.68

249.07

p95

817.05

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

over 890.3312653s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

31.850422

30.035473

p50

p95

68.11649

p99

136.719549

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.133431

0.1729

p50

0.003121

p95

0.373963

p99

0.923437

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.016465

0.074139

p50

0.002574

p95

0.107645

p99

0.46686

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.120272

0.06691

p50

0.076834

p95

0.205114

p99

0.254591

System validation workflow duration

Time spent running the sys validation workflow.

mean

18.621806

17.879191

p50

0.004204

p95

47.30791

p99

78.859508

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

1.397054

8.024771

p50

0.001838

p95

0.057609

p99

14.858962

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004817

0.004099

p50

0.003811

p95

0.011232

p99

0.013667

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.005411

0.006387

p50

0.003164

p95

0.008132

p99

0.01821

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000791

0.001619

p50

0.000582

p95

0.002728

p99

0.009401

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

19.648434

28.35882

p50

11.405604

p95

60.023013

p99

61.285535

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

1.022361

1.835053

p50

1.063703

p95

5.736293

p99

6.867363

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.51172

29.450587

p50

1.155367

p95

60.001562

p99

60.001778

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

1.396724

20.868121

p50

2.008504

p95

60.013099

p99

60.016892

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

44.15

Network receive rate (primary)

Rate of bytes received on primary network interface

5.19

KiB/s

mean

714.30

KiB/s

688.59

KiB/s

p95

1.85

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

B/s

mean

84.52

KiB/s

146.58

KiB/s

p95

394.96

KiB/s

Total bytes received

Total bytes received on primary network interface

count

89.11

GiB

mean

102.53

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.54

GiB

mean

12.13

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.0%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (72.2% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 33% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

2.57

mean

57.44

36.68

p95

94.82

System

13.29

CPU percentiles

CPU usage percentiles

p50

78.25

p95

94.82

p99

97.95

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

124

hosts

mean time

496.05

Memory used percentage

Percentage of memory used

7.11

mean

11.97

7.34

p95

23.85

Memory available percentage

Percentage of available memory

76.15

mean

88.03

7.34

p95

92.89

Max host memory used

Maximum memory usage percentage across all hosts

max

82.32

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

72.15

Memory growth rate

Rate of memory growth over time

growth

0.38

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.92

MiB/s

Disk write throughput

Disk write throughput in MiB/s

83.70

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.29

5 min

1.81

15 min

1.50

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

33.11

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

0.01

mean

28.0718

25.3118

p95

79.91

60 second average

26.4880

300 second average

20.9302

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0002

0.0194

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0002

0.0194

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

3.0286

4.0556

p95

8.83

60 second average

2.9148

300 second average

2.4251

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.2743

1.4705

p95

3.74

60 second average

1.2453

300 second average

1.0171

Holochain process CPU usage

CPU usage by Holochain process

mean

54.91

35.46

p95

89.9

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

204.33

KiB

mean

276.85

KiB

66.25

KiB

p95

404.15

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

29.79

threads

5.67

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

75.55

file descriptors

11.14

file descriptors

p95

file descriptors

Mixed Arc `must_get_agent_activity`

↑ Back to index

A scenario where write agents create entries in batches of 10, while must_get_agent_activity agents each pick a random write agent and repeatedly attempt to create an entry that references the chain top of their latest batch. This reference means that the entry's validation function needs to make a must_get_agent_activity call.

The purpose of this scenario is to measure the time it takes for published agent activity data to be gossiped among authorities and become available to peers that query it via must_get_agent_activity.

This test is similar to Write Validated must_get_agent_activity, but some write agents are zero-arc and all must_get_agent_activity agents are zero-arc. This forces the must_get_agent_activity host call to go to the network.

Started

Thu, 24 Sep 2026 07:01:36 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

full_write (1 agent)
zero_must_get_agent_activity (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

mixed_arc_must_get_agent_activity_35946754846_1

Highest observed action_seq

The maximum chain length observed per write agent, aggregated across all reading agents. The reading dimension is collapsed by taking the maximum observed value for each write agent (any successful read counts as propagation). This reflects the DHT’s ability to propagate agent activity ops and make them available to querying peers.

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

1288.33

Max

Highest chain head value observed for any single write agent.

1710

Write agents observed

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

134.850773

97.942854

Max

Highest per-partition mean latency

max mean

288.055652

Min

Lowest per-partition mean latency

min mean

15.373546

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

1.31

Max

Highest per-partition mean rate

max

2.58

Min

Lowest per-partition mean rate

min

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

1.338089

1.92567

Max

Highest per-partition mean latency

max mean

9.740088

Min

Lowest per-partition mean latency

min mean

0.100073

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

84273

Agents affected

43 / 43

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

1960

Max

Highest count in any single partition

6353

Min

Lowest count in any single partition

Open connections

The average number of open connections maintained by an agent over the run, broken down by behaviour.

behaviour: full_write

mean

80.25

33.07

p95

148

behaviour: zero_must_get_agent_activity

mean

37.43

19.45

p95

behaviour: zero_write

mean

35.61

18.75

p95

Final error count

The total number of all types of error accumulated over the run by all agents.

99065

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.105642

4.14882

p50

0.163255

p95

2.755891

p99

33.822739

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.010538

0.008145

p50

0.008778

p95

0.021847

p99

0.050521

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.207047

0.220494

p50

0.091101

p95

0.669603

p99

1.073256

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.007558

0.001626

p50

0.007487

p95

0.010399

p99

0.011107

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.600061

10.540367

p50

0.646981

p95

32.628261

p99

40.952576

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.014049

0.412485

p50

0.154811

p95

0.902457

p99

1.624152

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.008891

0.00775

p50

0.00735

p95

0.019829

p99

0.048936

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.205024

0.219729

p50

0.089372

p95

0.666061

p99

1.06858

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006227

0.001386

p50

0.006125

p95

0.008527

p99

0.009457

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.492502

7.977634

p50

0.521891

p95

19.045285

p99

40.951329

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013595

0.412424

p50

0.153591

p95

0.901017

p99

1.622745

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002277

0.000918

p50

0.001952

p95

0.003946

p99

0.005968

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.029016

0.069903

p50

0.009738

p95

0.117148

p99

0.153893

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

5.8e-05

3.9e-05

p50

4.8e-05

p95

0.000126

p99

0.000235

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00348

0.010075

p50

0.000629

p95

0.024773

p99

0.056104

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00601

0.010308

p50

0.000732

p95

0.021403

p99

0.061917

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.335934

5.810957

p50

0.452927

p95

8.35268

p99

40.946026

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.085474

0.485214

p50

0.065477

p95

0.39907

p99

0.526621

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.072393

0.068536

p50

0.025124

p95

0.19792

p99

0.22773

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

4e-06

7e-06

p50

1.7e-05

p95

3.2e-05

p99

3.9e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004774

0.003136

p50

0.003235

p95

0.009523

p99

0.018948

Post-commit duration

Time spent executing post-commit workflows.

mean

0.001705

0.002502

p50

0.00059

p95

0.006434

p99

0.013399

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.71

mean

429.47

248.32

p95

815.17

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

over 888.401669743s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

52.972076

19.006932

p50

p95

18.518763

p99

162.641841

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

2.556402

1.320419

p50

0.003229

p95

3.284475

p99

3.916394

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.725641

0.455366

p50

0.002793

p95

1.301571

p99

1.399155

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.181846

0.177565

p50

0.040593

p95

0.519645

p99

0.693405

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.443115

11.78811

p50

0.00435

p95

26.610269

p99

65.91348

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

14.608655

22.455543

p50

0.00175

p95

65.812208

p99

114.684015

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.002425

0.001899

p50

0.000234

p95

0.005334

p99

0.007211

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.005032

0.006421

p50

0.003471

p95

0.019905

p99

0.033427

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.003963

0.009556

p50

0.000562

p95

0.0224

p99

0.052681

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.912131

29.046801

p50

0.227536

p95

60.010347

p99

60.058818

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.210606

19.163516

p50

0.251016

p95

60.001509

p99

60.001969

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.200901

28.294633

p50

0.227821

p95

60.001975

p99

60.002376

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.512141

0.369133

p50

0.52743

p95

1.220459

p99

1.473593

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

24.82

Network receive rate (primary)

Rate of bytes received on primary network interface

3.93

KiB/s

mean

978.53

KiB/s

796.07

KiB/s

p95

2.28

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

B/s

mean

90.61

KiB/s

101.97

KiB/s

p95

276.73

KiB/s

Total bytes received

Total bytes received on primary network interface

count

121.37

GiB

mean

139.64

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

11.24

GiB

mean

12.93

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 92.4%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (69.9% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 35% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

3.27

mean

37.21

34.24

p95

89.53

System

12.39

CPU percentiles

CPU usage percentiles

p50

16.46

p95

89.53

p99

92.44

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

129

hosts

mean time

262.92

Memory used percentage

Percentage of memory used

7.59

mean

13.39

7.48

p95

25.49

Memory available percentage

Percentage of available memory

74.51

mean

86.61

7.48

p95

92.41

Max host memory used

Maximum memory usage percentage across all hosts

max

85.47

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

69.94

Memory growth rate

Rate of memory growth over time

growth

0.41

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.17

MiB/s

Disk write throughput

Disk write throughput in MiB/s

59.11

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.64

5 min

1.72

15 min

1.72

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

35.14

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

0.12

mean

19.8153

25.6434

p95

66.89

60 second average

19.0833

300 second average

19.9937

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

4.4864

6.9591

p95

16.82

60 second average

4.3016

300 second average

3.7828

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.6918

3.1011

p95

7.63

60 second average

1.6449

300 second average

1.5170

Holochain process CPU usage

CPU usage by Holochain process

1.35

mean

33.48

32.84

p95

82.94

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

233.23

KiB

mean

373.59

KiB

83.76

KiB

p95

489.14

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

31.15

threads

6.08

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

76.19

file descriptors

13.12

file descriptors

p95

file descriptors

Remote Call Rate

↑ Back to index

Test the throughput of remote_call operations. Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

Started

Thu, 24 Sep 2026 02:28:05 UTC

Duration

900s

Peer count

150

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_call_rate_35946754846_1

Dispatch timing

The time between sending a remote call and the remote handler being invoked.

Mean

mean

-1.005466

20.536432

Max

Highest per-partition mean latency

max mean

0.077506

Min

Lowest per-partition mean latency

min mean

-7.618815

Round-trip timing

The total elapsed time to get a response to the client.

Mean

mean

0.106198

0.212172

Max

Highest per-partition mean latency

max mean

0.295591

Min

Lowest per-partition mean latency

min mean

0.041854

Error count

The total number of errors accumulated during the run.

3761

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_call::call_echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

274506222

over 888.815473497s

mean rate

3.71607087597e+09

std

1.43311269514e+10

1.226615047e+07

p95

1.65483269092e+10

peak

5.17722824621e+11

WASM usage: 'remote_call::echo_timestamp'

Metered usage count of the WASM ribosome for this zome function.

total

45573906

over 888.815473497s

mean rate

2.23037100865e+09

std

2.7154039908e+10

349550.85

p95

3.11018780617e+09

peak

1.07474231401e+12

WASM usage: 'remote_call::init'

Metered usage count of the WASM ribosome for this zome function.

total

over 888.815473497s

Not enough time data to show a trend.

Zome call duration: 'remote_call::call_echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.113193

0.203696

p50

0.116994

p95

0.278547

p99

0.612463

Zome call duration: 'remote_call::echo_timestamp'

Duration of this zome call as measured by Holochain internally.

mean

0.003563

0.000953

p50

0.003858

p95

0.005026

p99

0.005527

WASM call duration: 'remote_call::call_echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.112099

0.203769

p50

0.115708

p95

0.276761

p99

0.613074

WASM call duration: 'remote_call::echo_timestamp'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002232

0.000616

p50

0.002448

p95

0.003214

p99

0.003656

WASM call duration: 'remote_call::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005529

0.002856

p50

0.005279

p95

0.010057

p99

0.016023

Host function call duration: '__hc__call_1'

Duration of this host function call invoked from within WASM.

mean

0.109453

0.203711

p50

0.113102

p95

0.274345

p99

0.609692

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.00085

0.000639

p50

0.000737

p95

0.001675

p99

0.003767

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

1.2e-05

6e-06

p50

1.2e-05

p95

2.2e-05

p99

2.5e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000333

0.000892

p50

0.000413

p95

0.001214

p99

0.00245

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000631

0.001072

p50

0.000265

p95

0.003102

p99

0.006347

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.99

mean

430.52

248.87

p95

815.85

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

1838

over 888.815493677s

mean rate

5617.18

std

21794.8

p95

27448

peak

437598.46

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

11.707245

7.099851

p50

9.229368

p95

27.803783

p99

40.376415

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.011962

0.010818

p50

0.012739

p95

0.036091

p99

0.054321

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.08298

0.039566

p50

0.082486

p95

0.152269

p99

0.209464

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.019278

0.008947

p50

0.018424

p95

0.035706

p99

0.04732

System validation workflow duration

Time spent running the sys validation workflow.

mean

4.021304

3.435198

p50

4.660226

p95

10.973729

p99

17.930467

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

1.139156

2.708819

p50

0.513117

p95

8.021054

p99

13.104894

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.000364

0.000136

p50

0.000343

p95

0.000632

p99

0.000786

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.003608

0.001726

p50

0.003221

p95

0.006916

p99

0.00847

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000549

0.000248

p50

0.000569

p95

0.000975

p99

0.001381

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.111636

23.132856

p50

0.077891

p95

60.002189

p99

60.00423

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.110472

0.074105

p50

0.047679

p95

0.246152

p99

0.279618

P2P request roundtrip: 'call_remote'

Time spent sending a call_remote request and awaiting its response.

mean

0.108758

4.613066

p50

0.113018

p95

0.281134

p99

1.272998

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

9.74

Network receive rate (primary)

Rate of bytes received on primary network interface

26.19

KiB/s

mean

243.35

KiB/s

510.66

KiB/s

p95

675.18

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

6.14

KiB/s

mean

15.64

KiB/s

15.04

KiB/s

p95

42.56

KiB/s

Total bytes received

Total bytes received on primary network interface

count

30.32

GiB

mean

34.89

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

1.95

GiB

mean

2.24

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Warning Moderate swap usage (18.9% swap used)

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

4.5

mean

16.76

10.68

p95

37.73

System

7.02

CPU percentiles

CPU usage percentiles

p50

13.93

p95

37.73

p99

64.72

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

hosts

mean time

10.12

Memory used percentage

Percentage of memory used

5.87

mean

10.45

6.98

p95

22.15

Memory available percentage

Percentage of available memory

77.85

mean

89.55

6.98

p95

94.13

Max host memory used

Maximum memory usage percentage across all hosts

max

77.72

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

18.86

Memory growth rate

Rate of memory growth over time

growth

0.20

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.06

MiB/s

Disk write throughput

Disk write throughput in MiB/s

28.70

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.52

5 min

0.40

15 min

0.23

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

mean

4.862

3.4513

p95

11.36

60 second average

4.6417

300 second average

3.5178

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

0.8028

4.5785

p95

1.74

60 second average

0.7730

300 second average

0.5647

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

0.6387

4.3766

p95

1.13

60 second average

0.6093

300 second average

0.4333

Holochain process CPU usage

CPU usage by Holochain process

3.05

mean

13.33

9.06

p95

25.63

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

201.68

KiB

mean

241.30

KiB

34.74

KiB

p95

286.34

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

32.27

threads

4.96

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

80.88

file descriptors

6.97

file descriptors

p95

file descriptors

Remote Signals

↑ Back to index

This scenario tests the throughput of remote_signals operations.

Started

Thu, 24 Sep 2026 02:45:17 UTC

Duration

900s

Peer count

300

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

remote_signals_35946754846_1

Round trip time

The time from origin signal dispatch to origin receive of the remote side's response signal.

mean

0.33695

1.555573

p50

0.099181

p95

0.50539

p99

5.073629

Timeouts

The number of timeouts waiting for the remote side's response signal (default timeout is 20 seconds).

total

330

over 1327.297518449s

mean rate

38913.61

std

3.81776076e+06

p95

13801.43

peak

4.75e+08

Timeout rate

The proportion of remote signal round-trips that timed out.

0.0016

Holochain MetricsClick to toggle visibility.

WASM usage: 'remote_signal::init'

Metered usage count of the WASM ribosome for this zome function.

total

over 887.583593093s

Not enough time data to show a trend.

WASM usage: 'remote_signal::recv_remote_signal'

Metered usage count of the WASM ribosome for this zome function.

total

25516821

over 887.583593093s

mean rate

4.86806886426e+09

std

1.01772827676e+11

2.6982526e+06

p95

1.07254677866e+10

peak

1.06584347826e+13

WASM usage: 'remote_signal::signal_request'

Metered usage count of the WASM ribosome for this zome function.

total

13071368

over 887.583593093s

mean rate

1.3730755102e+08

std

3.02833039934e+09

p95

1.5229945665e+08

peak

2.44887852144e+11

Zome call duration: 'remote_signal::recv_remote_signal'

Duration of this zome call as measured by Holochain internally.

mean

0.00396

0.001155

p50

0.004215

p95

0.005556

p99

0.006499

Zome call duration: 'remote_signal::signal_request'

Duration of this zome call as measured by Holochain internally.

mean

0.003751

0.000912

p50

0.003871

p95

0.005188

p99

0.005765

WASM call duration: 'remote_signal::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.020084

0.12581

p50

0.004788

p95

0.020046

p99

0.459882

WASM call duration: 'remote_signal::recv_remote_signal'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002545

0.000795

p50

0.002732

p95

0.00361

p99

0.00435

WASM call duration: 'remote_signal::signal_request'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002328

0.00059

p50

0.002407

p95

0.003312

p99

0.003699

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.001429

0.002604

p50

0.000675

p95

0.005826

p99

0.018107

Host function call duration: '__hc__emit_signal_1'

Duration of this host function call invoked from within WASM.

mean

7.3e-05

2.5e-05

p50

7.7e-05

p95

0.000105

p99

0.000126

Host function call duration: '__hc__send_remote_signal_1'

Duration of this host function call invoked from within WASM.

mean

6.9e-05

3e-05

p50

7.1e-05

p95

0.000104

p99

0.000136

Host function call duration: '__hc__sys_time_1'

Duration of this host function call invoked from within WASM.

mean

9e-06

6e-06

p50

8e-06

p95

1.6e-05

p99

2.5e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.012916

0.120142

p50

0.000478

p95

0.00979

p99

0.2911

Emitted local signals

Count of local signals emitted via the emit_signal host function.

total

1434

over 887.583555755s

mean rate

494179.01

std

1.511223653e+07

100.75

p95

549977.44

peak

1.22857142857e+09

Sent remote signals

Count of remote signals sent via the send_remote_signal host function.

total

1529

over 887.583563395s

mean rate

322214.78

std

7.96769938e+06

76.61

p95

491620.15

peak

6.3033448674e+08

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000794

0.001791

p50

0.000257

p95

0.003412

p99

0.007237

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

40.37

mean

427.9

248.94

p95

814.57

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

3379

over 888.822602222s

mean rate

1.2394424e+06

std

2.029663901e+07

458.62

p95

1.8007252e+06

peak

1.03846153846e+09

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

19.569461

9.821964

p50

16.416876

p95

38.364291

p99

47.122071

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.021693

0.025481

p50

0.022308

p95

0.068287

p99

0.132064

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.099708

0.058328

p50

0.102601

p95

0.208813

p99

0.321064

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.031477

0.029396

p50

0.026671

p95

0.074395

p99

0.126693

System validation workflow duration

Time spent running the sys validation workflow.

mean

7.50557

6.653576

p50

9.206012

p95

22.407267

p99

32.578881

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

2.233211

7.628848

p50

1.344885

p95

6.39548

p99

48.169083

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00069

0.000598

p50

0.000628

p95

0.001279

p99

0.002394

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.00419

0.003715

p50

0.003911

p95

0.010246

p99

0.022762

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.0008

0.000609

p50

0.000752

p95

0.001574

p99

0.002834

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.284848

29.349779

p50

0.321023

p95

60.004063

p99

60.008169

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.148303

14.866919

p50

0.122213

p95

60.000911

p99

60.00176

P2P received remote signals

Count of remote signals received.

total

1435

over 887.583598287s

mean rate

349766.37

std

8.93170035e+06

97.05

p95

520440.04

peak

7.8571428571e+08

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

11.74

Network receive rate (primary)

Rate of bytes received on primary network interface

28.22

KiB/s

mean

447.06

KiB/s

585.01

KiB/s

p95

1.16

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

15.42

KiB/s

mean

33.55

KiB/s

28.77

KiB/s

p95

84.67

KiB/s

Total bytes received

Total bytes received on primary network interface

count

52.71

GiB

mean

60.64

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

3.96

GiB

mean

4.55

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 93.7%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Warning Moderate swap usage (19.1% swap used)

System overload anomaly

System overload anomaly was detected

NotDetected

CPU usage

Total CPU usage and kernel CPU usage

Total

5.18

mean

18.22

15.78

p95

44.28

System

6.48

CPU percentiles

CPU usage percentiles

p50

13.70

p95

44.28

p99

93.70

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

125

hosts

mean time

46.21

Memory used percentage

Percentage of memory used

7.18

mean

13.27

7.45

p95

25.36

Memory available percentage

Percentage of available memory

74.64

mean

86.73

7.45

p95

92.82

Max host memory used

Maximum memory usage percentage across all hosts

max

81.78

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

19.08

Memory growth rate

Rate of memory growth over time

growth

0.39

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.07

MiB/s

Disk write throughput

Disk write throughput in MiB/s

32.07

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/122

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.65

5 min

0.61

15 min

0.47

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

0.00

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

0.08

mean

7.4369

12.2646

p95

21.69

60 second average

7.2331

300 second average

6.7455

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

0.5758

1.8339

p95

2.32

60 second average

0.5766

300 second average

0.6279

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

0.3635

1.5961

p95

1.44

60 second average

0.3596

300 second average

0.4263

Holochain process CPU usage

CPU usage by Holochain process

1.3

mean

7.44

7.64

p95

20.6

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

162.81

KiB

mean

225.96

KiB

43.72

KiB

p95

273.25

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

31.71

threads

6.12

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

79.28

file descriptors

11.71

file descriptors

p95

file descriptors

Validation Receipts

↑ Back to index

Creates an entry, wait for required validation receipts, then repeat. Records the amount of time it took to accumulate the required number of receipts for all DHT operations. This is measured to the nearest 20ms so that we don't keep the agent too busy checking for receipts.

Each agent in this scenario waits for a certain number of peers to be available or for up to two minutes, whichever happens first, before starting its behaviour.

By default, this scenario will wait for a complete set of validation receipts before committing the next record. If the NO_VALIDATION_COMPLETE environment variable is set, it will instead publish new records on every round, building up an ever-growing list of action hashes to check on.

Started

Thu, 24 Sep 2026 03:35:11 UTC

Duration

900s

Peer count

150

Behaviours

default (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

validation_receipts_35946754846_1

Receipts complete timing

The amount of time between publishing a record and receiving the required number of validation receipts.

Mean

mean

36.738661

24.004701

Max

Highest per-partition mean latency

max mean

109.09609

Min

Lowest per-partition mean latency

min mean

14.622249

Receipts complete rate

The number of complete validation receipt sets collected per second.

Mean

mean

1.01

Max

Highest per-partition mean rate

max

1.2

Min

Lowest per-partition mean rate

min

Error count

The total number of errors accumulated during the run.

Holochain MetricsClick to toggle visibility.

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 760.007424122s

Not enough time data to show a trend.

WASM usage: 'crud::create_sample_entry'

Metered usage count of the WASM ribosome for this zome function.

total

754710

over 888.400152183s

mean rate

6.619454566e+07

std

2.32292131246e+09

p95

7.699033654e+07

peak

1.51984177215e+11

WASM usage: 'crud::get_sample_entry_validation_receipts'

Metered usage count of the WASM ribosome for this zome function.

total

12328069

over 888.400152183s

mean rate

1.446661684e+08

std

1.60915006731e+09

394042.08

p95

3.0919542038e+08

peak

7.50927777467e+10

WASM usage: 'crud::init'

Metered usage count of the WASM ribosome for this zome function.

total

over 888.400152183s

Not enough time data to show a trend.

WASM usage: 'crud_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

93632

over 888.400152183s

mean rate

8.01515434e+06

std

2.8461343371e+08

p95

9.19351396e+06

peak

1.88556962025e+10

Zome call duration: 'crud::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.007025

0.0018

p50

0.007168

p95

0.010102

p99

0.011443

Zome call duration: 'crud::get_sample_entry_validation_receipts'

Duration of this zome call as measured by Holochain internally.

mean

0.026621

0.027553

p50

0.014776

p95

0.074939

p99

0.158731

WASM call duration: 'crud::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005725

0.00147

p50

0.005926

p95

0.008194

p99

0.009364

WASM call duration: 'crud::get_sample_entry_validation_receipts'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.025094

0.027278

p50

0.013285

p95

0.072224

p99

0.155927

WASM call duration: 'crud::init'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.004215

0.003597

p50

0.003567

p95

0.006761

p99

0.02263

WASM call duration: 'crud_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001863

0.000441

p50

0.00193

p95

0.002513

p99

0.00284

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000705

0.000882

p50

0.000734

p95

0.002206

p99

0.003619

Host function call duration: '__hc__get_validation_receipts_1'

Duration of this host function call invoked from within WASM.

mean

0.022842

0.027076

p50

0.011011

p95

0.069204

p99

0.152881

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002924

0.000798

p50

0.003023

p95

0.004236

p99

0.004995

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000714

0.000511

p50

0.000668

p95

0.001668

p99

0.002499

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.89

mean

430.17

248.68

p95

815.28

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

10700

over 888.400163674s

mean rate

127165.85

std

2.39731694e+06

129.88

p95

283257.42

peak

1.4115384615e+08

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

15.440495

8.698772

p50

12.011414

p95

31.401992

p99

40.037113

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.005573

0.049496

p50

0.009698

p95

0.111274

p99

0.287584

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.045935

0.112053

p50

0.077078

p95

0.33274

p99

0.519519

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.088193

0.071448

p50

0.050696

p95

0.203075

p99

0.405909

System validation workflow duration

Time spent running the sys validation workflow.

mean

0.435396

5.658584

p50

0.943701

p95

15.547932

p99

22.613237

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

22.345043

19.651525

p50

22.236442

p95

61.782135

p99

81.226202

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.001434

0.000821

p50

0.000991

p95

0.002743

p99

0.004709

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.004673

0.003245

p50

0.003446

p95

0.007635

p99

0.010928

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000654

0.000592

p50

0.000589

p95

0.00149

p99

0.00173

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.201459

29.130435

p50

0.140655

p95

60.004511

p99

60.009464

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.139017

0.061413

p50

0.130759

p95

0.269152

p99

0.310654

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

16.31

Network receive rate (primary)

Rate of bytes received on primary network interface

27.01

KiB/s

mean

287.32

KiB/s

511.25

KiB/s

p95

684.58

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

6.07

KiB/s

mean

22.92

KiB/s

23.63

KiB/s

p95

67.56

KiB/s

Total bytes received

Total bytes received on primary network interface

count

35.84

GiB

mean

41.24

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

2.86

GiB

mean

3.29

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

NotDetected

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (22.6% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 10% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

5.68

mean

24.01

18.6

p95

70.38

System

7.70

CPU percentiles

CPU usage percentiles

p50

17.67

p95

70.38

p99

87.34

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

101

hosts

mean time

37.19

Memory used percentage

Percentage of memory used

6.47

mean

11.19

6.82

p95

22.78

Memory available percentage

Percentage of available memory

77.22

mean

88.81

6.82

p95

93.53

Max host memory used

Maximum memory usage percentage across all hosts

max

76.20

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

22.63

Memory growth rate

Rate of memory growth over time

growth

0.33

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.60

MiB/s

Disk write throughput

Disk write throughput in MiB/s

34.07

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

0.88

5 min

1.45

15 min

2.32

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

10.14

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

0.08

mean

8.4432

13.0658

p95

38.14

60 second average

8.3771

300 second average

15.7015

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0011

0.0755

≤

p95

60 second average

0.0007

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0011

0.0755

≤

p95

60 second average

0.0007

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

2.2434

5.1248

p95

7.15

60 second average

2.2504

300 second average

3.8285

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.5103

4.5321

p95

3.32

60 second average

1.4910

300 second average

1.7460

Holochain process CPU usage

CPU usage by Holochain process

3.9

mean

21.12

17.85

p95

66.27

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

237.11

KiB

mean

282.00

KiB

36.23

KiB

p95

327.19

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

31.31

threads

5.01

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

79.85

file descriptors

7.55

file descriptors

p95

file descriptors

Write/`get_agent_activity`

↑ Back to index

A scenario where write peers write entries, while get_agent_activity peers each query a single write agent's activity with get_agent_activity.

Before a target write peer and the requesting get_agent_activity peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 24 Sep 2026 03:52:22 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_35946754846_1

Observed chain advancement

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

2244.7

Max

Highest chain head value observed for any single write agent.

6091

Write agents observed

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.917033

2.051273

Max

Highest per-partition mean latency

max mean

4.162531

Min

Lowest per-partition mean latency

min mean

0.016407

Error count

The number of errors accumulated.

177

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.056046

2.561107

p50

0.019984

p95

2.588722

p99

10.393854

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 814.052811626s

mean rate

2.22

std

6.1

p95

22.32

peak

27.01

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007239

0.00176

p50

0.007281

p95

0.009662

p99

0.011941

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.011278

0.008417

p50

0.011204

p95

0.030895

p99

0.042289

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.240941

3.352727

p50

0.438408

p95

3.244013

p99

7.765905

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.00647

1.652995

p50

0.007508

p95

0.920944

p99

10.399183

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005944

0.001382

p50

0.005971

p95

0.007763

p99

0.008538

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.009768

0.007729

p50

0.009551

p95

0.027741

p99

0.039011

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.240071

3.354826

p50

0.437135

p95

3.317494

p99

7.763417

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005346

1.652961

p50

0.005996

p95

0.919652

p99

10.398332

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001715

0.000591

p50

0.001824

p95

0.002901

p99

0.003519

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

6.4e-05

5.1e-05

p50

4e-05

p95

0.000176

p99

0.000224

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.003921

0.004971

p50

0.003662

p95

0.015254

p99

0.02322

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000587

0.000159

p50

0.000603

p95

0.000799

p99

0.000905

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.237722

3.354713

p50

0.434964

p95

3.314248

p99

7.760419

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.000586

1.652884

p50

0.00057

p95

0.915488

p99

10.393915

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.2e-05

6e-06

p50

1.3e-05

p95

2.2e-05

p99

3e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.003023

0.001606

p50

0.003093

p95

0.006602

p99

0.00933

Post-commit duration

Time spent executing post-commit workflows.

mean

0.001563

0.001293

p50

0.000342

p95

0.003296

p99

0.005754

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.68

mean

430.25

248.73

p95

815.3

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

500

over 889.429232077s

mean rate

643789.25

std

5.43861891e+06

526.18

p95

2.7233686e+06

peak

2.7237485886e+08

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

95.630978

60.425907

p50

20.560598

p95

183.293176

p99

273.16469

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.080957

0.321473

p50

0.098514

p95

0.840224

p99

1.594805

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.140806

0.431096

p50

0.245867

p95

1.27737

p99

1.564448

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.849276

0.743414

p50

0.119525

p95

2.100688

p99

3.219223

System validation workflow duration

Time spent running the sys validation workflow.

mean

10.23068

23.710051

p50

17.240043

p95

72.144082

p99

119.220566

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

9.696099

20.883316

p50

0.372342

p95

47.021426

p99

98.732457

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.004571

0.002734

p50

0.003987

p95

0.008868

p99

0.009761

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006042

0.003829

p50

0.00386

p95

0.013594

p99

0.017671

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.003489

0.003838

p50

0.001146

p95

0.010538

p99

0.019303

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.594062

28.776047

p50

1.002075

p95

60.0079

p99

60.024261

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

1.467538

2.247773

p50

0.454179

p95

8.030371

≤

p99

8.030371

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.182058

28.388345

p50

0.381852

p95

60.009444

p99

60.026538

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.799394

24.973064

p50

2.947561

p95

60.00775

p99

60.05053

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

47.67

Network receive rate (primary)

Rate of bytes received on primary network interface

103.09

KiB/s

mean

986.31

KiB/s

644.40

KiB/s

p95

1.86

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

10.43

KiB/s

mean

128.52

KiB/s

156.65

KiB/s

p95

452.16

KiB/s

Total bytes received

Total bytes received on primary network interface

count

122.98

GiB

mean

141.49

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

16.02

GiB

mean

18.44

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 97.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (22.6% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 65% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

18.32

mean

67.13

22.17

p95

93.62

System

19.46

CPU percentiles

CPU usage percentiles

p50

71.14

p95

93.62

p99

97.50

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

127

hosts

mean time

363.96

Memory used percentage

Percentage of memory used

6.94

mean

11.95

7.25

p95

24.09

Memory available percentage

Percentage of available memory

75.91

mean

88.05

7.25

p95

93.06

Max host memory used

Maximum memory usage percentage across all hosts

max

80.02

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

22.57

Memory growth rate

Rate of memory growth over time

growth

0.43

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.18

MiB/s

Disk write throughput

Disk write throughput in MiB/s

118.93

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

3.37

5 min

2.59

15 min

2.19

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

64.86

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

2.17

mean

43.9495

25.337

p95

82.55

60 second average

41.1115

300 second average

31.1033

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

0.001

≤

p95

60 second average

0.0001

300 second average

0.0004

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

0.001

≤

p95

60 second average

0.0001

300 second average

0.0004

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

10.7403

8.1367

p95

23.91

60 second average

10.1116

300 second average

7.6997

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

4.423

4.4314

p95

12.6

60 second average

4.1958

300 second average

3.3729

Holochain process CPU usage

CPU usage by Holochain process

25.71

mean

64.73

18.89

p95

89.52

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

229.48

KiB

mean

337.60

KiB

82.79

KiB

p95

455.81

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

32.58

threads

6.08

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

81.07

file descriptors

12.27

file descriptors

p95

file descriptors

Write/`get_agent_activity` with volatile conductors

↑ Back to index

A scenario where write peers write entries, while get_agent_activity_volatile peers each query a single write agents activity with get_agent_activity but shutdown and restart their conductors at semi-random intervals.

Before a target write peer and the requesting get_agent_activity_volatile peer are in sync, this will measure the get_agent_activity call performance over a network. Once a write peer reaches sync with a get_agent_activity peer, the write peer will publish their actions and entries, and so the get_agent_activity calls will likely have most of the data they need locally. At that point this measures the database query performance and code paths through host functions.

Started

Thu, 24 Sep 2026 04:09:24 UTC

Duration

1800s

Peer count

150

Peer count at end

121

Behaviours

get_agent_activity_volatile (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_get_agent_activity_volatile_35946754846_1

Highest observed action_seq

The rate at which zero-arc readers observe new chain heads on the writers' chains via get_agent_activity. This reflects the DHT's ability to propagate agent activity ops and make them available to querying peers.

Mean

mean

276.16

Max

Highest per-partition mean rate

max

893.42

Min

Lowest per-partition mean rate

min

132.87

get_agent_activity_full zome call timing

The time taken to call the zome function that retrieves information on a write peer's source chain.

Mean

mean

0.028505

0.144883

Max

Highest per-partition mean latency

max mean

0.052823

Min

Lowest per-partition mean latency

min mean

0.007551

Running volatile conductors count

The number of conductors running by get_agent_activity_volatile peers.

conductors

mean

70.22

conductors

13.51

conductors

p95

conductors

Volatile conductor total running duration

The total running duration of a get_agent_activity_volatile conductor.

Mean

mean

901.907734

444.275431

Max

Highest per-partition mean latency

max mean

1711.071429

Min

Lowest per-partition mean latency

min mean

315.733333

Volatile conductor running duration

The duration that a get_agent_activity_volatile conductor was running before being stopped.

Mean

mean

41.984766

44.684041

Max

Highest per-partition mean latency

max mean

100.971073

Min

Lowest per-partition mean latency

min mean

23.265896

Volatile conductor stopped duration

The duration that a get_agent_activity_volatile conductor was stopped before being started again.

Mean

mean

35.748506

42.406964

Max

Highest per-partition mean latency

max mean

82.258477

Min

Lowest per-partition mean latency

min mean

21.872318

Reached target arc

Did get_agent_activity_volatile peer reach their target arc in the moment before they were shutdown.

get_agent_activity_volatile_agent: uhCAk-MlZSATGMGo_N3Wjw3lN7KzAQ7yGJZHp56gDWf1xlsJJWgSj

mean

0.17

0.37

p95

get_agent_activity_volatile_agent: uhCAk0GrgYLDRzUSoQ0DX3vFDJureyZO8ZAWLPfhqP6BCCJ5xVMq4

mean

0.21

0.41

p95

get_agent_activity_volatile_agent: uhCAk0hpmYwu7yrm9G8awrwCllvaS0k-wUOz-7CqXPnVzHjHkDUIJ

mean

0.1

0.3

p95

get_agent_activity_volatile_agent: uhCAk0rt1TkXcc8pyAfmlOhjCb4cLUNFko_b_xQ99jLvv3LIbkr94

mean

0.21

0.41

p95

get_agent_activity_volatile_agent: uhCAk0uTOKVRBD-sEhgia83qumx6NB-sFkla7IsFZWn9174e60ata

mean

0.13

0.33

p95

get_agent_activity_volatile_agent: uhCAk1USQMrEw_X4Ep7oi5FoduuJKNm9PvNwuA7gZtzLn9EcUXlVI

mean

0.16

0.36

p95

get_agent_activity_volatile_agent: uhCAk2FMgn8CKv9SVmGi2VAUycFC3kS2S_-OrnAT8W1M8OLEZhLKv

mean

0.07

0.26

p95

get_agent_activity_volatile_agent: uhCAk2Vx6FZj_iBYopnnlx4dBnBqp980YSTJZhhKPG6ORHu1LFDHC

mean

0.21

0.41

p95

get_agent_activity_volatile_agent: uhCAk3XhcReGCAVGZ0J7sPwD1ZK3235LOJ1nkuYHyaNJ1VBrWWRXI

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAk3qt_jpD68ge3nUXPEuyQ1841UNDUQh9tiiPcTs2fd_XSSj1E

mean

0.17

0.37

p95

get_agent_activity_volatile_agent: uhCAk4CrdYSWB3D3FicWWh7M2cNfac1HXyyb9ee9oyWATmGwJmqJT

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAk4_-N6LIBz9zfNVNR98S_RofD1DLMwtiXaU1FVJQsEE4SQ275

mean

0.08

0.28

p95

get_agent_activity_volatile_agent: uhCAk4djHRpRBr-ta9ub2eadk71pgxt8gpVa0Hwcxf9wSZdpEY7g1

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAk59nsLp8JUTRnTKyQssJ298XAsdYzGrraDw4jULzB7Gg1Bos1

mean

0.13

0.34

p95

get_agent_activity_volatile_agent: uhCAk5Wo22gHj3LjDaRsa02MaAWMXjZd2NSh_A7CDQW_TMynAChVs

mean

0.15

0.35

p95

get_agent_activity_volatile_agent: uhCAk5f5sbu9j5C4FIChHWT1Se7b7Rgk-P1htYQFCOjOJwa47namy

mean

0.33

0.47

p95

get_agent_activity_volatile_agent: uhCAk6mQ0HTTgJaq-vuKiUXhYXOxS4t_pUyOquCGL8JE3uTfBavzh

mean

0.12

0.33

p95

get_agent_activity_volatile_agent: uhCAk7AY_FFooZDgBauTTz_zZ12EepwiiJ8kpMuITsOBghuOy5aZ_

mean

0.12

0.32

p95

get_agent_activity_volatile_agent: uhCAk7ytvKBIETiwwUeWXzEoYBPthAVfJaaoOwYLqJzqDUSPAHBu5

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAk8RjoHy0vWINpL64BySlXiemb5pXNiLoeluQ2Ad2a5uGO1D9G

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAk8e5kVSAjiEQMWqRcPvQAkITz52WLvh-gWiHtjW8aF3_CwzKU

mean

0.07

0.26

p95

get_agent_activity_volatile_agent: uhCAk8fhT06J9sXO4o-M5Q4sgwMiqAJaMQKjyLab8t0LPr6Q-Dpng

mean

0.19

0.4

p95

get_agent_activity_volatile_agent: uhCAkApkX22xoe6ay57elWE6ZMU7xVAcpxI6DzTje4ukQnE7A9wqz

mean

0.03

0.18

≤

p95

get_agent_activity_volatile_agent: uhCAkAq-Nei3tlJ8CEomGGuY2p1UVP-gkQPKknsL98nhzB5u4oDiD

mean

0.12

0.32

p95

get_agent_activity_volatile_agent: uhCAkC07Htrh_QLIaQjCEHGOPf9c4ODV5AsntWovf7jQHH8Z_1nL9

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkDEgnBAW2AfAY55YWT-IrYyVuLK5m95Shn1rMSNbaHGJZOrn6

mean

0.24

0.43

p95

get_agent_activity_volatile_agent: uhCAkDy-Fu1aJVox341G5xDiC6vSHT2OJUm8a4i3ynIKx_fkDZFVv

mean

0.12

0.33

p95

get_agent_activity_volatile_agent: uhCAkE5wNlhPfdpCMVBhbONXjWyePXtSj-b8CiYS6PZ7aau45EZy3

mean

0.12

0.33

p95

get_agent_activity_volatile_agent: uhCAkESDWTL4o92BglhWrv5e1ZMDj4hgKQ6o4S_yYsP6dHj9SgP3B

mean

0.06

0.25

p95

get_agent_activity_volatile_agent: uhCAkFSyoMu8zAac_fuuXm_MBrbH4XCpvIyWDOcHN4VGWJ1rxYkKn

mean

0.14

0.34

p95

get_agent_activity_volatile_agent: uhCAkGNopVAZK_ZsRd8LP6nRNdqs16ZdK7huev6sQ8OH4tN9iFY6I

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkGqRd0-S-BTDpoP0C8GjAh3azMYxzRMLaNkSIVtetAWXA28jy

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkH37Wym6Fw68mXqzqEcKdInwhZxWXnIB0hf7KrcOXvTi-EEe1

mean

0.13

0.33

p95

get_agent_activity_volatile_agent: uhCAkHD5atS7vph7PU0fwOTBjAcq9Sh7ssXbbmuqNPNRZW64Bd4TD

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkHc4XmcyBOhf8ohDnQIjWybeG9ssHxktCu4fREkB6G09GnEAd

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkIFct_QVArVOo-rh2P735hBx-9D6_0MrSylNi4hjZd7jSknLu

mean

0.12

0.33

p95

get_agent_activity_volatile_agent: uhCAkJEBEplpjLUdsDqVg2mHNOVckAPDrqLncGJMh3GqWF-zTuec7

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkJLV_PJoreG76bcItZo377pH6cwlyWyOt43C-aK12l4RQu_DW

mean

0.1

0.3

p95

get_agent_activity_volatile_agent: uhCAkK03uTTUvC-oh2pcRxzaQz7RGVb5-fXxu2SogQZdOrKKh8zSL

mean

0.08

0.27

p95

get_agent_activity_volatile_agent: uhCAkKkHVhqwnubhEI1ZsYp9BYv8b_eVoT6cMJokC1M07k3J-pAmP

mean

0.24

0.42

p95

get_agent_activity_volatile_agent: uhCAkKqo3MhEDwdx8oue-tXYYOAHy207yfmzvnnN3AieRDH7u5NMG

mean

0.08

0.28

p95

get_agent_activity_volatile_agent: uhCAkKr9JTzYKbZudiBTHKJod2Ioy8kuMgvFxGjMme9SCVSI8cVLM

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkKwavFx5oGt0QEeI5j5sYIWpG_2weE3oWB2rfQhCBWyvZza6T

mean

0.17

0.37

p95

get_agent_activity_volatile_agent: uhCAkL0j8sdSRitLFzmJG19iWNCayhFhjkzoxZa-AJUWxspsyFW27

mean

0.03

0.17

≤

p95

get_agent_activity_volatile_agent: uhCAkNE-uPK5BNv0uUglYJYmKF5zIA-zElHc-TI27MQR2ENv1DQjC

mean

0.21

0.41

p95

get_agent_activity_volatile_agent: uhCAkNGCQQ8UKc2TCGWEn-OuEndqSxGkhbP_wLeGYauthzi335uOL

mean

0.16

0.37

p95

get_agent_activity_volatile_agent: uhCAkOdy4KXFuVwdepz5Nyv7I_0fmYL1HgPEgFos-TrZpZQ83BJo-

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkP5VztxtxSbRJENiA1N9ZW2iHjDKuREyIWd6Yni4uVbztimzL

mean

0.21

0.4

p95

get_agent_activity_volatile_agent: uhCAkQ0WceJvi4JtZVbw7G7pYgHNQHKsFpu-iv63VSuscn6uD8tK5

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkQPEczNtmZvNruZBcQkIip8jGP8mEXWS367sUZAI3LPm5qQp8

mean

0.13

0.34

p95

get_agent_activity_volatile_agent: uhCAkQUZZ2uP0CtIQ5AvCRiweLE8lJhSe556WraYPNsDw_sVM5Kjj

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkQXDG1frjI8c-F-ild3HMElHxyMyViIeEQ4rEYrmSHm2bjKz9

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkRB_2m6d4cNIMqkh_5BrG4pxQJp3aGFYmhqP03AFPeLa53GUn

mean

0.17

0.38

p95

get_agent_activity_volatile_agent: uhCAkRPc7VQgtE-3JzIJ2axKfYT9xtSXrfXGiVZWUjyLyRWucuQuW

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkRsb9alnzkV4Jg0hTIc4hnjULIqXX0YQyaMLZIbjaLUbPPAgB

mean

0.05

0.23

p95

get_agent_activity_volatile_agent: uhCAkS0z4tUbNUbuqygzjBdux4CSlgCeichbni-0kkkmMd00AY__q

mean

0.16

0.36

p95

get_agent_activity_volatile_agent: uhCAkSsW3uRObkWqjKUdhXJUgTeXsJUxbhY2LvA36Zuutwwl0vydO

mean

0.1

0.3

p95

get_agent_activity_volatile_agent: uhCAkTVaGem0HI0DPc8b57rfJGe-677AZeRSpVBD5Fy87psOowj0D

mean

0.15

0.35

p95

get_agent_activity_volatile_agent: uhCAkU8W9mkMwuxJS8xh8e6UGK1u_XbujLgX3TQi_2bAjerzofiyM

mean

0.16

0.37

p95

get_agent_activity_volatile_agent: uhCAkUH4XeheLhcihg3zjz6zpS3aMjmwKMza_CMeB80Hb9hcBwz1i

mean

0.08

0.28

p95

get_agent_activity_volatile_agent: uhCAkV0gG1tYqvYOC3olVs6zjVHQjTofhqQxTFDBX9Dl-jfr0Dcku

mean

0.06

0.24

p95

get_agent_activity_volatile_agent: uhCAkVdqI9fELQ31X-lScQv4IiYAkSlF3ewalelKEcNwJuzsQ9wNQ

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAkX4tHDB9WGNe3XRPn4rGRmJOGFB_OBbtCyJpdADTo8F8VvLjw

mean

0.1

0.3

p95

get_agent_activity_volatile_agent: uhCAkXUtQiEpgoac0hnRgfVhSyVJZox2DcQd51lVYHgP3_b0IKVxz

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkZRJECHqjj97SMeXbhITO70tkDjrh1MahK5lDQGvUZovcUqlQ

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAk_ZpW3YSR8wUrfIQmLyEXYL4XYb_-WcXA2s0RbKkEhqtjwIsq

mean

0.14

0.34

p95

get_agent_activity_volatile_agent: uhCAkaXxqCdjRFweFulSCSZJvvca7wmmHmEbM3jAaC1Hzpxj65Ir-

mean

0.11

0.31

p95

get_agent_activity_volatile_agent: uhCAkbPt0XOqVaueJSZD3GWaepQEq9AR7YItBZtehN_brtPKlN0vB

mean

0.14

0.34

p95

get_agent_activity_volatile_agent: uhCAkbkSn6DPaMvVFcsYjbLdtUcvMihHOnLdLn7DfI7rH1Ge8_8uv

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAkdcq749wmi-NeGBMecBnNUOIa_gm4lX6PQu0m1FNil0YIa1nt

mean

0.05

0.23

p95

get_agent_activity_volatile_agent: uhCAke4OAZo4d_-EO4E982CfI02w-XpG3K_fdRXf76hFoqrnimqcW

mean

0.18

0.39

p95

get_agent_activity_volatile_agent: uhCAkeMK1IwN6pWk_T2HUTHMexrwdBTwO6hq8lNpJGpfvaNdVCaiw

mean

0.13

0.33

p95

get_agent_activity_volatile_agent: uhCAkfIGGCnpOSPQ_p3KUG9Em2Jdr3Qup-e1hiUoF7EuLVH0usXkx

mean

0.14

0.34

p95

get_agent_activity_volatile_agent: uhCAkg1OOEhUS6HF21lpJLNBXu9GRCBbeCzn7nOwQbvaXazYD3UC_

mean

0.1

0.3

p95

get_agent_activity_volatile_agent: uhCAkg7fs67_eX75aHgpQ1wwGulkuZ9-q5jZirwotZkmaFrdwRIxV

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkgPXDap85vyEoD7n82Y9IQ9tT1NpUbzP_58ZNQsAClN17fJBC

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkgv5gLJwwdh4ZL5g_8Q4MUk4O4QAbITZnzB8fJ_GS7GQGupSu

mean

0.11

0.31

p95

get_agent_activity_volatile_agent: uhCAkhUc_PPFiBY_nBzPS3s_2fr3VKjPVzTl3BxF8uNmcuRhMpGNb

mean

0.18

0.38

p95

get_agent_activity_volatile_agent: uhCAki2SmOcFH0yiGuNH0kRiCUwdOlJ2uQ9r17oy1Wr_KTtOvHMop

mean

0.06

0.25

p95

get_agent_activity_volatile_agent: uhCAkjdjy5tgQ0UIOSYQd4RWU0oRNLqC-QAGRcZ22onTXaSodO-nT

mean

0.17

0.38

p95

get_agent_activity_volatile_agent: uhCAkkRXBB8Jc_EONj2qbAcmqBGR7LirC3-OvO9qztwgJ_xuOCSxR

mean

0.13

0.34

p95

get_agent_activity_volatile_agent: uhCAklpGQfXIHo4P1TsDueKEas_4G05QeodQwH0w0mhBJLlkQ1QzB

mean

0.17

0.37

p95

get_agent_activity_volatile_agent: uhCAklqtl1Yey27XUrsJUGhufSowopB9TtmtXbU34gGXBm2f4ETJ0

mean

0.06

0.24

p95

get_agent_activity_volatile_agent: uhCAkm6nOeO3i76c0T85Okv9x8Pbm14Gw0OsmbOAq5dduODzLEOC4

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAkoCxGCJdnAZy9HRQbnTvJAaYfMe8MtqTs61lIY5Q8WfUNUJ5R

mean

0.11

0.31

p95

get_agent_activity_volatile_agent: uhCAkpFC26Wq6Ijf497oC8xWhxoKEG9EFgiwzlHYOOPjMMV70MU4c

mean

0.17

0.37

p95

get_agent_activity_volatile_agent: uhCAkprUjGEAru1t8GVXH_U_ta1Ckl-qjzJQy42vtLKjcYLTPTcmV

mean

0.13

0.33

p95

get_agent_activity_volatile_agent: uhCAkq0sDoZKypOCKCPEA94ftLM5EVX6VKJlepPOkq7c13HQ5ppWU

mean

0.14

0.35

p95

get_agent_activity_volatile_agent: uhCAkqzx8fkTQByE89eFQZAxBko42lNXWaqj0fRM9_TPv1ISl8Poe

mean

0.13

0.34

p95

get_agent_activity_volatile_agent: uhCAkrLHRCKdJmzQBfx31BPP9dy5d21C9vPHZ6-o8nmvoLnROLeJ6

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkrYd79dxDXKrbaYBZjjf0VENYhfArg0WdbZJSzomloTzPEOIu

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAkru4cQBE_3bv6n6U44R9PXic3RHArNP7Mv3YK9U-MArnAi2a7

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAksvCCS9IcRW9RNG1GzG_HuxCF5EraIVgJT6Uy26EovYhEl0O_

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAktwXhFwp4ZMvDg1LtSxB9mzmN5oebgwxkLQNpT4W8Glg2JE3M

mean

0.15

0.36

p95

get_agent_activity_volatile_agent: uhCAku0X1lB082NbiKMZgbW76NIlFOevJIhMXCs0MEBzmHZFLz1Ub

mean

0.09

0.29

p95

get_agent_activity_volatile_agent: uhCAkvXGSLPqAIBlLjHC6Z6eOX0jMsdh0AZb_AoeH0rr8qQcXWeAK

mean

0.13

0.34

p95

get_agent_activity_volatile_agent: uhCAkvvsQmXXvkp2uTLUyVQMSHzROAyresmWC-hcuDrGB37kAy4y3

mean

0.03

0.17

≤

p95

get_agent_activity_volatile_agent: uhCAkwBxqM8Or9_PTV4PJ4KkDBjvm1RT15e1YlkEYc7yYt0zV4AnB

mean

0.11

0.31

p95

get_agent_activity_volatile_agent: uhCAkwVXrcbHZSBNToK0De2U9DBpjyrxc2PD9bcIL1OE2OSbuMS4w

mean

0.14

0.34

p95

get_agent_activity_volatile_agent: uhCAkwamMTRDKyNaYT72Or_xVLSJ76qog-dWthmQs-nArVpVi0BAO

mean

0.08

0.28

p95

get_agent_activity_volatile_agent: uhCAkyW1Q-QZSseu_71Hg7PvhM6MGm--6frDSh_yeIscCDD52E0_S

mean

0.18

0.38

p95

get_agent_activity_volatile_agent: uhCAkzdmVeLC05JsTrfKcKwjUt2Ah7YrlTXoQlUkfPMCjJxC5-vI8

mean

0.09

0.28

p95

get_agent_activity_volatile_agent: uhCAkze25r7qH7duE5UYS7SXV61q9verf835SbziblKdm4OBXo6-b

mean

0.13

0.33

p95

get_agent_activity_volatile_agent: uhCAkzk6GhhC-EZZiUL506GLgmj1T4EjX8whHP3rVkvs7PRu65i2_

mean

0.09

0.29

p95

Error count

The number of errors accumulated.

756

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.019404

24.550216

p50

0.000781

p95

43.339476

p99

65.462223

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 1708.869992934s

mean rate

135.74

std

522

p95

524.08

peak

8513.25

Zome call duration: 'agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.007199

0.002272

p50

0.007128

p95

0.010932

p99

0.012317

Zome call duration: 'agent_activity::create_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.00623

0.00144

p50

0.006653

p95

0.008682

p99

0.009238

Zome call duration: 'agent_activity::get_agent_activity_full'

Duration of this zome call as measured by Holochain internally.

mean

0.008579

2.222285

p50

0.005943

p95

1.130416

p99

3.044278

Zome call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.006267

0.948917

p50

0.008579

p95

1.058585

p99

1.505767

WASM call duration: 'agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005847

0.0019

p50

0.005739

p95

0.009357

p99

0.010879

WASM call duration: 'agent_activity::create_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005004

0.001131

p50

0.005285

p95

0.006869

p99

0.007446

WASM call duration: 'agent_activity::get_agent_activity_full'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.021477

25.134584

p50

0.002934

p95

43.356088

p99

65.466362

WASM call duration: 'agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.005104

0.920098

p50

0.007186

p95

0.789418

p99

1.321877

WASM call duration: 'agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001647

0.000366

p50

0.001716

p95

0.002214

p99

0.002495

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

7.3e-05

5.5e-05

p50

5.1e-05

p95

0.000186

p99

0.000211

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000565

0.000165

p50

0.000601

p95

0.000866

p99

0.00093

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000669

0.000351

p50

0.000604

p95

0.001369

p99

0.002138

Host function call duration: '__hc__get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.01994

25.133418

p50

0.000837

p95

43.353832

p99

65.464788

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001295

0.920118

p50

0.001026

p95

0.779534

p99

1.317842

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1e-05

9e-06

p50

1.3e-05

p95

2.3e-05

p99

6.7e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002604

0.000895

p50

0.002783

p95

0.004419

p99

0.006814

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000363

0.000478

p50

0.000414

p95

0.000683

p99

0.001146

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

2.93

mean

344.85

502.24

p95

1505.42

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

25990

over 1789.361274367s

mean rate

904678.65

std

4.086630887e+07

p95

1.45052663e+06

peak

4.28493598475e+09

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

211.382549

145.63871

p50

70.853425

p95

446.256802

p99

665.921394

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.207813

0.286556

p50

0.041274

p95

0.748012

p99

1.384014

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.04937

0.452982

p50

0.030088

p95

0.721125

p99

2.090703

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.807321

1.428927

p50

0.182311

p95

4.022439

p99

5.895546

System validation workflow duration

Time spent running the sys validation workflow.

mean

45.77542

54.975272

p50

20.190237

p95

168.737657

p99

220.168494

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

36.378846

50.026537

p50

0.040873

p95

142.003216

p99

245.357568

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.005123

0.003279

p50

0.0011

p95

0.009926

p99

0.013797

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007949

0.148239

p50

0.003382

p95

0.038758

p99

0.407733

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000486

0.001123

p50

0.00086

p95

0.003098

p99

0.006087

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

4.169234

29.210575

p50

0.609256

p95

60.002746

p99

60.003766

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.575798

1.514174

p50

0.489589

p95

0.875907

p99

10.367271

P2P request roundtrip: 'get_agent_activity'

Time spent sending a get_agent_activity request and awaiting its response.

mean

0.077178

14.872093

p50

0.159386

p95

60.000897

p99

60.001706

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.281247

24.119045

p50

0.539825

p95

60.001541

p99

60.00224

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

31.06

Network receive rate (primary)

Rate of bytes received on primary network interface

7.87

KiB/s

mean

240.24

KiB/s

381.49

KiB/s

p95

805.48

KiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

B/s

mean

25.80

KiB/s

79.29

KiB/s

p95

124.51

KiB/s

Total bytes received

Total bytes received on primary network interface

count

59.55

GiB

mean

34.07

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

6.40

GiB

mean

3.66

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 96.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (22.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 24% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

2.38

mean

41.39

29.63

p95

89.79

System

10.33

CPU percentiles

CPU usage percentiles

p50

43.49

p95

89.79

p99

96.17

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

131

hosts

mean time

264.74

Memory used percentage

Percentage of memory used

6.4

mean

10.22

7.53

p95

20.92

Memory available percentage

Percentage of available memory

79.08

mean

89.78

7.53

p95

93.6

Max host memory used

Maximum memory usage percentage across all hosts

max

80.57

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

22.51

Memory growth rate

Rate of memory growth over time

growth

0.15

MiB/s

Disk read throughput

Disk read throughput in MiB/s

3.01

MiB/s

Disk write throughput

Disk write throughput in MiB/s

52.97

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.47

5 min

1.68

15 min

1.96

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

24.32

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

mean

14.9373

16.7862

p95

51.31

60 second average

14.8407

300 second average

17.9746

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

2.2199

5.4757

p95

11.32

60 second average

2.2923

300 second average

3.3957

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.1864

3.1413

p95

5.72

60 second average

1.2196

300 second average

1.6403

Holochain process CPU usage

CPU usage by Holochain process

mean

39.55

30.81

p95

87.48

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

50.59

KiB

mean

192.29

KiB

114.71

KiB

p95

385.08

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

27.4

threads

9.05

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

74.19

file descriptors

20.33

file descriptors

p95

file descriptors

Write Validated `must_get_agent_activity`

↑ Back to index

This test is similar to Mixed-Arc must_get_agent_activity, but all agents are full-arc.

Started

Thu, 24 Sep 2026 04:41:53 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

must_get_agent_activity (1 agent)
write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

write_validated_must_get_agent_activity_35946754846_1

Highest observed action_seq

Mean max (across write agents)

Mean of the highest chain head value observed per write agent. Each write agent contributes its maximum — the value seen by any reader.

432.63

Max

Highest chain head value observed for any single write agent.

1083

Write agents observed

Chain batch delay timing

The time between a write agent's creation of a batch and a must_get_agent_activity agent's successful discovery of the batch and creation/self-validation of a new entry that depends on it.

Mean

mean

22.120482

58.722153

Max

Highest per-partition mean latency

max mean

924.92775

Min

Lowest per-partition mean latency

min mean

-938.855

Chain batch delay rate

The rate at which a must_get_agent_activity agent was able to discover batches and create/self-validate new entries that depend on them.

Mean

mean

1.3

Max

Highest per-partition mean rate

max

7.33

Min

Lowest per-partition mean rate

min

create_validated_sample_entry zome call timing

The time taken to complete the zome function call that creates the entry that depends on a write agent's source chain.

Mean

mean

0.126455

0.068041

Max

Highest per-partition mean latency

max mean

0.470563

Min

Lowest per-partition mean latency

min mean

0.017036

Retrieval errors

A running accumulation of the errors encountered by an agent when attempting to self-validate actions that depend on must_get_agent_activity calls.

Total

38750

Agents affected

85 / 85

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

456

Max

Highest count in any single partition

729

Min

Lowest count in any single partition

Final error count

The total number of all types of error accumulated over the run by all agents.

121376

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.025912

0.700459

p50

0.001465

p95

0.077594

p99

4.864374

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 760.02460752s

Not enough time data to show a trend.

Zome call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.008222

0.004446

p50

0.00751

p95

0.01373

p99

0.032357

Zome call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.361984

0.264185

p50

0.27226

p95

0.869983

p99

1.244904

Zome call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.01603

0.009918

p50

0.010627

p95

0.031936

p99

0.05677

Zome call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of this zome call as measured by Holochain internally.

mean

0.030942

0.844802

p50

0.008407

p95

0.017138

p99

6.076634

Zome call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of this zome call as measured by Holochain internally.

mean

0.03565

1.002959

p50

0.008241

p95

0.577515

p99

4.873195

WASM call duration: 'validated_must_get_agent_activity::announce_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006787

0.003898

p50

0.006335

p95

0.011795

p99

0.028664

WASM call duration: 'validated_must_get_agent_activity::create_sample_entries_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.359154

0.263166

p50

0.269555

p95

0.865369

p99

1.239818

WASM call duration: 'validated_must_get_agent_activity::create_validated_sample_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.013487

0.008926

p50

0.008712

p95

0.027987

p99

0.052336

WASM call duration: 'validated_must_get_agent_activity::get_chain_top_for_batch'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007889

0.809137

p50

0.007024

p95

0.016787

p99

5.77559

WASM call duration: 'validated_must_get_agent_activity::get_random_agent_with_write_behaviour'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.033854

1.002985

p50

0.006886

p95

0.576979

p99

4.871823

WASM call duration: 'validated_must_get_agent_activity_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.002103

0.000998

p50

0.0022

p95

0.004139

p99

0.005658

WASM call duration: 'validated_must_get_agent_activity_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.021483

0.018071

p50

0.015402

p95

0.057461

p99

0.080276

Host function call duration: '__hc__agent_info_1'

Duration of this host function call invoked from within WASM.

mean

0.000108

8.2e-05

p50

4.9e-05

p95

0.000228

p99

0.000405

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.007992

0.00922

p50

0.002352

p95

0.02188

p99

0.050138

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.011603

0.011631

p50

0.000962

p95

0.028716

p99

0.064399

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001609

0.912449

p50

0.000944

p95

0.003404

p99

5.769211

Host function call duration: '__hc__must_get_agent_activity_1'

Duration of this host function call invoked from within WASM.

mean

0.027247

0.028241

p50

0.02338

p95

0.084791

p99

0.115812

Host function call duration: '__hc__query_1'

Duration of this host function call invoked from within WASM.

mean

0.133934

0.076816

p50

0.108051

p95

0.239708

p99

0.292864

Host function call duration: '__hc__random_bytes_1'

Duration of this host function call invoked from within WASM.

mean

1.3e-05

1e-05

p50

1.2e-05

p95

1.9e-05

p99

5.9e-05

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.004478

0.002956

p50

0.003712

p95

0.010425

p99

0.01565

Post-commit duration

Time spent executing post-commit workflows.

mean

0.002995

0.00286

p50

0.000543

p95

0.006484

p99

0.013794

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.7

mean

430.57

248.96

p95

817.01

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

3569

over 889.821927296s

mean rate

956801.72

std

1.038115698e+07

1191.48

p95

1.88062155e+06

peak

5.3211736717e+08

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

83.905979

58.238766

p50

20.911829

p95

157.234524

p99

291.29392

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

2.263917

1.546624

p50

2.162647

p95

5.493418

p99

8.462337

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

1e-06

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

1.048655

0.489032

p50

0.651273

p95

1.47653

p99

1.626374

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.607557

0.58434

p50

0.138351

p95

0.921286

p99

3.148891

System validation workflow duration

Time spent running the sys validation workflow.

mean

13.793712

17.775165

p50

15.561546

p95

58.73312

p99

87.768145

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

12.999893

26.265367

p50

0.104486

p95

55.178458

p99

141.97005

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00569

0.004389

p50

0.005319

p95

0.014778

p99

0.017557

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.006245

0.004425

p50

0.003908

p95

0.012401

p99

0.017056

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.006628

0.008588

p50

0.001884

p95

0.019835

p99

0.04766

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

1.000383

28.333077

p50

0.721936

p95

60.00755

p99

60.031221

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

2.025852

1.459765

p50

1.924538

p95

4.858661

≤

p99

4.858661

P2P request roundtrip: 'must_get_agent_activity'

Time spent sending a must_get_agent_activity request and awaiting its response.

mean

0.33053

2.922902

p50

0.397818

p95

6.378979

≤

p99

6.378979

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

1.82793

27.808156

p50

3.312859

p95

60.016079

p99

60.033129

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

48.05

Network receive rate (primary)

Rate of bytes received on primary network interface

13.80

KiB/s

mean

825.13

KiB/s

533.25

KiB/s

p95

1.51

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

B/s

mean

83.41

KiB/s

102.91

KiB/s

p95

280.71

KiB/s

Total bytes received

Total bytes received on primary network interface

count

102.88

GiB

mean

118.37

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.40

GiB

mean

11.97

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 98.3%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (49.6% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 89% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

20.66

mean

70.38

21.09

p95

93.07

System

22.33

CPU percentiles

CPU usage percentiles

p50

76.07

p95

93.07

p99

98.35

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

136

hosts

mean time

416.22

Memory used percentage

Percentage of memory used

7.62

mean

13.08

8.05

p95

25.46

Memory available percentage

Percentage of available memory

74.54

mean

86.92

8.05

p95

92.38

Max host memory used

Maximum memory usage percentage across all hosts

max

90.98

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

49.57

Memory growth rate

Rate of memory growth over time

growth

0.43

MiB/s

Disk read throughput

Disk read throughput in MiB/s

6.56

MiB/s

Disk write throughput

Disk write throughput in MiB/s

117.41

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

4.02

5 min

3.18

15 min

2.41

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

89.19

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

2.07

mean

53.6076

26.9863

p95

82.77

60 second average

50.2433

300 second average

39.0004

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.01

0.4794

≤

p95

60 second average

0.0095

300 second average

0.0068

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0092

0.4716

≤

p95

60 second average

0.0086

300 second average

0.0060

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

11.9041

8.732

p95

25.59

60 second average

11.2421

300 second average

8.3947

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

4.5453

4.3583

p95

12.7

60 second average

4.3476

300 second average

3.3919

Holochain process CPU usage

CPU usage by Holochain process

29.73

mean

68.07

18.03

p95

88.79

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

157.80

KiB

mean

385.76

KiB

103.62

KiB

p95

516.97

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

35.09

threads

6.43

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

82.27

file descriptors

12.57

file descriptors

p95

file descriptors

Zero-Arc Create and Read

↑ Back to index

A zero-arc/full-arc mixed scenario with two types of zero-arc nodes -- ones that create data and ones that read data -- as well as full arc nodes to "relay" the data. The scenario has three roles:

zero_write: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that zero_read nodes can retrieve them.
zero_read: A zero-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.
full: A full-arc conductor that is just here to serve entries to zero arc nodes.

Started

Thu, 24 Sep 2026 06:08:08 UTC

Duration

900s

Peer count

150

Peer count at end

148

Behaviours

full (1 agent)
zero_read (1 agent)
zero_write (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_and_read_35946754846_1

Create rate

The number of timed entries created by the zero-arc write node(s) per second.

Mean

mean

45.44

Max

Highest per-partition mean rate

max

82.22

Min

Lowest per-partition mean rate

min

29.16

Fetch lag timing

For each entry, the time lag between when it was created and when a zero-arc read node fetched it from the network via the get_timed_entries_network zome function.

Mean

mean

321.733356

115.311196

Max

Highest per-partition mean latency

max mean

1151.768345

Min

Lowest per-partition mean latency

min mean

17.545789

Fetch lag rate

The number of entries per second fetched by zero-arc read nodes.

Mean

mean

25.22

Max

Highest per-partition mean rate

max

275.14

Min

Lowest per-partition mean rate

min

Delivery ratio

The proportion of created entries successfully fetched by all zero-arc read nodes.

0.0008

Entry created count

Total number of entries created by zero-arc write nodes, by agent.

Total

247529

Agents affected

60 / 60

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4125

Max

Highest count in any single partition

7753

Min

Lowest count in any single partition

2598

Received count

Total number of entries fetched by zero-arc read nodes, by agent.

Total

12156

Agents affected

52 / 58

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

210

Max

Highest count in any single partition

2441

Min

Lowest count in any single partition

Open connections

The number of currently open connections to other conductors.

full-arc

mean

55.52

25.54

p95

125

zero-arc

mean

22.98

14.79

p95

Error count

The number of errors accumulated across all nodes.

611

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.061351

1.061843

p50

0.273195

p95

2.67193

p99

5.12872

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

over 790.016560671s

Not enough time data to show a trend.

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

206869536

over 876.640130353s

mean rate

7.54152047782e+09

std

7.76338070272e+10

6.17834664e+06

p95

9.12166800503e+09

peak

2.22561028939e+12

WASM usage: 'timed::get_timed_entries_network'

Metered usage count of the WASM ribosome for this zome function.

total

16089989

over 876.817163903s

mean rate

1.16486044437e+10

std

2.82249547375e+10

7.10008962e+06

p95

7.36009291247e+10

peak

1.69095448249e+11

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

over 888.465187158s

Not enough time data to show a trend.

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.009394

0.003887

p50

0.010522

p95

0.016773

p99

0.019758

Zome call duration: 'timed::get_timed_entries_network'

Duration of this zome call as measured by Holochain internally.

mean

3.145573

40.237524

p50

7.942451

p95

99.855006

p99

278.381722

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.00825

0.003403

p50

0.009159

p95

0.014809

p99

0.017449

WASM call duration: 'timed::get_timed_entries_network'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

3.145611

40.687944

p50

7.941399

p95

99.85345

p99

278.38019

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001559

0.00047

p50

0.001919

p95

0.002644

p99

0.003089

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000743

0.000554

p50

0.000664

p95

0.001736

p99

0.002896

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000681

0.000453

p50

0.000659

p95

0.001484

p99

0.002359

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.0607

1.456939

p50

0.259252

p95

2.898715

p99

6.028828

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.111735

0.866208

p50

0.176569

p95

2.139607

p99

4.763693

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002486

0.000797

p50

0.0031

p95

0.004251

p99

0.004906

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000504

0.002386

p50

0.000427

p95

0.003336

p99

0.010678

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.65

mean

423.38

245.1

p95

804.3

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

over 888.465128536s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

7.137897

24.110352

p50

p95

4.174369

p99

18.730104

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.228662

0.244624

p50

0.002067

p95

0.541706

p99

1.183367

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.068045

0.461978

p50

0.002265

p95

1.294224

p99

1.411433

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.161919

0.079717

p50

0.031862

p95

0.235399

p99

0.268796

System validation workflow duration

Time spent running the sys validation workflow.

mean

9.484236

9.039652

p50

0.003323

p95

21.822949

p99

35.139596

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

3.250089

8.659046

p50

0.001406

p95

0.514904

p99

60.802412

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.00305

0.002674

p50

0.001749

p95

0.007854

p99

0.011127

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007018

0.003535

p50

0.003303

p95

0.012203

p99

0.015338

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000677

0.006245

p50

0.000797

p95

0.006528

p99

0.023437

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

0.328168

29.661923

p50

0.708995

p95

60.019443

p99

60.038734

P2P request roundtrip: 'get_links'

Time spent sending a get_links request and awaiting its response.

mean

0.080086

14.832829

p50

0.120768

p95

60.001202

p99

60.001379

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

2.490954

5.300302

p50

0.602323

p95

7.184443

p99

11.75774

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

34.97

Network receive rate (primary)

Rate of bytes received on primary network interface

6.26

KiB/s

mean

600.90

KiB/s

569.23

KiB/s

p95

1.44

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

328

B/s

mean

89.47

KiB/s

151.65

KiB/s

p95

406.93

KiB/s

Total bytes received

Total bytes received on primary network interface

count

73.66

GiB

mean

84.75

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

10.97

GiB

mean

12.62

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (69.5% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 50% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

2.42

mean

47.97

37.04

p95

88.21

System

13.00

CPU percentiles

CPU usage percentiles

p50

67.99

p95

88.21

p99

95.18

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

hosts

mean time

447.92

Memory used percentage

Percentage of memory used

6.56

mean

11.65

7.41

p95

23.88

Memory available percentage

Percentage of available memory

76.12

mean

88.35

7.41

p95

93.44

Max host memory used

Maximum memory usage percentage across all hosts

max

82.41

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

69.55

Memory growth rate

Rate of memory growth over time

growth

0.35

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.64

MiB/s

Disk write throughput

Disk write throughput in MiB/s

69.89

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/122

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

1.84

5 min

1.84

15 min

1.97

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

50.34

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

mean

21.6766

20.9841

p95

53.96

60 second average

20.9501

300 second average

20.7385

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

3.6328

5.3547

p95

14.97

60 second average

3.5289

300 second average

3.3082

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.6886

2.985

p95

6.73

60 second average

1.6418

300 second average

1.6163

Holochain process CPU usage

CPU usage by Holochain process

0.95

mean

44.62

35.06

p95

84.71

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

203.85

KiB

mean

272.57

KiB

59.54

KiB

p95

372.61

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

29.96

threads

5.29

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

75.64

file descriptors

11.74

file descriptors

p95

file descriptors

Zero-Arc Create Data

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 24 Sep 2026 05:16:10 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data--high_zero_35946754846_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

47.3

Max

Highest per-partition mean rate

max

87.35

Min

Lowest per-partition mean rate

min

30.61

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

238.556494

15.101389

Max

Highest per-partition mean latency

max mean

381.570572

Min

Lowest per-partition mean latency

min mean

95.542416

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

Max

Highest per-partition mean rate

max

Min

Lowest per-partition mean rate

min

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0001

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

584172

Agents affected

140 / 140

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

4173

Max

Highest count in any single partition

7655

Min

Lowest count in any single partition

2727

Received count

Total number of entries received by full-arc nodes, by agent.

Total

285

Agents affected

2 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

Max

Highest count in any single partition

156

Min

Lowest count in any single partition

Open connections

The number of currently open connections to other conductors.

full-arc

mean

76.22

53.38

p95

148

zero-arc

mean

19.28

12.88

p95

Error count

The number of errors accumulated across all nodes.

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.004728

0.002855

p50

0.001002

p95

0.010427

p99

0.011027

WASM usage: 'timed::created_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

396256224

over 833.829786857s

mean rate

1.62503831257e+10

std

2.79933345238e+11

8.11236921e+06

p95

3.24917199887e+10

peak

1.37538582738e+13

WASM usage: 'timed::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

194928720

over 882.136648456s

mean rate

9.948059843e+08

std

1.67860214474e+09

6.30616699e+06

p95

4.41611763284e+09

peak

4.48835114367e+09

WASM usage: 'timed_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

54410562

over 889.486890656s

mean rate

1.9961986439e+09

std

3.56751770753e+10

p95

3.23547060022e+09

peak

1.85509480595e+12

Zome call duration: 'timed::created_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008707

0.003034

p50

0.008706

p95

0.0149

p99

0.015939

Zome call duration: 'timed::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.021172

0.183154

p50

0.017143

p95

0.607799

p99

0.832576

WASM call duration: 'timed::created_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007572

0.002619

p50

0.007526

p95

0.012983

p99

0.013869

WASM call duration: 'timed::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.019926

0.182505

p50

0.015201

p95

0.604179

p99

0.828949

WASM call duration: 'timed_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001527

0.000729

p50

0.001537

p95

0.002554

p99

0.005482

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000516

0.00022

p50

0.000502

p95

0.000983

p99

0.001121

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.000501

0.000221

p50

0.000475

p95

0.000952

p99

0.001113

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.007822

0.002318

p50

0.007448

p95

0.012142

p99

0.013538

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001122

0.008637

p50

0.001073

p95

0.030894

p99

0.037015

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002464

0.002942

p50

0.00237

p95

0.004393

p99

0.020628

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000458

0.001849

p50

0.000529

p95

0.001256

p99

0.004444

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.75

mean

430.16

248.71

p95

815.29

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

over 889.486893355s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

0.648145

22.79651

p50

≤

p95

p99

1.823529

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

0.156719

0.208187

p50

0.001876

p95

0.05855

p99

1.588066

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

2e-06

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.010368

0.267899

p50

0.001861

p95

0.981796

p99

1.257823

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.157401

0.075855

p50

0.137326

p95

0.265751

p99

0.303372

System validation workflow duration

Time spent running the sys validation workflow.

mean

6.910444

8.747587

p50

0.002848

p95

6.403344

p99

37.652236

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

0.03256

0.143391

p50

0.000832

p95

0.003956

p99

0.020743

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.007889

0.003633

p50

0.008377

p95

0.012947

p99

0.014039

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007943

0.004712

p50

0.00394

p95

0.014956

p99

0.01685

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000446

0.007547

p50

0.000413

p95

0.000926

p99

0.01993

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

13.309339

27.617803

p50

15.049915

p95

60.107108

p99

60.123849

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

1.694019

22.080756

p50

0.934165

p95

60.089409

p99

60.090703

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

50.65

Network receive rate (primary)

Rate of bytes received on primary network interface

15.65

KiB/s

mean

443.51

KiB/s

448.65

KiB/s

p95

1.32

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

958

B/s

mean

45.33

KiB/s

53.44

KiB/s

p95

156.19

KiB/s

Total bytes received

Total bytes received on primary network interface

count

55.33

GiB

mean

63.66

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

5.65

GiB

mean

6.51

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.2%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (67.7% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 57% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

18.09

mean

67.12

19.76

p95

89.59

System

16.47

CPU percentiles

CPU usage percentiles

p50

70.41

p95

89.59

p99

95.22

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

131

hosts

mean time

292.88

Memory used percentage

Percentage of memory used

6.96

mean

11.54

7.52

p95

23.41

Memory available percentage

Percentage of available memory

76.59

mean

88.46

7.52

p95

93.04

Max host memory used

Maximum memory usage percentage across all hosts

max

85.21

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

67.73

Memory growth rate

Rate of memory growth over time

growth

0.30

MiB/s

Disk read throughput

Disk read throughput in MiB/s

0.03

MiB/s

Disk write throughput

Disk write throughput in MiB/s

88.00

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.17

5 min

2.20

15 min

2.46

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

57.43

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

1.4

mean

23.3839

15.4325

p95

50.57

60 second average

22.5890

300 second average

23.7229

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0004

0.032

≤

p95

60 second average

0.0003

300 second average

0.0002

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

mean

0.0004

0.0303

≤

p95

60 second average

0.0003

300 second average

0.0001

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

3.6002

4.0005

p95

9.78

60 second average

3.5143

300 second average

3.7869

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

2.2458

2.8681

p95

6.05

60 second average

2.1975

300 second average

2.3331

Holochain process CPU usage

CPU usage by Holochain process

28.84

mean

64.48

17.09

p95

85.94

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

211.72

KiB

mean

280.79

KiB

53.14

KiB

p95

369.40

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

31.76

threads

3.97

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

79.23

file descriptors

8.31

file descriptors

p95

file descriptors

Zero-arc Create Data (Validated)

↑ Back to index

A zero-arc/full-arc mixed scenario where zero-arc nodes create data that gets validated and full-arc nodes read the data. The scenario has two roles:

zero: A zero-arc conductor that just creates entries with a timestamp field. Those entries are linked to a known base hash so that full-arc nodes can retrieve them.
full: A full-arc conductor that reads the entries created by the zero-arc node(s) and records the time lag between when the entry had been created and when it was first discovered.

Started

Thu, 24 Sep 2026 05:50:47 UTC

Duration

900s

Peer count

150

Peer count at end

149

Behaviours

full (1 agent)
zero (1 agent)

Holochain version

0.7.0

Wind Tunnel version

0.7.0

Run ID

zero_arc_create_data_validated--high_zero_35946754846_1

Create rate

The number of timed entries created by the zero-arc node(s) per second.

Mean

mean

42.68

Max

Highest per-partition mean rate

max

82.73

Min

Lowest per-partition mean rate

min

28.18

Sync lag timing

For each entry, the time lag between when it was created and when the full-arc node could read it via the get_timed_local_entries zome function.

Mean

mean

85.84155

31.418623

Max

Highest per-partition mean latency

max mean

281.84835

Min

Lowest per-partition mean latency

min mean

15.02008

Sync lag rate

The number of entries per second received by full nodes.

Mean

mean

Max

Highest per-partition mean rate

max

Min

Lowest per-partition mean rate

min

Delivery ratio

The proportion of created entries successfully received by all full-arc nodes.

0.0003

Entry created count

Total number of entries created by zero-arc nodes, by agent.

Total

512833

Agents affected

140 / 140

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

3663

Max

Highest count in any single partition

6929

Min

Lowest count in any single partition

2426

Received count

Total number of entries received by full-arc nodes, by agent.

Total

1203

Agents affected

3 / 9

Mean per agent

Total count divided by number of partitions, rounded to nearest whole number

134

Max

Highest count in any single partition

1201

Min

Lowest count in any single partition

Open connections

The number of currently open connections to other conductors.

full-arc

mean

80.14

50.31

p95

148

zero-arc

mean

18.81

15.81

p95

Error count

The number of errors accumulated across all nodes.

Holochain MetricsClick to toggle visibility.

Cascade duration

Time taken to execute a cascade (get) query inside Holochain.

mean

0.009463

0.004878

p50

0.009344

p95

0.01789

p99

0.020086

Cascade fetch errors

Count of network fetch errors during cascade calls.

total

4896

over 753.232983186s

mean rate

7054.65

std

11858.78

53.44

p95

38386.37

peak

48605.05

WASM usage: 'timed_and_validated::create_timed_entry'

Metered usage count of the WASM ribosome for this zome function.

total

196192092

over 828.366846463s

mean rate

2.85991317921e+09

std

9.74743706053e+09

6.64263275e+06

p95

1.44979358177e+10

peak

1.78379382503e+11

WASM usage: 'timed_and_validated::get_timed_entries_local'

Metered usage count of the WASM ribosome for this zome function.

total

114251334

over 821.344193788s

mean rate

1.94086030578e+09

std

3.46693301111e+09

3.540657205e+07

p95

1.03643625592e+10

peak

1.20263649609e+10

WASM usage: 'timed_and_validated_integrity::entry_defs'

Metered usage count of the WASM ribosome for this zome function.

total

116145582

over 888.36740837s

mean rate

2.24572330174e+09

std

1.26373122158e+10

p95

9.47881860385e+09

peak

7.95219524564e+11

WASM usage: 'timed_and_validated_integrity::validate'

Metered usage count of the WASM ribosome for this zome function.

total

918537856

over 888.36740837s

mean rate

2.61131010612e+10

std

3.90743048111e+11

5.49389876e+06

p95

8.66528110845e+10

peak

2.89187519493e+13

Zome call duration: 'timed_and_validated::create_timed_entry'

Duration of this zome call as measured by Holochain internally.

mean

0.008838

0.00296

p50

0.009226

p95

0.014923

p99

0.015523

Zome call duration: 'timed_and_validated::get_timed_entries_local'

Duration of this zome call as measured by Holochain internally.

mean

0.047348

0.21222

p50

0.021786

p95

0.639406

p99

1.089805

WASM call duration: 'timed_and_validated::create_timed_entry'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.007703

0.002556

p50

0.007989

p95

0.012969

p99

0.013449

WASM call duration: 'timed_and_validated::get_timed_entries_local'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.044706

0.212495

p50

0.018966

p95

0.637644

p99

1.088014

WASM call duration: 'timed_and_validated_integrity::entry_defs'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.001544

0.000748

p50

0.001547

p95

0.002925

p99

0.005131

WASM call duration: 'timed_and_validated_integrity::validate'

Duration of inner WASM calls for this function, excluding Holochain overhead.

mean

0.006994

0.004746

p50

0.006324

p95

0.013866

p99

0.030799

Host function call duration: '__hc__create_1'

Duration of this host function call invoked from within WASM.

mean

0.000485

0.000185

p50

0.00051

p95

0.000883

p99

0.000995

Host function call duration: '__hc__create_link_1'

Duration of this host function call invoked from within WASM.

mean

0.00047

0.000184

p50

0.000477

p95

0.000866

p99

0.000916

Host function call duration: '__hc__get_1'

Duration of this host function call invoked from within WASM.

mean

0.005097

0.001425

p50

0.003425

p95

0.006632

p99

0.006881

Host function call duration: '__hc__get_links_1'

Duration of this host function call invoked from within WASM.

mean

0.001793

0.003917

p50

0.001454

p95

0.013196

p99

0.019517

Host function call duration: '__hc__must_get_valid_record_1'

Duration of this host function call invoked from within WASM.

mean

0.010589

0.004646

p50

0.009816

p95

0.018046

p99

0.020229

Host function call duration: '__hc__zome_info_1'

Duration of this host function call invoked from within WASM.

mean

0.002694

0.002744

p50

0.002504

p95

0.007223

p99

0.017346

Post-commit duration

Time spent executing post-commit workflows.

mean

0.000562

0.002264

p50

0.000668

p95

0.001355

p99

0.016727

Conductor uptime

Conductor uptime gauge. A drop in the trend indicates a restart during the run.

43.64

mean

429.65

248.4

p95

815.13

Integrated ops

Count of DHT ops integrated since the conductor started. Resets on restart.

total

over 888.367422344s

Not enough time data to show a trend.

Integration delay

Delay between an op being stored and being integrated. High values indicate the pipeline is falling behind.

mean

3.348075

14.663436

p50

p95

0.357143

p99

1.890411

Validation attempts per op

Number of validation attempts required per op. Values consistently above 1 indicate retries.

mean

p50

≤

p95

≤

p99

App validation workflow duration

Time spent running the app validation workflow.

mean

4.755032

2.207314

p50

0.00198

p95

2.242507

p99

12.184418

Countersigning workflow duration

Time spent running the countersigning workflow.

mean

1e-06

p50

≤

p95

≤

p99

Integrate DHT ops workflow duration

Time spent running the integration workflow.

mean

0.011606

0.262474

p50

0.002292

p95

0.865456

p99

1.295021

Publish DHT ops workflow duration

Time spent running the publish workflow.

mean

0.135448

0.062592

p50

0.114129

p95

0.220946

p99

0.245705

System validation workflow duration

Time spent running the sys validation workflow.

mean

3.758686

2.341029

p50

0.003082

p95

2.27992

p99

17.645876

Validation receipt workflow duration

Time spent running the validation receipt workflow.

mean

1.093249

3.902157

p50

0.001325

p95

0.014106

p99

0.533483

DHT DB connection use time

Time spent holding DHT database connections.

mean

0.006918

0.003494

p50

0.006677

p95

0.012288

p99

0.013007

Write transaction duration

Duration of exclusive write transactions across all databases.

mean

0.007202

0.003692

p50

0.003716

p95

0.011941

p99

0.013702

Lair keystore: signing

Duration of Ed25519 signing requests to the Lair keystore.

mean

0.000465

0.003677

p50

0.00041

p95

0.00112

p99

0.023223

P2P request roundtrip: 'get'

Time spent sending a get request and awaiting its response.

mean

18.275931

27.708117

p50

13.036746

p95

60.075292

p99

60.087637

P2P request roundtrip: 'send_validation_receipts'

Time spent sending a send_validation_receipts request and awaiting its response.

mean

0.29851

0.722282

p50

0.274735

p95

2.10614

≤

p99

2.10614

Host MetricsClick to toggle visibility.

User CPU usage

CPU usage by user space

49.35

Network receive rate (primary)

Rate of bytes received on primary network interface

578

B/s

mean

374.69

KiB/s

356.48

KiB/s

p95

1.03

MiB/s

Network send rate (primary)

Rate of bytes sent on primary network interface

B/s

mean

37.89

KiB/s

44.32

KiB/s

p95

132.11

KiB/s

Total bytes received

Total bytes received on primary network interface

count

46.39

GiB

mean

53.37

MiB/s

Total bytes sent

Total bytes sent on primary network interface

count

4.69

GiB

mean

5.40

MiB/s

CPU spike anomaly

CPU spike anomaly was detected

Detected

Warning CPU p99 reached 95.5%

Memory leak anomaly

Memory leak anomaly was detected

NotDetected

Disk full anomaly

Disk full anomaly was detected

NotDetected

Swap thrashing anomaly

Swap thrashing anomaly was detected

Detected

Critical Heavy swap usage (70.6% swap used)

System overload anomaly

System overload anomaly was detected

Detected

Warning 45% of hosts overloaded (load5/ncpus > 1.0)

CPU usage

Total CPU usage and kernel CPU usage

Total

16.13

mean

67.03

20.05

p95

88.29

System

17.69

CPU percentiles

CPU usage percentiles

p50

73.06

p95

88.29

p99

95.49

CPU usage above 80%

Number of hosts above 80% CPU and mean time spent above threshold for those hosts

count

128

hosts

mean time

210.56

Memory used percentage

Percentage of memory used

7.77

mean

13.16

7.68

p95

25.6

Memory available percentage

Percentage of available memory

74.4

mean

86.84

7.68

p95

92.23

Max host memory used

Maximum memory usage percentage across all hosts

max

85.44

Max host swap used percentage

Maximum swap space usage percentage across all hosts

max

70.58

Memory growth rate

Rate of memory growth over time

growth

0.35

MiB/s

Disk read throughput

Disk read throughput in MiB/s

1.24

MiB/s

Disk write throughput

Disk write throughput in MiB/s

77.49

MiB/s

Disk space utilization risk

Number of hosts nearing disk space capacity by mount point

Mount Point /

0/123

hosts

Mount Point /efi-boot

0/2

hosts

Mount Point /etc/hostname

0/13

hosts

Mount Point /etc/hosts

0/13

hosts

Mount Point /etc/nomad.d/client-meta.json

0/12

hosts

Mount Point /etc/resolv.conf

0/13

hosts

Mount Point /nix/store

0/2

hosts

System load average

System load averages over 1, 5, and 15 minutes. This is an unnormalised value not divided by number of CPUs, so it is only meaningful if all machines have the same core count.

1 min

2.17

5 min

2.10

15 min

2.16

CPU overloaded hosts

Percentage of hosts that experienced CPU overload

44.59

CPU pressure

CPU pressure over 10, 60, and 300 second averages

10 second average

0.81

mean

25.5132

15.2203

p95

54.48

60 second average

24.5704

300 second average

23.5973

Memory pressure some

Memory pressure (some tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

Memory pressure full

Memory pressure (all tasks stalled) over 10, 60, and 300 second averages

10 second average

≤

mean

≤

p95

60 second average

0.0000

300 second average

0.0000

I/O pressure some

I/O pressure (some tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

3.2714

3.4716

p95

9.78

60 second average

3.1710

300 second average

3.0642

I/O pressure full

I/O pressure (all tasks stalled) over 10, 60 and 300 second averages

10 second average

mean

1.894

2.3875

p95

4.55

60 second average

1.8246

300 second average

1.7080

Holochain process CPU usage

CPU usage by Holochain process

25.46

mean

64.59

17.74

p95

82.28

Holochain process memory (PSS)

Proportional Set Size memory of Holochain process

186.76

KiB

mean

376.97

KiB

87.04

KiB

p95

494.35

KiB

Holochain process threads

Number of threads in Holochain process

threads

mean

33.16

threads

4.5

threads

p95

threads

Holochain process file descriptors

Number of file descriptors used by Holochain process

file descriptors

mean

79.29

file descriptors

8.96

file descriptors

p95

file descriptors

DHT Sync Lag

Mixed Arc get_agent_activity

Mixed Arc must_get_agent_activity

Remote Call Rate

Remote Signals

Validation Receipts

Write/get_agent_activity

Write/get_agent_activity with volatile conductors

Write Validated must_get_agent_activity

Zero-Arc Create and Read

Zero-Arc Create Data

Zero-arc Create Data (Validated)

Mixed Arc `get_agent_activity`

Mixed Arc `must_get_agent_activity`

Write/`get_agent_activity`

Write/`get_agent_activity` with volatile conductors

Write Validated `must_get_agent_activity`