Actor Troubleshooting
Before you follow this section, consider configuring the log level to debug for more information when debugging.
When developing actors, there are a few common errors that can cause your actor to fail to run or properly operate. Below we'll walk through a few of these scenarios with accompanying error messages.
Failing to start an actor from a local registry over HTTP
The following error can happen when attempting to start an actor from a local registry server over HTTP:
2023-12-19T22:31:26.154791Z ERROR wasmcloud_host::wasmbus: failed to scale actor actor_ref=127.0.0.1:5000/v2/hello:0.1.0 err=failed to fetch actor
Caused by:
0: failed to fetch actor under OCI reference `127.0.0.1:5000/v2/hello:0.1.0`
1: failed to fetch OCI path
2: failed to fetch OCI bytes
3: error sending request for url (https://127.0.0.1:5000/v2/): error trying to connect: received corrupt message of type InvalidContentType
To fix this error, you need to set the --allowed-insecure argument on the host.
When accessing a local registry from within a docker compose network, you should use registry:5000 (or the name of the registry container, if it differs) instead of localhost:5000.
Even though port 5000 may be available on your host's network, networking works differently within docker, and you should use the name of the container instead of localhost.
Failing to invoke an actor
Missing Capability Claims
When a wasmCloud actor needs access to a capability provider, we require cryptographically signed claims embedded in that actor. The host runtime operates on a deny-by-default basis and require claims to be an allow-list to ensure our actors are secure.
Adding these capability claims is mostly managed for you with wash when you build your actor and the list of capability claims in wasmcloud.toml included with any actor project.
Consider the KVCounter example actor, which uses:
- the
wasmcloud:httpservercapability to receive HTTP requests - the
wasmcloud:keyvaluecapability to issue requests to a keyvalue database
Omitting either of these capabilities from the actor's claims will cause errors at runtime, as the wasmCloud host will deny the invocation:
2023-12-19T22:34:56.110198Z ERROR handle_rpc_message: wasmcloud_host::wasmbus: failed to handle request origin=WasmCloudEntity { public_key: "VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M", link_name: "default", contract_id: "wasmcloud:httpserver" } target=WasmCloudEntity { public_key: "MAKSODTVGPWU7AJ7F6CW2L4YNDBHU7MH2F6TX363TZKID32V2RURFNLP", link_name: "", contract_id: "" } operation="HttpServer.HandleRequest" invocation_id="08555444d0e84c6593d06fc5abb66e3f" e=failed to handle invocation
Caused by:
actor does not have capability claim `wasmcloud:httpserver`
To fix this error, make sure the actor's wasmcloud.toml file includes the appropriate capability claims:
[actor]
claims = ["wasmcloud:httpserver"]
Missing Runtime Links
Invocations are only allowed between actors and providers that have been linked. Omitting a link will result in a runtime error when the host denies the invocation:
{"error":"Host send error failed to call `wasmcloud:bus/host.call`: failed to call linked provider: unknown actor call alias"}
To fix this error, make sure actors and providers are linked to each other appropriately. To inspect the current set of runtime links, run:
wash get links
The command line wasmCloud shell (aka wash) attempts to safeguard against misspellings, though it is unable to determine if a particular link is between the intended actor and provider.
If any of the actor ID, provider ID, link name, or contract ID are misspelled or omitted, then the host will deny the invocation.
Debugging Invocations
Note that the following commands are experimental and may change in the future. To use them, supply the --experimental flag with each command, or set the WASH_EXPERIMENTAL environment variable to true.
wash spy
wash spy monitors all invocations to/from a given actor. This can be useful for debugging communication issues and identifying logic bugs. To use it, run:
wash spy <actor name or ID>
You can pass the actual actor ID if you wish, but for ease of use, wash will attempt to resolve the input to an ID by checking if the actor_name or call_alias fields from the actor's claims contains the given string (case-insensitive). If more than one matches, then an error will be returned, indicating the options to choose from.
You can try out this command with the KV Counter example:
$ wget https://raw.githubusercontent.com/wasmCloud/examples/main/actor/kvcounter/wadm.yaml
$ wash app put wadm.yaml
Successfully put manifest
$ wash app deploy kvcounter
Deployed model
$ wash spy kvcounter
Spying on actor kvcounter
[2023-06-21 11:24:32.397138 -06:00]
From: HTTP Server To: kvcounter Host: NC7XZN5VJW2EOK6E565MSRY3JW2WU5ABUA6HTSPY6O7MVKGZTFCR3TDM
Operation: HttpServer.HandleRequest
Message: {
"method": "GET",
"path": "/api/counter",
"queryString": "",
"header": {
"host": [
"127.0.0.1:8081"
],
"user-agent": [
"curl/7.88.1"
],
"accept": [
"*/*"
]
},
"body": []
}
[2023-06-21 11:24:32.407787 -06:00]
From: kvcounter To: Redis KeyValue Store Host: CB6S44L4Q5IIRD4BJGQXN5TJXF4YFJXD5DOYBZOYRJP5U2GEZFKXNGD4
Operation: KeyValue.Increment
Message: {
"key": "counter:default",
"value": 1
}
This output has the timestamp the invocation was observed at, the source and target of the invocation, and the content of the invocation itself. wash spy will attempt to decode the invocation and output it to JSON, but if it is unable to, it will output the raw bytes instead.
wash capture
This command enables you to debug an issue after it occurs by capturing a window of all invocations in the lattice (1 hour by default). This is similar to many gaming tools that constantly capture your video while you game, and then with a press of a button you can save the last 60s for later use. wash capture can be useful for debugging issues that are difficult to reproduce only occur in specific environments.
To use it, run:
wash capture --enable
By default, this will set up a sliding capture window of 1 hour (older invocations will be deleted). You can change this by passing the --window-size flag. For example, to set up a window of 5 minutes, you can run:
wash capture --enable --window-size 5
Once you want to inspect previous invocations, you can capture the last window by running:
wash capture
No messages received in the last second. Ending capture
Completed capture and output to file 2023-06-14T11:11:03.476732-06:00.default.washcapture
This command will run until it detects a message that is past the time you started the capture, or no new messages appear for 1s (whichever comes first). It will then create a file that contains all the messages that occurred in the last window, along with a dump of all hosts in the lattice and their inventories. This file is just a tarball, but since the structure of the tarball should not be relied upon, it has a .washcapture extension. The filename will be of the form <rfc3339 timestamp>.<lattice>.washcapture.
Using a washcapture file
You can use the wash capture replay command to replay a .washcapture file. For example:
$ wash capture replay 2023-06-14T11:11:03.476732-06:00.default.washcapture
This command will output all invocations in the order they were received. The output will look similar to below:
[2023-06-14 17:10:15.131165 +00:00:00]
From: VAG3QITQQ2ODAOWB5TTQSDJ53XK3SHBEIFNK4AYJ5RKAX2UNSCAPHA5M To: MCFMFDWFHGKELOXPCNCDXKK5OFLHBVEWRAOXR5JSQUD2TOFRE3DFPM7E Host: NC7XZN5VJW2EOK6E565MSRY3JW2WU5ABUA6HTSPY6O7MVKGZTFCR3TDM
Operation: HttpServer.HandleRequest
Message: {
"method": "GET",
"path": "/api/counter",
"queryString": "",
"header": {
"host": [
"127.0.0.1:8081"
],
"user-agent": [
"curl/7.88.1"
],
"accept": [
"*/*"
]
},
"body": []
}
[2023-06-14 17:10:15.151724 +00:00:00]
From: MCFMFDWFHGKELOXPCNCDXKK5OFLHBVEWRAOXR5JSQUD2TOFRE3DFPM7E To: VAZVC4RX54J2NVCMCW7BPCAHGGG5XZXDBXFUMDUXGESTMQEJLC3YVZWB Host: CB6S44L4Q5IIRD4BJGQXN5TJXF4YFJXD5DOYBZOYRJP5U2GEZFKXNGD4
Operation: KeyValue.Increment
Message: {
"key": "counter:default",
"value": 1
}
This output has the timestamp the invocation was observed, the source and target of the invocation, and the content of the invocation itself. wash spy will attempt to decode the invocation and output it to JSON, but if it is unable to, it will output the raw bytes instead. Because this can be done offline, wash capture does not attempt to fetch friendly names for each of the providers and actors.
You can also filter the output by passing the --provider-id and --actor-id flags. If you pass just one of the flags, it will filter all invocations that were set to or from that actor/provider. If both flags are passed, it will only output invocations between that actor and provider.
Lastly, there is an interactive mode, which can be enabled by passing the --interactive flag. You can then step through each invocation by pressing the Enter key.