Jorrit Salverda
Jorrit Salverda
PostsComments
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
i've seemed to have stabilized things by reducing redundant list operations on KV, so is stable for quite a while now. many many thanks for all your help! I'll add some more detail to the github issues and close them if it remains solved.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
yup working on that, but when it does don't want it to panic. might be https://github.com/zebp/worker-kv/blob/3c53503d21248b0b00ac3d7802a94848f2e22178/src/builder.rs#L174 that throws a panic on the limit reached error.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
lol. that's sounds like a blunt approach. love it. i'll first try to avoid the panic that causes this if it turns out to be within my reach.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
although according to that issue it's been fixed by an update to wasm-bindgen but I might have another dependency bringing in a faulty version.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
Ah, here's a ticket for my issue! https://github.com/cloudflare/workers-rs/issues/166
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
is there a way to kill the instance on a panic and ensure the next run is a fresh instance?
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
This could be the race condition discussed in https://blog.cloudflare.com/wasm-coredumps/ due to a panic not rejecting the promise.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
what i did just notice though is that as soon as a queue event leads to an exception all following executions no longer log from the rust code. the particular exception i see is caused by too many kv invocations: 
 js error: JsValue(Error: Too many API requests by single worker invocation.\nError: Too many API requests by single worker invocation.
 js error: JsValue(Error: Too many API requests by single worker invocation.\nError: Too many API requests by single worker invocation.
after this exception my process doesn't stop until it times out 99 seconds later (with the 120s timeout I use). This might happen because I run multiple async tasks concurrently with: 
let group_by_futures: Vec<_> = MeasurementGroupBy::iter()
    .map(|group_by| {
        self.execute_by_group(
            &location,
            &filtered_assets,
            start,
            &timezone,
            group_by,
        )
    })
    .collect();

let kv_requests_vec = try_join_all(group_by_futures)
    .await
    .map_err(map_to_boxed_error)?;
let group_by_futures: Vec<_> = MeasurementGroupBy::iter()
    .map(|group_by| {
        self.execute_by_group(
            &location,
            &filtered_assets,
            start,
            &timezone,
            group_by,
        )
    })
    .collect();

let kv_requests_vec = try_join_all(group_by_futures)
    .await
    .map_err(map_to_boxed_error)?;
although I would expect the try_join_all to let the error bubble up.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
I think the core dump is not going to work well because as soon as I to a dev build the size of my wasm binary becomes too large. I already managed to shrink it by a factor 15 by no longer using chrono-tz's parse function but only support a couple of specific timezones.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
thx i'll give that a try to see if I can get the core dump to work. btw as soon as i comment out the function call that does most of the work in my queue consumer it executes fine, although it does very little. it massively shrinks the uploaded size of the wasm file.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
i don't have it in all of my queue consumers either, just in 2 of them. but intermittently.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
Nothing gets logged indeed. Not in the #[event(queue)] handler, nor the #[event(start)] function.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
request is not defined stems from the recordCoredump so isn't the actual reason it fails. it just times out without doing anything.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
Unfortunately it doesn't help. I get the following log: 
{
  "outcome": "exception",
  "scriptName": "***",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "ReferenceError",
      "message": "request is not defined",
      "timestamp": 1693580862741
    }
  ],
  "logs": [
    {
      "message": [
        "Queue"
      ],
      "level": "log",
      "timestamp": 1693580742711
    },
    {
      "message": [
        "timeout after 120s"
      ],
      "level": "error",
      "timestamp": 1693580862711
    }
  ],
  "eventTimestamp": 1693580742709,
  "event": {
    "batchSize": 1,
    "queue": "location-trigger-aggregate"
  },
  "id": 0
}
{
  "outcome": "exception",
  "scriptName": "***",
  "diagnosticsChannelEvents": [],
  "exceptions": [
    {
      "name": "ReferenceError",
      "message": "request is not defined",
      "timestamp": 1693580862741
    }
  ],
  "logs": [
    {
      "message": [
        "Queue"
      ],
      "level": "log",
      "timestamp": 1693580742711
    },
    {
      "message": [
        "timeout after 120s"
      ],
      "level": "error",
      "timestamp": 1693580862711
    }
  ],
  "eventTimestamp": 1693580742709,
  "event": {
    "batchSize": 1,
    "queue": "location-trigger-aggregate"
  },
  "id": 0
}
The Queue and timeout after 120s are both logged from my entry.mjs while the rust code doesn't log anything. It doesn't fail every time and redeploying it sometimes fixes it sometime it doesn't.
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
I understood though that the console error panic hook even further increases the size, but I can indeed try this although there might be no panic to log
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
During deployment I do get the following warning: 
Total Upload: 15789.56 KiB / gzip: 6358.20 KiB
▲ [WARNING] We recommend keeping your script less than 1MiB (1024 KiB) after gzip. Exceeding past this can affect cold start time
Total Upload: 15789.56 KiB / gzip: 6358.20 KiB
▲ [WARNING] We recommend keeping your script less than 1MiB (1024 KiB) after gzip. Exceeding past this can affect cold start time
36 replies
CDCloudflare Developers
Created by Jorrit Salverda on 9/1/2023 in #queues
RuntimeError: memory access out of bounds
i'm using wrangler 3.6.0, worker-rs 0.0.18, compatibility date 2023-08-15 and the following settings for the queue consumer: 
[[queues.consumers]]
  queue = "***"
  max_batch_size = 1
  max_concurrency = 1
  max_retries = 0
  max_batch_timeout = 0
[[queues.consumers]]
  queue = "***"
  max_batch_size = 1
  max_concurrency = 1
  max_retries = 0
  max_batch_timeout = 0
36 replies