Cloudflare Developers•4w ago

unexpected errors

Hey team, I'm seeing an elevated amount of Error: Connection terminated unexpectedly in our setup (hyperdrive enabled). We did ship some changes recently that increased some calls within a waitUntil function. Could that code be the reason why we are seeing more of these errors?

12 Replies

AJR•4w ago

Hi! We are not currently sending out a release, nor am I seeing any widespread instability right now.
Without knowing quite a lot about your setup and what exactly was added to your waitUntil calls I'm not sure I could answer that.

flyingshoesOP•4w ago

So our waitUntil setup is like this:

const telemetryCall = (dbResults) => {
      // where we actually call into the telemetry library
     waitUntil(actualTelemetryCall(dbResults)) // do I need to return here?
}

const telemetryCall = async () => {
      // make DB call to enrich data for our telemetry call
     Const dbResults = await dbCall()

     actualTelemetryCall(dbResults) // do I need to return here?
}

const controller = () => {
   // this is tied to an endpoint. 
   // at very end of logic, call telemetry
   waitFor(telemetryCall)
}

const telemetryCall = (dbResults) => {
      // where we actually call into the telemetry library
     waitUntil(actualTelemetryCall(dbResults)) // do I need to return here?
}

const telemetryCall = async () => {
      // make DB call to enrich data for our telemetry call
     Const dbResults = await dbCall()

     actualTelemetryCall(dbResults) // do I need to return here?
}

const controller = () => {
   // this is tied to an endpoint. 
   // at very end of logic, call telemetry
   waitFor(telemetryCall)
}

I’m not the best with nested waitUntil, but I’m wondering if I’m not awaiting somewhere to ensure the connection stays open until everything is finished? thanks again!

AJR•4w ago

Hmmmm. Yeah, this sort of JS code isn't my specialty either, tbh. On a quick read through nothing about this is obviously wrong to me, though. One way you could try to resolve this is to add a waitUntil(db.close()) (or whatever the correct syntax is for explicitly closing your connection with your driver/ORM) to ensure you properly disconnect after it's all said and done.

flyingshoesOP•4w ago

hmmm, yeah. that's helpful. I'm realizing it might be something with our db connection code rather than our busines logic. I'll do more investigating. thank you!

ivan•4w ago

Hey, I'm seeing something similar, in my case there was no update on the code, was there any change on hyperdrive limits? Every big insert (2500+ values) gets "write CONNECTION_CLOSED" on postgres.js, but previously (before dec 3) this very same inserts were working flawlessly. Also those inserts work ok on local dev using the same database as in production

AJR•4w ago

No, we didn't make any changes of that nature, but that's an excellent bit of info to dig into it. Thanks for sharing that. I'll take a look. Obvious question here, but no changes to driver version or anything like that? 2500+ here meaning rows or columns?

ivan•4w ago

Nope, always postgres 3.4.5 I'm using this code to test and apparently nailed it down to 2046+ values (2045 works ok, 2046+ hangs and ends up in "write CONNECTION_CLOSED"), I have other inserts failing with less rows (~500) but over 11 columns

async function test({ params }, env) {
  if (params.secret !== secret) return;
  const sql = postgres(env.TSCALE.connectionString);
  let n = parseInt(params.n || 1);
  let inserts = [];
  for (let i = 0; i < n; i++) inserts.push({ value: n });
  try {
    await sql`INSERT INTO sandbox ${sql(inserts)}`;
  } catch (e) {
    console.error(e?.message);
  }
  return new Response("👌");
}

async function test({ params }, env) {
  if (params.secret !== secret) return;
  const sql = postgres(env.TSCALE.connectionString);
  let n = parseInt(params.n || 1);
  let inserts = [];
  for (let i = 0; i < n; i++) inserts.push({ value: n });
  try {
    await sql`INSERT INTO sandbox ${sql(inserts)}`;
  } catch (e) {
    console.error(e?.message);
  }
  return new Response("👌");
}

AJR•4w ago

Ok, that's incredibly helpful, thank you. I'll pass all this over to the team and start digging.

ivan•4w ago

awesome, thank you!

AJR•4w ago

@Iván This should be fixed now. Please check and confirm, if you would? Thank you again for the incredibly valuable repro. I wish all bugs came with one of those. There'll be some followup work to do off this, but hopefully this unblocks folks for now.

ivan•4w ago

Yeees! Working now! thanks a lot!

AJR•3w ago

As a heads up, we need to walk back that fix. There's some other related challenges, and we need to rework how it was done to be more efficient. You're going to see that issue again for a bit until we can get it re-done in a more complete way. I apologize for this inconvenience. Alright, holistic fix is out now. Should be done with this issue.

Gaming

Programming

unexpected errors

Did you find this page helpful?