One of the queries that Synology makes

One of the queries that Synology makes results in a NextMarker loop - forever

31 Replies

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Yep R2 returns a Marker of foo and a NextMarker of foo so it keeps following NextMarker I'm going to move the files from my R2 bucket onto S3 & run the same request against S3 Specifically it's <NextMarker>Resilience_1.hbk/</NextMarker> (varies between upload jobs but you get the idea) Previously the job uploaded to a folder called sdk-example and it looped on this

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

It's basically being told 'to check this folder, use this marker' but that marker isn't progressing anything

kianOP•2y ago

kianOP•2y ago

so NextMarker is Resilience_1.hbk/ so I'll add the marker search param with that value

kianOP•2y ago

kianOP•2y ago

now I'm stuck forever, since it's just telling me to check where I already am

kianOP•2y ago

if I remove the delimiter and the marker, we can see there are files

kianOP•2y ago

including other folders after that

kianOP•2y ago

if I do the same against S3:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

Let me try and setup a repro that doesn't involve Synology Repro:

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

S3:

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

R2:

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

Both ran on empty buckets

kianOP•2y ago

kianOP•2y ago

Same file/folder path on both To clarify why Marker: "folder/" is used, that's what R2 returns as NextMarker if I omit the marker.

Unknown User•2y ago

Message Not Public

kianOP•2y ago

kianOP•2y ago

Let's just say Synology will, uh... ferrissweatmegaspin

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Yes - but we're also within free blobcatbusiness

itsmatteomanf•2y ago

@kiannh is great, yeah If you want I can keep going 😛 This varies as it’s the base folder name, which defaults to the NAS name with the hbk extension. You can select it.

kianOP•2y ago

I should apply as QA /s

James•2y ago

Glad this was got to the bottom of! Thanks so much for the debug y’all. Would love to use R2 with my synology at some point 👀

Unknown User•2y ago

Message Not Public

itsmatteomanf•2y ago

I have my job ready, if you need a test just ping me anytime. It’s a very easy click to test, and even if I need to set-it up again I did a separate token, so I roll and restart easily. It was fun, seeing @kiannh swear at Signatures in S3 CLI 😂

kianOP•2y ago

James•2y ago

Any news here? 👀 Is there a better place I could file this for tracking? Ticket to custesc maybe?

Unknown User•2y ago

Message Not Public

James•2y ago

Alrighty, will do 🙂

itsmatteomanf•2y ago

Should we do one and CC the other into it? 👀

James•2y ago

can do. I was gonna take a look this weekend, don't have much time this week. Will DM you beforehand in case you've already done one lul

itsmatteomanf•2y ago

Not sure I’m gonna find the time, as it requires a bunch of info to share and to prepare 😅 @sdnts so, we should CUSTESC it, right? No progress in the meantime?

Unknown User•2y ago

Message Not Public

itsmatteomanf•2y ago

Hopefully gonna do that today, finally...

Gaming

Programming

One of the queries that Synology makes