Cloudflare Developers•2y ago

```

2023/02/20 16:55:05 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "planet.pmtiles" "r2:/protomap" "-P" "--s3-upload-concurrency=3" "-vv"]
2023/02/20 16:55:05 DEBUG : Creating backend with remote "planet.pmtiles"
2023/02/20 16:55:05 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 16:55:05 DEBUG : fs cache: adding new entry for parent of "planet.pmtiles", "/mnt"
2023/02/20 16:55:05 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 16:55:05 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2:protomap"
2023-02-20 16:55:05 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:             0 / 54.779 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        24.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779G, 0/s, -

2023/02/20 16:55:05 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "planet.pmtiles" "r2:/protomap" "-P" "--s3-upload-concurrency=3" "-vv"]
2023/02/20 16:55:05 DEBUG : Creating backend with remote "planet.pmtiles"
2023/02/20 16:55:05 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 16:55:05 DEBUG : fs cache: adding new entry for parent of "planet.pmtiles", "/mnt"
2023/02/20 16:55:05 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 16:55:05 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2:protomap"
2023-02-20 16:55:05 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:             0 / 54.779 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        24.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779G, 0/s, -

104 Replies

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Here's what I got when I tried that:

alastair@EchoStar:/mnt$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap--s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv
2023/02/20 17:24:01 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv"]
2023/02/20 17:24:01 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:24:01 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:24:01 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:24:01 DEBUG : Creating backend with remote "r2:/protomap--s3-upload-cutoff=100M"
2023/02/20 17:24:01 DEBUG : fs cache: renaming cache item "r2:/protomap--s3-upload-cutoff=100M" to be canonical "r2:protomap--s3-upload-cutoff=100M"
2023/02/20 17:24:01 ERROR : Attempt 1/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 ERROR : Attempt 2/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 ERROR : Attempt 3/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 INFO  : 
Transferred:                0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.4s

2023/02/20 17:24:01 DEBUG : 4 go routines active
2023/02/20 17:24:01 Failed to copy: BadRequest: Bad Request
    status code: 400, request id: , host id:

alastair@EchoStar:/mnt$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap--s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv
2023/02/20 17:24:01 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv"]
2023/02/20 17:24:01 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:24:01 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:24:01 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:24:01 DEBUG : Creating backend with remote "r2:/protomap--s3-upload-cutoff=100M"
2023/02/20 17:24:01 DEBUG : fs cache: renaming cache item "r2:/protomap--s3-upload-cutoff=100M" to be canonical "r2:protomap--s3-upload-cutoff=100M"
2023/02/20 17:24:01 ERROR : Attempt 1/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 ERROR : Attempt 2/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 ERROR : Attempt 3/3 failed with 1 errors and: BadRequest: Bad Request
    status code: 400, request id: , host id: 
2023/02/20 17:24:01 INFO  : 
Transferred:                0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.4s

2023/02/20 17:24:01 DEBUG : 4 go routines active
2023/02/20 17:24:01 Failed to copy: BadRequest: Bad Request
    status code: 400, request id: , host id:

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

OOp With that issue fixed:

alastair@EchoStar:/mnt$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv -P
2023/02/20 17:27:26 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap" "--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv" "-P"]
2023/02/20 17:27:26 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:27:26 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:27:26 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:27:26 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 17:27:26 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2:protomap"
2023-02-20 17:27:26 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:             0 / 54.779 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        30.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779G, 0/s, -

alastair@EchoStar:/mnt$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv -P
2023/02/20 17:27:26 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap" "--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv" "-P"]
2023/02/20 17:27:26 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:27:26 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:27:26 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:27:26 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 17:27:26 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2:protomap"
2023-02-20 17:27:26 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:             0 / 54.779 GBytes, 0%, 0 Bytes/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        30.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779G, 0/s, -

It's weird. It almost seems like somewhere along the line, something just pauses it...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Is there a way to up the bandwidth it can use per chunk? Seems really odd that it is running ~2 MB/s when the connection can run 400+, even if other stuff is running simultaneously.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Nope, or at least not anything that I've enabled. Just running bog-standard Ubuntu. I have WARP, but it isn't on

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

864cdf76f8254fb5539425299984d766

Hello, I’m Allie!OP•2y ago

Wut?

Hello, I’m Allie!OP•2y ago

Why is it on? The icon shows it is off, and I can't click on it... Trying to see if I can get it to shut off So the 1.1.1.1/help says I'm connected, warp-cli says I am not

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

And no other website shows my being AS 13335, so I assume it is a bug

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Might this be related to how I accidentally created a bucket called protmap via rclone due to a typo?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No, I just meant that I accidentally ran an upload to protmap, and instead of throwing an error, it just auto-created a new bucket for it.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Might be my fault? https://forum.rclone.org/t/rclone-copy-is-trying-to-create-new-b2-bucket-with-each-copy-what-am-i-doing-wrong/20530/2

rclone forum

Rclone copy is trying to create new B2 bucket with each copy, what ...

hello, you left out the version of rclone but based on the go version it looks very old. so best of update and test again https://rclone.org/downloads/#script-download-and-install the output would look like rclone v1.53.2 - os/arch: linux/amd64 - go version: go1.15.3 if you post again, you need to post the exact command with debug l...

Hello, I’m Allie!OP•2y ago

Apparently the version is very old...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Did it do anything different this time on your end? Still seems to get stuck...

alastair@EchoStar:~/Downloads/rclone-v1.61.1-linux-amd64$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv -P
2023/02/20 17:55:18 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap" "--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv" "-P"]
2023/02/20 17:55:18 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:55:18 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:55:18 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:55:18 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 17:55:18 DEBUG : r2: detected overridden config - adding "{ZdEwv}" suffix to name
2023/02/20 17:55:18 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2{ZdEwv}:protomap"
2023-02-20 17:55:19 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:              0 B / 54.779 GiB, 0%, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        30.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779Gi, 0/s, -

alastair@EchoStar:~/Downloads/rclone-v1.61.1-linux-amd64$ rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-upload-cutoff=100M --s3-chunk-size=100M --s3-upload-concurrency=10 -vv -P
2023/02/20 17:55:18 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles" "r2:/protomap" "--s3-upload-cutoff=100M" "--s3-chunk-size=100M" "--s3-upload-concurrency=10" "-vv" "-P"]
2023/02/20 17:55:18 DEBUG : Creating backend with remote "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles"
2023/02/20 17:55:18 DEBUG : Using config file from "/home/alastair/.config/rclone/rclone.conf"
2023/02/20 17:55:18 DEBUG : fs cache: adding new entry for parent of "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles", "/media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507"
2023/02/20 17:55:18 DEBUG : Creating backend with remote "r2:/protomap"
2023/02/20 17:55:18 DEBUG : r2: detected overridden config - adding "{ZdEwv}" suffix to name
2023/02/20 17:55:18 DEBUG : fs cache: renaming cache item "r2:/protomap" to be canonical "r2{ZdEwv}:protomap"
2023-02-20 17:55:19 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination
Transferred:              0 B / 54.779 GiB, 0%, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:        30.0s
Transferring:
 *                                planet.pmtiles:  0% /54.779Gi, 0/s, -

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Huh... Any other good way to upload a 54 GB file in a decent amount of time? I'll try making an rclone forum post about it

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No: planet.pmtiles: Need to transfer - File not found at Destination I have a similar file: map.pmtiles, but it isn't the same size, nor the same name Just to verify, a CreateBucket on a bucket name that exists would return an error, right?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

https://forum.rclone.org/t/repeated-createbucket-requests-while-attempting-to-copy-file/36259 Tell me if I missed something, or said something wrong

rclone forum

Repeated CreateBucket Requests while attempting to copy file

What is the problem you are having with rclone? When attempting to copy a 54.779 GiB local file to the remote with --s3-upload-concurrency higher than 2, rclone continuously sends CreateBucket requests for a bucket that does exist. Because this bucket already exists, the remote returns an conflict, which rclone seems to handle silently, and then...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

So does it only make one CreateBucket then?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Updated post: https://forum.rclone.org/t/rclone-fails-to-switch-to-multi-part-uploads-when-a-file-is-too-large/36259

rclone forum

Rclone fails to switch to multi-part uploads when a file is too large

What is the problem you are having with rclone? When attempting to copy a 54.779 GiB local file to the remote with --s3-upload-concurrency higher than 2, rclone attempts to PutBucket a single time. When it fails(because the file is too large), it seems to just stop, instead of attempting a multi-part upload. rclone does not provide any visual i...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Did it actually start uploading? Usually it starts within a second or two, at least if you don't have the bug I have

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

head -50000000000 /dev/random > lol.file Hm...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

I think I tried 1:30, but I'll try again with the new settings

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

I'm on a HDD, so probably I'll try for 5 minutes It worked, and it only took about 4:40... I'll try with 50 Aw yeah, 165 MB/s!!! Huh, it ran fast for like 10 seconds, but now it’s running at half a mb… Actually, it’s still slowing down 84 KB/s

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No, I’m at 8% Running at 7 KB 4 Still slowing down I almost feel like it uploaded a few parts, and then got rate-limited… At 200 years ETA I think it's just dead now... Anything on your end?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Oh, so it's just measuring upload speed really weird? No, it came back up for a second, but now it is back down to 200 KB/s

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Yeah, it seems to only count it as "uploading" when it is generating new Multiparts. Right now it is at 700 B/s

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Weird...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

62 They come in 5-10 minute spurts Maybe it's slow to finalize an upload?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

I'm at 2.8 MiB/s

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Got some new logs... Rate limited, increasing sleep to 1.28s

pacer: low level retry 1/2 (error RequestCanceled: request context canceled
caused by: context canceled)

planet.pmtiles: Cancelling multipart upload
2023-02-20 19:09:17 DEBUG : pacer: Reducing sleep to 1.5s
2023-02-20 19:09:17 ERROR : planet.pmtiles: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2023-02-20 19:09:17 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2023-02-20 19:09:19 DEBUG : pacer: Reducing sleep to 1.125s
2023-02-20 19:09:19 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination

planet.pmtiles: Cancelling multipart upload
2023-02-20 19:09:17 DEBUG : pacer: Reducing sleep to 1.5s
2023-02-20 19:09:17 ERROR : planet.pmtiles: Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2023-02-20 19:09:17 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. 
        status code: 403, request id: , host id:
2023-02-20 19:09:19 DEBUG : pacer: Reducing sleep to 1.125s
2023-02-20 19:09:19 DEBUG : planet.pmtiles: Need to transfer - File not found at Destination

And it restarted itself...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Sure! Could be interesting to see.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

In ENAM? Also, to make sure, using a VPN should work? Or is there a flag I can flip to get the location picker?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Do you have a list of colos that work for ENAM? I'm struggling to get my VPN to actually route through any colos in the Middle East, so I want to know how far I can overshoot...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Oh... Eastern North America? I thought it was Egypt, North Africa and the Mediterranean...

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Newark work?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Two things. I can't seem to access the settings for the thread, but you might want to lock it so that no one else can hop in and read it. Secondly, is this going to be permanent? Ok got it. Is https://864cdf76f8254fb5539425299984d766.r2.cloudflarestorage.com/enamtest in the right place?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Nice!

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No, I mean, is there any plans to make it so that at some point, you would be able to change it? I know you say it won't be immediately possible, but will it ever be?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Either?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

So it won't be replicated, but more auto-migrating? Will it be possible to have it replicate too? Like, if I have a very large number of requests coming from multiple places in the world, having it auto-migrate would be nice, but it would be even nicer to have multiple closer copies to the end users. ENAM seems to be having the same issue btw

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Transferred:        4.883 GiB / 54.779 GiB, 9%, 138.544 KiB/s, ETA 4d8h54m
Transferred:            0 / 1, 0%
Elapsed time:      6m37.5s
Transferring:
 *                                planet.pmtiles:  8% /54.779Gi, 138.544Ki/s, 104h54m0s

Transferred:        4.883 GiB / 54.779 GiB, 9%, 138.544 KiB/s, ETA 4d8h54m
Transferred:            0 / 1, 0%
Elapsed time:      6m37.5s
Transferring:
 *                                planet.pmtiles:  8% /54.779Gi, 138.544Ki/s, 104h54m0s

Replication in cloud storage? Or on pixelcf

specifically?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

I was referring to you saying that "this kind of thing just doesn't exist". Did you mean that CF doesn't have any similar products to base a model off of, or that no one in-industry has done it before?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

So is it looking like an R2 issue, or an rclone issue?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

https://maps.apple.com/?address=N%C3%A6stved,%20N%C3%A6stved,%20Danmark&auid=15970389555748127252&ll=55.240106,11.763546&lsp=6489&q=N%C3%A6stved&t=h

Næstved

Hello, I’m Allie!OP•2y ago

Do you need my ISP too?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Telia/Arelion 5G Home Broadband Hey @sdnts, any update on this issue?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Yeah. I received a response on the forum post, to disable the checksum generation, and reduce the chunk sizes. This seems to still have the same errors. The person that responded guessed that the long wait might be rclone waiting for confirmation that the chunk had settled. Does that sound plausible?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Looks like just a successful response. So basically, R2 uploads the entire chunk, and then hangs for a minute or however long until R2 actually provides a response to the Request. That's just a guess on our part though

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No worries! Oh hey, just to add to the info you have, now it stalls when I don't have any upload-related config flags:

rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap -vv -P

rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap -vv -P

This used to work(really slowly), but now it doesn't.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

No. I specifically nuked my config and then set it up from scratch

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Should I retry it now? I have a few settings I can tweak/test from the forum, if that might work?

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

It did something different. I got about 5 seconds per chunk, with

rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-disable-checksum --s3-upload-cutoff=5G --s3-upload-concurrency=2 -vv -P

rclone copy /media/alastair/7a70a803-7d9e-4711-a945-5d1791ec1507/planet.pmtiles r2:/protomap --s3-disable-checksum --s3-upload-cutoff=5G --s3-upload-concurrency=2 -vv -P

but each chunk is very small, so it still takes a very log time.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

Checking the System Monitor, it does fluctuate, but it generally stays below 3 MB/s. It hit 4 MB/s a single time, and then went straight back down to 2 MB/s.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

It's literally the best(and basically only good option right now). The neighbors across the street have Fiber-to-the-home, but I'm stuck with 5G broadband, that I think fluctuates based on the weather outside.

Unknown User•2y ago

Message Not Public

Hello, I’m Allie!OP•2y ago

I gave up, and now I’m just slowboating it. Estimated 8 hours for a 50 GB upload

Gaming

Programming

```