Connection timeout in microservices log
I saw a few failed face recognition jobs, so I checked the microservices log. I found many ETIMEDOUT errors.
I'm on v 1.76.1, the version just before the release that had the machine learning timeout issue. My docker-compose and .env files are here: https://github.com/immich-app/immich/issues/4002
Error: connect ETIMEDOUT
at Socket.<anonymous> (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41)
at Object.onceWrapper (node:events:628:28)
at Socket.emit (node:events:514:28)
at Socket._onTimeout (node:net:571:8)
at listOnTimeout (node:internal/timers:569:17)
at process.processTimers (node:internal/timers:512:7) {
errorno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'connect'
}
I'm surprised that the redis container seems to be timing out, if I'm understanding the logs correctly. I've included details about my hardware in the linked github issue. I suspect the culprit might be a relatively slow CPU, but I'd greatly appreciate it if someone can point me in the right direction on how to fix this. Thank you!3 Replies
I'm wondering if it's the issue @Alex mentioned here: https://discord.com/channels/979116623879368755/991643870523830382/1142232314080923818
I saw timeout with the server but haven’t seen timeout from Redis 🤔 what is the system you are running Immich from?
Docker, Debian 12, 2 GB RAM, 1 GB ZRAM and 1 GB SWAP. Worth noting that the system is nowhere close to out of memory, as long as CLIP embeddings are disabled. The processor is an Intel Atom Z8350.
There are also several errors because of a missing file in the upload folder, which led me to think it might be the issue you mentioned in the off-topic thread.
[Nest] 7 - 09/10/2023, 2:33:24 AM ERROR [JobService] Error: ENOENT: no such file or directory, stat 'upload/upload/2e110b99-bec8-4004-8f7e-8ace538d4c31/fc2b7470-ad17-4427-9334-d8afb98aa1ac.jpeg'
at Object.statSync (node:fs:1690:3)
at MetadataExtractionProcessor.handlePhotoMetadataExtraction (/usr/src/app/dist/microservices/processors/metadata-extraction.processor.js:268:45)
at async /usr/src/app/dist/domain/job/job.service.js:105:37
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 7 - 09/10/2023, 2:33:24 AM ERROR [JobService] Object:
{
"id": "94a0fe2b-6051-485a-aabd-af7dc46734de"
}