Immich•3y ago

Old path structure in database

Hi. I am having issue with old path in DB for assets. A noticed it after I re run the extract metadata job. Microservice log:

[Nest] 1  - 02/22/2023, 8:53:44 PM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to: Error: File not found - /usr/src/app/upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg on file upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg
[Nest] 1  - 02/22/2023, 8:53:44 PM   ERROR [MetadataExtractionProcessor] Error extracting EXIF Error: ENOENT: no such file or directory, stat 'upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg'
Error: ENOENT: no such file or directory, stat 'upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg'
    at Object.statSync (node:fs:1551:3)
    at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:113:49)
[Nest] 1  - 02/22/2023, 8:53:44 PM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to: Error: File not found - /usr/src/app/upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937/14307ba0-8c50-4acf-bb75-a2e8431aa0b6.heic on file upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937/14307ba0-8c50-4acf-bb75-a2e8431aa0b6.heic

[Nest] 1  - 02/22/2023, 8:53:44 PM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to: Error: File not found - /usr/src/app/upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg on file upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg
[Nest] 1  - 02/22/2023, 8:53:44 PM   ERROR [MetadataExtractionProcessor] Error extracting EXIF Error: ENOENT: no such file or directory, stat 'upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg'
Error: ENOENT: no such file or directory, stat 'upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/5f3dff0bb075466b8a60f87b4689bd37/48c5de4c-016e-45d6-9c54-44a4765004c4.jpg'
    at Object.statSync (node:fs:1551:3)
    at MetadataExtractionProcessor.extractExifInfo (/usr/src/app/dist/apps/microservices/apps/microservices/src/processors/metadata-extraction.processor.js:113:49)
[Nest] 1  - 02/22/2023, 8:53:44 PM    WARN [MetadataExtractionProcessor] The exifData parsing failed due to: Error: File not found - /usr/src/app/upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937/14307ba0-8c50-4acf-bb75-a2e8431aa0b6.heic on file upload/d8c4c8e3-a6ee-4889-8148-4e7db21660f4/original/fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937/14307ba0-8c50-4acf-bb75-a2e8431aa0b6.heic

I put count on DB where the originalpath field is string original and it gives me 32000 (out of 34000)... so something went wrong

26 Replies

jrasm91•3y ago

Are all of your photos (in the file system) stored in the expected storage template folder structure?

Alex Tran•3y ago

Can you run storage migration job?

jrasm91•3y ago

Not sure if that'll fix this as the storage migration job still does expect that originalPath is valid/correct, but in this situation it looks like originalPath is wrong to begin with.

pedroOP•3y ago

the 'original' product currently has 97 assets. I don't understand where the problem could be. The other photos are according to the folder structure. I tried to run the migration job again but it still gives me a bunch of errors as it tries to find those files in the old location

Alex Tran•3y ago

hmm For example, does this file original/fad37e7d856746d217f5c6dba2fe381b3bed05da0ef0c585944c3503f2d95937/14307ba0-8c50-4acf-bb75-a2e8431aa0b6.heic exist?

pedroOP•3y ago

Not in the original folder

jrasm91•3y ago

Do you see all 34k files in your upload folder?

pedroOP•3y ago

Assets are displayed normally both in the app and on the web...

jrasm91•3y ago

The original isn't displayed anywhere

pedroOP•3y ago

I counted them and I see 120000

jrasm91•3y ago

What folders do you see?

pedroOP•3y ago

jrasm91•3y ago

2011-2023 is the "storage template" format and should contain the originals, but it seems like the database values for "originalPath" are incorrect.

pedroOP•3y ago

correct maybe I could copy all files from 2011-2023 back to the original and run the migration job again

jrasm91•3y ago

It's not just the folder name that has changed, but the whole path, including the filename probably. I don't really understand how the database and the filesystem could have become out of sync in the first place. Are you using an NFS share with async by chance?

pedroOP•3y ago

no, I don't use that... Today it is quite late for me. Tomorrow I will take a good look at it from my side and let you know.

jrasm91•3y ago

Did you manually run the storage migration at some point recently? I mean, before you noticed all these errors

pedroOP•3y ago

dont remember

jrasm91•3y ago

There are like 2k records in the database that don't have a path with original/ though?

pedroOP•3y ago

yes... probably assets after mid dec It's weird, because if it wasn't for the new livephoto feature I wouldn't have even noticed it. Because the assets are nicely displayed in app and WEB too

jrasm91•3y ago

Yeah, I assume it's been broken for awhile and I think downloading something from the web would fail. After an original has been uploaded we don't really ever do anything with it except for the one case of moving it via the storage migration job. Seems somehow the job moved the files but didn't update the database record.

pedroOP•3y ago

You're right, they don't download . Now I understand why my wife always complained that she couldn't download anything 😄 and when I tried it I always had to choose the "right" photo of a newer date I'll take a look tomorrow, if I come across something I'll be sure to let you know. So far it seems to be a bug only for me.

jrasm91•3y ago

Sounds good. I think this an immich bug, although not exactly sure what caused it. Moving forward you probably have two options: 1. Start over in a new instance an import everything again 2. SQL updates or manually renaming files to get database and filesystem back in sync

pedroOP•3y ago

I'd probably rather choose option 2. 😅 We'll see. Thank you very much for your help so far

jrasm91•3y ago

We're pretty sure this was a bug . Probably in a situation where after the storage migration job started, an issue happened with the database, which resulted in the files moving new a new location, but their paths not being updated in the database. We've fixed 🤞 the initial bug and I will probably work on a server command to repair (re-match/correct) the file locations with their corresponding records in the database in the next week or so. So if you want to wait until that's done, you could test it and that should resolve your situation.

pedroOP•3y ago

I reuploaded the whole library again 😅 but thanks for the info

Gaming

Programming

Old path structure in database

Did you find this page helpful?