N
Novu6mo ago
Kedar

Self hosted novu deployment does not store any integrations

We have deployed 0.24 on a k8s cluster. We were able to setup and login; howver, none of the operations seems to work like adding brand logo, integrations etc. The UI shows a toast that operation completed successfully, however, after that there is no entry of integration or for that matter even the logo that is uploaded. We are using RDS postgres db for the same. Any clues what might be wrong?
16 Replies
Ask Inkeep
Ask Inkeep6mo ago
Hi @Kedar, I'm an AI assistant for Novu. While a team member reviews this question, I'll see if I can find relevant sources for your question. ----------------------------- While I generate an answer, here are some sources that might be helpful: * Kubernetes * Deploy with Docker * Commercial Self Host * Introduction ----------------------------- Answer It sounds like you may be experiencing issues related to the configuration of your Novu deployment or how it interacts with your database setup. Here are a few troubleshooting steps you can consider: 1. Check Database Connection: Verify if the RDS Postgres database is correctly configured in your environment variables. Ensure that Novu has the necessary permissions to read from and write to the database. 2. Review Logs: Check the logs for any errors or warnings that might indicate what is going wrong. This can give you clues about database connection issues or errors in the application itself. 3. Configuration Verification: Double-check your configuration settings, especially those related to storage and persistence. You might want to verify the environment variables and configuration files used in your Kubernetes (K8s) setup. 4. Version Compatibility: Ensure that the version of Novu you are deploying is compatible with your current infrastructure, including database and Kubernetes versions. Unfortunately, the information provided in the available documents does not directly address issues related to the integration storage or UI operations for self-hosted Novu on Kubernetes. For more detailed investigation, you might need to examine the specific logs and configuration of your deployment. If the problem persists, consider reaching out to the Novu community or support for more in-depth assistance. These steps might help you diagnose and address the issue regarding storing integrations and other operations not working as expected. Got it. Tagging @Support for additional help.
Zac Clifton
Zac Clifton6mo ago
@Kedar have you connected object storage? https://docs.novu.co/self-hosting-novu/object-storage This is where we store branding items
Kedar
KedarOP6mo ago
Yes, we do have s3 object storage I tried adding Email integration; the toast says the operation was success, however, the new integration is not listed.
Kedar
KedarOP6mo ago
This is the issue that I am encountering - very slow UI - operations seem to complete but it does not. Version 0.23.0
Novu_Bot
Novu_Bot6mo ago
@Kedar, you just advanced to level 3!
Kedar
KedarOP6mo ago
@Zac Clifton
Zac Clifton
Zac Clifton6mo ago
@Kedar are you able to see any errors in the api logs? @Joel so you are aware with loading
Kedar
KedarOP6mo ago
@Zac Clifton I see below error on api pod
"context":"ExceptionsHandler","err":{"type":"Error","message":"error:1C800064:Provider routines::bad decrypt","stack":"Error: error:1C800064:Provider routines::bad decrypt\n at Decipheriv.final (node:internal/crypto/cipher:193:29)\n at decrypt (/usr/src/app/packages/application-generic/build/main/encryption/cipher.js:22:52)\n at decryptSecret (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:16:33)\n at decryptCredentials (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:34:19)\n at Function.getDecryptedCredentials (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:53:71)\n at /usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:49:66\n at Array.map (<anonymous>)\n at GetDecryptedIntegrations.<anonymous> (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:49:18)\n at Generator.next (<anonymous>)\n at fulfilled (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:14:58)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},"msg":"error:1C800064:Provider routines::bad decrypt"}
"context":"ExceptionsHandler","err":{"type":"Error","message":"error:1C800064:Provider routines::bad decrypt","stack":"Error: error:1C800064:Provider routines::bad decrypt\n at Decipheriv.final (node:internal/crypto/cipher:193:29)\n at decrypt (/usr/src/app/packages/application-generic/build/main/encryption/cipher.js:22:52)\n at decryptSecret (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:16:33)\n at decryptCredentials (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:34:19)\n at Function.getDecryptedCredentials (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:53:71)\n at /usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:49:66\n at Array.map (<anonymous>)\n at GetDecryptedIntegrations.<anonymous> (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:49:18)\n at Generator.next (<anonymous>)\n at fulfilled (/usr/src/app/packages/application-generic/build/main/usecases/get-decrypted-integrations/get-decrypted-integrations.usecase.js:14:58)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},"msg":"error:1C800064:Provider routines::bad decrypt"}
We had upgrade the version from 0.23 to 0.24. The issue was observed on both versions
(node:internal/process/task_queues:95:5)","library":"Provider routines","reason":"bad decrypt","code":"ERR_OSSL_BAD_DECRYPT"},"responseTime":33,"msg":"request errored"}
(node:internal/process/task_queues:95:5)","library":"Provider routines","reason":"bad decrypt","code":"ERR_OSSL_BAD_DECRYPT"},"responseTime":33,"msg":"request errored"}
Kedar
KedarOP6mo ago
Redeployed novu, but gets this error while adding email integration
No description
Zac Clifton
Zac Clifton6mo ago
@Richard Fontein or @Paweł T. would you have an idea what changed with env from v0.23 to v0.24
Kedar
KedarOP6mo ago
We redeployed the entire stack from scratch again; the only thing that does not work is storage (we use AWS S3 directly). Below is the error
No description
No description
Kedar
KedarOP6mo ago
Any pointers what might be wrong here? @Zac Clifton any thoughts on this one?
Zac Clifton
Zac Clifton6mo ago
I am sorry I do not have any insight and the platform team curently unable to help with this at this time. @Paweł T. @Emil
Kedar
KedarOP4mo ago
@Zac Clifton @Paweł T. @Pawan Jain we ran into this again, any clue what might be happening?
Zac Clifton
Zac Clifton4mo ago
I am unable to help with this @sokratis may be able to say of this bug was fixed or not in the upcoming version.
Kedar
KedarOP4mo ago
I think it's the storage encryption that is has the issue; however, we have not changed the key for almost last 3 months. The decrpyt method failed failing with below
"err":{"type":"Error","message":"error:1C800064:Provider routines::bad decrypt","stack":"Error: error:1C800064:Provider routines::bad decrypt\n at Decipheriv.final (node:internal/crypto/cipher:193:29)\n at decrypt (/usr/src/app/packages/application-generic/build/main/encryption/cipher.js:22:52)\n at decryptSecret (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:16:33)\n
"err":{"type":"Error","message":"error:1C800064:Provider routines::bad decrypt","stack":"Error: error:1C800064:Provider routines::bad decrypt\n at Decipheriv.final (node:internal/crypto/cipher:193:29)\n at decrypt (/usr/src/app/packages/application-generic/build/main/encryption/cipher.js:22:52)\n at decryptSecret (/usr/src/app/packages/application-generic/build/main/encryption/encrypt-provider.js:16:33)\n
We found the issue and looks for some reason the storage encryption that we had used earlier was replaced with a new one (we yet to find why this happened). We have moved to using a key that is stored externally and not rely on the one that is generated in helm.
Want results from more Discord servers?
Add your server