Upgrade to v0.33 sync of standard objects failed
Hi Team!
I've just completed the migration from the old PostgreSQL image over to Spilo. I've restored my old database to the new Spilo container and everything is looking good (I think) except that I've seen errors that the metadata is missing it's version and the migration command to 0.33 is failing.
Here are the images I'm using:
I've attached a screenshot of the errored logs as well hoping that it helps.
When attempting to login, I do get a 500 server error (second screenshot)
37 Replies
I can't see those screens.
Sorry I was really hoping I could capture all of the output..
Here's what's left of my terminal scrollback for the migration script:
Here's the Server 500 generated on login:
what is your container health looking like?
Also, what version did you upgrade from?
All containers are up, no restarts, I don't have the worker running at the moment until the migration script has completed successfully.
upgrading from:
twentycrm/twenty:v0.32.3 -> twentycrm/twenty:v0.33.0
twentycrm/twenty-postgres:v0.32.3 -> twentycrm/twenty-postgres-spilo:v0.33.0@Skyfox675 did you follow https://twenty.com/developers/section/self-hosting/upgrade-guide#v0.32.0-to-v0.33.0 ?
I think the schema has evolved between O.32 and 0.33. So only restoring your old DB will lead to schema errors
I did do the update and scripts for 0.33
looking at your first screen, I see invalid input syntax for type json
looks like this is trying to parse a note
could you make sure all your notes have valid json bodies ? Or maybe just try without the notes just for testing
Is there a way to delete notes that might be cuasing issues?
I've been hitting bugs with Note objects exceeding 8 nested items that I think broke the schema and would like to remove all notes that might be causing this
For now, probably only delete directly in database
Also, could you write down how to replicate your problem? I'd like to reproduce it
And if you're not sure how to delete it safely, feel free to ask
Yeah if you don't mind I would really appreciate help not mucking up a psql query with deletes. I'm thinking I could start by a delete where note char length is greater than 100 which should keep the basic (good quality notes that would be helpful to keep) and remove the lengthy broken JSON schemas of the test notes I was doing to inject form responses into notes
Do you maybe know about transactions in SQL?
I'm out running errands at the moment, what info can I provide to help you reproduce? I have a copy of my N8n "node" that is the JavaScript responsible for creating the looped note structure that apparently has the broken JSON
I'm good fumbing through JOINs and inserts but WHERE with DELETES and DROPs make me pretty nervous lol
Okay, so here's a short explanation
If you want to make some change but you're not sure about the query (like forgotten WHERE or wrong column), you can use transactions as a precaution
In order to "enable" transaction mode, all you have to do is simply type
BEGIN TRANSACTION; and then the "cursor" in your CLI changes from => to =*>
After that, you can type whatever query you want, in order to revert your changes, you have to type ROLLBACK;
And if you want to save your changes, you have to type COMMIT;
Last thing, you can't revert committed changesDoes your notes look like this by any chance?
So it might, but I can't view them via the UI it throws 500's and after the upgrade I can't login to the UI so hoping to get fixed so the UI can login, I can triage what else is going on from there
The note structure is suppose to be multi-line (line 1 the form question, and line 2 the user response) but I found after 8 items in a note, it throws 500's and the note can only be deleted by API
This was the GH issue I opened for it a while back
https://github.com/twentyhq/twenty/issues/7444
GitHub
Notes REST API (POST & PUT) - Internal Server Errors for all Notes ...
Bug Description Hi Team, After a while of debugging I found that when creating notes (or updating them) the maximum body length can be 8 items consisting of: { "type": "heading"...
Ah, no wonder I couldn't find this issue in open
I guess I've made note with more than double the limit and app still works normally
I'd like to help you with testing the solution but now I've broke my local instance somehow and can't recover it 😆
So I basically stopped using twenty when I encountered this and left it for a few months and left it in a broken state and just now making it a priority to fix it
Think deleting the suspect notes that are exceeding long is a good first place to start?
It's worth trying for sure
Do you by chance know the table location of where the notes are stored?
workspace_<id>.noteThank you!
Am I safe to delete rows in that table or will there be broken references from doing so that might cause more problems?
I'd say you're safe deleting them
Basically records in table are the same as records in UI
I'll work on this in the next few days, have a cert exam on Tuesday afternoon I should priortize 😦
Thank you for the help, seriously have been stuck, thank you
@ɃØĦɆᵾS I've gotten my notes table cleaned up and gone through each note I've kept to double check it has valid syntax. These are the errors I'm seeing now in the
yarn command:prod upgrade-0.33 command:
Sadly I don't have enough knowledge about upgrading process to help you, @thomast could you take a look?
Looks like it now complains about the json in a task 😅
Oh goodness, ok going to run through those now
Weird, dozens of empty bodied tasks, I ended up just deleting all tasks lol
Thanks @thomast that looks like that did the trick 🙂 successful sync (I think?), showing 43 errors that exist after the successful run of
yarn command:prod upgrade-0.33
Is there anything I need to do before I bump the container images up to 0.34?
Glad that worked! Make sure to follow https://twenty.com/developers/section/self-hosting/upgrade-guide#v0.33.0-to-v0.34.0 and I hope it will be easier this time!
Looks like the server is still having issues. It looks like all of my custom objects are failing to return and the UI is on a coffee break with
t is undefined
@charles have you already seen that error? 🤔
FYI I haven't upgraded yet, still sitting on v0.33, I'm able to login now (I wasn't before with the previous errors) and I do see all of these custom object types on the left nav bar, but clicking on them or any object including the default ones returns these errors above
I'm honestly considering starting over 😕 I don't know if it's reason to consider it, but it's been down so long at this point, not sure if there's anything worth recovering. I liked the schema/object structures I've setup though so it would be nice to not have all of that wasted
Hi @Skyfox675, that could indeed be a solution because every time we fix something we just reach another issue 😅
You do have a complex schema, hard to setup again?
do you have a table named automation? looks like this is the one that is failing
if you have, the step to debug would be to:
- dive into the code to reach the query that is done in postgres
- log the query
- try to run the query by itself to understand the issue
If you don't have an automation custom object, this is the actual issue. We need to understand why this query is done
@thomast @bob thank you both for your help, after a few drinks 🍷 and a long talk with @xSwampxFox we decided to just backup data and nuke the instance and start over lol we're on 0.42.15 now and up and running again and will be working on the data load and rebuilding the schema in the next week or so. Thanks again for helping us try to recover it