Does it make sense to use workers to host a chat endpoint?
Since workers are a server-less platform an instance needs to boot up every time you make a request to the endpoint, from a latency POV this is fine because its super fast, but I just don't know if it makes sense for a chat endpoint. Is it wasteful to use it for an app where you are constantly talking back and forth with a chatbot and for every message you send a new instance is being spun up and closed right after?
P.S. this isn't for a peer to peer chat, but with an ai powered chatbot so no need to keep an open connection (e.g. sockets) or is there?
2 Replies
There is typically no need for a websocket for an AI chat since it only responds to messages you sent and doesnt send random messages of its own
if you implement this with Workers AI or similar service Im sure you'll have no problem doing it on workers
Thanks for the response @Erisa , that makes me feel more at ease! I understand there is no need for the open socket connection since I just use the message the user sent as a prompt to query Open AI's api and return the response after doing a little bit of server side processing in the middle. The whole thing takes just a few seconds over the course of a standard https connection (as it would take hitting any regular REST api endpoint that wasn't AI related.
But I am still a little unsure if I should be using a serverless solution like workers here or a server that's running 24/7? Obviously the workers solution works great, it's super fast too (I already tried) but for something like this, would I be saving a ton more money/resources with the regular continuously running server option vs serverless? Assuming we're talking about massive scale?
Obviously the live server solution will always be cheaper at the expense of not having any of the great dev ops that is auto handled for you with serverless, but is it about the same as it would be for a regular REST endpoint, or would it be a lot worse since in a chat app you'd have lots of people constantly hitting the message endpoint non stop throughout the course of a regular convo?
I would feel a bit better if it's about the same or if anyone can say other big chat apps use serverless and the costs aren't astronomical relative to regular servers, then I want to go with the workers/serverless route and call it a day. Just want to confirm it's appropiate for this kind fo app, with this kind of usage characteristics..