Wasp•2w ago

Caching data with wasp

Just set up file upload/download to S3 as done in openSaaS. I'm guessing that every time a S3 url is generated with a key, you would get billed on AWS - is there a good way to cache data that works well with prisma/wasp?

11 Replies

kapa.ai•2w ago

Hi @pawnpusher12. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- Based on the knowledge sources provided, I can offer some insights on caching data with Wasp, particularly in the context of S3 file uploads and downloads. However, it's important to note that the sources don't directly address AWS billing for S3 URL generation. Wasp uses react-query for query management, which provides built-in caching mechanisms. Here are some relevant points: 1. Automatic Entity-based Query cache invalidation: Wasp offers automatic cache invalidation for queries that use the same Entity as an Action. This helps keep queries "fresh" without manual intervention. Wasp Documentation on Cache Invalidation 2. Optimistic updates: You can use optimistic updates to set cache values after performing an Action. This is supported through Wasp's useAction hook. Wasp Documentation on Cache Invalidation 3. Manual cache management: For more complex caching scenarios, you can use react-query's mechanisms directly. Wasp currently only natively supports optimistic updates for manual cache invalidation. Wasp Documentation on Cache Invalidation Regarding S3 URL generation, the OpenSaaS example uses short-lived presigned URLs:

export const getDownloadFileSignedURLFromS3 = ({ key }: { key: string }) => {
  const s3Params = {
    Bucket: process.env.AWS_S3_FILES_BUCKET,
    Key: key,
    Expires: 30, // <----
  };
  return s3Client.getSignedUrl("getObject", s3Params);
}

export const getDownloadFileSignedURLFromS3 = ({ key }: { key: string }) => {
  const s3Params = {
    Bucket: process.env.AWS_S3_FILES_BUCKET,
    Key: key,
    Expires: 30, // <----
  };
  return s3Client.getSignedUrl("getObject", s3Params);
}

Discord Support Thread The expiration time (30 seconds in this example) can be adjusted. Some community members suggested increasing it to 120 seconds or longer to account for slower server responses. To optimize caching and potentially reduce AWS costs: 1. Increase the expiration time of presigned URLs to a reasonable duration based on your use case. 2. Implement caching on your server for frequently accessed files, storing the presigned URLs for their duration. 3. Use Wasp's query caching mechanisms to store and serve these URLs on the client side. Remember that while caching can help reduce the number of S3 URL generations, you'll need to balance this with security considerations, especially for sensitive or frequently changing data. For a more comprehensive caching solution specific to your needs, you might need to implement additional logic using react-query or other caching libraries compatible with Wasp. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @pawnpusher12, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

pawnpusher12OP•2w ago

@kapa.ai why is the expiry set to only 30s?

kapa.ai•2w ago

Let me check the knowledge base for you again. One moment! :books: ---------------------- The expiry time of 30 seconds for the presigned S3 URLs in the OpenSaaS example is indeed quite short. According to the knowledge sources, this short expiration time is set for a specific reason: Discord Support Thread explains: 1. In the current implementation, the link is generated and opened directly after creation. 2. A new download link is generated each time a user requests to download a file. Community Member 8 states:

in the current implementation the link is opened directly after generating
so a new download link is generated each time

in the current implementation the link is opened directly after generating
so a new download link is generated each time

This approach ensures that each download request gets a fresh, short-lived URL, which can be beneficial for security purposes. However, as noted in the same discussion, this short expiration time might be problematic if: 1. The server response is slow 2. The user is slow to click on the generated link Community members in the discussion suggested increasing the expiration time to account for these potential issues. For example:

like Martin said, maybe just increase the `Expires` property to `120` or something a bit longer

like Martin said, maybe just increase the `Expires` property to `120` or something a bit longer

It's worth noting that you can adjust this expiration time based on your specific needs and use case. The 30-second expiration is not a hard requirement, but rather a default setting in the OpenSaaS example that you can modify as needed. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @pawnpusher12, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

MEE6•2w ago

Wohooo @pawnpusher12, you just became a Waspeteer level 1!

pawnpusher12OP•2w ago

Is this the best method to display user-uploaded images with a wasp app? That may be viewed many times, at unpredictable intervals? @kapa.ai

kapa.ai•2w ago

Let me check the knowledge base for you again. One moment! :books: ---------------------- Based on the knowledge sources provided, I can offer some insights on handling user-uploaded images in a Wasp app, but it's important to note that there isn't a definitive "best method" explicitly stated for your specific use case. The current approach used in the OpenSaaS example, which generates short-lived presigned URLs for S3, might not be the most efficient for images that are viewed frequently at unpredictable intervals. Here's why: 1. Short expiration times: The default 30-second expiration time is very short for images that may be viewed multiple times. As discussed in the Discord Support Thread, this could lead to unnecessary URL regeneration. 2. Frequent regeneration: Generating a new URL for each view could potentially increase AWS costs and add latency to your application. For frequently viewed user-uploaded images, you might want to consider the following alternatives: 1. Longer-lived presigned URLs: Increase the expiration time for presigned URLs to reduce the frequency of regeneration. However, be mindful of security implications for sensitive content. 2. Public URLs: If the images don't require access control, you could consider making them publicly accessible in S3 and using permanent URLs. As mentioned by Community Member 2 in the Discord Support Thread: "if you want anybody to be able to download a file, meaning it is public, I believe you can even get a permanent url from S3 (AWS)." 3. Caching: Implement caching mechanisms on both server and client sides to store and serve frequently accessed image URLs. 4. Content Delivery Network (CDN): For improved performance and reduced latency, consider using a CDN to serve your images. 5. Custom route: As suggested by Community Member 2, you could create a custom route in your Wasp app that handles the image requests and generates presigned URLs as needed: "have them share a link that is not a S3 presigned URL -> instead you share a link to a route in your Wasp app (probably on client), for example https://myapp.com/downloadFile/fileId where fileId is something that will allow you to know which file they want" This approach gives you more control over the process and allows you to implement caching or other optimizations. To implement the best solution for your specific use case, you'll need to consider factors such as security requirements, expected traffic patterns, and performance needs. The current OpenSaaS example provides a starting point, but you may need to adapt and optimize it for displaying frequently viewed user-uploaded images. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @pawnpusher12, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

pawnpusher12OP•2w ago

So, for private images - generate presigned urls with longer expiration time, and do checks for non-expired cached url before generating new one. For public images - perma s3 urls? is this a reasonable approach, could it be done better? @kapa.ai

kapa.ai•2w ago

Let me check the knowledge base for you again. One moment! :books: ---------------------- Your approach is indeed reasonable and aligns well with the best practices discussed in the knowledge sources. Let's break it down and see if there are any potential improvements: 1. For private images: - Generating presigned URLs with longer expiration times is a good strategy. This reduces the frequency of URL regeneration while maintaining access control. - Checking for non-expired cached URLs before generating new ones is an excellent optimization. This can significantly reduce the number of S3 API calls and potentially lower costs. 2. For public images: - Using permanent S3 URLs for public images is a valid approach. This eliminates the need for presigned URLs altogether for content that doesn't require access control. This approach is solid and addresses the main concerns of efficiency and security. However, based on the information from our knowledge sources, we can suggest a few potential improvements or considerations: 1. Implement a custom route: As suggested in the Discord Support Thread, you could create a custom route in your Wasp app to handle image requests. This gives you more control over the process and allows for easier implementation of caching and access control. For example:

   <https://myapp.com/downloadFile/fileId>

   <https://myapp.com/downloadFile/fileId>

This route could then determine whether to serve a cached URL, generate a new presigned URL, or serve a public URL based on the file's properties and your access control logic. 2. Use Wasp's caching mechanisms: Leverage Wasp's built-in caching capabilities (which use react-query under the hood) to cache URLs on the client side. This can improve performance for frequently accessed images. 3. Consider a CDN: For improved performance and reduced latency, especially for public images, consider using a Content Delivery Network (CDN) in conjunction with your S3 storage. 4. Implement server-side caching: In addition to client-side caching, implement server-side caching of presigned URLs to further reduce S3 API calls. 5. Adjust expiration times dynamically: Consider implementing a system that adjusts presigned URL expiration times based on usage patterns. Frequently accessed images could have longer expiration times, while less frequently accessed ones could have shorter times. 6. Security considerations: Ensure that your access control logic is robust, especially when dealing with longer-lived presigned URLs for private content. In conclusion, your proposed approach is sound and can be effectively implemented within a Wasp app. The suggestions above can potentially optimize it further, but the core idea of using longer-lived presigned URLs for private content and permanent URLs for public content is a good starting point. As always, the best approach may vary depending on your specific use case and requirements. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @pawnpusher12, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

miho•2w ago

This is not really a Wasp specific question, you can do whatever works with Node.js. I'd go for some sort of cache either: in-memory, using Redis or maybe even in the database. Just store the URL and the until it's valid. Then if the cached URL is expired, regenerate a new one and store it again in the cache.

pawnpusher12OP•2w ago

Thank you!

AAA•2w ago

import { create } from 'zustand';
import { persist } from 'zustand/middleware';
import { getDownloadFileSignedURL } from 'wasp/client/operations';

type SignedUrlCache = {
  url: string;
  expiresAt: number;
};

type SignedUrlStore = {
  urls: Record<string, SignedUrlCache>;
  getSignedUrl: (key: string) => Promise<string | null>;
  refreshSignedUrl: (key: string) => Promise<string | null>;
  clearCache: () => void;
};

const URL_EXPIRY_TIME = 45 * 60 * 1000; // 45 minutes

export const useSignedUrlStore = create<SignedUrlStore>()(
  persist(
    (set, get) => ({
      urls: {},
      
      getSignedUrl: async (key: string) => {
        const state = get();
        const cached = state.urls[key];

        // Return cached URL if valid
        if (cached && cached.expiresAt > Date.now()) {
          return cached.url;
        }

        // Fetch new URL if no cache or expired
        try {
          const signedUrl = await getDownloadFileSignedURL({ key });
          set((state) => ({
            urls: {
              ...state.urls,
              [key]: {
                url: signedUrl,
                expiresAt: Date.now() + URL_EXPIRY_TIME
              }
            }
          }));
          return signedUrl;
        } catch (err) {
          console.error('Failed to get signed URL for:', key);
          return null;
        }
      },

[truncated]

import { create } from 'zustand';
import { persist } from 'zustand/middleware';
import { getDownloadFileSignedURL } from 'wasp/client/operations';

type SignedUrlCache = {
  url: string;
  expiresAt: number;
};

type SignedUrlStore = {
  urls: Record<string, SignedUrlCache>;
  getSignedUrl: (key: string) => Promise<string | null>;
  refreshSignedUrl: (key: string) => Promise<string | null>;
  clearCache: () => void;
};

const URL_EXPIRY_TIME = 45 * 60 * 1000; // 45 minutes

export const useSignedUrlStore = create<SignedUrlStore>()(
  persist(
    (set, get) => ({
      urls: {},
      
      getSignedUrl: async (key: string) => {
        const state = get();
        const cached = state.urls[key];

        // Return cached URL if valid
        if (cached && cached.expiresAt > Date.now()) {
          return cached.url;
        }

        // Fetch new URL if no cache or expired
        try {
          const signedUrl = await getDownloadFileSignedURL({ key });
          set((state) => ({
            urls: {
              ...state.urls,
              [key]: {
                url: signedUrl,
                expiresAt: Date.now() + URL_EXPIRY_TIME
              }
            }
          }));
          return signedUrl;
        } catch (err) {
          console.error('Failed to get signed URL for:', key);
          return null;
        }
      },

[truncated]

Here is a store you could use

Gaming

Programming

Caching data with wasp

Did you find this page helpful?