Rob Comments - Answer Overflow

Rob

Posts Comments

CCConvex Community

•Created by Rob on 5/7/2024 in #support-community

Crawlee in Convex Action

Ah, I see. Thanks. Any plans to support it in the future?

5 replies

CCConvex Community

•Created by Rob on 5/7/2024 in #support-community

Crawlee in Convex Action

Here is my action code:

"use node";
// For more information, see https://crawlee.dev/
import { Configuration, PlaywrightCrawler, ProxyConfiguration } from "crawlee";
import { router } from "./routes.js";
import { action } from "./_generated/server.js";

export const runCrawlee = action({
    args: {},
    handler: async () => {
        const startUrls = ["https://crawlee.dev/"];

        const crawler = new PlaywrightCrawler(
            {
                // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
                requestHandler: router,
                // Comment this option to scrape the full website.
                maxRequestsPerCrawl: 20,
            },
            new Configuration({ persistStorage: false })
        );

        await crawler.run(startUrls);

        return await crawler.getData();
    },
});

"use node";
// For more information, see https://crawlee.dev/
import { Configuration, PlaywrightCrawler, ProxyConfiguration } from "crawlee";
import { router } from "./routes.js";
import { action } from "./_generated/server.js";

export const runCrawlee = action({
    args: {},
    handler: async () => {
        const startUrls = ["https://crawlee.dev/"];

        const crawler = new PlaywrightCrawler(
            {
                // proxyConfiguration: new ProxyConfiguration({ proxyUrls: ['...'] }),
                requestHandler: router,
                // Comment this option to scrape the full website.
                maxRequestsPerCrawl: 20,
            },
            new Configuration({ persistStorage: false })
        );

        await crawler.run(startUrls);

        return await crawler.getData();
    },
});

"use node";
import { createPlaywrightRouter } from "crawlee";

export const router = createPlaywrightRouter();

router.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ["https://crawlee.dev/**"],
        label: "detail",
    });
});

router.addHandler("detail", async ({ request, page, log, pushData }) => {
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });

    await pushData({
        url: request.loadedUrl,
        title,
    });
});

"use node";
import { createPlaywrightRouter } from "crawlee";

export const router = createPlaywrightRouter();

router.addDefaultHandler(async ({ enqueueLinks, log }) => {
    log.info(`enqueueing new URLs`);
    await enqueueLinks({
        globs: ["https://crawlee.dev/**"],
        label: "detail",
    });
});

router.addHandler("detail", async ({ request, page, log, pushData }) => {
    const title = await page.title();
    log.info(`${title}`, { url: request.loadedUrl });

    await pushData({
        url: request.loadedUrl,
        title,
    });
});

This code is essentially the template for Playwright + Typescript provided by crawlee, adapted to be in a Convex action, with some slight adjustments based on documentation of deploying on an AWS lambda https://crawlee.dev/docs/deployment/aws-cheerio

5 replies

PD🧩 Plasmo Developers

•Created by frostman on 2/9/2024 in #👟framework

Pin Side Panel like google search

If you get rid of your popup file and put the following code in your BGSW (background.ts), it will make it so the "action" button open the sidepanel directly, which can then be pinned

chrome.action.onClicked.addListener((tab) =>
    chrome.sidePanel.open({ windowId: tab.windowId })
);

chrome.action.onClicked.addListener((tab) =>
    chrome.sidePanel.open({ windowId: tab.windowId })
);

5 replies

PD🧩 Plasmo Developers

•Created by Erskine Williams on 4/26/2024 in #🔰newbie

Loading a content script on a specific tab

No problem!

5 replies

PD🧩 Plasmo Developers

•Created by Erskine Williams on 4/26/2024 in #🔰newbie

Loading a content script on a specific tab

In my BGSW, I did this:

import contentTs from "url:~contents/content.ts";

const getFilenameFromImport = (fullFileName: string) => {
    const split = fullFileName.split("/").pop();
    if (split === undefined)
        throw new Error(
            `Issue parsing filename in background.ts: ${fullFileName}`
        );
    const filename = split.split("?")[0];
    return filename;
};

const contentTsFilename = getFilenameFromImport(contentTs);

import contentTs from "url:~contents/content.ts";

const getFilenameFromImport = (fullFileName: string) => {
    const split = fullFileName.split("/").pop();
    if (split === undefined)
        throw new Error(
            `Issue parsing filename in background.ts: ${fullFileName}`
        );
    const filename = split.split("?")[0];
    return filename;
};

const contentTsFilename = getFilenameFromImport(contentTs);