Workers design advice - running multiple curls and processing responses #coding-help
Hi all, I need to run ten curls in a row consecutively, could you please advise me on design?
Final goal is having one single page with
1. a few input fields like email, key, etc
2. hitting the bulk run button
3. processing responses and reporting (un)success on same page
It is a migration from postman runner, so the only differences between requests is path, request body, and test criteria
Not a developer myself and would highly appreciate advice on how to built it in simplest way on full Cloudflare stack
Already got a worker running for submitting authed request and getting the response, and playing around on VSS + Wrangler and CF tenant
3 Replies
Based on scope so far you could implement all this in one worker pretty easily. There is effectively 2 'routes', so far. (1) Renders the form with email, key and 'bulk run' button (2) Responds to the form submission, does the work and returns the results
You could separate the code these 2 routes simply by checking for 'GET' and 'POST' in the request.
If you plan to expand on this functionality, then I highly recommend using an actual router e.g. itty-router (easy, small, Cloudflare appropriate)
Given that you plan to make multiple requests, assuming the responses are not co-dependant then this situation is ripe for a parallel requests, meaning you can get all the responses simultaneously, up to 50 I think on the free plan and join them back together.
Here is some sample worker code that will do effectively what I have described:
const html_form =
<form method=post action='/' >
Key : <input type=text name=key value=''><br/>
Email : <input type=text name=email value=''><br/>
<button type=submit>Run</button>
</form>
;
const jobs = [
"/process?value=1",
"/process?value=2",
"/process?value=3",
];
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
// test route to test remote requests
if (url.pathname === '/process') {
const value = url.searchParams.get('value');
return new Response(parseInt(value)*2, {
status: 200,
headers: { 'content-type': 'text/plain' },
});
}
if (request.method === 'POST') {
const data = await request.formData();
// do what is required with these values
const key = data.get('key');
const email = data.get('email');
const job_promises = [];
for (const job of jobs) {
job_promises.push(fetch(url.origin+job,));
}
const job_responses = await Promise.all(job_promises);
const job_results = await Promise.all(job_responses.map(r => r.text()));
return new Response(job_results.join("\n"), {
status: 200,
headers: { 'content-type': 'text/plain' },
});
}
return new Response(html_form, {
headers: { 'content-type': 'text/html' },
});
}
};
Having a bit of trouble formatted the code, so you'll need to add backticks around the html_form variable contentsresult is
It's not actually the best implementation of parallel requests (fetch + r.text()) should be moved into an independant function for each request, but easy to read this way.