Kevin Powell - Community•16mo ago

Failed to Fetch (JS)

A friend of mine sent me a website and I was curious as to how they were getting data. I found an API which I'm playing around with fetch data using JavaScript. I'm attempting to read an API, but it returns "TypeError: Failed to fetch" Here's my code:

let api = "https://users.roblox.com/v1/users/4060137991";

function fetchData() {
    fetch(api)
    .then(response => console.log(response))
    .catch(err => console.log("error:" + err))
}

fetchData()

let api = "https://users.roblox.com/v1/users/4060137991";

function fetchData() {
    fetch(api)
    .then(response => console.log(response))
    .catch(err => console.log("error:" + err))
}

fetchData()

Typing the API in browser returns data, but when I run this function it returns an error. What am I doing wrong?

41 Replies

ἔρως•16mo ago

you're using let don't think that the variable exists inside the function nope, i was wrong

MattOP•16mo ago

I read on the documentation that I need to use GET

MattOP•16mo ago

MattOP•16mo ago

Do I need to do something like this, with method, mode, cache, credentials, headers, etc?

ἔρως•16mo ago

you're being bitten by cors

ἔρως•16mo ago

if you paste your code in a random tab, you get this:

ἔρως•16mo ago

if you paste it into roblox, then it works

MattOP•16mo ago

Would that mean I have to do something like this? or do I have to use a proxy server? Something like https://corsproxy.github.io/

ἔρως•16mo ago

don't think you're supposed to access this api

MattOP•16mo ago

Hmm It's on their website https://catalog.roblox.com/docs/index.html and it's not under their robots.txt

ἔρως•16mo ago

i have no idea then

MattOP•16mo ago

okay

Jochem•16mo ago

the error you get suggests the solution:

    fetch(api, {mode: 'no-cors'})

    fetch(api, {mode: 'no-cors'})

MattOP•16mo ago

This got rid of the error

ἔρως•16mo ago

but doesn't fetch https://users.roblox.com/docs/index.html <-- i did find the api

Jochem•16mo ago

it does fetch

MattOP•16mo ago

I don't know much about scraping, just playing around to see if I can get data CORS is used to prevent scraping, adding mode no-cors bypasses this?

Jochem•16mo ago

CORS can be used for that, but it's up to the server to enforce it properly

MattOP•16mo ago

How could I get this to console log?

MattOP•16mo ago

and when I try to log as json:

Jochem•16mo ago

mode: "no-cors" doesn't magically hack the server or anything, it just sends a request without CORS headers

MattOP•16mo ago

ἔρως•16mo ago

it didn't for me

ἔρως•16mo ago

you use it like this: [...].then((response) => response.json()).[...]

MattOP•16mo ago

yeah thats what I wrote that spit out that log

ἔρως•16mo ago

it automatically parses the json code

MattOP•16mo ago

Do people scrape data that is blocked by CORS using something like Selenium?

ἔρως•16mo ago

depends on what you're familiar with

Jochem•16mo ago

it's being weird, I see the response JSON in the console, but parsing it with response.json() produces a warning that it isn't valid JSON

ἔρως•16mo ago

probably the russian characters should be escaped but it should work anyways

MattOP•16mo ago

Just reading some random stackoverflow posts, and people say to get around this kind of issue, you need to request through a server? Similar to a proxy? I found a github regarding scraping from the site --> https://github.com/noblox/noblox.js Seems like what they're doing is using node.js and getting a user account's cookie which they then use to request from the account so maybe to successfully scrape this site, you'd need to make the api calls while logged into an account (on their website) to avoid CORS

ἔρως•16mo ago

https://noblox.js.org/ <-- there's stuff like this, so, you probably don't need

MattOP•16mo ago

This NPM package enables operations froms the Roblox website to be executed via NodeJS; this package requests setCookie() which is an account login

ἔρως•16mo ago

https://www.npmjs.com/package/roblox

npm

roblox

This module is intended to be used to access the Roblox platform and handle (the best it can) mumbojumbo like X-CSRF-Token headers and throttling/retries.. Latest version: 1.0.40, last published: a year ago. Start using roblox in your project by running npm i roblox. There is 1 other project in the npm registry using roblox.

ἔρως•16mo ago

and this?

MattOP•16mo ago

This stuff is too far out of my scope I figured I would be able to just call the api and get data

ἔρως•16mo ago

read the code and try to figure out what they did, and try to reproduce it yourself

MattOP•16mo ago

I am but it requires node I haven't touched node at all I figured I could do this with vanilla js

ἔρως•16mo ago

node is just javascript without the browser bits

Gaming

Programming

Failed to Fetch (JS)