Web Scraping
Hi guys, recently I've been thinking about inserting a function to import a certain product into my project, so I need to know web scraping, does anyone know content, maybe in videos can someone teach this area better?, I want to do it in javascript
9 Replies
"certain product"? A lot of the time, scraping is used for copyright infringement and TOS abuse, instructions for which aren't allowed to be discussed on Discord. Can you clarify what you're scraping and why?
instead of web scraping, try to see if there's an api
if you scrape a website, and you intend to make it public, you may be required to give attribution to the website and have a link to the license terms
Fireship did a bit on the legal issues.
Personally, I did a Puppeteer script to pull images from Vecteezy. When I started out I was pulling down like 10–20k a day. Then they added some checksums to the download links to prevent scraping. I fixed my program & began again. There was about four rounds of this until my account was suspended & I was told to call support to get it reactivated.
Surprisingly, the tech I talked to had no idea what I had been doing & just turned my account back on. There was now a rate limit to where if I pull down more than 200 images a day, I get automatically suspended for 24 hours. (I'm authenticated to have full access to all the images, so I can't pull tricks like rotating my IP to hide my traffic.)
I've not tried them, but there's a new generation of AI scrapers that are supposedly pretty powerful.
Fireship
YouTube
Am I going to jail for web scraping?
Is it legal to mine data from a public website? Let's take a detailed look at recent lawsuits and controversies surrounding the practice of web scraping.
#programming #webdevelopment #thecodereport
💬 Chat with Me on Discord
https://discord.gg/fireship
🔗 Resources
Booking vs Ryanair https://corporate.ryanair.com/news/ryanair-wins-its-case-...
Bardeen
YouTube
Comparing Top 5 AI Web Scraping Tools (updated 2024)
Explore how web scraping can transform your data collection process, whether you're a small business or a large corporation.
Try it for free: https://s.bardeen.ai
This video covers various types of web scrapers, including browser-based, cloud-based, and hybrid solutions, helping you choose the right tool.
We highlight the benefits of AI web ...
that genuinely sounds like it would be against Vecteezy's TOS, which makes it inappropriate to discuss on this server
I have "unlimited" downloads with a paid account.
I just can't republish the images. Believe you me, I read the legalese.
stackoverflow is a good candidate for web scraping, but they are (possibly) breaking license agreements by blocking everything except google
so, that instantly is a very gray one
if you want to do web scraping, do so on websites that specifically allow it
exactly. And please limit the discussion of this topic to sites that specifically allow it, as per Discord TOS
(and "unlimited" downloads don't count, those can still have fair use policies and policies against scraping)
and alternatively to scraping, use the api
Like I said, I read the terms.