Railway•3mo ago

Connection closed while reading from the driver

Hey so my code works fine (code below) locally but in Railway hosted I am getting the below error, anyway I can fix this? Essentially I am using playwright to scrape some data and return it via the python API

curl -X POST "xxxxxxxxxx" \
     -H "Content-Type: application/json" \
     -d '{"url": "xxxxxxx
{"detail":"Connection closed while reading from the driver"}%

curl -X POST "xxxxxxxxxx" \
     -H "Content-Type: application/json" \
     -d '{"url": "xxxxxxx
{"detail":"Connection closed while reading from the driver"}%

Code attached

message.txt

36 Replies

Percy•3mo ago

Project ID: 77fedaf9-b72c-45fa-b96e-6c1f7f459653

ozOP•3mo ago

77fedaf9-b72c-45fa-b96e-6c1f7f459653

Brody•3mo ago

you can check your HTTP logs for more information

ozOP•3mo ago

/opt/venv/lib/python3.9/site-packages/playwright/driver/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /nix/store/xpxln7rqi3pq4m0xpnawhxb2gs0mn1s0-gcc-12.3.0-lib/lib/libstdc++.so.6)

/opt/venv/lib/python3.9/site-packages/playwright/driver/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /nix/store/xpxln7rqi3pq4m0xpnawhxb2gs0mn1s0-gcc-12.3.0-lib/lib/libstdc++.so.6)

Is the only way to solve this is to use my systemw ith docker?

Brody•3mo ago

you mean write your own dockerfile?

ozOP•3mo ago

Yeah Dont see how else I can add this dependency

Brody•3mo ago

its complicated for the most part, i'd recommend moving to a dockerfile, i can give your Dockerfile a once over after you write it?

ozOP•3mo ago

Appreciate it, never really used docker tbh, I should learn it at some point. WIll build one and send it if I run into issue sthanks

Brody•3mo ago

sounds good! additionally, would you be interested in using browserless? not having to install and bundle chrome into your container would decrease the build time, and also decrease the publish times!

ozOP•3mo ago

So I use something similar, I use BrightData's bypass browser Much better overall experience and cheaper

Brody•3mo ago

ah perfect, i just saw playwright and thought chrome install, my bad

ozOP•3mo ago

I seem to have got it all working though thanks!

Brody•3mo ago

with a Dockerfile? that was fast

ozOP•3mo ago

I used perplexity tbh haha

FROM mcr.microsoft.com/playwright:v1.38.0-focal

# Install Python and pip
RUN apt-get update && apt-get install -y python3 python3-pip

# Set the working directory
WORKDIR /app

# Copy the requirements file
COPY requirements.txt .

# Install the Python dependencies
RUN pip3 install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Install Playwright browsers and dependencies
RUN playwright install chromium
RUN playwright install-deps

# Set the command to run the application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

FROM mcr.microsoft.com/playwright:v1.38.0-focal

# Install Python and pip
RUN apt-get update && apt-get install -y python3 python3-pip

# Set the working directory
WORKDIR /app

# Copy the requirements file
COPY requirements.txt .

# Install the Python dependencies
RUN pip3 install --no-cache-dir -r requirements.txt

# Copy the rest of the application code
COPY . .

# Install Playwright browsers and dependencies
RUN playwright install chromium
RUN playwright install-deps

# Set the command to run the application
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Brody•3mo ago

not ideal, you are using BrightData yet you are installing chrome now you are also installing python from apt-get, thats not a super stable install path for installing programming languages

ozOP•3mo ago

Ah your right, stupid of me to confuse that. I wonder if I am not using brightdata correctly which prompted the need for this

Brody•3mo ago

well first off, simply try removing those two playwright commands

ozOP•3mo ago

Yes your right .. it still worked 🙂

Brody•3mo ago

okay now instead of FROM playwright and then installing python with apt-get, you can do FROM python with a version

ozOP•3mo ago

Hey sorry will check thatbut quick question

{"detail":"BrowserType.connect_over_cdp: WebSocket error: wss://[email protected]:9222/ 401 Auth Failed (ip_blacklisted: 34.105.6.0/24)\nAuth Failed ip_blacklisted: 34.105.6.0/24\nCall log:\n<ws connecting> wss://[email protected]:9222/\n  - <ws unexpected response> wss://[email protected]:9222/ 401 Auth Failed (ip_blacklisted: 34.105.6.0/24)\nAuth Failed ip_blacklisted: 34.105.6.0/24\n  - <ws error> wss://[email protected]:9222/ error WebSocket was closed before the connection was established\n  - <ws connect error> wss://[email protected]:9222/ WebSocket was closed before the connection was established\n  - <ws disconnected> wss://[email protected]:9222/ code=1006 reason=\n"}%

{"detail":"BrowserType.connect_over_cdp: WebSocket error: wss://[email protected]:9222/ 401 Auth Failed (ip_blacklisted: 34.105.6.0/24)\nAuth Failed ip_blacklisted: 34.105.6.0/24\nCall log:\n<ws connecting> wss://[email protected]:9222/\n  - <ws unexpected response> wss://[email protected]:9222/ 401 Auth Failed (ip_blacklisted: 34.105.6.0/24)\nAuth Failed ip_blacklisted: 34.105.6.0/24\n  - <ws error> wss://[email protected]:9222/ error WebSocket was closed before the connection was established\n  - <ws connect error> wss://[email protected]:9222/ WebSocket was closed before the connection was established\n  - <ws disconnected> wss://[email protected]:9222/ code=1006 reason=\n"}%

Is this saying BrightData blacklisted you guys? Or the other way arround? Every request was working at first

Brody•3mo ago

very possible, but its also a 401 instead of 403, so maybe your credentials are wrong / expired?

ozOP•3mo ago

Cant be, working locally fine

Brody•3mo ago

Gotcha, do you have static Ips enabled?

ozOP•3mo ago

Apologies for my lack of experience in this. On Railway or BrightData?

Brody•3mo ago

on your railway service, check the service settings

ozOP•3mo ago

Disabled Static Outbound IPs

Brody•3mo ago

try enabling that

ozOP•3mo ago

I did but they get banned fast it seems

Brody•3mo ago

ah then you would need to get in contact with them

ozOP•3mo ago

So its them blocking you? (BrightData)

Brody•3mo ago

correct

ozOP•3mo ago

Thanks for info Brody. Any info I can provide that will help them? Like any info about yoru servr or IP ranges or etc?

Brody•3mo ago

you have a static IP now, so that's all you'd need to give them

ozOP•3mo ago

Perfect thanks again. I always feel confident in my knowledge until I run into these issues then remeber I have so muc more to learn 🙂 Appreciate your pateince with me.

Brody•3mo ago

I wish I could help here but unfortunately an IP ban is outside of what we can control

ozOP•3mo ago

Of course I understand You helped a lot already

Gaming

Programming

Connection closed while reading from the driver