Data Fetching Issue: Outdated Content from External TXT file Despite Cache Control Measures
[Project ID: 9628a8da-e395-4f61-9904-4850be48eac0]
TL;DR
I'm trying to find out why my Railway-hosted application is not fetching the latest data from my external TXT file despite successful fetches in other environments and having implemented standard cache control measures. Any insights into Railway-specific configurations, caching, or network behaviours that might be affecting outbound HTTP requests would be helpful.
Detail:
I have an application hosted on Railway which is not fetching the latest data from an external TXT file. When the same request is made from local development, Postman, or via a browser, it retrieves the latest data. Railway continues to receive an older version of the data.
It's a really simple request:
app.get('/downloadproductfeed', async (req, res) => {
try {
const response = await fetch('https://www.example.com/feed.txt', {
headers: {
'Content-Type': 'text/plain',
'Cache-Control': 'no-cache, no-store, must-revalidate',
'Pragma': 'no-cache',
'Expires': '0'
}
});
if (!response.ok) {
throw new Error(Error fetching data: ${response.statusText});
}
const data = await response.text();
console.log(data);
res.send(data);
} catch (error) {
console.error('Error:', error);
res.status(500).send('An error occurred while fetching the product feed.');
}
});
Steps Taken to Resolve:
- Implemented Cache Control Headers
- Redeployed Application on Railway to ensure that the latest code is running.
- Confirmed that the external TXT file is updated and contains the latest data.
- Checked with Different Clients: Made requests to the same endpoint using various clients (like Postman and browsers), which successfully fetched the latest data, indicating the problem is specific to the Railway environment.
Any pointers would be hugely appreciated.Solution:Jump to solution
@jack1994 - please check this link https://appliance-world-online.up.railway.app/get-product?id=36741
11 Replies
Project ID:
9628a8da-e395-4f61-9904-4850be48eac0this is super odd, are you able to provide the link to the txt file so we can try to reproduce this?
Yeah it is strange. If you search for the id: 36741 in the txt file you'll notice a field saying "in_stock" which is for availability. When I query the same ID within my railway app I get "backorder". https://www.applianceworldonline.com/media/feeds/feed_15.txt
Snippet that I used to log the line:
app.get('/downloadproductfeed', async (req, res) => {
try {
const response = await fetch('https://www.applianceworldonline.com/media/feeds/feed_15.txt', {
headers: {
'Content-Type': 'text/plain',
'Cache-Control': 'no-cache, no-store, must-revalidate',
'Pragma': 'no-cache',
'Expires': '0'
}
});
const data = await response.text();
const lines = data.split('\n');
lines.forEach(line => {
const fields = line.split('\t');
if (fields[0] === '36741') {
console.log('Line with ID 36741:', line);
}
});
this is a tabular formatted text file right?
Yes
do you have the url for your code that youve shown?
Solution
@jack1994 - please check this link https://appliance-world-online.up.railway.app/get-product?id=36741
Thank you for that. Noticed my app is pulling in the correct data for that ID now but still unsure what caused it? Did you do anything different? I wonder if there's some sort of delay with the feed data. I'll have to keep an eye out for discrepancies this week
in my app i am also setting no cache headers when i return the data, you might not be doing that
@Brody Can you share your code? I even started a fresh project using Railway's own express template and it still caches with headers disabled?
Testing with this ID: 33242. The sale_price on the feed is £499.00. The sale_price locally on the app is £499. Yet logging in Railway returns sale_price £599.00. Your app returns the correct value so it's clearly something I'm doing wrong despite it working locally.
https://express-production.up.railway.app/downloadproductfeed
EDIT: added a timestamp to the URL and it seems to have fixed it. Definitely worth adding something to docs about caching ...
railway is not catching anything, they never have, it is your browser that is doing the cacheing