Theo's Typesafe Cult•2y ago

pagination from multiple sources?

So I've been trying to figure out a way to combine two paginated sources into 1 paginated feed. Essentially the current functionality pulls in a list of things from an outside api. We also have our own list of things that we'd like to start including in the same feed that we display the outside data. Both lists have the same fields, and come from paginated api endpoints, but I'm struggling to figure out a way to combine the lists on the front end that grabs data from both sources as needed based on a date field on the things. Any ideas?

15 Replies

oadster•2y ago

I’m not sure I fully understand you are fetching both sources on frontend then want to combine them? Like do each of the items have an associated date for each item couldn’t you sort by that?

SlamerzOP•2y ago

I was attempting to fetch from front end, but might move it to it's own endpoint on the backend. Basic problem in a more visual sense is that there are 2 sources with similar data that I'm attempting to combine into 1 paginated feed. (Example data structures below) Both sources are paginated. I'm looking for a strategy to combine them into a feed that gives a consistent number of results while pages are available from these 2 sources. The issue is that sometimes these 2 sources data are a bit far from each other sort wise. Example is if Source A's data for a particular month is very active but Source B's is not. When I currently pull in page 1 from each and combine/sort them the date range (in the example below) would be something like June 20-june 4th But then when page 2 of Source A is pulled in the dates are back up to June 19th. But I'm having a hard time visualizing or at least putting together a way to pull from both sources, check date ranges to see about pulling the next page of one of the sources and see if those better match the date range Source A:

{
  Page: 1,
  Decks: [{
    date: June 20
  },
  {
    date: June 20
  },
   {
    date: June 20
  },
  {
    date: June 19,
  },
  {
    date: June 19
  }
]
}

{
  Page: 1,
  Decks: [{
    date: June 20
  },
  {
    date: June 20
  },
   {
    date: June 20
  },
  {
    date: June 19,
  },
  {
    date: June 19
  }
]
}

Source B:

{
  Page: 1,
  Decks: [{
    date: June 20
  },
  {
    date: June 18
  },
   {
    date: June 18
  },
  {
    date: June 16,
  },
  {
    date: June 4
  }
]
}

{
  Page: 1,
  Decks: [{
    date: June 20
  },
  {
    date: June 18
  },
   {
    date: June 18
  },
  {
    date: June 16,
  },
  {
    date: June 4
  }
]
}

Keef•2y ago

Is it possible for you to scrape and store these to build out an order ahead of time (storing them in a database for example) or are you trying to do these things on the fly The hard part here is building out some order between the two “meta” data sets and managing that between request contexts The storing ahead to create some order is what I’ve implemented before and it really really makes things simpler

Keef•2y ago

https://softwareengineering.stackexchange.com/questions/362585/implementing-paging-with-multiple-data-sources

Software Engineering Stack Exchange

Implementing paging with multiple data sources

I have multiple data sources that I need to search across and return back to the client (web app). For example the sources are: an elastic search index a sql database Is there an efficient way to

Keef•2y ago

https://softwareengineering.stackexchange.com/questions/279115/how-to-paginate-and-combine-results-that-come-from-different-sources-sorted

Software Engineering Stack Exchange

how to paginate and combine results that come from different source...

I have the following scenario: I have a user that needs to see some news feeds from different sites and I want to paginate them and sort them by date. The problem is I don't understand how to do ...

Keef•2y ago

Those two look helpful in your situation idk if you’ve seen them

SlamerzOP•2y ago

One of the datasources is an API that is completely out of my control, that updates often so storing the results from it in a database doesn't really work. If the source didn't update very often I'd pry just clone it into my database and combining it all into 1 source

Keef•2y ago

Eh I’ve consumed apis that are third party and maybe caching plus managing updating is a simpler problem then what you are trying to do. So it could still be an option

SlamerzOP•2y ago

At this point the strategy seems to be (Based on these articles and a few others I saw) make an endpoint that gives paginated values for say 20 results. That endpoint pulls in 40 results from each source. Page 1 [0-40] Page 2 [0-40] Combine them, serve the top 20, and hope that the next pages when those 40 run out weren't super out of synce date whise Yeah, I'm definetly gonna have to do some caching and a few other things

Keef•2y ago

It’s gonna be a fun problem to tackle peepoAing

SlamerzOP•2y ago

Yeah, with all the aggrigation websites out there I'm just more shocked there aren't more well defined solutions for a similar issue lol

Keef•2y ago

They are probably doing what I mentioned Creating some order from multiple sources and just relying on eventual consistency Since the topics are the same I’m sure you could find a way to bind your results together for storage

SlamerzOP•2y ago

Yeah, I've seen some fuzzy pagination examples

Keef•2y ago

@ me when you get it down 😊

SlamerzOP•2y ago

Gaming

Programming

pagination from multiple sources?