Jdyn Posts - Answer Overflow

Jdyn

•Created by Jdyn on 7/4/2024 in #membrane-help

Live video effects in fishjam

Hey, I am curious about how possible it is to implement live effects into fishjam webrtc streams from the server using a custom endpoint with something like video-compositor in real time. How possible is this with membrane currently? Excuse the video but here is an example of an effect I am talking about https://www.youtube.com/shorts/4YaE5u1mjlc

6 replies

SMSoftware Mansion

•Created by Jdyn on 3/11/2024 in #membrane-help

Developing an advanced Jellyfish use case

Hey I've been using jellyfish to develop a platform for essentially one-on-one calls between two people and it works really well. I'd like to now bring in something more advanced. I essentially want to: 1. take the two audio streams of the two peers from jellyfish and convert everything their saying into text using something like bumblebee whisper. 2. After a certain amount of audio has been translated into words (or maybe just X amount of time), feed the text into a gpt model and get some new text response 3. take the gpt text response and convert it back into audio using text-to-speech (using bumblebee again?) 4. feed the text-to-speech audio back directly into the jellyfish room for both peers to hear it. 5. all this would essentially need to be happening while the room is still ongoing and the entire process ideally not taking too long. I am willing to get into the weeds of membrane and jellyfish to make it work but I am new to this entire multimedia space and not sure where to start. Im hoping I can use the infrastructure around jellyfish as much as possible. So far I know that I'll need a third JF component to play the gpt audio to both peers, but not sure which one is suitable or how to get the audio to the component. I am also unsure how I can get access to the webRTC peer audio streams where I would potentially need to feed them into a custom membrane pipeline? I would appreciate any ideas you guys. Thank you for your time

53 replies

SMSoftware Mansion

•Created by Jdyn on 3/6/2024 in #membrane-help

On JF Tracks and Reconnecting (in React)

So I noticed a few things about the react-sdk and JF tracks in general. Note I have react code that works identical to the videoroom demo. If you're connected to a jellyfish room and then abruptly refresh the browser, a new set of media device ids are created which causes a new set of addTrack calls. I'm not sure if I am doing something wrong or this is intended, but since new ids are created, new tracks are added to the peer on refresh without being able to remove the old ones since any clean-up code is never fired on refresh. And even when I disconnect gracefully, the removeTrack call fails as described below. I'm not able to reliably call removeTrack() despite giving it valid local Ids. during development, I have consistently hit these bang operators https://github.com/jellyfish-dev/membrane-webrtc-js/blob/7fbb3123fc1870b974061a9309d4f2512721b253/src/webRTCEndpoint.ts#L1244 in remove and replace tracks, but the tracks are there locally and on the server. I guess the behavior I am expecting is if the peer disconnects, either the old tracks are removed, or they are somehow reused again when they reconnect, but neither happens

14 replies

SMSoftware Mansion

•Created by Jdyn on 1/17/2024 in #membrane-help

Extending the jellyfish video room demo with a queue

Hey I am curious what a scalable way would be to add a queue in front of jellyfish rooms. I have a setup very similar to the jellyroom demo. My current attempt is adding additional state for the queue to the RoomService module and somehow using the max_children option of the DynamicSupervisor to add to the queue but its getting super convoluted to manage the state Would creating another GenServer like RoomQueue to manage queueing rooms be a good idea? Any ideas would be appreciated

11 replies

SMSoftware Mansion

•Created by Jdyn on 12/19/2023 in #membrane-help

Spinning up a new GenServer for each room

I have been learning from the videoroom demo and I have a few questions.

# meeting.ex
  @impl true
  def init(%{name: name, jellyfish_address: jellyfish_address}) do
    Logger.metadata(room_name: name)

    client = Jellyfish.Client.new(server_address: jellyfish_address)

    with {:ok, room, jellyfish_address} <- create_new_room(client, name) do
      peer_timeout = Application.fetch_env!(:videoroom, :peer_join_timeout)

      client = Jellyfish.Client.update_address(client, jellyfish_address)

      Logger.info("Created meeting room id: #{room.id}")

      {:ok,
       %State{
         client: client,
         name: name,
         room_id: room.id,
         peer_timers: %{},
         peer_timeout: peer_timeout,
         jellyfish_address: jellyfish_address
       }}
    else
      {:error, reason} ->
        Logger.error("Failed to create a meeting, reason: #{inspect(reason)}")
        raise "Failed to create a meeting, reason: #{inspect(reason)}"
    end
  end

# meeting.ex
  @impl true
  def init(%{name: name, jellyfish_address: jellyfish_address}) do
    Logger.metadata(room_name: name)

    client = Jellyfish.Client.new(server_address: jellyfish_address)

    with {:ok, room, jellyfish_address} <- create_new_room(client, name) do
      peer_timeout = Application.fetch_env!(:videoroom, :peer_join_timeout)

      client = Jellyfish.Client.update_address(client, jellyfish_address)

      Logger.info("Created meeting room id: #{room.id}")

      {:ok,
       %State{
         client: client,
         name: name,
         room_id: room.id,
         peer_timers: %{},
         peer_timeout: peer_timeout,
         jellyfish_address: jellyfish_address
       }}
    else
      {:error, reason} ->
        Logger.error("Failed to create a meeting, reason: #{inspect(reason)}")
        raise "Failed to create a meeting, reason: #{inspect(reason)}"
    end
  end

I am wondering what the overhead would be for starting a GenServer for each room at a large scale and if there is a better way? My first impression which may not be correct or smart, is that we could skip the overhead of GenServers and interface directly with the jellyfish sdk directly without a GenServer. All of the nice state like peer_timers could be put into a in-memory key-value store would it work to store a single map in an in-memory key-value store of the form

%{
"jelly_address_1": %Jellyfish.Client{ ... }
"jelly_address_2": %Jellyfish.Client{ ... }
}

%{
"jelly_address_1": %Jellyfish.Client{ ... }
"jelly_address_2": %Jellyfish.Client{ ... }
}

This way a jellyfish client is only created once for each jellyfish instance instead of creating one for each room or is this a bad idea? Is it wrong / a bottle neck if many rooms try to use the same jellyfish client?

1 replies

Gaming

Programming