SM
Software Mansion•2y ago
nickdichev_fw

Intermittent Failures with RTMP Sink

We are running into some intermittent failures with the RTMP sink. What we notice is that sometimes a given pipeline will stream all the way through and sometimes the RTMP sink will raise when writing frames and crash the pipeline. 
22:01:58.807 [error] <0.5365.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 139, 207, 65, 154, 128, 54, 188, 23, 73, 255, 152, 130, 98, 10, 43, 88, 28, 64, 176, 10, 39, 247, 233, 179, 54, 27, 17, 168, 97, 24, 82, 152, 175, 21, 138, 252, 216, 108, 205, 134, ...>>, pts: 27240000000, dts: 27240000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:01:58.808 [error] GenServer #PID<0.5369.0> terminating
** (RuntimeError) writing audio frame failed with reason: "End of file"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:269: Membrane.RTMP.Sink.write_frame/3
22:01:58.807 [error] <0.5365.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 139, 207, 65, 154, 128, 54, 188, 23, 73, 255, 152, 130, 98, 10, 43, 88, 28, 64, 176, 10, 39, 247, 233, 179, 54, 27, 17, 168, 97, 24, 82, 152, 175, 21, 138, 252, 216, 108, 205, 134, ...>>, pts: 27240000000, dts: 27240000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:01:58.808 [error] GenServer #PID<0.5369.0> terminating
** (RuntimeError) writing audio frame failed with reason: "End of file"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:269: Membrane.RTMP.Sink.write_frame/3
this one crashed ~20 seconds into a 1 hour long stream, so I don't understand how it could have been the end of file. Additionally, in the error log I see an error handling a video buffer, however, the crash is in the write audio frame path. 
22:57:02.461 [error] <0.3134.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 0, 7, 65, 154, 0, 122, 0, 63, 204>>, pts: 3626320000000, dts: 3626320000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:57:02.461 [error] GenServer #PID<0.3138.0> terminating
** (RuntimeError) writing video frame failed with reason: "Invalid argument"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:285: Membrane.RTMP.Sink.write_frame/3
22:57:02.461 [error] <0.3134.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 0, 7, 65, 154, 0, 122, 0, 63, 204>>, pts: 3626320000000, dts: 3626320000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:57:02.461 [error] GenServer #PID<0.3138.0> terminating
** (RuntimeError) writing video frame failed with reason: "Invalid argument"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:285: Membrane.RTMP.Sink.write_frame/3
3 Replies
nickdichev_fw
nickdichev_fwOP•2y ago
Any ideas on how to debug or fix this? Is there a constraint on the protocol which makes the element raise/crash on an error writing a frame? In our use case, I think we would rather drop the frame than crash the stream. However, I can understand there can be problems with dropping a keyframe. After some additional testing today, we notice there seems to be some correlation between the chunk size passed to the file source we are reading the h264/aac from and the file size. I also noticed that on each crash the buffer starts with the binary <<0, 0, ....>> -- is there maybe some correlation/ratio between the file size, chunk size, bit/sample rate we need to be concerned with?
shuntrho
shuntrho•2y ago
Hi nick, unfortunately I am not sure what could be causing your issue. The error you get is thrown by ffmpeg's function av_write_frame in the native code. You can get the error codes from the messages that were printed here: https://github.com/FFmpeg/FFmpeg/blob/512bba3d1bd404627fb2799191605258654b693e/libavutil/error.c#L108 . So "Invalid argument" corresponds to EINVAL error in the ffmpeg, and "End of file" to AVERROR_EOF. Another step would be looking at the definition of av_write_frame in the ffmpeg code: https://github.com/FFmpeg/FFmpeg/blob/512bba3d1bd404627fb2799191605258654b693e/libavformat/mux.c#LL1198C6-L1198C6 You can try to figure out where those errors come from. I would assume that one of the errors you get is causing the other one, presumably the error with writing video frame causes error with audio. But that's just a guess.
GitHub
FFmpeg/mux.c at 512bba3d1bd404627fb2799191605258654b693e · FFmpeg/F...
Mirror of https://git.ffmpeg.org/ffmpeg.git. Contribute to FFmpeg/FFmpeg development by creating an account on GitHub.
GitHub
FFmpeg/error.c at 512bba3d1bd404627fb2799191605258654b693e · FFmpeg...
Mirror of https://git.ffmpeg.org/ffmpeg.git. Contribute to FFmpeg/FFmpeg development by creating an account on GitHub.
odingrail
odingrail•2d ago
Hey! Any updates on this? We are running in the same issue when doing the forwarding to the RTMP sink. But unlike your case, our payload is actually the audio payload: %Membrane.Buffer{payload: <<255, 241, 80, 128, 1, 191, 252, 33, 16, 4, 96, 140, 28>>, pts: 4318911564, dts: nil, metadata: %{name: :rtsp_source_tcp_receiver, socket: #Port<0.26100>, rtp: %{timestamp: 3337790470, extensions: [], marker: true, sequence_number: 911, payload_type: 97, ssrc: 3256846846, padding_size: 0, csrcs: [], total_header_size: 12}}} So it is somehow trying to call Native.write_video_frame with the audio payload

Did you find this page helpful?