Software Mansion•2y ago

Intermittent Failures with RTMP Sink

We are running into some intermittent failures with the RTMP sink. What we notice is that sometimes a given pipeline will stream all the way through and sometimes the RTMP sink will raise when writing frames and crash the pipeline.

22:01:58.807 [error] <0.5365.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 139, 207, 65, 154, 128, 54, 188, 23, 73, 255, 152, 130, 98, 10, 43, 88, 28, 64, 176, 10, 39, 247, 233, 179, 54, 27, 17, 168, 97, 24, 82, 152, 175, 21, 138, 252, 216, 108, 205, 134, ...>>, pts: 27240000000, dts: 27240000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:01:58.808 [error] GenServer #PID<0.5369.0> terminating
** (RuntimeError) writing audio frame failed with reason: "End of file"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:269: Membrane.RTMP.Sink.write_frame/3

22:01:58.807 [error] <0.5365.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 139, 207, 65, 154, 128, 54, 188, 23, 73, 255, 152, 130, 98, 10, 43, 88, 28, 64, 176, 10, 39, 247, 233, 179, 54, 27, 17, 168, 97, 24, 82, 152, 175, 21, 138, 252, 216, 108, 205, 134, ...>>, pts: 27240000000, dts: 27240000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:01:58.808 [error] GenServer #PID<0.5369.0> terminating
** (RuntimeError) writing audio frame failed with reason: "End of file"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:269: Membrane.RTMP.Sink.write_frame/3

this one crashed ~20 seconds into a 1 hour long stream, so I don't understand how it could have been the end of file. Additionally, in the error log I see an error handling a video buffer, however, the crash is in the write audio frame path.

22:57:02.461 [error] <0.3134.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 0, 7, 65, 154, 0, 122, 0, 63, 204>>, pts: 3626320000000, dts: 3626320000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:57:02.461 [error] GenServer #PID<0.3138.0> terminating
** (RuntimeError) writing video frame failed with reason: "Invalid argument"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:285: Membrane.RTMP.Sink.write_frame/3

22:57:02.461 [error] <0.3134.0>/:rtmp_sink/ Error handling action {:split, {:handle_write, [[:video, %Membrane.Buffer{payload: <<0, 0, 0, 7, 65, 154, 0, 122, 0, 63, 204>>, pts: 3626320000000, dts: 3626320000000, metadata: %{h264: %{key_frame?: false}, mp4_payload: %{key_frame?: false}}}]]}} returned by callback Membrane.RTMP.Sink.handle_write_list


22:57:02.461 [error] GenServer #PID<0.3138.0> terminating
** (RuntimeError) writing video frame failed with reason: "Invalid argument"
    (membrane_rtmp_plugin 0.11.2) lib/membrane_rtmp_plugin/rtmp/sink/sink.ex:285: Membrane.RTMP.Sink.write_frame/3

2 Replies

nickdichev_fwOP•2y ago

Any ideas on how to debug or fix this? Is there a constraint on the protocol which makes the element raise/crash on an error writing a frame? In our use case, I think we would rather drop the frame than crash the stream. However, I can understand there can be problems with dropping a keyframe. After some additional testing today, we notice there seems to be some correlation between the chunk size passed to the file source we are reading the h264/aac from and the file size. I also noticed that on each crash the buffer starts with the binary <<0, 0, ....>> -- is there maybe some correlation/ratio between the file size, chunk size, bit/sample rate we need to be concerned with?

shuntrho•2y ago

Hi nick, unfortunately I am not sure what could be causing your issue. The error you get is thrown by ffmpeg's function av_write_frame in the native code. You can get the error codes from the messages that were printed here: https://github.com/FFmpeg/FFmpeg/blob/512bba3d1bd404627fb2799191605258654b693e/libavutil/error.c#L108 . So "Invalid argument" corresponds to EINVAL error in the ffmpeg, and "End of file" to AVERROR_EOF. Another step would be looking at the definition of av_write_frame in the ffmpeg code: https://github.com/FFmpeg/FFmpeg/blob/512bba3d1bd404627fb2799191605258654b693e/libavformat/mux.c#LL1198C6-L1198C6 You can try to figure out where those errors come from. I would assume that one of the errors you get is causing the other one, presumably the error with writing video frame causes error with audio. But that's just a guess.

GitHub

FFmpeg/mux.c at 512bba3d1bd404627fb2799191605258654b693e · FFmpeg/F...

Mirror of https://git.ffmpeg.org/ffmpeg.git. Contribute to FFmpeg/FFmpeg development by creating an account on GitHub.

GitHub

FFmpeg/error.c at 512bba3d1bd404627fb2799191605258654b693e · FFmpeg...

Mirror of https://git.ffmpeg.org/ffmpeg.git. Contribute to FFmpeg/FFmpeg development by creating an account on GitHub.

Gaming

Programming

Intermittent Failures with RTMP Sink