Fix timestamps between contiguous mixed audio packets may vary #199

XueshiQiao · 2019-07-24T02:43:49Z

We found timestamps between contiguous mixed audio packets may vary, which will mislead the client WebRTC neteq module make too much accelerations and expansions, and lead to the rhythm of music is very unstable, so I do this fix to make the timestamps between contiguous mixed audio packets stable as much as possible.

daijh · 2019-07-24T05:10:55Z

source/agent/audio/audioMixer/AcmEncoder.cpp

-    frame.timeStamp = (AudioTime::currentTime()) * m_rtpSampleRate / 1000;
+
+    int64_t computedTimestamp =  AudioTime::currentTime() * m_rtpSampleRate / 1000; 
+    int64_t sizeFor10Ms = m_rtpSampleRate * 10 / 1000 * frame.additionalInfo.audio.channels; //size for 10 ms, such as 48000 * 10/1000 * 2 = 960


multiply by frame.additionalInfo.audio.channels is not correct.

Thx, would you please tell me why it's not correct?

daijh

The interval of OPUS frame is 20ms, not 10ms.
Please try to use timestamp in SendData() callback.

XueshiQiao · 2019-07-24T07:13:53Z

@daijh Thanks for your review. Maybe I misunderstood the code. I tried to print timestamp parameter in SendData() function, it looks like this, the diff between two contiguous timestamps is 2880, I don't know what does it meaning and how I should use them: (In our case, we use samplerate 48000)

2019-07-24 14:57:46,954  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2375760, diff with last Timestamp: 2880
2019-07-24 14:57:46,964  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2377680, diff with last Timestamp: 2880
2019-07-24 14:57:46,974  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2378640, diff with last Timestamp: 2880
2019-07-24 14:57:46,984  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2380560, diff with last Timestamp: 2880
2019-07-24 14:57:46,994  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2381520, diff with last Timestamp: 2880
2019-07-24 14:57:47,004  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2383440, diff with last Timestamp: 2880
2019-07-24 14:57:47,014  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2384400, diff with last Timestamp: 2880
2019-07-24 14:57:47,024  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2386320, diff with last Timestamp: 2880
2019-07-24 14:57:47,034  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2387280, diff with last Timestamp: 2880
2019-07-24 14:57:47,044  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2389200, diff with last Timestamp: 2880
2019-07-24 14:57:47,054  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2390160, diff with last Timestamp: 2880
2019-07-24 14:57:47,064  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2392080, diff with last Timestamp: 2880
2019-07-24 14:57:47,074  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2393040, diff with last Timestamp: 2880
2019-07-24 14:57:47,084  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2394960, diff with last Timestamp: 2880
2019-07-24 14:57:47,094  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2395920, diff with last Timestamp: 2880
2019-07-24 14:57:47,104  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2397840, diff with last Timestamp: 2880
2019-07-24 14:57:47,114  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2398800, diff with last Timestamp: 2880
2019-07-24 14:57:47,124  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2400720, diff with last Timestamp: 2880
2019-07-24 14:57:47,134  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2401680, diff with last Timestamp: 2880
2019-07-24 14:57:47,144  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2403600, diff with last Timestamp: 2880
2019-07-24 14:57:47,154  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2404560, diff with last Timestamp: 2880
2019-07-24 14:57:47,165  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2406480, diff with last Timestamp: 2880
2019-07-24 14:57:47,174  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2407440, diff with last Timestamp: 2880
2019-07-24 14:57:47,184  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2409360, diff with last Timestamp: 2880
2019-07-24 14:57:47,195  - INFO: mcu.media.AcmEncoder - ====!===this: 0x30da890, timestamp passed in: 2410320, diff with last Timestamp: 2880
2019-07-24 14:57:47,205  - INFO: mcu.media.AcmEncoder - ====!===this: 0x2e8ea50, timestamp passed in: 2412240, diff with last Timestamp: 2880

daijh · 2019-07-24T08:06:10Z

Could you provide more info so I can reproduce on my side and try to improve it?
How many audio inputs? What is the input audio codec, sample rate and channels?
What is the audio output?
Thank you!

XueshiQiao · 2019-07-24T10:11:33Z

In our case, to reproduce this problem is very simple , and we can reproduce it by 100%, it only need 2 mobile clients, the first one publish a audio stream which plays a music, and if the second one subscribe the first one's single stream, the music sounds good, but if the second one subscribe the mixed stream, the music sounds very bad, the beats is a mass.

My fix can resolve this problem, its key goal is to keep the diff of two timestamp constantly, but if it's not correct for all cases, would you please improve it for us and other users?

XueshiQiao · 2019-07-24T10:13:23Z

Extra info:
input/output audio codec: opus
sample rate: 48000
channels: maybe 2

XueshiQiao · 2019-08-06T07:27:58Z

Could you provide more info so I can reproduce on my side and try to improve it?
How many audio inputs? What is the input audio codec, sample rate and channels?
What is the audio output?
Thank you!

Hi @daijh, is there any progresses?

daijh · 2019-08-09T03:00:41Z

We prepared two patches to improve the timestamp, still under review and testing.

XueshiQiao · 2019-08-09T03:19:47Z

We prepared two patches to improve the timestamp, still under review and testing.

Thanks very much!

duan-xiande · 2019-08-13T08:02:44Z

@qiaoxueshi We are about to close this PR since @daijh has fixed this issue with PR #213(#213) on this repo and PR #14(open-webrtc-toolkit/owt-deps-webrtc#14) on owt-deps-webrtc.

XueshiQiao · 2019-08-13T08:08:34Z

@qiaoxueshi We are about to close this PR since @daijh has fixed this issue with PR #213(#213) on this repo and PR #14(open-webrtc-toolkit/owt-deps-webrtc#14) on owt-deps-webrtc.

Thanks!

XueshiQiao added 2 commits July 24, 2019 10:32

Use uint64_t for sTimestampOffset, coz original uint32_t will overflowed

2522bff

Fix timestamps between contiguous miexed audio packets may vary

54a7b2a

starwarfan requested a review from daijh July 24, 2019 02:45

daijh reviewed Jul 24, 2019

View reviewed changes

duan-xiande closed this Aug 13, 2019

Fix timestamps between contiguous mixed audio packets may vary #199

Fix timestamps between contiguous mixed audio packets may vary #199

Uh oh!

Conversation

XueshiQiao commented Jul 24, 2019

Uh oh!

daijh Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XueshiQiao Jul 24, 2019

Choose a reason for hiding this comment

Uh oh!

daijh left a comment

Choose a reason for hiding this comment

Uh oh!

XueshiQiao commented Jul 24, 2019

Uh oh!

daijh commented Jul 24, 2019

Uh oh!

XueshiQiao commented Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

XueshiQiao commented Jul 24, 2019

Uh oh!

XueshiQiao commented Aug 6, 2019

Uh oh!

daijh commented Aug 9, 2019

Uh oh!

XueshiQiao commented Aug 9, 2019

Uh oh!

duan-xiande commented Aug 13, 2019

Uh oh!

XueshiQiao commented Aug 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

daijh Jul 24, 2019 •

edited

Loading

XueshiQiao commented Jul 24, 2019 •

edited

Loading