Skip to content

Commit

Permalink
fix: handle rollover and don't set wrong timing info for segments wit…
Browse files Browse the repository at this point in the history 
…h high PTS/DTS values (#1040)
  • Loading branch information
@gesinger
gesinger authored Jan 6, 2021
1 parent 6c337e1 commit 9919b85
Show file tree
Hide file tree
Showing 21 changed files with 503 additions and 42 deletions.
77 changes: 77 additions & 0 deletions docs/creating-content.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,83 @@ Copy only the first two audio frames, leave out video.
$ ffmpeg -i index0.ts -aframes 2 -vn -acodec copy audio.ts
```

### videoMinOffset.ts

video.ts but with an offset of 0

```
$ ffmpeg -i video.ts -muxpreload 0 -muxdelay 0 -vcodec copy videoMinOffset.ts
```

### audioMinOffset.ts

audio.ts but with an offset of 0. Note that muxed.ts is used because ffmpeg didn't like
the use of audio.ts

```
$ ffmpeg -i muxed.ts -muxpreload 0 -muxdelay 0 -acodec copy -vn audioMinOffset.ts
```

### videoMaxOffset.ts

This segment offsets content such that it ends at exactly the max timestamp before a rollover occurs. It uses the max timestamp of 2^33 (8589934592) minus the segment duration of 6006 (0.066733 seconds) in order to not rollover mid segment, and divides the value by 90,000 to convert it from media time to seconds.

(2^33 - 6006) / 90,000 = 95443.6509556

```
$ ffmpeg -i videoMinOffset.ts -muxdelay 95443.6509556 -muxpreload 95443.6509556 -output_ts_offset 95443.6509556 -vcodec copy videoMaxOffset.ts
```

### audioMaxOffset.ts

This segment offsets content such that it ends at exactly the max timestamp before a rollover occurs. It uses the max timestamp of 2^33 (8589934592) minus the segment duration of 11520 (0.128000 seconds) in order to not rollover mid segment, and divides the value by 90,000 to convert it from media time to seconds.

(2^33 - 11520) / 90,000 = 95443.5896889

```
$ ffmpeg -i audioMinOffset.ts -muxdelay 95443.5896889 -muxpreload 95443.5896889 -output_ts_offset 95443.5896889 -acodec copy audioMaxOffset.ts
```

### videoLargeOffset.ts

This segment offsets content by the rollover threshhold of 2^32 (4294967296) found in the rollover handling of mux.js, adds 1 to ensure there aren't any cases where there's an equal match, then divides the value by 90,000 to convert it from media time to seconds.

(2^32 + 1) / 90,000 = 47721.8588556

```
$ ffmpeg -i videoMinOffset.ts -muxdelay 47721.8588556 -muxpreload 47721.8588556 -output_ts_offset 47721.8588556 -vcodec copy videoLargeOffset.ts
```

### audioLargeOffset.ts

This segment offsets content by the rollover threshhold of 2^32 (4294967296) found in the rollover handling of mux.js, adds 1 to ensure there aren't any cases where there's an equal match, then divides the value by 90,000 to convert it from media time to seconds.

(2^32 + 1) / 90,000 = 47721.8588556

```
$ ffmpeg -i audioMinOffset.ts -muxdelay 47721.8588556 -muxpreload 47721.8588556 -output_ts_offset 47721.8588556 -acodec copy audioLargeOffset.ts
```

### videoLargeOffset2.ts

This takes videoLargeOffset.ts and adds the duration of videoLargeOffset.ts (6006 / 90,000 = 0.066733 seconds) to its offset so that this segment can act as the second in one continuous stream.

47721.8588556 + 0.066733 = 47721.9255886

```
$ ffmpeg -i videoLargeOffset.ts -muxdelay 47721.9255886 -muxpreload 47721.9255886 -output_ts_offset 47721.9255886 -vcodec copy videoLargeOffset2.ts
```

### audioLargeOffset2.ts

This takes audioLargeOffset.ts and adds the duration of audioLargeOffset.ts (11520 / 90,000 = 0.128 seconds) to its offset so that this segment can act as the second in one continuous stream.

47721.8588556 + 0.128 = 47721.9868556

```
$ ffmpeg -i audioLargeOffset.ts -muxdelay 47721.9868556 -muxpreload 47721.9868556 -output_ts_offset 47721.9868556 -acodec copy audioLargeOffset2.ts
```

### caption.ts

Copy the first two frames of video out of a ts segment that already includes CEA-608 captions.
Expand Down
6 changes: 3 additions & 3 deletions package-lock.json
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
"global": "^4.4.0",
"m3u8-parser": "4.5.0",
"mpd-parser": "0.15.0",
"mux.js": "5.7.0",
"mux.js": "5.8.0",
"video.js": "^6 || ^7"
},
"devDependencies": {
Expand Down
43 changes: 43 additions & 0 deletions src/media-segment-request.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,7 @@ const transmuxAndNotify = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -350,6 +351,9 @@ const transmuxAndNotify = ({
onVideoSegmentTimingInfo: (videoSegmentTimingInfo) => {
videoSegmentTimingInfoFn(videoSegmentTimingInfo);
},
onAudioSegmentTimingInfo: (audioSegmentTimingInfo) => {
audioSegmentTimingInfoFn(audioSegmentTimingInfo);
},
onId3: (id3Frames, dispatchType) => {
id3Fn(segment, id3Frames, dispatchType);
},
Expand All @@ -375,6 +379,7 @@ const handleSegmentBytes = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -505,6 +510,7 @@ const handleSegmentBytes = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand All @@ -520,6 +526,13 @@ const handleSegmentBytes = ({
* @param {Object} segment - a simplified copy of the segmentInfo object
* from SegmentLoader
* @param {Function} trackInfoFn - a callback that receives track info
* @param {Function} timingInfoFn - a callback that receives timing info
* @param {Function} videoSegmentTimingInfoFn
* a callback that receives video timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} audioSegmentTimingInfoFn
* a callback that receives audio timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} dataFn - a callback that is executed when segment bytes are available
* and ready to use
* @param {Function} doneFn - a callback that is executed after decryption has completed
Expand All @@ -530,6 +543,7 @@ const decryptSegment = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand All @@ -553,6 +567,7 @@ const decryptSegment = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -595,6 +610,12 @@ const decryptSegment = ({
* routines
* @param {Function} trackInfoFn - a callback that receives track info
* @param {Function} timingInfoFn - a callback that receives timing info
* @param {Function} videoSegmentTimingInfoFn
* a callback that receives video timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} audioSegmentTimingInfoFn
* a callback that receives audio timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} id3Fn - a callback that receives ID3 metadata
* @param {Function} captionsFn - a callback that receives captions
* @param {Function} dataFn - a callback that is executed when segment bytes are available
Expand All @@ -608,6 +629,7 @@ const waitForCompletion = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -653,6 +675,7 @@ const waitForCompletion = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand All @@ -667,6 +690,7 @@ const waitForCompletion = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -701,6 +725,13 @@ const handleLoadEnd = ({ loadendState, abortFn }) => (event) => {
* @param {Function} progressFn - a callback that is executed each time a progress event
* is received
* @param {Function} trackInfoFn - a callback that receives track info
* @param {Function} timingInfoFn - a callback that receives timing info
* @param {Function} videoSegmentTimingInfoFn
* a callback that receives video timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} audioSegmentTimingInfoFn
* a callback that receives audio timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} dataFn - a callback that is executed when segment bytes are available
* and ready to use
* @param {Event} event - the progress event object from XMLHttpRequest
Expand All @@ -711,6 +742,7 @@ const handleProgress = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -747,6 +779,7 @@ const handleProgress = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn
Expand Down Expand Up @@ -811,6 +844,13 @@ const handleProgress = ({
* @param {Function} progressFn - a callback that receives progress events from the main
* segment's xhr request
* @param {Function} trackInfoFn - a callback that receives track info
* @param {Function} timingInfoFn - a callback that receives timing info
* @param {Function} videoSegmentTimingInfoFn
* a callback that receives video timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} audioSegmentTimingInfoFn
* a callback that receives audio timing info based on media times and
* any adjustments made by the transmuxer
* @param {Function} id3Fn - a callback that receives ID3 metadata
* @param {Function} captionsFn - a callback that receives captions
* @param {Function} dataFn - a callback that receives data from the main segment's xhr
Expand All @@ -830,6 +870,7 @@ export const mediaSegmentRequest = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand All @@ -843,6 +884,7 @@ export const mediaSegmentRequest = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down Expand Up @@ -909,6 +951,7 @@ export const mediaSegmentRequest = ({
trackInfoFn,
timingInfoFn,
videoSegmentTimingInfoFn,
audioSegmentTimingInfoFn,
id3Fn,
captionsFn,
dataFn,
Expand Down
60 changes: 39 additions & 21 deletions src/segment-loader.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2067,26 +2067,31 @@ export default class SegmentLoader extends videojs.EventTarget {
});
}

handleVideoSegmentTimingInfo_(requestId, videoSegmentTimingInfo) {
handleSegmentTimingInfo_(type, requestId, segmentTimingInfo) {
if (!this.pendingSegment_ || requestId !== this.pendingSegment_.requestId) {
return;
}

const segment = this.pendingSegment_.segment;

if (!segment.videoTimingInfo) {
segment.videoTimingInfo = {};
}

segment.videoTimingInfo.transmuxerPrependedSeconds =
videoSegmentTimingInfo.prependedContentDuration || 0;
segment.videoTimingInfo.transmuxedPresentationStart =
videoSegmentTimingInfo.start.presentation;
segment.videoTimingInfo.transmuxedPresentationEnd =
videoSegmentTimingInfo.end.presentation;
const timingInfoProperty = `${type}TimingInfo`;

if (!segment[timingInfoProperty]) {
segment[timingInfoProperty] = {};
}

segment[timingInfoProperty].transmuxerPrependedSeconds =
segmentTimingInfo.prependedContentDuration || 0;
segment[timingInfoProperty].transmuxedPresentationStart =
segmentTimingInfo.start.presentation;
segment[timingInfoProperty].transmuxedDecodeStart =
segmentTimingInfo.start.decode;
segment[timingInfoProperty].transmuxedPresentationEnd =
segmentTimingInfo.end.presentation;
segment[timingInfoProperty].transmuxedDecodeEnd =
segmentTimingInfo.end.decode;
// mainly used as a reference for debugging
segment.videoTimingInfo.baseMediaDecodeTime =
videoSegmentTimingInfo.baseMediaDecodeTime;
segment[timingInfoProperty].baseMediaDecodeTime =
segmentTimingInfo.baseMediaDecodeTime;
}

appendData_(segmentInfo, result) {
Expand Down Expand Up @@ -2209,7 +2214,8 @@ export default class SegmentLoader extends videojs.EventTarget {
progressFn: this.handleProgress_.bind(this),
trackInfoFn: this.handleTrackInfo_.bind(this),
timingInfoFn: this.handleTimingInfo_.bind(this),
videoSegmentTimingInfoFn: this.handleVideoSegmentTimingInfo_.bind(this, segmentInfo.requestId),
videoSegmentTimingInfoFn: this.handleSegmentTimingInfo_.bind(this, 'video', segmentInfo.requestId),
audioSegmentTimingInfoFn: this.handleSegmentTimingInfo_.bind(this, 'audio', segmentInfo.requestId),
captionsFn: this.handleCaptions_.bind(this),
id3Fn: this.handleId3_.bind(this),

Expand Down Expand Up @@ -2264,12 +2270,24 @@ export default class SegmentLoader extends videojs.EventTarget {
gopsToAlignWith: segmentInfo.gopsToAlignWith
};

const previousSegment = segmentInfo.playlist.segments[segmentInfo.mediaIndex];

if (previousSegment &&
previousSegment.end &&
previousSegment.timeline === segment.timeline) {
simpleSegment.baseStartTime = previousSegment.end + segmentInfo.timestampOffset;
const previousSegment = segmentInfo.playlist.segments[segmentInfo.mediaIndex - 1];

if (previousSegment && previousSegment.timeline === segment.timeline) {
// The baseStartTime of a segment is used to handle rollover when probing the TS
// segment to retrieve timing information. Since the probe only looks at the media's
// times (e.g., PTS and DTS values of the segment), and doesn't consider the
// player's time (e.g., player.currentTime()), baseStartTime should reflect the
// media time as well. transmuxedDecodeEnd represents the end time of a segment, in
// seconds of media time, so should be used here. The previous segment is used since
// the end of the previous segment should represent the beginning of the current
// segment, so long as they are on the same timeline.
if (previousSegment.videoTimingInfo) {
simpleSegment.baseStartTime =
previousSegment.videoTimingInfo.transmuxedDecodeEnd;
} else if (previousSegment.audioTimingInfo) {
simpleSegment.baseStartTime =
previousSegment.audioTimingInfo.transmuxedDecodeEnd;
}
}

if (segment.key) {
Expand Down
4 changes: 4 additions & 0 deletions src/segment-transmuxer.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ export const processTransmux = ({
onAudioTimingInfo,
onVideoTimingInfo,
onVideoSegmentTimingInfo,
onAudioSegmentTimingInfo,
onId3,
onCaptions,
onDone
Expand Down Expand Up @@ -111,6 +112,9 @@ export const processTransmux = ({
if (event.data.action === 'videoSegmentTimingInfo') {
onVideoSegmentTimingInfo(event.data.videoSegmentTimingInfo);
}
if (event.data.action === 'audioSegmentTimingInfo') {
onAudioSegmentTimingInfo(event.data.audioSegmentTimingInfo);
}
if (event.data.action === 'id3Frame') {
onId3([event.data.id3Frame], event.data.id3Frame.dispatchType);
}
Expand Down
Loading

0 comments on commit 9919b85

Please sign in to comment.