Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get available subtitles text from subtitle track #65

Open
basemosama opened this issue Jan 22, 2024 · 29 comments
Open

How to get available subtitles text from subtitle track #65

basemosama opened this issue Jan 22, 2024 · 29 comments
Assignees
Labels
enhancement New feature or request

Comments

@basemosama
Copy link

I have a video with some subtitle tracks and i want to allow the user to choose the subtitle track from the video and display it.

How can I have the current available subtitle text or all text of the current selected subtitle track?

And Can we render it automatically using fvp?

Here is some videos with a subtitle track
https://storage.googleapis.com/exoplayer-test-media-1/mp4/dizzy-with-tx3g.mp4
http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8

For example in Exoplayer there's oncues function that return current text to display based on current playback time

public void onCues(@NonNull CueGroup cueGroup) {
   final Cue cue = cueGroup.cues.get(0);
   final text = cue.text.toString() ;
    }

Thanks

@wang-bin
Copy link
Owner

wang-bin commented Jan 23, 2024

subtitle rendering requires libass, it's already supported on windows, linux, iOS and macOS. I haven't tested on android.

@wang-bin
Copy link
Owner

run dart pub cache clean and build again, then libass will be added for android. for iOS, you have to download https://sourceforge.net/projects/mdk-sdk/files/deps/dep.7z/download and add ass.framework manually in xcode project. for linux, installing system libass is enough.

@basemosama
Copy link
Author

Currently I am focusing in Android.

Can I have access to the selected subtitle track text?
And Is there any configuration to render subtitle?

@wang-bin
Copy link
Owner

Currently I am focusing in Android.

Can I have access to the selected subtitle track text?

No. which api are you using? video_player via fvp.dart or Player via mdk.dart?

And Is there any configuration to render subtitle?

No configuration is required.

@basemosama
Copy link
Author

Player via mdk.dart

@basemosama
Copy link
Author

When trying the latest version of fvp, The app doesn't open and I get this error when using the app.

[ +775 ms] E/flutter (12374): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Invalid argument(s): Failed to load dynamic library 'libfvp_plugin.so': dlopen failed: library "libass.so" not found
[        ] E/flutter (12374): #0      _open (dart:ffi-patch/ffi_dynamic_library_patch.dart:11:43)
[        ] E/flutter (12374): #1      new DynamicLibrary.open (dart:ffi-patch/ffi_dynamic_library_patch.dart:22:12)
[        ] E/flutter (12374): #2      Libfvp._load (package:fvp/src/lib.dart:52:29)
[        ] E/flutter (12374): #3      Libfvp.instance (package:fvp/src/lib.dart:58:27)
[        ] E/flutter (12374): #4      Libfvp.instance (package:fvp/src/lib.dart)
[        ] E/flutter (12374): #5      Libfvp.isEmulator (package:fvp/src/lib.dart:80:29)
[        ] E/flutter (12374): #6      Libfvp.isEmulator (package:fvp/src/lib.dart)
[        ] E/flutter (12374): #7      PlatformEx.isAndroidEmulator (package:fvp/src/extensions.dart:16:19)
[        ] E/flutter (12374): #8      MdkVideoPlayerPlatform.registerVideoPlayerPlatformsWith (package:fvp/src/video_player_mdk.dart:138:42)
[        ] E/flutter (12374): #9      registerWith (package:fvp/fvp.dart:39:26)
[        ] E/flutter (12374): #10     main (package:untitled6/main.dart:15:7)
[        ] E/flutter (12374): #11     _runMain.<anonymous closure> (dart:ui/hooks.dart:301:23)
[        ] E/flutter (12374): #12     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:297:19)
[        ] E/flutter (12374): #13     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:184:12)
[        ] E/flutter (12374): 

@wang-bin
Copy link
Owner

can you upload your apk? no error in my tests

@basemosama
Copy link
Author

basemosama commented Jan 24, 2024

I'm testing on the default example app.

Steps to reproduce:

  • clean cache using dart pub cache clean
  • delete pubspec.lock
  • run flutter clean
  • run flutter pub get and run the app.

There is an issue when in the latest mdk sdk version that is downloaded to the cache from here
https://sourceforge.net/projects/mdk-sdk/files/nightly/mdk-sdk-android.7z

Here is the code

import 'package:flutter/material.dart';
import 'package:fvp/fvp.dart' as fvp;
import 'package:video_player/video_player.dart';

void main() {
  fvp.registerWith();

  runApp(
    MaterialApp(
      home: _App(),
    ),
  );
}

class _App extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      key: const ValueKey<String>('home_page'),
      appBar: AppBar(
        title: const Text('Video player example'),
      ),
      body: _BumbleBeeRemoteVideo(),
    );
  }
}

class _BumbleBeeRemoteVideo extends StatefulWidget {
  @override
  _BumbleBeeRemoteVideoState createState() => _BumbleBeeRemoteVideoState();
}

class _BumbleBeeRemoteVideoState extends State<_BumbleBeeRemoteVideo> {
  late VideoPlayerController _controller;

  Future<ClosedCaptionFile> _loadCaptions() async {
    final String fileContents = await DefaultAssetBundle.of(context)
        .loadString('assets/bumble_bee_captions.vtt');
    return WebVTTCaptionFile(
        fileContents); // For vtt files, use WebVTTCaptionFile
  }

  String subtitle = "Subtitle";

  @override
  void initState() {
    super.initState();

    _controller = VideoPlayerController.networkUrl(
      Uri.parse(
          'http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8'),
    );
    _controller.setLooping(true);
    _controller.initialize();
  }

  @override
  void dispose() {
    _controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return SingleChildScrollView(
      child: Column(
        children: <Widget>[
          Container(padding: const EdgeInsets.only(top: 20.0)),
          const Text('With remote mp4'),
          Container(
            padding: const EdgeInsets.all(20),
            child: AspectRatio(
              aspectRatio: _controller.value.aspectRatio,
              child: Stack(
                alignment: Alignment.bottomCenter,
                children: <Widget>[
                  VideoPlayer(_controller),
                  ClosedCaption(text: subtitle),
                  _ControlsOverlay(controller: _controller),
                  VideoProgressIndicator(_controller, allowScrubbing: true),
                ],
              ),
            ),
          ),
        ],
      ),
    );
  }
}

class _ControlsOverlay extends StatelessWidget {
  const _ControlsOverlay({required this.controller});

  static const List<Duration> _exampleCaptionOffsets = <Duration>[
    Duration(seconds: -10),
    Duration(seconds: -3),
    Duration(seconds: -1, milliseconds: -500),
    Duration(milliseconds: -250),
    Duration.zero,
    Duration(milliseconds: 250),
    Duration(seconds: 1, milliseconds: 500),
    Duration(seconds: 3),
    Duration(seconds: 10),
  ];
  static const List<double> _examplePlaybackRates = <double>[
    0.25,
    0.5,
    1.0,
    1.5,
    2.0,
    3.0,
    5.0,
    10.0,
  ];

  final VideoPlayerController controller;

  @override
  Widget build(BuildContext context) {
    return Stack(
      children: <Widget>[
        AnimatedSwitcher(
          duration: const Duration(milliseconds: 50),
          reverseDuration: const Duration(milliseconds: 200),
          child: controller.value.isPlaying
              ? const SizedBox.shrink()
              : Container(
                  color: Colors.black26,
                  child: const Center(
                    child: Icon(
                      Icons.play_arrow,
                      color: Colors.white,
                      size: 100.0,
                      semanticLabel: 'Play',
                    ),
                  ),
                ),
        ),
        GestureDetector(
          onTap: () {
            controller.value.isPlaying ? controller.pause() : controller.play();
          },
        ),
        Align(
          alignment: Alignment.topLeft,
          child: PopupMenuButton<Duration>(
            initialValue: controller.value.captionOffset,
            tooltip: 'Caption Offset',
            onSelected: (Duration delay) {
              controller.setCaptionOffset(delay);
            },
            itemBuilder: (BuildContext context) {
              return <PopupMenuItem<Duration>>[
                for (final Duration offsetDuration in _exampleCaptionOffsets)
                  PopupMenuItem<Duration>(
                    value: offsetDuration,
                    child: Text('${offsetDuration.inMilliseconds}ms'),
                  )
              ];
            },
            child: Padding(
              padding: const EdgeInsets.symmetric(
                // Using less vertical padding as the text is also longer
                // horizontally, so it feels like it would need more spacing
                // horizontally (matching the aspect ratio of the video).
                vertical: 12,
                horizontal: 16,
              ),
              child: Text('${controller.value.captionOffset.inMilliseconds}ms'),
            ),
          ),
        ),
        Align(
          alignment: Alignment.topRight,
          child: PopupMenuButton<double>(
            initialValue: controller.value.playbackSpeed,
            tooltip: 'Playback speed',
            onSelected: (double speed) {
              controller.setPlaybackSpeed(speed);
            },
            itemBuilder: (BuildContext context) {
              return <PopupMenuItem<double>>[
                for (final double speed in _examplePlaybackRates)
                  PopupMenuItem<double>(
                    value: speed,
                    child: Text('${speed}x'),
                  )
              ];
            },
            child: Padding(
              padding: const EdgeInsets.symmetric(
                // Using less vertical padding as the text is also longer
                // horizontally, so it feels like it would need more spacing
                // horizontally (matching the aspect ratio of the video).
                vertical: 12,
                horizontal: 16,
              ),
              child: Text('${controller.value.playbackSpeed}x'),
            ),
          ),
        ),
      ],
    );
  }
}

@wang-bin
Copy link
Owner

i need your apk because i can't reproduce your error

@basemosama
Copy link
Author

basemosama commented Jan 24, 2024

Here is the apk
https://drive.google.com/file/d/1aoEyq15vNhf9xdKjcbefBmqrHi-gx06A/view?usp=sharing

I tested it on Huawei Mate 20

@wang-bin
Copy link
Owner

show me the result of flutter doctor --verbose

github action build is ok too, libass is automatically added in to apk, so i guess the error is from your build environment.

@basemosama
Copy link
Author

[✓] Flutter (Channel stable, 3.16.8, on Microsoft Windows [Version 10.0.22621.3007], locale ar-EG)
    • Flutter version 3.16.8 on channel stable at E:\flutter\flutter
    • Upstream repository https://github.com/flutter/flutter.gitFramework revision 67457e669f (8 days ago), 2024-01-16 16:22:29 -0800Engine revision 6e2ea58a5c
    • Dart version 3.2.5DevTools version 2.28.5

[✓] Windows Version (Installed version of Windows is version 10 or higher)

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at E:\SdkPlatform android-34, build-tools 34.0.0ANDROID_HOME = E:\SdkANDROID_SDK_ROOT = E:\SdkJava binary at: C:\Program Files\Android\Android Studio\jbr\bin\java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[✗] Visual Studio - develop Windows apps
    ✗ Visual Studio not installed; this is necessary to develop Windows apps.
      Download at https://visualstudio.microsoft.com/downloads/.
      Please install the "Desktop development with C++" workload, including all of its default components

[✓] Android Studio (version 2022.3)
    • Android Studio at C:\Program Files\Android\Android StudioFlutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutterDart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dartJava version OpenJDK Runtime Environment (build 17.0.6+0-b2043.56-10027231)

[✓] VS Code, 64-bit edition (version 1.79.2)
    • VS Code at C:\Program Files\Microsoft VS CodeFlutter extension can be installed from:
      🔨 https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[✓] Connected device (4 available)
    • LYA L29 (mobile)  • LHS7N19216018187 • android-arm64  • Android 10 (API 29)
    • Windows (desktop) • windows          • windows-x64    • Microsoft Windows [Version 10.0.22621.3007]
    • Chrome (web)      • chrome           • web-javascript • Google Chrome 120.0.6099.225Edge (web)        • edge             • web-javascript • Microsoft Edge 120.0.2210.144

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.
The Flutter CLI developer tool uses Google Analytics to report usage and diagnostic
data along with package dependencies, and crash reporting to send basic crash
reports. This data is used to help improve the Dart platform, Flutter framework,
and related tools.

Telemetry is not sent on the very first run. To disable reporting of telemetry,
run this terminal command:

    flutter --disable-analytics

If you opt out of telemetry, an opt-out event will be sent, and then no further
information will be sent. This data is collected in accordance with the Google
Privacy Policy (https://policies.google.com/privacy).

Please note that analytics reporting was already disabled, and will continue to be disabled.

@wang-bin
Copy link
Owner

The difference is I use macos and ubuntu to build android apps. Please open fvp project in vscode, and run task(ctrl + p, then type task and a space), select Build APK to build the example, copy all the terminal output and paste here.

@basemosama
Copy link
Author

basemosama commented Jan 24, 2024

Can you try rerun the apk github action build again is it was built 3 days ago as the issue has something to do with the nightly mdk- sdk here https://sourceforge.net/projects/mdk-sdk/files/nightly/mdk-sdk-android.7z that was update recently.

I will try to build the apk and send the ouptut

@wang-bin
Copy link
Owner

i ran the jobs again a few hours ago.

@basemosama
Copy link
Author

Yes, The action apk is working, but when building on my side it's not working.
Here is the log from vscode
https://drive.google.com/file/d/1fQc8wcZXBkStRePHF24QmRtS4Bfy0gAV/view?usp=sharing

@wang-bin
Copy link
Owner

I add android on windows in github actions, it works as expected. From your build log a can't see libass.so, it's weired. libass.so is processed exactly the same as libffmpeg.so, I don't know why no libass.so in your build. Other differences are:

  • ndk version, mime is ndk 25 or later
  • you use fvp 0.13.1, what about building fvp/example directly(open fvp project in vscode, run Build APK task)?

@wang-bin
Copy link
Owner

I used a cmake feature not supported in old versions. run flutter clean and build again should fix the issue

@basemosama
Copy link
Author

Yes, It's working again. Thanks.
But I still can't manage to render the subtitle track.

@wang-bin
Copy link
Owner

https://storage.googleapis.com/exoplayer-test-media-1/mp4/dizzy-with-tx3g.mp4 this subtitle requires a font does not exist on android. I see some other players(e.g. mpv-android) provides a font file, it's about 7M. I have to find out a way to optionally adding the font

@wang-bin
Copy link
Owner

http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8 hls subtitle is disabled by ffmpeg by default. There is an option to enable it.

@wang-bin
Copy link
Owner

Try the master branch code, add assets/subfont.ttf in pubspec.yaml, then subtitle can be renderer required font is not found in the system

@basemosama
Copy link
Author

Thanks, It's working now.

I have some questions.
How can we enable subs for hls?
And Can we configure the subtitle style like font, font size and color?
And for the future It will be nice to have an event for subtitles that shows the current subtitle text need to be displayed.

@wang-bin
Copy link
Owner

I have some questions. How can we enable subs for hls?

You can enable it via player.setProperty('avformat.strict', 'experimental');. But not sure if it works for http://sample.vodobox.com/planete_interdite/planete_interdite_alternate.m3u8 because i fail to download segments

And Can we configure the subtitle style like font, font size and color? And for the future It will be nice to have an event for subtitles that shows the current subtitle text need to be displayed.

Will be supported in a future version.

@basemosama
Copy link
Author

Is teletext subtitles supported? I am trying to test some samples with teletext subtitles but can't manage to show subs.
Here is some samples :
https://mega.nz/folder/dBQSwaJL#MUgvR9DNiqkM59MRCSLhzQ

And some logs related to subtitle

19:21:02.270  I  mdk.INFO: CopyToAppFilesDir: assets://flutter_assets/assets/subfont.ttf => /data/user/0/com.mediadevkit.fvp_example/files/mdk/subfont.ttf
19:21:02.270  I  mdk.INFO: CopyToAppFilesDir: /data/user/0/com.mediadevkit.fvp_example/files/mdk/subfont.ttf already exists
19:21:02.347  I  fvp.FINE: 783605235 player507254545920 onEvent: thread.subtitle 1
19:21:02.347  I  mdk.INFO: subtitle stream#3 starting decoding loop from decoder index 0...
19:21:02.347  I  mdk.INFO: creating subtitle decoder: auto...
19:21:02.348  I  mdk.INFO: opening subtitle decoder: FFmpeg...
19:21:02.349  I  mdk.INFO: subtitle decoder not found for codec: dvb_teletext/force: 
19:21:02.350  I  mdk.WARNING: ERROR! failed to setup decoder: subtitle
19:21:02.350  I  fvp.FINE: 783605235 player507254545920 onEvent: decoder.subtitle -1
19:21:02.350  I  mdk.INFO: 0x76226d5800 1st subtitle frame @0.000000
19:21:02.350  I  mdk.INFO: 0x76226d5800 seek end subtitle frame @0.000000 seek_pos_: -1
19:21:02.350  I  mdk.INFO: EOS subtitle frame of track 0
19:21:02.351  I  mdk.INFO: subtitle stream#3 decoding loop is finished. packets: 8, pts: [34930.641944, 34930.921944]
19:21:02.351  I  fvp.FINE: 783605235 player507254545920 onEvent: thread.subtitle 0

@wang-bin
Copy link
Owner

teletext requires another external library

@basemosama
Copy link
Author

Teletext is the main subtitle in the app I am building.
Can you try to add it to the package?

@wang-bin
Copy link
Owner

teletext requires libzvbi, it's license is GPL, it's not acceptable because the whole app must be open source, is it ok for you? You can build ffmpeg yourself with libzvbi enabled, then replace libffmpeg.so. I can also try to build a new libffmpeg.so for you, but I won't redistribute it with my library.

@basemosama
Copy link
Author

Ok, No problem.
Can you build the libffmpeg.so with libzvbi enabled?

@wang-bin wang-bin self-assigned this Feb 6, 2024
@wang-bin wang-bin added the enhancement New feature or request label Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants