Merge dev to refactor

.github/CONTRIBUTING.md

@@ -6,7 +6,7 @@ NewPipe contribution guidelines
 ## Crash reporting
 
 Report crashes through the **automated crash report system** of NewPipe.
-This way all the data needed for debugging is included in your bugreport for GitHub.
+This way all the data needed for debugging is included in your bug report for GitHub.
 You'll see *exactly* what is sent, be able to add **your comments**, and then send it.
 
 ## Issue reporting/feature requests

.github/workflows/build-release-apk.yml

@@ -0,0 +1,38 @@
+name: "Build unsigned release APK on master"
+
+on:
+  workflow_dispatch:
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: 'master'
+
+      - uses: actions/setup-java@v4
+        with:
+          distribution: 'temurin'
+          java-version: '21'
+          cache: 'gradle'
+
+      - name: "Build release APK"
+        run: ./gradlew assembleRelease --stacktrace
+
+      - name: "Rename APK"
+        run: |
+          VERSION_NAME="$(jq -r ".elements[0].versionName" "app/build/outputs/apk/release/output-metadata.json")"
+          echo "Version name: $VERSION_NAME" >> "$GITHUB_STEP_SUMMARY"
+          echo '```json' >> "$GITHUB_STEP_SUMMARY"
+          cat "app/build/outputs/apk/release/output-metadata.json" >> "$GITHUB_STEP_SUMMARY"
+          echo >> "$GITHUB_STEP_SUMMARY"
+          echo '```' >> "$GITHUB_STEP_SUMMARY"
+          # assume there is only one APK in that folder
+          mv app/build/outputs/apk/release/*.apk "app/build/outputs/apk/release/NewPipe_v$VERSION_NAME.apk"
+
+      - name: "Upload APK"
+        uses: actions/upload-artifact@v4
+        with:
+          name: app
+          path: app/build/outputs/apk/release/*.apk

README.md

@@ -20,7 +20,7 @@
 <p align="center"><a href="https://newpipe.net">Website</a> &bull; <a href="https://newpipe.net/blog/">Blog</a> &bull; <a href="https://newpipe.net/FAQ/">FAQ</a> &bull; <a href="https://newpipe.net/press/">Press</a></p>
 <hr>
 
-*Read this document in other languages: [Deutsch](doc/README.de.md), [English](README.md), [Español](doc/README.es.md), [Français](doc/README.fr.md), [हिन्दी](doc/README.hi.md), [Italiano](doc/README.it.md), [한국어](doc/README.ko.md), [Português Brasil](doc/README.pt_BR.md), [Polski](doc/README.pl.md), [ਪੰਜਾਬੀ ](doc/README.pa.md), [日本語](doc/README.ja.md), [Română](doc/README.ro.md), [Soomaali](doc/README.so.md), [Türkçe](doc/README.tr.md), [正體中文](doc/README.zh_TW.md), [অসমীয়া](doc/README.asm.md), [Српски](doc/README.sr.md)*
+*Read this document in other languages: [Deutsch](doc/README.de.md), [English](README.md), [Español](doc/README.es.md), [Français](doc/README.fr.md), [हिन्दी](doc/README.hi.md), [Italiano](doc/README.it.md), [한국어](doc/README.ko.md), [Português Brasil](doc/README.pt_BR.md), [Polski](doc/README.pl.md), [ਪੰਜਾਬੀ ](doc/README.pa.md), [日本語](doc/README.ja.md), [Română](doc/README.ro.md), [Soomaali](doc/README.so.md), [Türkçe](doc/README.tr.md), [正體中文](doc/README.zh_TW.md), [অসমীয়া](doc/README.asm.md), [Српски](doc/README.sr.md), [العربية](README.ar.md)* 
 
 > [!warning]
 > <b>THIS APP IS IN BETA, SO YOU MAY ENCOUNTER BUGS. IF YOU DO, OPEN AN ISSUE IN OUR GITHUB REPOSITORY BY FILLING OUT THE ISSUE TEMPLATE.</b>
@@ -96,7 +96,7 @@ Also, since they are free and open source software, neither the app nor the Extr
 ## Installation and updates
 You can install NewPipe using one of the following methods:
  1. Add our custom repo to F-Droid and install it from there. The instructions are here: https://newpipe.net/FAQ/tutorials/install-add-fdroid-repo/
- 2. Download the APK from [GitHub Releases](https://github.com/TeamNewPipe/NewPipe/releases) and install it.
+ 2. Download the APK from [GitHub Releases](https://github.com/TeamNewPipe/NewPipe/releases), [compare the signing key](#apk-info) and install it.
  3. Update via F-Droid. This is the slowest method of getting updates, as F-Droid must recognize changes, build the APK itself, sign it, and then push the update to users.
  4. Build a debug APK yourself. This is the fastest way to get new features on your device, but is much more complicated, so we recommend using one of the other methods.
  5. If you're interested in a specific feature or bugfix provided in a Pull Request in this repo, you can also download its APK from within the PR. Read the PR description for instructions. The great thing about PR-specific APKs is that they're installed side-by-side the official app, so you don't have to worry about losing your data or messing anything up.
@@ -109,7 +109,15 @@ In the meanwhile, if you want to switch sources for some reason (e.g. NewPipe's
 3. Download the APK from the new source and install it
 4. Import the data from step 1 via Settings > Backup and Restore > Import Database
 
-<b>Note: when you're importing a database into the official app, always make sure that it is the one you exported _from_ the official app. If you import a database exported from an APK other than the official app, it may break things. Such an action is unsupported, and you should only do so when you're absolutely certain you know what you're doing.</b>
+> [!Note]
+> When you're importing a database into the official app, always make sure that it is the one you exported _from_ the official app. If you import a database exported from an APK other than the official app, it may break things. Such an action is unsupported, and you should only do so when you're absolutely certain you know what you're doing.
+
+### APK Info
+
+This is the SHA fingerprint of NewPipe's signing key to verify downloaded APKs which are signed by us. The fingerprint is also available on [NewPipe's website](https://newpipe.net#download). This is relevant for method 2.
+```
+CB:84:06:9B:D6:81:16:BA:FA:E5:EE:4E:E5:B0:8A:56:7A:A6:D8:98:40:4E:7C:B1:2F:9E:75:6D:F5:CF:5C:AB
+```
 
 ## Contribution
 Whether you have ideas, translations, design changes, code cleaning, or even major code changes, help is always welcome. The app gets better and better with each contribution, no matter how big or small! If you'd like to get involved, check our [contribution notes](.github/CONTRIBUTING.md).

app/build.gradle

@@ -24,8 +24,15 @@ android {
         resValue "string", "app_name", "NewPipe"
         minSdk 21
         targetSdk 33
-        versionCode 999
-        versionName "0.27.2"
+        if (System.properties.containsKey('versionCodeOverride')) {
+            versionCode System.getProperty('versionCodeOverride') as Integer
+        } else {
+            versionCode 1001
+        }
+        versionName "0.27.4"
+        if (System.properties.containsKey('versionNameSuffix')) {
+            versionNameSuffix System.getProperty('versionNameSuffix')
+        }
 
         testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
 

app/src/main/java/org/schabi/newpipe/DownloaderImpl.java

@@ -29,7 +29,7 @@
 
 public final class DownloaderImpl extends Downloader {
     public static final String USER_AGENT =
-            "Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0";
+            "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0";
     public static final String YOUTUBE_RESTRICTED_MODE_COOKIE_KEY =
             "youtube_restricted_mode_key";
     public static final String YOUTUBE_RESTRICTED_MODE_COOKIE = "PREF=f2=8000000";

app/src/main/java/org/schabi/newpipe/database/playlist/dao/PlaylistStreamDAO.java

@@ -154,6 +154,6 @@ default Flowable<List<PlaylistStreamEntity>> listByService(final int serviceId)
             + " AND :streamUrl = :streamUrl"
 
             + " GROUP BY " + JOIN_PLAYLIST_ID
-            + " ORDER BY " + PLAYLIST_DISPLAY_INDEX)
+            + " ORDER BY " + PLAYLIST_DISPLAY_INDEX + ", " + PLAYLIST_NAME)
     Flowable<List<PlaylistDuplicatesEntry>> getPlaylistDuplicatesMetadata(String streamUrl);
 }

app/src/main/java/org/schabi/newpipe/info_list/dialog/StreamDialogDefaultEntry.java

@@ -114,7 +114,10 @@ public enum StreamDialogDefaultEntry {
     DOWNLOAD(R.string.download, (fragment, item) ->
             fetchStreamInfoAndSaveToDatabase(fragment.requireContext(), item.getServiceId(),
                     item.getUrl(), info -> {
-                        if (fragment.getContext() != null) {
+                        // Ensure the fragment is attached and its state hasn't been saved to avoid
+                        // showing dialog during lifecycle changes or when the activity is paused,
+                        // e.g. by selecting the download option and opening a different fragment.
+                        if (fragment.isAdded() && !fragment.isStateSaved()) {
                             final DownloadDialog downloadDialog =
                                     new DownloadDialog(fragment.requireContext(), info);
                             downloadDialog.show(fragment.getChildFragmentManager(),

app/src/main/java/org/schabi/newpipe/ktx/SharedPreferences.kt

@@ -0,0 +1,7 @@
+package org.schabi.newpipe.ktx
+
+import android.content.SharedPreferences
+
+fun SharedPreferences.getStringSafe(key: String, defValue: String): String {
+    return getString(key, null) ?: defValue
+}

app/src/main/java/org/schabi/newpipe/local/dialog/PlaylistAppendDialog.java

@@ -160,7 +160,8 @@ private void onPlaylistSelected(@NonNull final LocalPlaylistManager manager,
                 .subscribe(ignored -> {
                     successToast.show();
 
-                    if (playlist.thumbnailUrl.equals(PlaylistEntity.DEFAULT_THUMBNAIL)) {
+                    if (playlist.thumbnailUrl != null
+                            && playlist.thumbnailUrl.equals(PlaylistEntity.DEFAULT_THUMBNAIL)) {
                         playlistDisposables.add(manager
                                 .changePlaylistThumbnail(playlist.getUid(), streams.get(0).getUid(),
                                         false)

app/src/main/java/org/schabi/newpipe/local/feed/service/FeedLoadManager.kt

@@ -17,8 +17,10 @@ import org.schabi.newpipe.database.subscription.NotificationMode
 import org.schabi.newpipe.database.subscription.SubscriptionEntity
 import org.schabi.newpipe.extractor.Info
 import org.schabi.newpipe.extractor.NewPipe
+import org.schabi.newpipe.extractor.ServiceList
 import org.schabi.newpipe.extractor.feed.FeedInfo
 import org.schabi.newpipe.extractor.stream.StreamInfoItem
+import org.schabi.newpipe.ktx.getStringSafe
 import org.schabi.newpipe.local.feed.FeedDatabaseManager
 import org.schabi.newpipe.local.subscription.SubscriptionManager
 import org.schabi.newpipe.util.ChannelTabHelper
@@ -69,12 +71,10 @@ class FeedLoadManager(private val context: Context) {
         val outdatedThreshold = if (ignoreOutdatedThreshold) {
             OffsetDateTime.now(ZoneOffset.UTC)
         } else {
-            val thresholdOutdatedSeconds = (
-                defaultSharedPreferences.getString(
-                    context.getString(R.string.feed_update_threshold_key),
-                    context.getString(R.string.feed_update_threshold_default_value)
-                ) ?: context.getString(R.string.feed_update_threshold_default_value)
-                ).toInt()
+            val thresholdOutdatedSeconds = defaultSharedPreferences.getStringSafe(
+                context.getString(R.string.feed_update_threshold_key),
+                context.getString(R.string.feed_update_threshold_default_value)
+            ).toInt()
             OffsetDateTime.now(ZoneOffset.UTC).minusSeconds(thresholdOutdatedSeconds.toLong())
         }
 
@@ -91,6 +91,10 @@ class FeedLoadManager(private val context: Context) {
             else -> feedDatabaseManager.outdatedSubscriptionsForGroup(groupId, outdatedThreshold)
         }
 
+        // like `currentProgress`, but counts the number of YouTube extractions that have begun, so
+        // they can be properly throttled every once in a while (see doOnNext below)
+        val youtubeExtractionCount = AtomicInteger()
+
         return outdatedSubscriptions
             .take(1)
             .doOnNext {
@@ -106,6 +110,15 @@ class FeedLoadManager(private val context: Context) {
             .observeOn(Schedulers.io())
             .flatMap { Flowable.fromIterable(it) }
             .takeWhile { !cancelSignal.get() }
+            .doOnNext { subscriptionEntity ->
+                // throttle YouTube extractions once every BATCH_SIZE to avoid being rate limited
+                if (subscriptionEntity.serviceId == ServiceList.YouTube.serviceId) {
+                    val previousCount = youtubeExtractionCount.getAndIncrement()
+                    if (previousCount != 0 && previousCount % BATCH_SIZE == 0) {
+                        Thread.sleep(DELAY_BETWEEN_BATCHES_MILLIS.random())
+                    }
+                }
+            }
             .parallel(PARALLEL_EXTRACTIONS, PARALLEL_EXTRACTIONS * 2)
             .runOn(Schedulers.io(), PARALLEL_EXTRACTIONS * 2)
             .filter { !cancelSignal.get() }
@@ -329,7 +342,19 @@ class FeedLoadManager(private val context: Context) {
         /**
          * How many extractions will be running in parallel.
          */
-        private const val PARALLEL_EXTRACTIONS = 6
+        private const val PARALLEL_EXTRACTIONS = 3
+
+        /**
+         * How many YouTube extractions to perform before waiting [DELAY_BETWEEN_BATCHES_MILLIS]
+         * to avoid being rate limited
+         */
+        private const val BATCH_SIZE = 50
+
+        /**
+         * Wait a random delay in this range once every [BATCH_SIZE] YouTube extractions to avoid
+         * being rate limited
+         */
+        private val DELAY_BETWEEN_BATCHES_MILLIS = (6000L..12000L)
 
         /**
          * Number of items to buffer to mass-insert in the database.

app/src/main/java/org/schabi/newpipe/local/history/StatisticsPlaylistFragment.java

@@ -332,10 +332,6 @@ private void showInfoItemDialog(final StreamStatisticsEntry item) {
                             StreamDialogDefaultEntry.DELETE,
                             (f, i) -> deleteEntry(
                                     Math.max(itemListAdapter.getItemsList().indexOf(item), 0)))
-                    .setAction(
-                            StreamDialogDefaultEntry.START_HERE_ON_BACKGROUND,
-                            (f, i) -> NavigationHelper.playOnBackgroundPlayer(
-                                    context, getPlayQueueStartingAt(item), true))
                     .create()
                     .show();
         } catch (final IllegalArgumentException e) {

app/src/main/java/org/schabi/newpipe/local/subscription/services/SubscriptionsExportService.java

@@ -76,7 +76,10 @@ public int onStartCommand(final Intent intent, final int flags, final int startI
 
         try {
             outFile = new StoredFileHelper(this, path, "application/json");
-            outputStream = new SharpOutputStream(outFile.getStream());
+            // truncate the file before writing to it, otherwise if the new content is smaller than
+            // the previous file size, the file will retain part of the previous content and be
+            // corrupted
+            outputStream = new SharpOutputStream(outFile.openAndTruncateStream());
         } catch (final IOException e) {
             handleError(e);
             return START_NOT_STICKY;

app/src/main/java/org/schabi/newpipe/player/Player.java

@@ -454,7 +454,6 @@ public void handleIntent(@NonNull final Intent intent) {
         if (oldPlayerType != playerType && playQueue != null) {
             // If playerType changes from one to another we should reload the player
             // (to disable/enable video stream or to set quality)
-            setRecovery();
             reloadPlayQueueManager();
         }
 

app/src/main/java/org/schabi/newpipe/player/seekbarpreview/SeekbarPreviewThumbnailHolder.java

@@ -133,17 +133,8 @@ private void generateDataFrom(final Frameset frameset, final UUID updateRequestI
 
                 // Get the bounds where the frame is found
                 final int[] bounds = frameset.getFrameBoundsAt(currentPosMs);
-                generatedDataForUrl.put(currentPosMs, () -> {
-                    // It can happen, that the original bitmap could not be downloaded
-                    // In such a case - we don't want a NullPointer - simply return null
-                    if (srcBitMap == null) {
-                        return null;
-                    }
-
-                    // Cut out the corresponding bitmap form the "srcBitMap"
-                    return Bitmap.createBitmap(srcBitMap, bounds[1], bounds[2],
-                            frameset.getFrameWidth(), frameset.getFrameHeight());
-                });
+                generatedDataForUrl.put(currentPosMs,
+                                        createBitmapSupplier(srcBitMap, bounds, frameset));
 
                 currentPosMs += frameset.getDurationPerFrame();
                 pos++;
@@ -166,6 +157,45 @@ private void generateDataFrom(final Frameset frameset, final UUID updateRequestI
         }
     }
 
+    private Supplier<Bitmap> createBitmapSupplier(final Bitmap srcBitMap,
+                                                  final int[] bounds,
+                                                  final Frameset frameset) {
+        return () -> {
+            // It can happen, that the original bitmap could not be downloaded
+            // (or it was recycled though that should not happen)
+            // In such a case - we don't want a NullPointer/
+            // "cannot use a recycled source in createBitmap" Exception -> simply return null
+            if (srcBitMap == null || srcBitMap.isRecycled()) {
+                return null;
+            }
+
+            // Under some rare circumstances the YouTube API returns slightly too small storyboards,
+            // (or not the matching frame width/height)
+            // This would lead to createBitmap cutting out a bitmap that is out of bounds,
+            // so we need to adjust the bounds accordingly
+            if (srcBitMap.getWidth() < bounds[1] + frameset.getFrameWidth()) {
+                bounds[1] = srcBitMap.getWidth() - frameset.getFrameWidth();
+            }
+
+            if (srcBitMap.getHeight() < bounds[2] + frameset.getFrameHeight()) {
+                bounds[2] = srcBitMap.getHeight() - frameset.getFrameHeight();
+            }
+
+            // Cut out the corresponding bitmap form the "srcBitMap"
+            final Bitmap cutOutBitmap = Bitmap.createBitmap(srcBitMap, bounds[1], bounds[2],
+                    frameset.getFrameWidth(), frameset.getFrameHeight());
+
+            // If the cut out bitmap is identical to its source,
+            // we need to copy the bitmap to create a new instance.
+            // createBitmap allows itself to return the original object that is was created with
+            // this leads to recycled bitmaps being returned (if they are identical)
+            // Reference: https://stackoverflow.com/a/23683075 + first comment
+            // Fixes: https://github.com/TeamNewPipe/NewPipe/issues/11461
+            return cutOutBitmap == srcBitMap
+                    ? cutOutBitmap.copy(cutOutBitmap.getConfig(), true) : cutOutBitmap;
+        };
+    }
+
     @Nullable
     private Bitmap getBitMapFrom(final String url) {
         if (url == null) {