Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Download file > 512MB from SMB external storage not possible (NC 23.0.2) #31308

Closed
5 of 8 tasks
spiezmaestro opened this issue Feb 21, 2022 · 28 comments
Closed
5 of 8 tasks
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug

Comments

@spiezmaestro
Copy link

⚠️ This issue respects the following points: ⚠️

  • This is a bug, not a question or a configuration/webserver/proxy issue.
  • This issue is not already reported on Github (I've searched it).
  • Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
  • I agree to follow Nextcloud's Code of Conduct.

Bug description

When I try to download files larger than 512 MB from SMB/CIFS shares (attached via the "External storage support" Nextcloud), the download fails (downloading starts then stops at 512 MB and finally results in failure).

This is on a NC 23.0.2. / External Storage app version 1.15.0 (latest). When I copy the directory /var/www/html/apps/files_external/3rdparty/icewind from the backup of NC 23.0.0 than it works again.

Therefore there is an issue at least with the combination of the external storage app with the used icewind 3rdparty app.

Steps to reproduce

  1. Upgrade to NC 23.0.2 with external storage app 1.15.0 (most likely fresh install will have the same issue)
  2. Try to download a file larger than 512 MB from a via SMB connected share

Expected behavior

Download without issue

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

No response

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

No response

List of activated Apps

Enabled:
  - accessibility: 1.9.0
  - activity: 2.15.0
  - admin_audit: 1.13.0
  - announcementcenter: 6.1.1
  - apporder: 0.14.0
  - bookmarks: 10.0.3
  - breezedark: 23.2.0
  - bruteforcesettings: 2.3.0
  - calendar: 3.0.6
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contacts: 4.0.8
  - contactsinteraction: 1.4.0
  - cospend: 1.4.3
  - dav: 1.21.0
  - dicomviewer: 1.2.3
  - epubreader: 1.4.7
  - event_update_notification: 1.4.0
  - federatedfilesharing: 1.13.0
  - files: 1.18.0
  - files_downloadactivity: 1.12.0
  - files_external: 1.15.0
  - files_markdown: 2.3.5
  - files_pdfviewer: 2.4.0
  - files_readmemd: 1.2.1
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - firstrunwizard: 2.12.0
  - forms: 2.4.0
  - gpoddersync: 3.2.0
  - health: 1.5.0
  - impersonate: 1.10.0
  - joplin: 0.0.25
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - mail: 1.11.6
  - maps: 0.1.10
  - metadata: 0.15.0
  - news: 17.0.1
  - nextcloud_announcements: 1.12.0
  - notes: 4.3.0
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - occweb: 0.0.7
  - onlyoffice: 7.3.0
  - password_policy: 1.13.0
  - phonetrack: 0.7.0
  - photos: 1.5.0
  - polls: 3.5.4
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - quota_warning: 1.13.0
  - rainloop: 7.1.3
  - ransomware_protection: 1.12.0
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - suspicious_login: 4.1.0
  - systemtags: 1.13.0
  - tasks: 0.14.2
  - text: 3.4.0
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - twofactor_nextcloud_notification: 3.3.1
  - twofactor_totp: 6.2.0
  - twofactor_webauthn: 0.2.15
  - updatenotification: 1.13.0
  - user_status: 1.3.1
  - viewer: 1.7.0
  - weather_status: 1.3.0
  - welcome: 1.0.1
  - workflowengine: 2.5.0
Disabled:
  - circles: 0.20.2
  - cms_pico: 1.0.17
  - dashboard: 7.2.0
  - encryption
  - extract: 1.3.3
  - federation: 1.7.0
  - files_antivirus: 3.2.2
  - files_texteditor: 2.14.0
  - files_trackdownloads: 1.11.0
  - files_trashbin: 1.7.0
  - flowupload: 1.1.3
  - gpxedit: 0.0.14
  - gpxmotion: 0.1.0
  - gpxpod: 4.3.0
  - groupfolders: 11.1.2
  - guests: 2.1.0
  - integration_discourse: 1.0.2
  - integration_github: 1.0.2
  - integration_gitlab: 1.0.3
  - integration_twitter: 1.0.2
  - integration_whiteboard: 0.0.14
  - music: 1.5.1
  - ncdownloader: 0.7.1
  - ocsms: 2.2.0
  - previewgenerator: 4.0.0
  - quickaccesssorting: 1.1.3
  - radio: 1.0.3
  - ransomware_detection: 0.10.0
  - recommendations: 0.8.0
  - richdocuments: 5.0.2
  - richdocumentscode: 21.11.103
  - smb_test: 0.3.3
  - support: 1.3.0
  - survey_client: 1.5.0
  - terms_of_service: 1.9.2
  - twofactor_u2f: 6.3.0
  - unsplash: 1.2.4
  - user_ldap

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

@spiezmaestro spiezmaestro added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Feb 21, 2022
@XueSheng-GIT
Copy link

I had the same issue using Ubuntu 20.04 and smbclient. After installing php-smbclient (using an additional ppa) this issue was solved.

@spiezmaestro
Copy link
Author

Thanks, tried that and solved my issue as well.

@dd4y
Copy link

dd4y commented Feb 28, 2022

Samo issue here, max 512mb download on NC 23.0.2 with external storage app 1.15.0 on Ubuntu 20.04 . Installed PPA php-smbclient 7.4 1.0.6 from https://dannyda.com/tag/ubuntu-20-04-1-lts-php-smbclient/

Still not working.

@plaing1
Copy link

plaing1 commented Mar 16, 2022

Same issue here, nextcloud 22.2.5, external storage app (files_external) version 1.13.1, OS Ubuntu 20.04

Works with php7.4-smbclient version 1.0.6-6+ubuntu20.04.1+deb.sury.org+10 as per instructions from https://dannyda.com/tag/ubuntu-20-04-1-lts-php-smbclient/

Restart php-fpm with systemctl restart php7.4-fpm after installing php7.4-smbclient.

@icsy7867
Copy link

icsy7867 commented Apr 19, 2022

Not sure if this is helpful for anyone. I am running the latest nextcloud 23.0.3 docker container. Ran into this as well.

I simply installed libsmbclient-dev and then ran pecl:

apt install libsmbclient-dev
pecl install smbclient
echo "extension=smbclient.so" >> /usr/local/etc/php/conf.d/nextcloud.ini

Anyone feeling lazy and running docker, I made a docker container (I only update this every few minor releases):
docker pull icsy7867/nextcloud:23

You can see the dockerfile here, pretty simple:
https://github.com/icsy7867/nextcloud/blob/master/nextcould-23/Dockerfile

@maxbader
Copy link

maxbader commented May 4, 2022

@icsy7867 Thanks using libsmbclient-dev and then ran pecl solved my issue.
I am using the official Nextcloud Hub II (23.0.3) inside docker

@Oroo9CeL
Copy link

Oroo9CeL commented May 16, 2022

I can confirm this bug with version 24.0.0
Stumbled across this when I wanted to move files larger than 512 MB between external SMB drives.
Installing and enabling php-smbclient as a workaround solves my problem.

@v1-valux
Copy link

I'm using the docker Image, but pecl isn't working for me:

After installing libsmclient-dev and smbclient within apt I get the following error:

No releases available for package "pecl.php.net/smbclient"
install failed

Also echo "extension=smbclient.so" >> /usr/local/etc/php/conf.d/nextcloud.ini
causes nextcloud not to start anymore.

Any ideas what might be wrong?

@icsy7867
Copy link

icsy7867 commented May 20, 2022

I'm using the docker Image, but pecl isn't working for me:

After installing libsmclient-dev and smbclient within apt I get the following error:

No releases available for package "pecl.php.net/smbclient"
install failed

Also echo "extension=smbclient.so" >> /usr/local/etc/php/conf.d/nextcloud.ini causes nextcloud not to start anymore.

Any ideas what might be wrong?

I wasnt able to reproduce your issue. Also you shouldnt run the echo line if you werent able to run PECL as the .so file wont exist.

I pulled the latest nextcloud continaer:
docker run -itd nextcloud:latest

And then I exec'd into the container:
docker exec -it docker_name bash

And I was able to successfully run:

apt-get update && apt-get install libsmbclient-dev smbclient
pecl install smbclient

...
Build process completed successfully
Installing '/usr/local/lib/php/extensions/no-debug-non-zts-20200930/smbclient.so'
install ok: channel://pecl.php.net/smbclient-1.0.6
configuration option "php_ini" is not set to php.ini location
You should add "extension=smbclient.so" to php.ini

@v1-valux
Copy link

v1-valux commented May 21, 2022

My Host is a RaspberryPi with Pi OS arm64 (Bullseye) with Samba 4.13.13 installed.

I am using a Portainer Stack to run the containers:

  • nextcloud:latest
  • postgres:14.2

My ENV file:

TZ=Europe/Berlin
DISABLE_IPV6=true
POSTGRES_DB=***
POSTGRES_HOST=***
POSTGRES_PASSWORD=***
POSTGRES_USER=***
NEXTCLOUD_ADMIN_PASSWORD=***
NEXTCLOUD_ADMIN_USER=***
NEXTCLOUD_DATA_DIR=/var/www/html/data
NEXTCLOUD_TRUSTED_DOMAINS=***
NEXTCLOUD_UPDATE=1
TRUSTED_PROXIES=<nginx-container>
PHP_MEMORY_LIMIT=512M
PHP_UPLOAD_LIMIT=10G 

I can reproduce my issue after deleting the image and running it from nextcloud:latest:

root@nextcloud:/var/www/html# apt-get update && apt-get install libsmbclient-dev smbclient

[...]
The following additional packages will be installed:
  dbus dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libapparmor1 libarchive13 libassuan0 libavahi-client3 libavahi-common-data libavahi-common3 libcap2 libcups2 libdbus-1-3 libgpgme11 libksba8
  libldb2 liblmdb0 libmpdec3 libnpth0 libpython3-stdlib libpython3.9 libpython3.9-minimal libpython3.9-stdlib libsmbclient libtalloc2 libtdb1 libtevent0 libwbclient0 pinentry-curses python3 python3-gpg python3-ldb python3-minimal python3-samba
  python3-talloc python3-tdb python3.9 python3.9-minimal samba-common samba-common-bin samba-dsdb-modules samba-libs

Suggested packages:
  default-dbus-session-bus | dbus-session-bus dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage scdaemon lrzip cups-common pinentry-doc python3-doc python3-tk python3-venv python3.9-venv python3.9-doc binfmt-support
  heimdal-clients python3-markdown python3-dnspython cifs-utils

The following NEW packages will be installed:
  dbus dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libapparmor1 libarchive13 libassuan0 libavahi-client3 libavahi-common-data libavahi-common3 libcap2 libcups2 libdbus-1-3 libgpgme11 libksba8
  libldb2 liblmdb0 libmpdec3 libnpth0 libpython3-stdlib libpython3.9 libpython3.9-minimal libpython3.9-stdlib libsmbclient libsmbclient-dev libtalloc2 libtdb1 libtevent0 libwbclient0 pinentry-curses python3 python3-gpg python3-ldb python3-minimal
  python3-samba python3-talloc python3-tdb python3.9 python3.9-minimal samba-common samba-common-bin samba-dsdb-modules samba-libs smbclient

0 upgraded, 51 newly installed, 0 to remove and 4 not upgraded.
Need to get 26.6 MB of archives.
After this operation, 101 MB of additional disk space will be used.
Do you want to continue? [Y/n] 

[...]
Fetched 26.6 MB in 2s (12.5 MB/s)           
debconf: delaying package configuration, since apt-utils is not installed

[...]
Setting up samba-common (2:4.13.13+dfsg-1~deb11u3) ...
**debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline**
Creating config file /etc/samba/smb.conf with new version

[...]
Setting up samba-common-bin (2:4.13.13+dfsg-1~deb11u3) ...
Checking smb.conf with testparm
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Weak crypto is allowed
Server role: ROLE_STANDALONE

Done
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Processing triggers for mailcap (3.69) ...
root@nextcloud:/var/www/html# pecl install smbclient
No releases available for package "pecl.php.net/smbclient"
install failed

Could this be related to the RaspberryPi only?

@icsy7867
Copy link

My Host is a RaspberryPi with Pi OS arm64 (Bullseye) with Samba 4.13.13 installed.

I am using a Portainer Stack to run the containers:

  • nextcloud:latest
  • postgres:14.2

My ENV file:

TZ=Europe/Berlin
DISABLE_IPV6=true
POSTGRES_DB=***
POSTGRES_HOST=***
POSTGRES_PASSWORD=***
POSTGRES_USER=***
NEXTCLOUD_ADMIN_PASSWORD=***
NEXTCLOUD_ADMIN_USER=***
NEXTCLOUD_DATA_DIR=/var/www/html/data
NEXTCLOUD_TRUSTED_DOMAINS=***
NEXTCLOUD_UPDATE=1
TRUSTED_PROXIES=<nginx-container>
PHP_MEMORY_LIMIT=512M
PHP_UPLOAD_LIMIT=10G 

I can reproduce my issue after deleting the image and running it from nextcloud:latest:

root@nextcloud:/var/www/html# apt-get update && apt-get install libsmbclient-dev smbclient

[...]
The following additional packages will be installed:
  dbus dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libapparmor1 libarchive13 libassuan0 libavahi-client3 libavahi-common-data libavahi-common3 libcap2 libcups2 libdbus-1-3 libgpgme11 libksba8
  libldb2 liblmdb0 libmpdec3 libnpth0 libpython3-stdlib libpython3.9 libpython3.9-minimal libpython3.9-stdlib libsmbclient libtalloc2 libtdb1 libtevent0 libwbclient0 pinentry-curses python3 python3-gpg python3-ldb python3-minimal python3-samba
  python3-talloc python3-tdb python3.9 python3.9-minimal samba-common samba-common-bin samba-dsdb-modules samba-libs

Suggested packages:
  default-dbus-session-bus | dbus-session-bus dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage scdaemon lrzip cups-common pinentry-doc python3-doc python3-tk python3-venv python3.9-venv python3.9-doc binfmt-support
  heimdal-clients python3-markdown python3-dnspython cifs-utils

The following NEW packages will be installed:
  dbus dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libapparmor1 libarchive13 libassuan0 libavahi-client3 libavahi-common-data libavahi-common3 libcap2 libcups2 libdbus-1-3 libgpgme11 libksba8
  libldb2 liblmdb0 libmpdec3 libnpth0 libpython3-stdlib libpython3.9 libpython3.9-minimal libpython3.9-stdlib libsmbclient libsmbclient-dev libtalloc2 libtdb1 libtevent0 libwbclient0 pinentry-curses python3 python3-gpg python3-ldb python3-minimal
  python3-samba python3-talloc python3-tdb python3.9 python3.9-minimal samba-common samba-common-bin samba-dsdb-modules samba-libs smbclient

0 upgraded, 51 newly installed, 0 to remove and 4 not upgraded.
Need to get 26.6 MB of archives.
After this operation, 101 MB of additional disk space will be used.
Do you want to continue? [Y/n] 

[...]
Fetched 26.6 MB in 2s (12.5 MB/s)           
debconf: delaying package configuration, since apt-utils is not installed

[...]
Setting up samba-common (2:4.13.13+dfsg-1~deb11u3) ...
**debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline**
Creating config file /etc/samba/smb.conf with new version

[...]
Setting up samba-common-bin (2:4.13.13+dfsg-1~deb11u3) ...
Checking smb.conf with testparm
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Weak crypto is allowed
Server role: ROLE_STANDALONE

Done
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Processing triggers for mailcap (3.69) ...
root@nextcloud:/var/www/html# pecl install smbclient
No releases available for package "pecl.php.net/smbclient"
install failed

Could this be related to the RaspberryPi only?

I'm not sure.
https://hub.docker.com/_/php mentions deleting
/etc/apt/preferences.d/no-debian-php

But just a guess.

@icsy7867
Copy link

nextcloud/vm#1232

Someone also mentioned pihole blocking pecl

@v1-valux
Copy link

Thanks a million times for pointing me that direction! It was the pihole blocking..

@itstevey
Copy link

itstevey commented Jun 8, 2022

Same issue here, nextcloud 22.2.5, external storage app (files_external) version 1.13.1, OS Ubuntu 20.04

Works with php7.4-smbclient version 1.0.6-6+ubuntu20.04.1+deb.sury.org+10 as per instructions from https://dannyda.com/tag/ubuntu-20-04-1-lts-php-smbclient/

Restart php-fpm with systemctl restart php7.4-fpm after installing php7.4-smbclient.

This worked for me:

sudo apt install php7.4-smbclient
sudo service nginx restart

Thanks.

@eivamu
Copy link

eivamu commented Aug 27, 2022

I had the same issue using Ubuntu 20.04 and smbclient. After installing php-smbclient (using an additional ppa) this issue was solved.

I had the same issue with NC 24.0.4, External Storage Support 1.16.1, Ubuntu 20.04.4 LTS, PHP 8.1.8.
Installing the package php-smbclient using apt immediately solved the issue.

@Decstasy
Copy link

Same Issue here...

@szaimen
Copy link
Contributor

szaimen commented Nov 29, 2022

Please simply install php-smbclient then

@szaimen szaimen closed this as completed Nov 29, 2022
@Decstasy
Copy link

@szaimen
Copy link
Contributor

szaimen commented Nov 29, 2022

Please alter the Documentation then https://github.com/nextcloud/documentation/blob/master/admin_manual/configuration_files/external_storage/smb.rst

PRs are welcome!

@v1-valux
Copy link

Case closed.. lol

@szaimen
Copy link
Contributor

szaimen commented Nov 29, 2022

I've created this issue for you to not forget it: nextcloud/documentation#9428

@Oroo9CeL
Copy link

Please simply install php-smbclient then

Installing php-smbclient can only be a workaround, since not all linux distribution provides a package for this. Installation via pecl or adding external repositories may affect the consistency of the installation and its maintainability. This is not acceptable in professional environments. Since smbclient is the CIFS client which is present on all linux distributions, the focus should be on finding the actual bug instead of using workarounds and closing bugreports. So please reopen the bug. Thanks!

@maxi1134
Copy link

maxi1134 commented Jan 30, 2023

I also have this issue!

And before being told to install php-smbclient:

E: Unable to locate package php7.4-smbclient
E: Couldn't find any package by glob 'php7.4-smbclient'
root@33831c23a033:/var/www/html#

@icsy7867
Copy link

Unable

Is this in a docker container? Try doing and apt-get update first.

I also used a different set of commands personally.
#31308 (comment)

@maxi1134
Copy link

Unable

Is this in a docker container? Try doing and apt-get update first.

I also used a different set of commands personally. #31308 (comment)

That worked!

Could you share the docker file? It does not exist anymore

rossmaclean pushed a commit to rossmaclean/cloudron-base-image that referenced this issue Feb 6, 2023
this only becomes valid in ubuntu 22

required for nc - nextcloud/server#31308
@Ryushin
Copy link

Ryushin commented Feb 14, 2023

Today I noticed php-smbclient is not working with Samba 4.17.5 from Debian. Recompiled again as well against the new lib-smbclient:
pecl uninstall smbclient ; pecl install smbclient
Found other are having the same problem:
https://www.spinics.net/lists/centos/msg181966.html

So I've reverted to using smbclient and now have the 512MB bug again. Really, smbclient is standard and should be the path forward. So whatever bug is causing this really needs to be looked at. If the ticket is not re-opened in the next couple of days, I'll open a new ticket as I don't know if the admins are notified if a comment takes place on a closed ticket.

@icsy7867
Copy link

Unable

Is this in a docker container? Try doing and apt-get update first.
I also used a different set of commands personally. #31308 (comment)

That worked!

Could you share the docker file? It does not exist anymore

https://github.com/icsy7867/nextcloud

I also finally got around the playing with gitlab. I am automatically rebuilding these weekly with the latest major versions (24 and 25):
https://hub.docker.com/r/icsy7867/nextcloud/tags
icsy7867/nextcloud:25
icsy7867/nextcloud:24

Feel free to use them if you like.

@FrancisHGR
Copy link

Had the same bug, got it solved by running the following commands within the docker container:

apt update && apt -y install libsmbclient-dev libmagickwand-dev && pecl install smbclient && echo “extension=smbclient.so” > /usr/local/etc/php/conf.d/docker-php-ext-smbclient.ini && pecl install inotify

Still, this bug should be fixed, ideally php-smbclient directly integrated in Nextcloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug
Projects
None yet
Development

No branches or pull requests