Connecting Nextcloud, Calibre-Web and External Storage

HelenaGwyn

I've been thinking about setting up my own private cloud for a while now and when I saw the tutorial on running Nextcloud on Opalstack, I felt overloaded with joy. I have a few questions for use case I want to work towards.

Is it possible to connect several nextcloud-instances, hosted on different servers and domains?
I'm looking for a cost-effective way of having at least 1TB of cloudspace.
Hetzner offers Storage Boxes (https://www.hetzner.com/storage/storage-box). They come without Nextcloud, so I assume I could just 'mount' them in the interface?
Hetzner also offers Storage Shares (https://www.hetzner.com/storage/storage-share). Those instances are Nextcloud, but without the other connectionprotocols. Can they connect and share data?
Or maybe Opalstack could offer storagesolutions in the future as well 😉?

Is it possible to run a Calibre-Web-app on Opalstack and connect to a folder within a Nextcloud-app?
I would like to manage and host my own ebooklibrary, or maybe multiple libraries for different users.
And I'm wondering if the ebook-files could be shared between users, without sharing the annotations made by users.

I know some questions might be beyond the scope of traditional support. But maybe someone already figured these things out. I haven't actively used Nextcloud yet. I tried it out years ago, but more or less new to the whole setup.

sean

HelenaGwyn if you migrate to VPS hosting we can attach a 1TB storage volume to the VPS for $160/month (in addition to the normal VPS hosting fees). This would be very fast local storage which you could use as your Nextcloud data directory. We've some other storage-only plans in the works but I can't discuss them at the moment 🙂

Nextcloud supports a variety of external storage providers, so if you wanted to use something like Hetzner then that's a possibility: https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.htm

It's possible to run Calibre-Web on Opalstack and based on this you should be able to keep it in sync with Nextcloud.

HelenaGwyn

sean
Those VPS's sounds great but a bit out of my current budget 🙂. Looking forward to the new storage-only plans though.

For Calibre-web I create a python-app and run pip3 install calibreweb and point it to the nextcloud directory for storage (which I will have to share between them)? I plan on storing my local calibre-database in a nextcloud-folder which syncs to the server.

For reading on my devices I thought about Marvin, but Calibre-sync looks even better.

sean

HelenaGwyn For Calibre-web I create a python-app and run pip3 install calibreweb and point it to the nextcloud directory for storage (which I will have to share between them)?

Yes, I think that's basically it although Caibre-web needs a couple of extra steps:

Install a Python/UWSGI app via your Opalstack dashboard and route it to a site. Make a note of the app's assigned port as shown in the app list the dashboard.

SSH to the app's shell user and run the following commands, replacing name_of_app and XXXXX with the app's name and port assignment:

cd ~/apps/name_of_app
source env/bin/activate
pip3 install calibreweb
export CALIBRE_PORT=XXXXX
./stop
( nohup ~/apps/name_of_app/env/bin/cps -i 127.0.0.1 > ~/logs/apps/name_of_app/name_of_app.log 2>&1 & )

At that point Calibre-web should be up and running and available at the site you set up in step 1.

HelenaGwyn

sean I'm just getting "Hello World!" right now. In the log-file it says

usage: cps.py [-h] [-p path] [-g path] [-c path] [-k path] [-v]
[-i ip-address] [-s user:pass] [-f] [-l] [-d] [-r]
cps.py: error: unrecognized arguments: 127.0.0.1

I tried adding -i or another IP, then it started, but I don't know how to actually reach it.

And when creating a nextcloud application, it failed and the log showed:

Started at 1970-01-01 00:00:00
UUID validation and server lookup OK.
Admin email lookup OK.
DB user creation failed.

sean

HelenaGwyn You'll have to stop the uwsgi server before you run cps, and it looks like there was a typo in the final step (-l instead of -i). I've corrected the instructions.

Sorry about that, I'm making this up as I go! 🤣

Regarding Nextcloud, it's likely that you used a common app name (like "nextcloud", "cloud", etc) that was already in use on your shared server, so when the installer attempted to create a matching database for the application the install failed. To work around that, delete the failed app and then create a new one with a unique name that's less likely to conflict with the name of a pre-existing DB.

HelenaGwyn

sean Sorry, still not working for me. Without starting I get a 502 although the cps-process is running.
I'm not that much familiar with Python, so I might be overlooking something.

Nextcloud is working, thanks!

sean

HelenaGwyn I think your app got auto-configured to use the wrong port. To fix this, delete your ~/.calibre-web directory on the server then repeat the commands in step 2 above. If that doesn't solve it then please email support so that we can take a closer look.

HelenaGwyn

sean I'm getting 'malformed database schema' when trying to read my calibre-db. This could be related to the version of Sqlite3 which is 3.7.17 on the server. Is there a fix for this?

HelenaGwyn

Solved, thank you!

sean

HelenaGwyn there is a newer version of sqlite available in /usr/sqlite330/bin/so try running export PATH=/usr/sqlite330/bin/:$PATH then repeat whatever it is you were attempting to do.

HelenaGwyn

sean Still no luck. Same error. No idea why. It just won't read the database, even an empty one newly created with Calibre.

sean

HelenaGwyn where exactly do you see the error message?

HelenaGwyn

sean When trying to load the database it gives me 'DB Location is not Valid, Please Enter Correct Path' in the interface. If I check $HOME/.calibre-web/calibre-web.log I see the details that lead up to this message. I did some searching and came across this page https://github.com/janeczku/calibre-web/issues/1633 but updating sqlite didn't solve it for me sadly.

sean

HelenaGwyn ok, please try this:

Activate your app's environment.
Install pysqlite3-binary: pip3 install pysqlite3-binary

Edit env/bin/cps to add the following lines just after import sys:

__import__('pysqlite3')
sys.modules['sqlite3'] = sys.modules.pop('pysqlite3')

Deactivate the environment and then re-activate it.
Stop the running cps process and restart it.

HelenaGwyn

sean Thank you, that solved it, Calibre-Web up and running.

sean

HelenaGwyn Nice, I honestly wasn't sure if that would work!

Note that you'll need to make that last change again if you ever update the calibreweb Python package.

Eventually we'll have an updated OS with newer software so that things like these sqlite3 gymnastics won't be necessary 🙂

HelenaGwyn

sean Hi again,

I've been toying with nextcloud, apps, calibreweb, syncing, ... on opalstack and on hetzner and I came to a few conclusions and a new question.

It turns out hetzner doesn't support external storage because of security reasons, so connecting another server to their nextcloud isn't possible.

Connecting a hetzner-nextcloud to an opalstack-instance is possible. But from what I understand it doesn't sync an an external storage locally. It just provides a webinterface to the external storage and temporarily caches it.
If I want to use Calibreweb on opalstack the files have to be somewhere within my user on opalstack.

I saw there is a nextcloudmd-client (https://docs.nextcloud.com/desktop/3.4/advancedusage.html#install-nextcloudcmd) that can be triggered to sync a nextcloud-account. But I assume using rclone with a cronjob might to the same thing?

My setup would be installing calibreweb on opalstack and using rclone to daily sync my ebooklibrary with my nextcloud at hetzner which syncs with calibre on my laptop amongst other things.

Do you consider that the best solution given the current situation? (until opalstack comes with similar storage solutions 🙂).

sean

HelenaGwyn I don't think nextcloudcmd would help with this. That tool syncs from a Nextcloud instance to a client directory, but it doesn't sync files between 2 Nextcloud instances, which I think is what you want to do?

If you use rclone (or any other external sync tool) to sync files directly into the Nextcloud filesystem, ie into ~/apps/nextcloud/data/username/files then you'll need to tell Nextcloud to sync up its database after the file sync is complete. The commands to do that are:

cd ~/apps/nextcloud
php74 occ files:scan --all

So you'd need to incorporate that into whatever cron job you run for your sync.

HelenaGwyn

sean I want one location to sync all my files with (on hetzner in this case) and serve my ebooklibrary with calibreweb on opalstack. Connecting two nextcloud-instances doesn't work for this use case from what I understand. Calibreweb needs to be pointed to the folder with the database and files. Adding an external storage in nextcloud, doesn't sync the external files, it just shows them in the webinterface. Or I don't know where to look for them.
At the moment there's no need to sync directly with a nextcloud on opalstack, I just want my ebooklibrary served on opalstack.

sean

HelenaGwyn in that case I think you should be able to use any sync tool (rclone, rsync, whatever) to sync the Calibre library directories on each end.