Canuck Audio Mart Hifi and Audio ForumCanuck Audio Mart Hifi and Audio Forum
It is currently Tue Dec 12, 2017 5:22 pm

All times are UTC - 8 hours




Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Mon Sep 04, 2017 4:47 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
I figured I could help share with this community, my current setup which gives me unlimited, always available and fast cloud storage for all of my music (currently 12tbs, not including the non cloud mirrored 11tbs at home) how I built it, and some tips for how you can too.

GSuite, formerly Google Apps For Business is so far the best unlimited cloud storage solution on the market. At the moment marketing material states 5 users at $10 a month is the threshold for getting unlimited, however google has decided to not enforce it (and hasnt for the last year or so) meaning you can get away with a single user and still gain unlimited storage.

How does my system work?

Let me preface this by saying this is about to get technical.

1. I upload music, in an encrypted form using rclone which is a command line application to interface with various cloud storage providers. The reason for encrypted is that I would rather not have google prying at my activities and as this is a business account, client side encryption of my data should be standard anyways. Most people will use their home computers, I use a series of 3 servers hosted with Hetzner to automatically scrape my sites for FLACs I am interested in, and then upload those files to my Google Drive.

2. Using rclone my Google Drive is mounted on another, small and fast SSD based server with OVH. It is decrypted by rclone giving me access to the files I uploaded in step 1.

3. This server runs media indexing software, such as JRiver Media Center or Roon, they use the mounted folder containing the music as a library to pull from

4. I then connect to said server via client software and viola. Infinite storage space


Common questions:

1. Is Google okay with this? Yes I have had numerous conversations with their engineers, they have no problem with companies storing and accessing their encrypted data. I currently have over 200tbs on a single account and they still want my business.
2.Is it fast? In my experience its almost as lag free as playing a locally hosted file. I can play 4k Blu Ray Remuxes which clock in at around 60gbs with no problem. Upload speeds can reach 500MBPS with 10gigabit servers.
3. Is this illegal? If you own the CDs, its perfectly fine. I'm not responsible for how you get your music but saving money always makes sense.
4.Is unlimited Google Cloud here to stay? Only time will tell. Somebody uploaded 2 Petabytes of porn to Amazon and they took it away, they also werent pleased with my 400tbs but that's small potatoes. However, Google's service is for businesses and the most they will likely do is begin enforcing the 5 users for unlimited restriction. I would even possibly be okay with paying $100 a month for this.
5.Who do I ask for help? Basic things, me. Don't ask me to teach you how to use linux. forum.rclone.org and /r/datahoarder are your friend. Hire a nerd to set things up.


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 04, 2017 6:19 am 
Offline

Joined: Wed Nov 21, 2007 6:43 am
Posts: 2025
Location: Québec, QC, CA
:idea:


Attachments:
zcPdUbh.png
zcPdUbh.png [ 101.29 KiB | Viewed 1587 times ]
Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 04, 2017 8:35 am 
Offline
User avatar

Joined: Wed May 30, 2007 7:54 am
Posts: 3468
Location: Vernon, BC, CA
Or you could just setup your own home based music server:
http://www.subsonic.org/pages/index.jsp

With that I can stream both my music and TV shows to any internet connected computer.


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 04, 2017 8:47 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
Jared Rachwalski wrote:
Or you could just setup your own home based music server:
http://www.subsonic.org/pages/index.jsp

With that I can stream both my music and TV shows to any internet connected computer.

I still do that (all 95tbs of locally stored media) but I just cant beat always on and hosted servers. It's just financially unfeasible to store 200tbs on a bare metal server, or through Amazon S3 so I had to hack together my own solution.

Plus its a little hard to get a home connection in Canada which will stream a 4k remux or multiple untouched blu rays to various external clients. In the US with Google Fiber or another enterprise level service, Id be in great shape and be able to invest a few grand a year into my home lab but due to our internet, it doesnt exactly make sense. A 90tb server is usually around 600ish a month, that would add up fast if I didnt use Google Drive as my storage backend


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 11, 2017 10:15 am 
Offline

Joined: Thu Feb 10, 2005 5:25 pm
Posts: 125
Location: Toronto, ON, CA
EveDoesAudioThings wrote:
I figured I could help share with this community, my current setup which gives me unlimited, always available and fast cloud storage for all of my music (currently 12tbs, not including the non cloud mirrored 11tbs at home) how I built it, and some tips for how you can too.

GSuite, formerly Google Apps For Business is so far the best unlimited cloud storage solution on the market. At the moment marketing material states 5 users at $10 a month is the threshold for getting unlimited, however google has decided to not enforce it (and hasnt for the last year or so) meaning you can get away with a single user and still gain unlimited storage.

How does my system work?

Let me preface this by saying this is about to get technical.

1. I upload music, in an encrypted form using rclone which is a command line application to interface with various cloud storage providers. The reason for encrypted is that I would rather not have google prying at my activities and as this is a business account, client side encryption of my data should be standard anyways. Most people will use their home computers, I use a series of 3 servers hosted with Hetzner to automatically scrape my sites for FLACs I am interested in, and then upload those files to my Google Drive.

2. Using rclone my Google Drive is mounted on another, small and fast SSD based server with OVH. It is decrypted by rclone giving me access to the files I uploaded in step 1.

3. This server runs media indexing software, such as JRiver Media Center or Roon, they use the mounted folder containing the music as a library to pull from

4. I then connect to said server via client software and viola. Infinite storage space


Common questions:

1. Is Google okay with this? Yes I have had numerous conversations with their engineers, they have no problem with companies storing and accessing their encrypted data. I currently have over 200tbs on a single account and they still want my business.
2.Is it fast? In my experience its almost as lag free as playing a locally hosted file. I can play 4k Blu Ray Remuxes which clock in at around 60gbs with no problem. Upload speeds can reach 500MBPS with 10gigabit servers.
3. Is this illegal? If you own the CDs, its perfectly fine. I'm not responsible for how you get your music but saving money always makes sense.
4.Is unlimited Google Cloud here to stay? Only time will tell. Somebody uploaded 2 Petabytes of porn to Amazon and they took it away, they also werent pleased with my 400tbs but that's small potatoes. However, Google's service is for businesses and the most they will likely do is begin enforcing the 5 users for unlimited restriction. I would even possibly be okay with paying $100 a month for this.
5.Who do I ask for help? Basic things, me. Don't ask me to teach you how to use linux. forum.rclone.org and /r/datahoarder are your friend. Hire a nerd to set things up.


This is fascinating. I signed up for the program and now have my unlimited account. I was paying $3/month for 100GB, so an extra $7 for a full backup of my 4TB server is a no-brainer.

Unfortunately my upload speed is pathetic. The Google Backup and Synch program has been running on my WHS v1 system for two days now, and I'm only 20GB through. It'll take months at this rate.

I did not encrypt my data, so let's see what transpires.

Jason


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 18, 2017 7:36 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
doctor_big wrote:
EveDoesAudioThings wrote:
I figured I could help share with this community, my current setup which gives me unlimited, always available and fast cloud storage for all of my music (currently 12tbs, not including the non cloud mirrored 11tbs at home) how I built it, and some tips for how you can too.

GSuite, formerly Google Apps For Business is so far the best unlimited cloud storage solution on the market. At the moment marketing material states 5 users at $10 a month is the threshold for getting unlimited, however google has decided to not enforce it (and hasnt for the last year or so) meaning you can get away with a single user and still gain unlimited storage.

How does my system work?

Let me preface this by saying this is about to get technical.

1. I upload music, in an encrypted form using rclone which is a command line application to interface with various cloud storage providers. The reason for encrypted is that I would rather not have google prying at my activities and as this is a business account, client side encryption of my data should be standard anyways. Most people will use their home computers, I use a series of 3 servers hosted with Hetzner to automatically scrape my sites for FLACs I am interested in, and then upload those files to my Google Drive.

2. Using rclone my Google Drive is mounted on another, small and fast SSD based server with OVH. It is decrypted by rclone giving me access to the files I uploaded in step 1.

3. This server runs media indexing software, such as JRiver Media Center or Roon, they use the mounted folder containing the music as a library to pull from

4. I then connect to said server via client software and viola. Infinite storage space


Common questions:

1. Is Google okay with this? Yes I have had numerous conversations with their engineers, they have no problem with companies storing and accessing their encrypted data. I currently have over 200tbs on a single account and they still want my business.
2.Is it fast? In my experience its almost as lag free as playing a locally hosted file. I can play 4k Blu Ray Remuxes which clock in at around 60gbs with no problem. Upload speeds can reach 500MBPS with 10gigabit servers.
3. Is this illegal? If you own the CDs, its perfectly fine. I'm not responsible for how you get your music but saving money always makes sense.
4.Is unlimited Google Cloud here to stay? Only time will tell. Somebody uploaded 2 Petabytes of porn to Amazon and they took it away, they also werent pleased with my 400tbs but that's small potatoes. However, Google's service is for businesses and the most they will likely do is begin enforcing the 5 users for unlimited restriction. I would even possibly be okay with paying $100 a month for this.
5.Who do I ask for help? Basic things, me. Don't ask me to teach you how to use linux. forum.rclone.org and /r/datahoarder are your friend. Hire a nerd to set things up.


This is fascinating. I signed up for the program and now have my unlimited account. I was paying $3/month for 100GB, so an extra $7 for a full backup of my 4TB server is a no-brainer.

Unfortunately my upload speed is pathetic. The Google Backup and Synch program has been running on my WHS v1 system for two days now, and I'm only 20GB through. It'll take months at this rate.

I did not encrypt my data, so let's see what transpires.

Jason


How many concurrent transfers are you running? use the switch --transfers=x, for movies / tv I usually do 10-15 and for something like music with hundreds of thousands of small files, I usually put it up to 25-35. See what works for you. Also --checkers=16 is always a good idea too.

My typical rclone command to copy a folder over to my chosen remote.
rclone copy "x" "GEnc:" --transfers=20 --stats=3s --checkers=16 -v
So that will copy a folder, do 20 files in parallel, update my terminal view every 3 seconds and any existing files will be checked, 16 at a time concurrently, -v is for verbose output which is very helpful. Stay away from -vv as that will spam your terminal to no end, it's great for tracking down problems though.


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 18, 2017 8:50 am 
Offline

Joined: Thu Feb 10, 2005 5:25 pm
Posts: 125
Location: Toronto, ON, CA
EveDoesAudioThings wrote:

How many concurrent transfers are you running? use the switch --transfers=x, for movies / tv I usually do 10-15 and for something like music with hundreds of thousands of small files, I usually put it up to 25-35. See what works for you. Also --checkers=16 is always a good idea too.

My typical rclone command to copy a folder over to my chosen remote.
rclone copy "x" "GEnc:" --transfers=20 --stats=3s --checkers=16 -v
So that will copy a folder, do 20 files in parallel, update my terminal view every 3 seconds and any existing files will be checked, 16 at a time concurrently, -v is for verbose output which is very helpful. Stay away from -vv as that will spam your terminal to no end, it's great for tracking down problems though.


I think you're assuming much more technical knowledge than I have. While I do get the gist of what you're suggesting, I'm using Google's Backup and Sync program running on a WHS v1 box.

I'll check the settings tonight, but I don't think it offers the levels of granularity that you suggest.

Jason


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 18, 2017 10:03 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
doctor_big wrote:
EveDoesAudioThings wrote:

How many concurrent transfers are you running? use the switch --transfers=x, for movies / tv I usually do 10-15 and for something like music with hundreds of thousands of small files, I usually put it up to 25-35. See what works for you. Also --checkers=16 is always a good idea too.

My typical rclone command to copy a folder over to my chosen remote.
rclone copy "x" "GEnc:" --transfers=20 --stats=3s --checkers=16 -v
So that will copy a folder, do 20 files in parallel, update my terminal view every 3 seconds and any existing files will be checked, 16 at a time concurrently, -v is for verbose output which is very helpful. Stay away from -vv as that will spam your terminal to no end, it's great for tracking down problems though.


I think you're assuming much more technical knowledge than I have. While I do get the gist of what you're suggesting, I'm using Google's Backup and Sync program running on a WHS v1 box.

I'll check the settings tonight, but I don't think it offers the levels of granularity that you suggest.

Jason


Oh I thought you were using the command line rclone application. Id suggest you try running that instead of the google apps. They *suck*.

Whats your upload speed (like what you pay for from your ISP)?


Top
 Profile  
Reply with quote  
PostPosted: Mon Sep 18, 2017 11:55 am 
Offline

Joined: Thu Feb 10, 2005 5:25 pm
Posts: 125
Location: Toronto, ON, CA
EveDoesAudioThings wrote:

Oh I thought you were using the command line rclone application. Id suggest you try running that instead of the google apps. They *suck*.

Whats your upload speed (like what you pay for from your ISP)?


My ISP quotes 1mb, which seems reasonable based on a) speed tests and b) the amount of data I've uploaded in 9 days (79mb). At this rate it'll take me a year.

I've asked teksavvy to bump me to 10mb, which theoretically should get me done in a month, but they have a 10-day lead time on making the change.

Jason


Top
 Profile  
Reply with quote  
PostPosted: Wed Sep 20, 2017 7:10 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
doctor_big wrote:
EveDoesAudioThings wrote:

Oh I thought you were using the command line rclone application. Id suggest you try running that instead of the google apps. They *suck*.

Whats your upload speed (like what you pay for from your ISP)?


My ISP quotes 1mb, which seems reasonable based on a) speed tests and b) the amount of data I've uploaded in 9 days (79mb). At this rate it'll take me a year.

I've asked teksavvy to bump me to 10mb, which theoretically should get me done in a month, but they have a 10-day lead time on making the change.

Jason

That will make a big difference :3

Enjoy the cloud storage. Google is going nowhere and unlike alot of the other providers, I have a feeling they won't start restricting their service.
However there are 2 limits in place, per 24 hour period you can retrieve 10tbs before a temporary api ban (only means you cant retrieve a file over a few mb) and a 750gb a day, per account, per IP upload limit which has been a b*tch for me as I dump everything on here from servers. I ended up moving to 5 users and then buying an extra block of 4 new IPs for one of my servers.


Top
 Profile  
Reply with quote  
PostPosted: Wed Sep 20, 2017 9:07 am 
Offline

Joined: Thu Feb 10, 2005 5:25 pm
Posts: 125
Location: Toronto, ON, CA
EveDoesAudioThings wrote:
doctor_big wrote:
EveDoesAudioThings wrote:

Oh I thought you were using the command line rclone application. Id suggest you try running that instead of the google apps. They *suck*.

Whats your upload speed (like what you pay for from your ISP)?


My ISP quotes 1mb, which seems reasonable based on a) speed tests and b) the amount of data I've uploaded in 9 days (79mb). At this rate it'll take me a year.

I've asked teksavvy to bump me to 10mb, which theoretically should get me done in a month, but they have a 10-day lead time on making the change.

Jason

That will make a big difference :3

Enjoy the cloud storage. Google is going nowhere and unlike alot of the other providers, I have a feeling they won't start restricting their service.
However there are 2 limits in place, per 24 hour period you can retrieve 10tbs before a temporary api ban (only means you cant retrieve a file over a few mb) and a 750gb a day, per account, per IP upload limit which has been a b*tch for me as I dump everything on here from servers. I ended up moving to 5 users and then buying an extra block of 4 new IPs for one of my servers.


Thanks Eve! I don't think there's any chance of me hitting either of those limits.

As an aside, and back to the original intent of this thread, (in a year's time) when my uploads are finished, hopefully I can figger out some way to play these files back via my android phone...?


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 22, 2017 7:33 am 
Offline

Joined: Mon Aug 28, 2017 7:13 am
Posts: 58
Location: Toronto, ON, CA
doctor_big wrote:

Thanks Eve! I don't think there's any chance of me hitting either of those limits.

As an aside, and back to the original intent of this thread, (in a year's time) when my uploads are finished, hopefully I can figger out some way to play these files back via my android phone...?



That you can, there's 2 options. Since you're uploading the files unencrypted, you can use the 'Google Drive' application (its probably already on your phone) to download and then play them with the player of your choosing. OR, the superior solution but harder is to buy a very cheap VPS server (its pretty much a computer in the cloud you connect to) and run media server software on it to serve the files to your phone directly. The beauty of that is, you can stream them without actually downloading and you can transcode your FLACs to say MP3 320 to save data.
If you want to setup the media server I can probably link you a guide or find you someone who can quickly and cheaply set it up for you. I use OVH for my media server and it costs me I think 10 - 20 bucks a month, I can't remember which one I purchased. They're a very reliable Canadian company with great speed, service and hardware. For bigger jobs Hetzner is my go to, I have a pair of 6TB servers I use to download my 'linux ISOs' and then dump it to google drive, for what works out to be 35 - 40 a month canadian each I'm very happy.


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 22, 2017 9:02 am 
Offline
Moderator
User avatar

Joined: Tue Feb 22, 2005 2:06 pm
Posts: 4910
Location: Montreal, QC, CA
This is the thing that gives me pause.

doctor_big wrote:
Unfortunately my upload speed is pathetic. The Google Backup and Synch program has been running on my WHS v1 system for two days now, and I'm only 20GB through. It'll take months at this rate.

I did not encrypt my data, so let's see what transpires.


I've been considering an online backup service for my music files. These things are precious, they represent a lot of time, money, and even effort, in the tagging. Right now I only have two backups, in the form of spinning drives.

But the upload time, even for my paltry 2TB of music, is unworkably long, even if it didn't represent eight times my monthly data cap.

Maybe I can work out a one-time deal with my ISP... @doctor_big, let us know how your upload goes, if you get the chance.


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 22, 2017 9:09 am 
Offline

Joined: Thu Feb 10, 2005 5:25 pm
Posts: 125
Location: Toronto, ON, CA
Toby wrote:
This is the thing that gives me pause.

doctor_big wrote:
Unfortunately my upload speed is pathetic. The Google Backup and Synch program has been running on my WHS v1 system for two days now, and I'm only 20GB through. It'll take months at this rate.

I did not encrypt my data, so let's see what transpires.


I've been considering an online backup service for my music files. These things are precious, they represent a lot of time, money, and even effort, in the tagging. Right now I only have two backups, in the form of spinning drives.

But the upload time, even for my paltry 2TB of music, is unworkably long, even if it didn't represent eight times my monthly data cap.

Maybe I can work out a one-time deal with my ISP... @doctor_big, let us know how your upload goes, if you get the chance.


My plan is to bump up my speed to 10mbps, and cap to unlimited, for the period it takes to complete the upload. At 1mbps, it will take me about a year. At 10mbps, it'll be a little over a month. After that's done, I'll lower it back down.

Still way cheaper than another hard drive.


Top
 Profile  
Reply with quote  
PostPosted: Fri Sep 22, 2017 9:25 am 
Offline
Moderator
User avatar

Joined: Tue Feb 22, 2005 2:06 pm
Posts: 4910
Location: Montreal, QC, CA
Thanks! Maybe my ISP will go for something like that.

doctor_big wrote:
My plan is to bump up my speed to 10mbps, and cap to unlimited, for the period it takes to complete the upload. At 1mbps, it will take me about a year. At 10mbps, it'll be a little over a month. After that's done, I'll lower it back down.

Still way cheaper than another hard drive.


Cheaper -- good point.

Hope this info is useful for folks who are considering the OP's measures too.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 17 posts ]  Go to page 1, 2  Next

All times are UTC - 8 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group