Friday, September 12, 2014

Synology and bittorrent sync




What is Bittorent Sync?

This is an efficient and secure way for groups of users to have read only  or read/write access to a shared group of files and or directories.

Once a connection is made each user has a read-only or read-write copy of the files kept in sync 

In the simplest case

  • User A and B install the BitTorrent Sync
  • User A and B are on the same Network at home
  • User A sets up a directory X to share and sends a link to user B
  • User B loads the link into their BitTorrent Sync
  • User A is asked to confirm that they wish user B to have access to directory X
  • The Synchronisation begins and quite soon the files are in a directory X on both users computers
  • Assuming read/write shared access users can edit files on either side and when any changes are made and saved from the editor to the filesystem in directory X then are transmitted to the other side. 

Lan or Internet
Oh yes, one more thing computers can be interconnected on your home Wifi or Wired Lan or connected thru the Internet!

So as an example Marcus in Lausanne Switzerland shared his P:\linux directory on his Windows server to Bob in Texas where Bob stored his files on his Apple OSX laptop in /Users/Bob/Linux.  We tested and achieved synchronisation speeds of about 2MB/second. That is a distance of over 5000 miles apart.


BitTorent Sync on Synology

It's available on Qnap as an application but not yet for Synology so here is the manual process to install it.

This tutorial assumes that you will setup your Synology NAS storage with a user btsync which has read write access to a number of directories that you will share out to other users, usually read/write or readonly as you prefer.


  • From Synology Gui create user to dedicate to Torrent sync.  I chose user  btsync and created it via Synology Control Panel GUI
  • Give user btsync  read/write permissions to the area you want it to manipulate
  • On the Synology Control Panel, Terminal enable ssh service
  • Use putty or the OSX equivalent to ssh from your controlling PC/OS to Synology and logon as root  which means using the password for the user admin
  • Change my user btsync to allow login
vi /etc/passwd

# Allow btsync user to logon
# note I also had to change home directory manually as 
# somehow it keeps pointing to a now removed volume1 ! btsync:x:1030:100:Bittorrent Sync:/volume2/homes/btsync:/bin/sh

  • Create some directories for user btsync


cd /volume2/homes/btsync
mkdir bin conf
ls -ltr
drwxr-xr-x    2 btsync   users         4096 Aug 27 15:48 conf
drwxr-xr-x    2 btsync   users         4096 Aug 27 15:48 bin

  • Check library version installed

uname -a
Linux Cyclopes 3.2.40 #4493 SMP Thu Aug 21 21:46:06 CST 2014 x86_64 GNU/Linux synology_cedarview_1812+
ls -l /lib/libc.so.6
-r-xr-xr-x    1 root     root       1528336 May 30 18:23 /lib/libc.so.6
/lib/libc.so.6
GNU C Library stable release version 2.13, by Roland McGrath et al.
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 4.6.3.
Compiled on a Linux 3.6.3 system on 2012-12-03.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B

libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.


  • So Download btsync 64 bit version
cd /volume2/homes/btsync/bin
pwd
/volume2/homes/btsync/bin
wget http://download-new.utorrent.com/endpoint/btsync/os/linux-glibc23-x64/track/stable
pathconf: Value too large for defined data type
pathconf: Value too large for defined data type
--2014-08-27 16:00:25--  http://download-new.utorrent.com/endpoint/btsync/os/linux-glibc23-x64/track/stable
Resolving download-new.utorrent.com... 67.215.238.66
Connecting to download-new.utorrent.com|67.215.238.66|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3864813 (3.7M) [application/octet-stream]
Saving to: 'stable'

100%[=====================================================================================================>] 3,864,813   1.15MB/s   in 3.2s

2014-08-27 16:00:29 (1.15 MB/s) - 'stable' saved [3864813/3864813]

ls -ltr
-rw-r--r--    1 btsync   users      3864813 Aug 26 03:30 stable
# file called stable is not downloaded

# This is a tar file so unpack it

tar -tvf stable
-rw-rw-r-- jenkins/jenkins 227 2014-08-25 10:53 LICENSE.TXT
-rwxrwxr-x jenkins/jenkins 6662048 2014-08-25 11:00 btsync
tar -xvpf stable
LICENSE.TXT
btsync
ls -ltr
-rw-rw-r--    1 btsync   users          227 Aug 25 10:53 LICENSE.TXT
-rwxrwxr-x    1 btsync   users      6662048 Aug 25 11:00 btsync
-rw-r--r--    1 btsync   users      3864813 Aug 26 03:30 stable
pwd
/volume2/homes/btsync/bin

  • Create  a configuration file then edit it

./btsync --dump-sample-config > ../conf/sync.conf
cat ../conf/sync.conf
{
  "device_name": "My Sync Device",
  "listening_port" : 0, // 0 - randomize port

/* storage_path dir contains auxilliary app files if no storage_path field: .sync dir created in the directory
   where binary is located. otherwise user-defined directory will be used */
// "storage_path" : "/home/user/.sync",

/* set location of pid file */
// "pid_file" : "/var/run/btsync/btsync.pid",

/* use UPnP for port mapping */
  "use_upnp" : true,

/* limits in kB/s. 0 - no limit */
  "download_limit" : 0,
  "upload_limit" : 0,

/* proxy configuration */
// "proxy_type" : "socks4", // Valid types: "socks4", "socks5", "http_connect". Any other value means no proxy
// "proxy_addr" : "192.168.1.2", // IP address of proxy server.
// "proxy_port" : 1080,
// "proxy_auth" : false, // Use authentication for proxy. Note: only username/password for socks5 (RFC 1929) is supported, and it is not really secure
// "proxy_username" : "user",
// "proxy_password" : "password",

  "webui" :
  {
    "listen" : "0.0.0.0:8888" // remove field to disable WebUI

/* preset credentials. Use password or password_hash */
//  ,"login" : "admin"
//  ,"password" : "password"
//  ,"password_hash" : "some_hash" // password hash in crypt(3) format

/* ssl configuration */
//  ,"force_https" : true // disable http
//  ,"ssl_certificate" : "/path/to/cert.pem"
//  ,"ssl_private_key" : "/path/to/private.key"

/* directory_root path defines where the WebUI Folder browser starts (linux only). Default value is / */
//  ,"directory_root" : "/home/user/MySharedFolders/"

/* dir_whitelist defines which directories can be shown to user or have folders added (linux only)
   relative paths are relative to directory_root setting */
//  ,"dir_whitelist" : [ "/home/user/MySharedFolders/personal", "work" ]
  }

/* !!! if you set shared folders in config file WebUI will be DISABLED !!!
   shared directories specified in config file  override the folders previously added from WebUI. */
/*,
  "shared_folders" :
  [
    {
      "secret" : "MY_SECRET_1", // required field - use --generate-secret in command line to create new secret
      "dir" : "/home/user/bittorrent/sync_test", // * required field
      "use_relay_server" : true, //  use relay server when direct connection fails
      "use_tracker" : true,
      "use_dht" : false,
      "search_lan" : true,
      "use_sync_trash" : true, // enable SyncArchive to store files deleted on remote devices
      "overwrite_changes" : false, // restore modified files to original version, ONLY for Read-Only folders
      "known_hosts" : // specify hosts to attempt connection without additional search
      [
        "192.168.1.2:44444"
      ]
    }
  ]
*/

/* Advanced preferences can be added to config file. Info is available at http://sync-help.bittorrent.com */

}

  • Edit the file sync.conf


  "webui" :
  {
    "listen" : "0.0.0.0:1234" // remove field to disable WebUI
 

/* preset credentials. Use password or password_hash */
  ,"login" : "admins"
  ,"password" : "put_ascii_password here"
  ,"password_hash" : "put_password hash here" // password hash in crypt(3) format


Hint:  Create a user in Synology Control panel GUI with the password that you want then look into /etc/shadow


  • Now create an automatic startup file in /etc/rc.local

cat /etc/rc.local

#!/bin/sh
cd /volume2/homes/btsync
nohup su -c "/volume2/homes/btsync/bin/btsync --config /volume2/homes/btsync/conf/sync.conf" btsync



  • Make it executable

chmod 755 /etc/rc.local 
chown root.root /etc/rc.local


  • Shutdown Synology and reboot and check that it started up automatically

# from ssh logon as root I see
ps -w | grep bts
 8393 root      3816 S    grep bts
11019 btsync    466m S    /volume2/homes/btsync/bin/btsync --config /volume2/homes/btsync/conf/sync.conf



  • Start GUI from your controlling PC and access  btsync


Goto the Web address and port that you configured  (mine shown above are clearly faked for my security!).  If you are asked for a userid first use the userid and password from sync.conf that you created. (permission to use the Gui logon),   if you get asked a second time it needs btsync, the name of the UNIX level user.

Then you can add one or more directories on this Synology server, read writable to user btsync that can be served out i.e. synchronised to remote users.   In the above picture I created a Synology directory called  DOS  and a Torrent share called dos and in it put some 16 bit DOS programs that I am testing on some old Windows computers. 

 Backup Suitability?
BitTorrent Sync creates an online mirror of the source on one more targets.  So watch out!  If you accidentally delete a file on a source, or a read/write target mirror then the deletion propagates instantly.   Be warned!

In other words this is NOT a great mechanism for backups because it does not protect against accidental deletion. Instead  I'd recommend SyncBack  (Windows) or Carbon Copy Cloner (OSX)

What a great use
Hmm let me see.  Suppose there are two friends living thousands of miles away who often share technical files.  We each have a small server running BitTorrent Sync, I read and write to C:\share\marcus and read from my friends area which is C:\share\bob,  he sees these directories at his end via OSX as /users/Bob/share/marcus and /users/Bob/share/bob.  Ah wonderful!



Links
Wikipedia BitTorrent Sync