Syncing a Local Directory With a Remote Directory via FTP

Overview

We can use the following Sequence Diagram to depict the process of modifying the Files within a Local Directory and a Remote Directory before syncing the two Directories via the File Transfer Protocol (FTP), a standard communication protocol used on computer networks to send files between a Client and a Server.

sequenceDiagram
    loop
        Machine With the FTP Client ->> Machine With the FTP Client: Modify Files

        Machine With the FTP Server ->> Machine With the FTP Server: Modify Files

        Machine With the FTP Server ->> Machine With the FTP Server: Start FTP Server

        Machine With the FTP Client ->> Machine With the FTP Client: Start FTP Client

        Machine With the FTP Client ->> Machine With the FTP Server: Connect to FTP Server

        Machine With the FTP Client ->> Machine With the FTP Server: Mirror Remote Directory -> Local Directory
        Machine With the FTP Server -->> Machine With the FTP Client: Update Local Directory With Changed Files in Remote Directory

        Machine With the FTP Client ->> Machine With the FTP Server: Reverse Mirror Local Directory -> Remote Directory
        Machine With the FTP Client -->> Machine With the FTP Server: Update Remote Directory With Changed Files in Local Directory
    end

Setting Up the Machine With the FTP Client

Obviously, we need to install an FTP Client on this Machine. Personally, I recommend installing lftp, a "sophisticated" file transfer program. Unlike a standard FTP Client, which only enables you to upload or download files, lftp additionally enables you to maintain file synchronisation using its built-in mirror command.

Setting Up the Machine With the FTP Server

We would also need an FTP Server on the other Machine. Although there are many existing UNIX FTP servers, such as proftpd and vsftpd, they are usually tricky to compile, configure, and set up and require Root Privileges to run, which is tedious, if not impossible, in many situations.

As an alternative, we write our own FTP Server using pyftpdlib, a pure Python FTP server library written which offers a high-level interface to creating portable and efficient FTP servers. Such a solution requires us to have a Python environment running on the Machine With the FTP Server and install pyftpdlib, which is very simple in today's world where the Python ecosystem is ubiquitous.

After setting up a Python environment and installing pyftpdlib, we can write a script for an FTP server. Below is the script that I am using. By default, it sets up a user user with password 12345 and listens on port 2121 of the Machine With the FTP Server's Outbound IP Address, but these settings can all be tweaked by providing command-line arguments.

#!/usr/bin/env python3

import argparse
import os
import socket

from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer


def get_outbound_ip_address():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        # doesn't even have to be reachable
        s.connect(('10.255.255.255', 1))
        IP = s.getsockname()[0]
    except Exception:
        IP = '127.0.0.1'
    finally:
        s.close()
    return IP


def parse_command_line_arguments():
    parser = argparse.ArgumentParser()

    parser.add_argument('--user', type=str, required=False, default='user', help='Define a user having full read/write permissions')
    parser.add_argument('--password', type=str, required=False, default='12345', help='The password of the user having full read/write permissions')
    parser.add_argument('--anonymous', action='store_true', required=False, default=False, help='Add read-only anonymous user')
    parser.add_argument('--root', type=str, required=False, default=os.getcwd(), help='Root directory in FTP server')
    parser.add_argument('--host', type=str, required=False, default=get_outbound_ip_address(), help='Host to listen on')
    parser.add_argument('--port', type=int, required=False, default=2121, help='Port to listen on')

    args = parser.parse_args()

    return args.user, args.password, args.anonymous, args.root, args.host, args.port


def main():
    # Parse command line arguments
    user, password, anonymous, root, host, port = parse_command_line_arguments()

    # Instantiate a dummy authorizer for managing 'virtual' users
    authorizer = DummyAuthorizer()

    # Define a new user having full r/w permissions
    authorizer.add_user(user, password, root, perm='elradfmwMT')

    # Add anonymous user
    if anonymous:
        authorizer.add_anonymous(root)

    # Instantiate FTP handler class
    handler = FTPHandler
    handler.authorizer = authorizer

    # Define a customized banner (string returned when client connects)
    handler.banner = "pyftpdlib based ftpd ready."

    # Instantiate FTP server class and listen on <host>:<port>
    server = FTPServer((host, port), handler)

    # Start FTP server
    server.serve_forever()


if __name__ == '__main__':
    main()

Save this to a file, and run chmod +x on the file to make it executable.

abbas@abbas-ThinkPad-X1-Carbon-Gen-9:~$ ./pyftpd -h
usage: pyftpd [-h] [--user USER] [--password PASSWORD] [--anonymous] [--root ROOT] [--host HOST] [--port PORT]

optional arguments:
  -h, --help           show this help message and exit
  --user USER          Define a user having full read/write permissions
  --password PASSWORD  The password of the user having full read/write permissions
  --anonymous          Add read-only anonymous user
  --root ROOT          Root directory in FTP server
  --host HOST          Host to listen on
  --port PORT          Port to listen on
abbas@abbas-ThinkPad-X1-Carbon-Gen-9:~$ ./pyftpd 
[I 2023-02-16 15:47:39] >>> starting FTP server on 10.43.111.144:2121, pid=11969 <<<
[I 2023-02-16 15:47:39] concurrency model: async
[I 2023-02-16 15:47:39] masquerade (NAT) address: None
[I 2023-02-16 15:47:39] passive ports: None

Demonstration

Last but not least, we will present a demonstration of syncing a Local Directory With a Remote Directory via FTP.

Our Local Directory is a directory named mirror_ubuntu with two files mirror_ubuntu_1.txt and mirror_ubuntu_2.txt on the Machine With the FTP Client.

Our Remote Directory is a directory named mirror_ipad with two files mirror_ipad_1.txt and mirror_ipad_2.txt on the Machine With the FTP Server, which is the iSH app running within an iPad.

We start the FTP Server on the Machine With the FTP Server, and start the FTP Client on the Machine With the FTP Client.

As depicted in Overview, we first Mirror Remote Directory to Local Directory, which can be accomplished by running mirror --continue --no-perms <Remote Directory> <Local Directory> within lftp.

Mirroring Remote Directory to Local Directory

The Local Directory will now contain Files that were modified the Remote Directory.

Local Directory After Mirroring Remote Directory to Local Directory

Afterwards, we Reverse Mirror Local Directory to Remote Directory, which can be accomplished by running mirror --continue --no-perms --reverse <Local Directory> <Remote Directory> within lftp.