Featured image of post Golang Snippet for SFTP Client
Golang

Golang Snippet for SFTP Client

Learn Golang From Example

Motivation

During my work in Geniox mobile, I face responsibility to maintain Golang service that connects to the SFTP server. The existing code is pretty advanced, therefore I decided to learn from basic, main source of Golang package (https://pkg.go.dev/). Moreover, It’s better for us to understand the fundamentals of File Transfer Protocol before jumping to the code.

FTP Concept

Fundamental FTP is very important to understand. If somehow you should use other programming languages or face other problems that force you to interact with SFTP, this information will help you since the concept remains the same.

What is SFTP

SFTP is stand for SSH File Transfer Protocol (or sometimes called ‘Secure’ File Tranfer Protocol). In ordinary FTP, file tranfered without encryption, making data readable for anyone who intercepts it. While this is fine if you’re just sending unimportant files, this could lead to major data compromises if you’re sending crucial data (ftptoday).

Advantages of using (s)FTP

  1. It allows you to transfer multiple files and folders
  2. When the connection is lost then it has the ability to resume the transfer
  3. There is no limitation on the size of the file to be transferred. The browsers allow a transfer of only up to 2GB.
  4. Many FTP clients like FileZilla have the ability to schedule the transfers.
  5. The data transfer is faster than HTTP.
  6. The items that are to be uploaded or downloaded are added to the ‘queue’. The FTP client can add items to the ‘queue’.

source: afteracademy

Glossary

SSH Key

An SSH key is a secure access credential used in Secure Shell (SSH) protocol. SSH keys use key pairs based on public key infrastructure (PKI) technology, the gold standard for digital identity authentication and encryption, to provide a secure and scalable method of authentication. (source: sectigo)

Public Key and Private Key

An SSH key relies upon the use of two related keys, a public key and a private key, that together create a key pair that is used as the secure access credential. The private key is secret, known only to the user, and should be encrypted and stored safely. The public key can be shared freely with any SSH server to which the user wishes to connect. (source: sectigo)

Uploading

Transferring files from a client computer to a server computer (source: deskshare)

Downloading

Transferring files from a server to a client (source: deskshare)

Control Connection

The FTP client, for example, FileZilla sends a connection request usually to server port number 21. Typically a user needs to log on to the FTP server for estabilishing the connection but there are some servers that make all their content available without login. These servers are known as anonymous FTP

Data Connection

For transferring the files and folder we use a separate connection called data connection (source: afteracademy)

known_hosts

a client file containing all remotely connected hosts, and the ssh client uses this file. This file authenticates for the client to the server they are connecting to. The known_hosts file contains the host public key for all known hosts. Usually this file created automatically when client try to connect server at the first time ssh user@host or ssh user@host -p custom_port (note: you’re only be able to connect after creating ssh key e.g. ssh-keygen -t rsa). To remove host public key, we can use ssh-keygen -R "hostname". (source: linuxhint, howtouselinux, serverfault).

authorized_keys

a server file that houses are all SSH authentication keys that were copied to the server, from remote clients. Usually this file is located in ~/home/username/.ssh/. The ssh id can be copied by running ssh-copy-id user@host if ssh id located in the default folder, usually $HOME/.ssh/id_rsa.pub. If not, you can use ssh-copy-id path/to/id_rsa.pub user@host

Steps to generate SSH key

  1. Client generate the key using command ssh-keygen -t rsa
  2. Enter location in which to save the keys. Typically, the keys stored in ~/home/username/.ssh/ in Linux or ~/Users/username/.ssh/ in Mac.
  3. Enter in an optional passphrase or leave empty for no passphrase.
  4. Once the pair is generated, the next step is to put the public one on the remote server using command ssh-copy-id command.

(source: sectigo)

Golang Snippet

I combine example code in the sftp here and ssh here and also a little help from stackoverflow here

SFTP Example

This example is written as package sftp. The code I copied directly from here

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

...
func main() {
	var conn *ssh.Client

	// open an SFTP session over an existing ssh connection.
	client, err := sftp.NewClient(conn)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// walk a directory
	w := client.Walk("/home/user")
	for w.Step() {
		if w.Err() != nil {
			continue
		}
		log.Println(w.Path())
	}

	// leave your mark
	f, err := client.Create("hello.txt")
	if err != nil {
		log.Fatal(err)
	}
	if _, err := f.Write([]byte("Hello world!")); err != nil {
		log.Fatal(err)
	}
	f.Close()

	// check it's there
	fi, err := client.Lstat("hello.txt")
	if err != nil {
		log.Fatal(err)
	}
	log.Println(fi)
}

On the line number #3, a SSH connection is estabilished. Then the connection is used in the next code ln #6. The code for generating this SSH connection will be explained on the next session.

SSH Example

To get SSH connection, we should take a look at other examples from pkg.go.dev. There are two ways of example how to make SSH connection, password (example-dial) and public key (example-PublicKeys). I combine both auth method and get the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

...
func main() {
	var hostKey ssh.PublicKey
	// An SSH client is represented with a ClientConn.
	//
    // A public key may be used to authenticate against the remote
	// server by using an unencrypted PEM-encoded private key file.
	//
	// If you have an encrypted private key, the crypto/x509 package
	// can be used to decrypt it.
	key, err := ioutil.ReadFile("/home/user/.ssh/id_rsa")
	if err != nil {
		log.Fatalf("unable to read private key: %v", err)
	}
	// Create the Signer for this private key.
	signer, err := ssh.ParsePrivateKey(key)
	if err != nil {
		log.Fatalf("unable to parse private key: %v", err)
	}
	// To authenticate with the remote server you must pass at least one
	// implementation of AuthMethod via the Auth field in ClientConfig,
	// and provide a HostKeyCallback.
	config := &ssh.ClientConfig{
		User: "username",
		Auth: []ssh.AuthMethod{
			ssh.Password("yourpassword"),
            // Use the PublicKeys method for remote authentication.
			ssh.PublicKeys(signer),
		},
		HostKeyCallback: ssh.FixedHostKey(hostKey),
	}
	client, err := ssh.Dial("tcp", "yourserver.com:22", config)
	if err != nil {
		log.Fatal("Failed to dial: ", err)
	}
	defer client.Close()

	// Each ClientConn can support multiple interactive sessions,
	// represented by a Session.
	session, err := client.NewSession()
	if err != nil {
		log.Fatal("Failed to create session: ", err)
	}
	defer session.Close()

	// Once a Session is created, you can execute a single command on
	// the remote side using the Run method.
	var b bytes.Buffer
	session.Stdout = &b
	if err := session.Run("/usr/bin/whoami"); err != nil {
		log.Fatal("Failed to run: " + err.Error())
	}
	fmt.Println(b.String())
}

The auth method using PublicKey doesn’t require password but the server should have public key of client on server’s authorized_keys.

Getting Host PublicKey

From stackoverflow, I get this lines

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

func getHostKey(host string) (ssh.PublicKey, error) {
    file, err := os.Open(filepath.Join(os.Getenv("HOME"), ".ssh", "known_hosts"))
    if err != nil {
        return nil, err
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    var hostKey ssh.PublicKey
    for scanner.Scan() {
        fields := strings.Split(scanner.Text(), " ")
        if len(fields) != 3 {
            continue
        }
        if strings.Contains(fields[0], host) {
            var err error
            hostKey, _, _, _, err = ssh.ParseAuthorizedKey(scanner.Bytes())
            if err != nil {
                return nil, errors.New(fmt.Sprintf("error parsing %q: %v", fields[2], err))
            }
            break
        }
    }

    if hostKey == nil {
        return nil, errors.New(fmt.Sprintf("no hostkey for %s", host))
    }
    return hostKey, nil
}

Entire Code

Mixing up the code together, finally I get this code running and hit the SFTP server.

Next Question

Still, I have some of question in my mind regarding SFTP and SSH.
Firstly, How if someone steal the known_hosts file and put it onto his/her laptop and try to access the server. Could he/she impersonante the origin client and access the server?. Second, Is there anyway to make our the known_hosts safer? by adding salt or hash perhaps, so the thief couldn’t see what hostname server connecting to the client.

References:

SFTP
Licensed under CC BY-NC-SA 4.0
comments powered by Disqus
© 2020 - 2022 Coding Notes
Built with Hugo
Theme Stack designed by Jimmy