Lumen Cloud Guide to Access Object Storage as a Filesystem Using s3fs

Updated by Gavin Lai on Sep 27, 2021
Article Code: kb/1362

Table of contents

Overview

s3fs is a FUSE filesystem that allows you to mount a S3 compatible bucket as a local filesystem. It stores files natively and transparently in S3 compatible Objective Storage (i.e., you can use other programs to access the same files). Maximum file size=64GB (as of this article, limited by s3fs, not Lumen Cloud Object Storage). The limitations of s3fs is listed here. For pricing of Object Storage, please see the pricing page.

Prerequisites

  • Access to the Lumen Cloud platform as an authorized user

  • Identify a CentOS Linux platform that you would like to have s3fs installed

  • Understanding the functions of Object Storage (beyond the scope of this article), to learn more, please see this knowledge article

  • Ability to install packages on the server

  • Access to Lumen Cloud Object Storage

Use Case Scenarios

As Object Storage is accessed using the Internet, it can be used in many scenarios. While mounting Object Storage as a Linux file system makes consuming Object Storage much easier, keep in mind that it is only accessed via the Internet, hence the speed and latency is not the same as local storage. With this in mind, using Object Storage in a file system can be a good alternative to expensive local storage. Use cases can be:

  • Data archiving

  • Backup

  • Internet speed data storage

  • Long term storage

If encryption is a requirement, please see s3QL, files created with S3QL can only be accessed using S3QL.

Preparation

In preparation. There are several factors need to be considered:

  • Deployment of s3fs will install Git, gcc along with Make and Fuse (for full list of packages, please refer to s3fs GitHub site)

  • Mountpoint for the Object Storage filesystem

Deployment

There are two methods to deploy s3fs in Lumen Cloud, it can be done manually or using a package.

Manual deployment

  1. s3fs deployment is done with the following commands on the CentOS 7 server:

    sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
    git clone https://github.com/s3fs-fuse/s3fs-fuse.git
    cd s3fs-fuse
    ./autogen.sh
    ./configure
    make
    sudo make install
    
  2. Once installed, access to the Object Storage screen in the Control Portal

    s3fs-objectstorage

  3. Locate the user and bucket for this deployment and record both the "Access Key ID" and "Secret Access Key"

  4. Create a password file with the "Access Key ID" and "Secret Access Key" and make proper permission on the file, for example:

    echo access_key_id:secret_access_key > /path_to_password_file/password_s3fs
    chmod 666 /path_to_password_file/password_s3fs
    
  5. Once the above steps are completed, create a directory for the mount point of the new filesystem

Execute using a package

A script (S3fs on CentOS 7) is created to speed up the deployment of s3fs on Lumen Cloud. The detail of the package is available here. The package can be install via the portal, API or CLI.

For API, please refer to the Execute Package API.

The CLI deployment can be done following the command in the Lumen Cloud Guide to CLI.

When using the portal, this Knowledge article

In the portal, the package can be found by searching for s3fs:

Search Window

The parameters required for this script are:

s3fs parameters

Testing

  • Test the installation with the command below. For the current list of Object Storage end points, please see this knowledge article:

    s3fs mybucket:/ /path_to_mountpoint -o passwd_file=/path_to_password_file/password_s3fs -o url=https://canada.os.ctl.io/
    
  • If the command ran successfully, then a /etc/fstab entry can be inserted to mount the Object Storage bucket permanently, a fstab_entry file is created in the /root directory if the package is used (debug level set at info, it can be removed to minimized logging):

    s3fs#mybucket:/ /path_to_mountpoint fuse _netdev,dbglevel=info,allow_other,nodnscache,retries=5,url=https://canada.os.ctl.io/,passwd_file=/path_to_password_file/password_s3fs
    

    or

    mybucket:/ /path_to_mountpoint fuse.s3fs _netdev,dbglevel=info,allow_other,nodnscache,retries=5,url=https://canada.os.ctl.io/,passwd_file=/path_to_password_file/password_s3fs
    

Troubleshooting

  • Debug and run s3fs in the foreground can help during troubleshooting (-d x 2 to direct messages to STDOUT):

    s3fs mybucket:/ /path_to_mountpoint -o passwd_file=/path_to_password_file/password -o url=https://canada.os.ctl.io/ -d -d -f -o f2 -o curldbg
    

Support

  • For issues related to deploying s3fs or accessing the deployed software, please visit the s3fs Website
  • For issues related to cloud infrastructure (VM's, network, etc), or if you experience a problem deploying any Blueprint or Script Package, please open a Lumen Cloud Support ticket by emailing help@ctl.io or through the Lumen Cloud Support website.