Jun 13, 2011

Apache Commons VFS for SFTP




In Java, using FTP to upload, download file is very easy. You can see this post: Upload Files to FTP Server using Java. Java API support FTP operation, but does not support SFTP operation.

Apache Commons VFS provides a single API for accessing various different file systems. It presents a uniform view of the files from various different sources, such as the files on local disk, on an HTTP server, or SFTP server.

Before start, Please download below library.

When download complete, please unzip and add .jar to class path.

First, create a method to concatenate Host name, Username, Password and Remote File Path in one String:
public static String createConnectionString(String hostName,
        String username, String password, String remoteFilePath) {
    // result: "sftp://user:123456@domainname.com/resume.pdf
    return "sftp://" + username + ":" + password + "@" + hostName + "/"
            + remoteFilePath;
}

Then, create a method to setup default SFTP config:
public static FileSystemOptions createDefaultOptions()
        throws FileSystemException {
    // Create SFTP options
    FileSystemOptions opts = new FileSystemOptions();

    // SSH Key checking
    SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
            opts, "no");

    // Root directory set to user home
    SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);

    // Timeout is count by Milliseconds
    SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 10000);

    return opts;
}

Write a upload file function:
public static void upload(String hostName, String username,
        String password, String localFilePath, String remoteFilePath) {

    File f = new File(localFilePath);
    if (!f.exists())
        throw new RuntimeException("Error. Local file not found");

    StandardFileSystemManager manager = new StandardFileSystemManager();

    try {
        manager.init();

        // Create local file object
        FileObject localFile = manager.resolveFile(f.getAbsolutePath());

        // Create remote file object
        FileObject remoteFile = manager.resolveFile(
                createConnectionString(hostName, username, password,
                        remoteFilePath), createDefaultOptions());

        // Copy local file to sftp server
        remoteFile.copyFrom(localFile, Selectors.SELECT_SELF);

        System.out.println("File upload success");
    } catch (Exception e) {
        throw new RuntimeException(e);
    } finally {
        manager.close();
    }
}

Now, We can test upload function is it work.
/*
 * args[0]: hostName 
 * args[1]: username 
 * args[2]: password 
 * args[3]:localFilePath 
 * args[4]: remoteFilePath
 */
public static void main(String[] args) {
    if (args.length < 5)
        throw new RuntimeException(
                "Error. Please enter "
                        + "args[0]: hostName, args[1]: username, args[2]: password, "
                        + "args[3]: localFilePath, args[4]: remoteFilePath.");

    upload(args[0], args[1], args[2], args[3], args[4]);
}

If upload file success, console will print "File upload success".

Download file function:
public static void download(String hostName, String username,
        String password, String localFilePath, String remoteFilePath) {

    StandardFileSystemManager manager = new StandardFileSystemManager();

    try {
        manager.init();

        String downloadFilePath = localFilePath.substring(0,
                localFilePath.lastIndexOf("."))
                + "_downlaod_from_sftp"
                + localFilePath.substring(localFilePath.lastIndexOf("."),
                        localFilePath.length());

        // Create local file object
        FileObject localFile = manager.resolveFile(downloadFilePath);

        // Create remote file object
        FileObject remoteFile = manager.resolveFile(
                createConnectionString(hostName, username, password,
                        remoteFilePath), createDefaultOptions());

        // Copy local file to sftp server
        localFile.copyFrom(remoteFile, Selectors.SELECT_SELF);

        System.out.println("File download success");
    } catch (Exception e) {
        throw new RuntimeException(e);
    } finally {
        manager.close();
    }
}

Check remote file is exist function:
public static boolean exist(String hostName, String username,
        String password, String remoteFilePath) {
    StandardFileSystemManager manager = new StandardFileSystemManager();

    try {
        manager.init();

        // Create remote object
        FileObject remoteFile = manager.resolveFile(
                createConnectionString(hostName, username, password,
                        remoteFilePath), createDefaultOptions());

        System.out.println("File exist: " + remoteFile.exists());

        return remoteFile.exists();
    } catch (Exception e) {
        throw new RuntimeException(e);
    } finally {
        manager.close();
    }
}

Delete remote file function:
public static void delete(String hostName, String username,
        String password, String remoteFilePath) {
    StandardFileSystemManager manager = new StandardFileSystemManager();

    try {
        manager.init();

        // Create remote object
        FileObject remoteFile = manager.resolveFile(
                createConnectionString(hostName, username, password,
                        remoteFilePath), createDefaultOptions());

        if (remoteFile.exists()) {
            remoteFile.delete();
            System.out.println("Delete remote file success");
        }
    } catch (Exception e) {
        throw new RuntimeException(e);
    } finally {
        manager.close();
    }
}

Complete program:
package sftp.sample;

import java.io.File;

import org.apache.commons.vfs.FileObject;
import org.apache.commons.vfs.FileSystemException;
import org.apache.commons.vfs.FileSystemOptions;
import org.apache.commons.vfs.Selectors;
import org.apache.commons.vfs.impl.StandardFileSystemManager;
import org.apache.commons.vfs.provider.sftp.SftpFileSystemConfigBuilder;

public class Main {

    /*
     * args[0]: hostName 
     * args[1]: username 
     * args[2]: password
     * args[3]: localFilePath 
     * args[4]: remoteFilePath
     */
    public static void main(String[] args) {
        if (args.length < 5)
            throw new RuntimeException(
                    "Error. Please enter "
                            + "args[0]: hostName, args[1]: username, args[2]: password, "
                            + "args[3]: localFilePath, args[4]: remoteFilePath.");

        upload(args[0], args[1], args[2], args[3], args[4]);
        exist(args[0], args[1], args[2], args[4]);
        download(args[0], args[1], args[2], args[3], args[4]);
        delete(args[0], args[1], args[2], args[4]);
    }

    public static void upload(String hostName, String username,
            String password, String localFilePath, String remoteFilePath) {

        File f = new File(localFilePath);
        if (!f.exists())
            throw new RuntimeException("Error. Local file not found");

        StandardFileSystemManager manager = new StandardFileSystemManager();

        try {
            manager.init();

            // Create local file object
            FileObject localFile = manager.resolveFile(f.getAbsolutePath());

            // Create remote file object
            FileObject remoteFile = manager.resolveFile(
                    createConnectionString(hostName, username, password,
                            remoteFilePath), createDefaultOptions());

            // Copy local file to sftp server
            remoteFile.copyFrom(localFile, Selectors.SELECT_SELF);

            System.out.println("File upload success");
        } catch (Exception e) {
            throw new RuntimeException(e);
        } finally {
            manager.close();
        }
    }

    public static void download(String hostName, String username,
            String password, String localFilePath, String remoteFilePath) {

        StandardFileSystemManager manager = new StandardFileSystemManager();

        try {
            manager.init();

            String downloadFilePath = localFilePath.substring(0,
                    localFilePath.lastIndexOf("."))
                    + "_downlaod_from_sftp"
                    + localFilePath.substring(localFilePath.lastIndexOf("."),
                            localFilePath.length());

            // Create local file object
            FileObject localFile = manager.resolveFile(downloadFilePath);

            // Create remote file object
            FileObject remoteFile = manager.resolveFile(
                    createConnectionString(hostName, username, password,
                            remoteFilePath), createDefaultOptions());

            // Copy local file to sftp server
            localFile.copyFrom(remoteFile, Selectors.SELECT_SELF);

            System.out.println("File download success");
        } catch (Exception e) {
            throw new RuntimeException(e);
        } finally {
            manager.close();
        }
    }

    public static void delete(String hostName, String username,
            String password, String remoteFilePath) {
        StandardFileSystemManager manager = new StandardFileSystemManager();

        try {
            manager.init();

            // Create remote object
            FileObject remoteFile = manager.resolveFile(
                    createConnectionString(hostName, username, password,
                            remoteFilePath), createDefaultOptions());

            if (remoteFile.exists()) {
                remoteFile.delete();
                System.out.println("Delete remote file success");
            }
        } catch (Exception e) {
            throw new RuntimeException(e);
        } finally {
            manager.close();
        }
    }

    public static boolean exist(String hostName, String username,
            String password, String remoteFilePath) {
        StandardFileSystemManager manager = new StandardFileSystemManager();

        try {
            manager.init();

            // Create remote object
            FileObject remoteFile = manager.resolveFile(
                    createConnectionString(hostName, username, password,
                            remoteFilePath), createDefaultOptions());

            System.out.println("File exist: " + remoteFile.exists());

            return remoteFile.exists();
        } catch (Exception e) {
            throw new RuntimeException(e);
        } finally {
            manager.close();
        }
    }

    public static String createConnectionString(String hostName,
            String username, String password, String remoteFilePath) {
        // result: "sftp://user:123456@domainname.com/resume.pdf
        return "sftp://" + username + ":" + password + "@" + hostName + "/"
                + remoteFilePath;
    }

    public static FileSystemOptions createDefaultOptions()
            throws FileSystemException {
        // Create SFTP options
        FileSystemOptions opts = new FileSystemOptions();

        // SSH Key checking
        SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
                opts, "no");

        // Root directory set to user home
        SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);

        // Timeout is count by Milliseconds
        SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 10000);

        return opts;
    }

}


11 comments:

Anonymous said...

very good work..thanks for your efforts

Prash

Nestor Urquiza said...

Code that just works. Just what the community needs.

Thanks for sharing this!

-Nestor

Anonymous said...

i integrate this into my code (upload operation) and it throws an exception at the following line

FileObject remoteFile = manager.resolveFile( remotePath, opts );

It says:

org.apache.commons.vfs.FileSystemException: Invalid absolute URI "sftp://username:***@170.40.164.100:22/input/pdf2.pdf".

I'm not sure what this means. I tried different locations, and i still get the same error.
Any thoughts on that?

Lawrence Cheung said...

If your config have this code:
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);

Assume file placed at "/home/user/input/pdf2.pdf"

Your path should be like that: "sftp://user:***@127.0.0.1:22/input/pdf2.pdf"

if without this code:
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, true);

Your path should be like that: "sftp://user:***@127.0.0.1:22/home/user/input/pdf2.pdf"

Anonymous said...

Hi,
can i upload directory also with above code

Anonymous said...

hey I am getting an error

org.apache.commons.vfs.FileSystemException: Could not delete "sftp://ploaduser:***@abcw22/ae".
at org.apache.commons.vfs.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:818)
at org.apache.commons.vfs.provider.AbstractFileObject.delete(AbstractFileObject.java:877)
at org.apache.commons.vfs.provider.AbstractFileObject.copyFrom(AbstractFileObject.java:1015)
at sftp.sample.SFTPCTest.upload(SFTPCTest.java:63)
at sftp.sample.SFTPCTest.main(SFTPCTest.java:30)
Caused by: 3: Permission denied
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2569)
at com.jcraft.jsch.ChannelSftp.rmdir(ChannelSftp.java:1942)
at org.apache.commons.vfs.provider.sftp.SftpFileObject.doDelete(SftpFileObject.java:253)
at org.apache.commons.vfs.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:807)
... 4 more

any ideas appreciated..

Satish Kotla said...

I am having the same issue. I have the JSCH jar in claspath. Any help is greatly appreciated. I am using vfs 1.0 and i am testing in mac as junit test.

org.apache.commons.vfs.FileSystemException: Invalid absolute URI "sftp://xx:xx@xx.xx.com/share/Details.xlsx".
at org.apache.commons.vfs.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:58)
at org.apache.commons.vfs.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:641)
at org.apache.commons.vfs.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:582)
at com.fb.service.file.util.SFTP.transferFileToDeloitte(SFTP.java:150)
at com.fb.service.impl.DeloitteTargetFileServiceImpl.encryptAndTransferFileToSFTP(DeloitteTargetFileServiceImpl.java:146)
at com.fb.service.impl.TransactionDetailsFileGenerator.generateTargetFileContent(TransactionDetailsFileGenerator.java:269)
at com.fb.service.impl.TransactionDetailsFileGenerator.execute(TransactionDetailsFileGenerator.java:40)
at com.fb.service.impl.SchwabFileProcessorTest.testGenerateTransactionDetailsFile(SchwabFileProcessorTest.java:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)

Expecting / to follow the hostname in URI "sftp://
at org.apache.commons.vfs.provider.HostFileNameParser.extractToPath(HostFileNameParser.java:139)
at org.apache.commons.vfs.provider.URLFileNameParser.parseUri(URLFileNameParser.java:48)
at org.apache.commons.vfs.provider.AbstractFileProvider.parseUri(AbstractFileProvider.java:170)
at org.apache.commons.vfs.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:54)
... 26 more

Mike Levine said...

I was running into the same problem. Tracked it down to this code in HostFileNameParser. The problem is I had an '@' in my password which causes this to stop parsing early:

protected String extractUserInfo(final StringBuilder name)
{
final int maxlen = name.length();
for (int pos = 0; pos < maxlen; pos++)
{
final char ch = name.charAt(pos);
if (ch == '@')
{
// Found the end of the user info
String userInfo = name.substring(0, pos);
name.delete(0, pos + 1);
return userInfo;
}
if (ch == '/' || ch == '?')
{
// Not allowed in user info
break;
}
}

// Not found
return null;

Anonymous said...

@Mike Levine: Thank you! I had the same problem

Anonymous said...

when i try to upload file from local machine to remote directory it replaces the exisiting remote directory and copies the content from local machine file but not the file.

Anonymous said...

Where can we use .key file and .cert file while downloading file form FTPS .

Post a Comment

Twitter Delicious Facebook Digg Google Favorites More

 
Design by Free WordPress Themes | Bloggerized by Lasantha - Premium Blogger Themes | Best Hostgator Coupon Code