Scala Example for FTPS server

This is an example of how to use Generic Connector with Apache Spark in order to process files stored in a FTPS server directory.

Load dependencies for Apache Spark 2.x



In [ ]:

    
%dep

z.load("alvsanand:spark-generic-connector:0.2.0-spark_2x-s_2.11")

Import dependencies



In [ ]:

    
import es.alvsanand.sgc.ftp.{FTPCredentials, FTPSlot}
import es.alvsanand.sgc.ftp.secure.{FTPSSgcConnectorFactory, FTPSParameters, KeystoreConfig}
import org.apache.spark.streaming.sgc._

Create the SgcConnectorParameters with the desired parameters

Using user and password authentication:



In [ ]:

    
val parameters = FTPSParameters("HOST", PORT, "DIRECTORY", FTPCredentials("USER", Option("PASSWORD")))

Using active mode:



In [ ]:

    
val parameters = FTPSParameters("HOST", PORT, "DIRECTORY", FTPCredentials("USER", Option("PASSWORD"), activeMode = true)

Using a keystore for client certificate connection:



In [ ]:

    
val parameters = FTPSParameters("HOST", PORT, "DIRECTORY", FTPCredentials("USER", Option("PASSWORD")),
    kconfig = Option(KeystoreConfig(Option("KEYSTORE_URL"), keystorePassword = Option("KEYSTORE_PASSWORD"))))

Using a keystore for client certificate connection and a truststore for validating the server certificate:



In [ ]:

    
val parameters = FTPSParameters("HOST", PORT, "DIRECTORY", FTPCredentials("USER", Option("PASSWORD")),
    kconfig = Option(KeystoreConfig(Option("KEYSTORE_URL"), keystorePassword = Option("KEYSTORE_PASSWORD"))),
    tconfig = Option(KeystoreConfig(Option("TRUSTSTORE_URL"), keystorePassword = Option("TRUSTSTORE_PASSWORD"))))

Create the RDD passing the SgcConnectorFactory and the parameters



In [ ]:

    
val rdd = sc.createSgcRDD(FTPSSgcConnectorFactory, parameters)

Use the RDD as desired



In [ ]:

    
rdd.partitions.map(_.asInstanceOf[SgcRDDPartition[FTPSlot]].slot)
rdd.take(10).foreach(println)