write generic function that calls generic functions in scala -


i'm using spark datasets read in csv files. wanted make polymorphic function number of files. here's function:

def loadfile[m](file: string):dataset[m] = {     import spark.implicits._     val schema = encoders.product[m].schema     spark.read       .option("header","false")       .schema(schema)       .csv(file)       .as[m] } 

the errors are:

[error] <myfile>.scala:45: type arguments [m] not conform method product's type parameter bounds [t <: product] [error]     val schema = encoders.product[m].schema [error]                                  ^ [error] <myfile>.scala:50: unable find encoder type stored in dataset.  primitive types (int, string, etc) , product types (case classes) supported importing spark.implicits._  support serializing other types added in future releases. [error]       .as[m] [error]          ^ [error] 2 errors found 

i don't know first error. tried adding same variance product definition (m <: product), error "no typetag available m"

if pass in schema produced encoder, error:

[error] unable find encoder type stored in dataset  

you need require calling loadfile[m] provide evidence there such encoder m. can using context bounds on m requires encoder[m]:

def loadfile[m : encoder](file: string): dataset[m] = {   import spark.implicits._   val schema = implicitly[encoder[m]].schema   spark.read    .option("header","false")    .schema(schema)    .csv(file)    .as[m] } 

Comments

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

html - How to custom Bootstrap grid height? -

transpose - Maple isnt executing function but prints function term -