write generic function that calls generic functions in scala -
i'm using spark datasets read in csv files. wanted make polymorphic function number of files. here's function:
def loadfile[m](file: string):dataset[m] = { import spark.implicits._ val schema = encoders.product[m].schema spark.read .option("header","false") .schema(schema) .csv(file) .as[m] } the errors are:
[error] <myfile>.scala:45: type arguments [m] not conform method product's type parameter bounds [t <: product] [error] val schema = encoders.product[m].schema [error] ^ [error] <myfile>.scala:50: unable find encoder type stored in dataset. primitive types (int, string, etc) , product types (case classes) supported importing spark.implicits._ support serializing other types added in future releases. [error] .as[m] [error] ^ [error] 2 errors found i don't know first error. tried adding same variance product definition (m <: product), error "no typetag available m"
if pass in schema produced encoder, error:
[error] unable find encoder type stored in dataset
you need require calling loadfile[m] provide evidence there such encoder m. can using context bounds on m requires encoder[m]:
def loadfile[m : encoder](file: string): dataset[m] = { import spark.implicits._ val schema = implicitly[encoder[m]].schema spark.read .option("header","false") .schema(schema) .csv(file) .as[m] }
Comments
Post a Comment