aws sdk - Why can't I load data from Amazon S3 in DSX notebook? -


i used following code load data amazon s3:

from ingest import connectors pyspark.sql import sqlcontext sqlcontext = sqlcontext(sc)  s3loadoptions = {                    connectors.amazons3.access_key          : 'akiajycjafzyennpacna',               connectors.amazons3.secret_key          : 'a6voqu3caccbfi0peqlkwqxkrquqyxqqnousondy',               connectors.amazons3.source_bucket       : 'ngpconnector',               connectors.amazons3.source_file_name    : 'addresses3.csv',               connectors.amazons3.source_infer_schema : '1',               connectors.amazons3.source_file_format  : 'csv'}  s3df = sqlcontext.read.format('com.ibm.spark.discover').options(**s3loadoptions).load() s3df.printschema() s3df.show(5)

but when run code snippet, following error. similar error message when load data source, such dashdb. 

attributeerrortraceback (most recent call last) <ipython-input-1-9da344857d7e> in <module>()       4        5 s3loadoptions = {  ----> 6                   connectors.amazons3.access_key          : 'akiajycjafzyennpacna',       7                   connectors.amazons3.secret_key          : 'a6voqu3caccbfi0peqlkwqxkrquqyxqqnousondy',       8                   connectors.amazons3.source_bucket       : 'ngpconnector',  attributeerror: 'nonetype' object has no attribute 'amazons3'

please use alternative ingest if like.

https://datascience.ibm.com/blog/use-ibm-data-science-experience-to-read-and-write-data-stored-on-amazon-s3/

for spark 1.6

hconf = sc._jsc.hadoopconfiguration()

for spark 2.0

spark = sparksession.builder.getorcreate() hconf = spark.sparkcontext._jsc.hadoopconfiguration()

set s3 parameters in hadoop configuration

#replace accesskey amazon accesskey , secret amazon secret  hconf.set("fs.s3a.access.key", "<put-your-access-key>")   hconf.set("fs.s3a.secret.key", "<put-your-secret-key>")

then read

spark = sparksession.builder.getorcreate()   df_data_1 = spark.read\    .format('org.apache.spark.sql.execution.datasources.csv.csvfileformat')\   .option('header', 'true')\   .load('s3a://<your-bucket-name>/<foldername>/<filename>.csv') df_data_1.take(5)

to write back

df_data_1.write.save("s3a://charlesbuckets31/folderb/users.parquet")

thanks, charles.


Comments

Post a Comment

Popular posts from this blog

python - Selenium remoteWebDriver (& SauceLabs) Firefox moseMoveTo action exception -

while performing simple hover test on firefox through selenium (3.4.0) python bindings (3.4.3) using of available os in saucelabs (except linux, latest available firefox version old), performing following actionchain: hover = actionchains(driver) hover.move_to_element(elm_men_menu).perform() it throws following error: taceback: file [...] in testhover hover.perform() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 80, in perform action() file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/action_chains.py", line 290, in <lambda> command.move_to, {'element': to_element.id})) file "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute self.error_handler.check_response(response) file "/usr/local/lib/python2.7/...
Read more

html - How to custom Bootstrap grid height? -

why bootstrap grid has height 591.067 ? did height came ? .main-post { padding: 20px; background-color: #fff; border: 1px solid #d8d8d8; margin-bottom: 100px; } .main-post .post-categories { margin-bottom: 10px } .main-post h3 { margin: 0 0 10px; color: #777; letter-spacing: -1px; font-weight: bold } .main-post .post-author, .main-post .post-date, .main-post .post-comments { font-size: 12px } .main-post img { display: block; margin: 10px 0; } .main-post .post-summary { line-height: 1.7; color: #888 } .main-post { color: #999 } <div class="col-sm-6"> <div class="main-post"> <h3 class="post-title"> <a href="#"> post title </a> </...
Read more

transpose - Maple isnt executing function but prints function term -

im using maple im in worksheetmode tried using maple input , 2d input , want transpose matrix a: := `<|>`(`<,>`(1, .5, -2), `<,>`(.5, 9/4+b, 5+3*b), `<,>`(-2, 5+3*b, 18+9*b+4*a)); b:= transpose(a); when exectute sheet dont translated values, there same input. matrix looks same input matrix plus function term. you can see picture in following link: why arent functions executed? meanwhile b:=a^+ doing right way , transposed matrix. other functions return function body instead needed values... if using 2d input mode (the default) space you've got between transpose , bracketed (b) interpreted multiplcation. rid of such space. also, either load package @ start of document like, with(linearalgebra): before calling transpose command package, or call it's full name like, linearalgebra:-transpose(b);
Read more