scala - Spark: OutOfMemory despite MEMORY_AND_DISK_SER -

-

i wrote piece of code reads multiple parquet files , caches them subsequent use. code looks simplified this

val data = sparkstartup.sqlcontext.read.parquet(...) data.setname(...).persist(storagelevel.memory_and_disk_ser).collect() map += data

the parquet file in total 11g. config application by:

 val sparkconfig = new sparkconf().setappname(...).setmaster("local[128]") sparkconfig.set("spark.serializer", "org.apache.spark.serializer.kryoserializer") sparkconfig.set("spark.kryoserializer.buffer.max", "512m"); sparkconfig.set("spark.kryoserializer.buffer", "256"); sparkconfig.set("spark.driver.maxresultsize", "0"); sparkconfig.set("spark.driver.memory", "9g");

i thought using memory_and_disk_ser, spark spill out disk if memory used. however, `java.lang.outofmemoryerror: java heap space errors @ 

at java.util.arrays.copyof(arrays.java:3230) @ java.io.bytearrayoutputstream.grow(bytearrayoutputstream.java:113) @ java.io.bytearrayoutputstream.ensurecapacity(bytearrayoutputstream.java:93) @ java.io.bytearrayoutputstream.write(bytearrayoutputstream.java:140) @ java.io.bufferedoutputstream.flushbuffer(bufferedoutputstream.java:82) @ java.io.bufferedoutputstream.write(bufferedoutputstream.java:126) @ com.esotericsoftware.kryo.io.output.flush(output.java:155) @ com.esotericsoftware.kryo.io.output.require(output.java:135) @ com.esotericsoftware.kryo.io.output.writeascii_slow(output.java:446) @ com.esotericsoftware.kryo.io.output.writestring(output.java:306) @ com.esotericsoftware.kryo.util.defaultclassresolver.writename(defaultclassresolver.java:105) @ com.esotericsoftware.kryo.util.defaultclassresolver.writeclass(defaultclassresolver.java:81) @ com.esotericsoftware.kryo.kryo.writeclass(kryo.java:472)

why this? start application -xmx9g -dspark.executor.memory=9g -dspark.executor.cores=3. files read before crashes, can see in sparkui parquet files takes 9x size when read memory.

it because calling collect() in driver application. returns array of data items, need fit memory.

you should instead work data rdd , map, reduce, group, etc large set of data desired result, , collect() smaller amount of data.


Comments

Post a Comment

Popular posts from this blog

java - Date formats difference between yyyy-MM-dd'T'HH:mm:ss and yyyy-MM-dd'T'HH:mm:ssXXX -

-
i trying parse date 2014-12-03t10:05:59.5646+08:00 using these 2 formats: yyyy-mm-dd't'hh:mm:ss yyyy-mm-dd't'hh:mm:ssxxx when parse using yyyy-mm-dd't'hh:mm:ss works fine, when parse yyyy-mm-dd't'hh:mm:ssxxx parseexception thrown. which correct format parse date , difference between these 2 formats? note : cannot use joda :( those valid formats: yyyy-mm-dd't'hh:mm:ss.sssz >>> e.g.: 2001-07-04t12:08:56.235-0700 yyyy-mm-dd't'hh:mm:ss.sssxxx >>> e.g.: 2001-07-04t12:08:56.235-07:00 edit: btw, "x" refer (iso 8601 time zone)
Read more

c# - Get rid of xmlns attribute when adding node to existing xml -

-
i have xml-file: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:d>some content</d> </a> </root> i need add new node in specific place, code is: var doc = xdocument.load(file); xnamespace ns2 = "namespace"; doc.element(ns2 + "root").element(ns2 + "a").element(ns2 + "c").addafterself( new xelement(ns2+"new", new xelement("new1", new xelement("new2","some content"), new xelement("new3", "some content")))); the output is: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:new> <new1 xmlns=""> <new2>some content</new2> ...
Read more