In the spark-shell, running collect() on wordCounts transforms it from an RDD to an Array = Array which itself can be sorted on the second field of each Tuple2 element using: Array.sortBy(_._2) The main difference in the output of the spark and python version of wordCount is that where spark outputs (word,3) python outputs (u'word', 3).įor more information on spark RDD methods see for python and for scala. Since python captures leading and trailing whitespace as data, strip() is inserted before splitting each line on spaces, but this is not necessary using spark-shell/scala. In order to sortbyKey in descending order its first arg should be 0. sortByKey(1, 1) \ # 1st arg configures ascending sort, 2nd configures 1 task reduceByKey(lambda a, b: a + b, 1) \ # last arg configures one reducer task WordCounts = file.flatMap(lambda line: line.strip().split(" ")) \ Here is the pyspark version demonstrating sorting a collection by value: file = sc.textFile("file:some_local_text_file_pathname") Using pyspark a python script very similar to the scala script shown above produces output that is effectively the same. (starting with part-00000) depending on the number of reducers configured for the job (1 output data file per reducer), a _SUCCESS file depending on if the job succeeded or not and. Its second argument is the number of tasks (equivilent to number of partitions) which is set to 1 for testing with a small input file where only one output data file is desired reduceByKey also takes this optional argument.Īfter this the wordCounts RDD can be saved as text files to a directory with saveAsTextFile(directory_pathname) in which will be deposited one or more part-xxxxx files In order to reverse the ordering of the sort use sortByKey(false,1) since its first arg is the boolean value of ascending. sortByKey(true, 1) // 1st arg configures ascending sort, 2nd arg configures one task map(item => item.swap) // interchanges position of entries in each tuple reduceByKey(_ + _, 1) // 2nd arg configures one task (same as number of partitions) Val wordCounts = file.flatMap(line => line.split(" ")) Val file = sc.textFile("some_local_text_file_pathname") Below is an example of how this is scripted in spark-shell: // this whole block can be pasted in spark-shell in :paste mode followed by D Using spark's scala API sorting before collect() can be done following eliasah's suggestion and using Tuple2.swap() twice, once before sorting and once after in order to produce a list of tuples sorted in increasing or decreasing order of their second field (which is named _2) and contains the count of number of words in their first field (named _1). With the spark API this approach provides the flexibility of writing the output in "raw" form where you want, such as to a file where it could be used as input for further processing.
X word sort driver#
The sorting usually should be done before collect() is called since that returns the dataset to the driver program and also that is the way an hadoop map-reduce job would be programmed in java so that the final output you want is written (typically) to HDFS.
0 kommentar(er)
