With keyed data a common use case is grouping our data by key - for example, viewing all of a customer's order together. If our data is already keyed in the way we want, groupByKey() will group our data using the key in our RDD. On an RDD consisting of keys of type K and values of type V, we get back an RDD of type [K, Iterable[V]].
groupBy() works on unpaired data or data where we want to use a different condition besides equality on the current key. It takes a function that it applies to every element in the source RDD and uses the result to determine the key.
In addition to grouping data from a single RDD, we can group data sharing the same key from multiple RDDs using a function called cogroup(). This function over two RDDs sharing the same key type K, with the respective value types Vand W gives us back RDD[(K, (Iterable[V], Iterable[W]))]. If one of the RDDs doesn't have element for a given key that is present in the other RDD, the corresponding Iterable is simply empty. cogroup() gives us the power to group data from multiple RDDs. Below is a simple example:
cogroup() is used as a building block for the joins we discuss in next section.
Joins
Some of the most useful operations we get with keyed data comes from using it together with other keyed data. Joining data together is probably one of the most common operations on a pair RDD, and we have a full range of options including right and left outer joins, cross joins, and inner joins. The simple join operator is an inner join. Only keys that are present in both pair RDDs are output. When there are multiple values for the same key in one of the inputs the resulting pair RDD will have an entry for every possible pair of values with that key from the two input RDDs. A simple to understand this is by look at below example:
- Example 4-17. inner join in Python
Sometimes we don't need the key to be present in both RDDs to want it in our result. For example, if we were joining customer information with recommendations we might not want to drop customers if there were not any recommendations yet. leftOuterJoin(other) and rightOuterJoin(other) both join pair RDDs together by key, where one of the pair RDDs can be missing the key.
We can revisit Example 4-17 and do a leftOuterJoin(other) and a rightOuterJoin(other) between two pair RDDs we used to illustrate join() in below example:
- Example 4-18. leftOuterJoin() and rightOuterJoin()
Sorting Data
Having sorted data is quite useful in many cases, especially when you're producing downstream output. We can sort an RDD with key/value pairs provided that there is an ordering defined on the key. Once we have sorted our data, any subsequent call on the sorted data to collect() or save() will result in ordered data. Since we often want our RDDs in the reverse order, the sortByKey() function takes a parameter called ascending indicating whether we want it in ascending order (in default to true). Sometimes we want a different sort order entirely, and to support this we can provide our own comparison function. In below example, we will sort our RDD by converting the integers to strings and using the string comparison functions:
- Example 4-19. Custom sort order in Python, sorting integers as if strings
As with the transformations, all of the traditional actions available on the base RDD are also available on pair RDDs. Some additional actions are available on pair RDDs to take advantage of the key/value nature of the data; these are listed below:
- countByKey(): Count the number of elements for each key
- collectAsMap(): Collect the result as a map to provide easy lookup.
- lookup(key): Return all values associated with the provided key.
There are also multiple other actions on pair RDDs that save the RDD, which we will describe in Chapter 5.
Supplement
* Ch4. Working with Key/Value Pairs - Part1
* Ch4. Working with Key/Value Pairs - Part2
* Ch4. Working with Key/Value Pairs - Part3
沒有留言:
張貼留言