Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Another way to implement map reduce in Python is to use the Hadoop Streaming Jar (http://hadoop.apache.org/common/docs/r0.20.1/streaming.html). You basically write a mapper and reducer as python scripts and ship them out with the job in a tarball. The scripts just need to read input from stdin and write to stdout. You then pass arguments to the streaming jar when you kick off the job with the shell command to invoke your python scripts for the map and reduce tasks.


Yes, for Python you can use Dumbo for this (http://yz.mit.edu/wp/no-nonsense-standalone-hadoop-and-dumbo...).




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: