当前位置: 动力学知识库 > 问答 > 编程问答 >

mongodb - Save rdd into mongo database using java

问题描述:

I am attempting to save tweets in MongoDB using Java, This is what I have;

JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(

3000));

JavaDStream<Status> tweets = TwitterUtils.createStream(ssc);

JavaDStream<String> statuses = tweets

.map(new Function<Status, String>() {

public String call(Status status) {

return status.getUser().getName() + ":"

+ status.getText();

}

});

JavaDStream<String> users = tweets.map(new Function<Status, String>() {

public String call(Status status) {

return status.getUser().getName();

}

});

users.foreachRDD(new Function<JavaRDD<String>, Void>() {

public Void call(JavaRDD<String> rdd) throws Exception {

if (rdd.count() > 0)

rdd.saveAsTextFile("storage/users/test" + rdd.id()

+ "_.txt");

return null;

}

});

Like you see I can store users in text file using rdd.saveAsTextFile but what I need is a way to save this rdd to a data base (MongoDB).

网友答案:

You can use the MongoDB Hadoop Connector to store save an RDD to MongoDB using com.mongodb.hadoop.MongoOutputFormat:

Configuration config = new Configuration();
config.set("mongo.output.format", "com.mongodb.hadoop.MongoOutputFormat");
config.set("mongo.output.uri", "mongodb://host:port/database.collection");
rdd.saveAsNewAPIHadoopFile("file://this-is-not-used",
                           <keyClass>,
                           <valueClass>,
                           MongoOutputFormat.class,
                           config);

It may also be helpful to look at an example project for how to do this.

分享给朋友:
您可能感兴趣的文章:
随机阅读: