Quantcast

add Field to mongo db

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

add Field to mongo db

lsroudi
Hello nutchers !
i'm trying hard to build a plugin allow me to add some fields into database
(Mongo)
i use nutch 2.x
i update webpage.avsc with my customs fields like

{

      "name": "pageLength",

      "type": [

        "null",

        "string"

      ],

      "doc": "description",

      "default": null

    }

and also i add

<field name="pageLength" docfield="pageLength" type="string"/>

into gora-mongodb-mapping.xml

and finaly this my plugin code:

package org.apache.nutch.indexer;



import java.util.Collection;

import java.util.HashSet;


import org.apache.commons.logging.Log;

import org.apache.commons.logging.LogFactory;

import org.apache.hadoop.conf.Configuration;

import org.apache.nutch.indexer.IndexingFilter;

import org.apache.nutch.indexer.NutchDocument;

import org.apache.nutch.storage.WebPage;

import org.apache.nutch.storage.WebPage.Field;

import java.lang.Integer;



public class AddField implements IndexingFilter {



private static final Log LOG = LogFactory.getLog(AddField.class);

private Configuration conf;






//Boilerplate

public Configuration getConf() {

return conf;

}



//Boilerplate

public void setConf(Configuration conf) {

this.conf = conf;

}


@Override

public Collection<Field> getFields() {

// TODO Auto-generated method stub

return null;

}


@Override

public NutchDocument filter(NutchDocument doc, String url, WebPage page)
throws IndexingException {

// TODO Auto-generated method stub

String content = page.getText().toString();

//adds the new field to the document

String len = Integer.toString(content.length());

doc.add("pageLength", len);

return doc;

}

}


the problem is when i run

ant eclipse

the build is failed and this is the output error:

[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

  [taskdef] Could not load definitions from resource
org/sonar/ant/antlib.xml. It could not be found.


copy-libs:


compile-core:

    [javac] Compiling 257 source files to
/Users/lsroudiSpinergie/Documents/workspace/2.x/build/classes

    [javac] warning: [options] bootstrap class path not set in conjunction
with -source 1.7

    [javac]
/Users/lsroudiSpinergie/Documents/workspace/2.x/src/java/org/apache/nutch/parse/ParserChecker.java:112:
error: cannot find symbol

    [javac]     if (!protocolOutput.getStatus().isSuccess()) {

    [javac]                                    ^

    [javac]   symbol:   method isSuccess()

    [javac]   location: class ProtocolStatus

    [javac] Note: Some input files use unchecked or unsafe operations.

    [javac] Note: Recompile with -Xlint:unchecked for details.

    [javac] 1 error

    [javac] 1 warning

BUILD FAILED


any help would be appreciated
Loading...