Problems

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems

Michenux
Hi,
 
I have integrated nutch in my application.
 
In the admin part, i can launch the indexation.
In the client part, i can launch searches.
 
After a search,  i can't do an indexation again
because the index files are used by the NutchBean.
 
How can i do that please ?
 
thx
Reply | Threaded
Open this post in threaded view
|

Re: Problems

Thomas Delnoij-3
Unfortunately, in the 0.7 release, the NutchBean does not clean up properly
after itself, so some SegementReaders and IndexReaders remain open. I think
this is fixed in the current code line. I had similar problems in my app
based on 0.7 - all that helped was killing the processes blocking the index.

Rgrds, Thomas


On 3/13/06, Laurent Michenaud <[hidden email]> wrote:

>
> Hi,
>
> I have integrated nutch in my application.
>
> In the admin part, i can launch the indexation.
> In the client part, i can launch searches.
>
> After a search,  i can't do an indexation again
> because the index files are used by the NutchBean.
>
> How can i do that please ?
>
> thx
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Problems

Michenux
In reply to this post by Michenux
It would be interesting to have a fix for 0.7

-----Message d'origine-----
De : TDLN [mailto:[hidden email]]
Envoyé : mardi 14 mars 2006 12:32
À : [hidden email]
Objet : Re: Problems

Unfortunately, in the 0.7 release, the NutchBean does not clean up properly after itself, so some SegementReaders and IndexReaders remain open. I think this is fixed in the current code line. I had similar problems in my app based on 0.7 - all that helped was killing the processes blocking the index.

Rgrds, Thomas


On 3/13/06, Laurent Michenaud <[hidden email]> wrote:

>
> Hi,
>
> I have integrated nutch in my application.
>
> In the admin part, i can launch the indexation.
> In the client part, i can launch searches.
>
> After a search,  i can't do an indexation again because the index
> files are used by the NutchBean.
>
> How can i do that please ?
>
> thx
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Problems

Thomas Delnoij-3
I think so too, maybe as part of the upcoming 0.7.2 release.

In the meantime, you can try to implement the fix suggested in this post

http://www.mail-archive.com/nutch-user@.../msg03011.html

If it works, for you, please let me know, I need the fix as well :)

Rgrds, Thomas



On 3/14/06, Laurent Michenaud <[hidden email]> wrote:

>
> It would be interesting to have a fix for 0.7
>
> -----Message d'origine-----
> De : TDLN [mailto:[hidden email]]
> Envoyé : mardi 14 mars 2006 12:32
> À : [hidden email]
> Objet : Re: Problems
>
> Unfortunately, in the 0.7 release, the NutchBean does not clean up
> properly after itself, so some SegementReaders and IndexReaders remain open.
> I think this is fixed in the current code line. I had similar problems in my
> app based on 0.7 - all that helped was killing the processes blocking the
> index.
>
> Rgrds, Thomas
>
>
> On 3/13/06, Laurent Michenaud <[hidden email]> wrote:
> >
> > Hi,
> >
> > I have integrated nutch in my application.
> >
> > In the admin part, i can launch the indexation.
> > In the client part, i can launch searches.
> >
> > After a search,  i can't do an indexation again because the index
> > files are used by the NutchBean.
> >
> > How can i do that please ?
> >
> > thx
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: Problems

Michenux
In reply to this post by Michenux
Here where i am :

*** In the NutchBean

I have made the segments and localFileSystem
Variables as instance variables.

private FetchedSegments segments ;
private LocalFileSystem localFileSystem ;

In the init method :
localFileSystem = new LocalFileSystem();
segments = new FetchedSegments(localFileSystem,segmentsDir.toString());

And i've created the method closeSegment :

public void closeSegments() throws IOException {
        segments.closeSegments();
        localFileSystem.close();
}

** In the class FetchSegments

I have added the method :
public void closeSegments() throws IOException {
  Iterator iterSegments = segments.values().iterator();
  while( iterSegments.hasNext()) {
          Segment oSegment = (Segment) iterSegments.next();
          oSegment.close();
        }
}

** In the subclass FetchSegments.Segment

I have added the method :
public void close() throws IOException {
    if ( fetcher != null )
    fetcher.close();
    if ( content != null )
    content.close();
    if ( text != null )
    text.close();
    if ( parsedata != null )
    parsedata.close();
  }




My test program :
I instanciate the NutchBean.
I make a search.
I call oNutchBean.closeSegments()
Finally, I try to delete the index dir and i get :

java.io.IOException: Unable to delete file: D:\Projets\application-data\recherche\indexations\indexes-20060314-17h33.54\segments\20060314173333\parse_text\data
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:951)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:713)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:680)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:713)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:680)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:713)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:680)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:713)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:680)

So, it is not ok yet.


 

-----Message d'origine-----
De : TDLN [mailto:[hidden email]]
Envoyé : mardi 14 mars 2006 12:39
À : [hidden email]
Objet : Re: Problems

I think so too, maybe as part of the upcoming 0.7.2 release.

In the meantime, you can try to implement the fix suggested in this post

http://www.mail-archive.com/nutch-user@.../msg03011.html

If it works, for you, please let me know, I need the fix as well :)

Rgrds, Thomas



On 3/14/06, Laurent Michenaud <[hidden email]> wrote:

>
> It would be interesting to have a fix for 0.7
>
> -----Message d'origine-----
> De : TDLN [mailto:[hidden email]] Envoyé : mardi 14 mars 2006
> 12:32 À : [hidden email] Objet : Re: Problems
>
> Unfortunately, in the 0.7 release, the NutchBean does not clean up
> properly after itself, so some SegementReaders and IndexReaders remain open.
> I think this is fixed in the current code line. I had similar problems
> in my app based on 0.7 - all that helped was killing the processes
> blocking the index.
>
> Rgrds, Thomas
>
>
> On 3/13/06, Laurent Michenaud <[hidden email]> wrote:
> >
> > Hi,
> >
> > I have integrated nutch in my application.
> >
> > In the admin part, i can launch the indexation.
> > In the client part, i can launch searches.
> >
> > After a search,  i can't do an indexation again because the index
> > files are used by the NutchBean.
> >
> > How can i do that please ?
> >
> > thx
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Problems

Thomas Delnoij-3
That's about where I got stuck as well. I didn't have time to look into this
problem any further then. I think it's best to either check the trunk and
see how it is solved there or just to move to 0.8 altogether.

Rgrds, Thomas


On 3/14/06, Laurent Michenaud <[hidden email]> wrote:

>
> Here where i am :
>
> *** In the NutchBean
>
> I have made the segments and localFileSystem
> Variables as instance variables.
>
> private FetchedSegments segments ;
> private LocalFileSystem localFileSystem ;
>
> In the init method :
> localFileSystem = new LocalFileSystem();
> segments = new FetchedSegments(localFileSystem,segmentsDir.toString());
>
> And i've created the method closeSegment :
>
> public void closeSegments() throws IOException {
>        segments.closeSegments();
>        localFileSystem.close();
> }
>
> ** In the class FetchSegments
>
> I have added the method :
> public void closeSegments() throws IOException {
> Iterator iterSegments = segments.values().iterator();
> while( iterSegments.hasNext()) {
>          Segment oSegment = (Segment) iterSegments.next();
>          oSegment.close();
>        }
> }
>
> ** In the subclass FetchSegments.Segment
>
> I have added the method :
> public void close() throws IOException {
>        if ( fetcher != null )
>                fetcher.close();
>        if ( content != null )
>                content.close();
>        if ( text != null )
>                text.close();
>        if ( parsedata != null )
>                parsedata.close();
> }
>
>
>
>
> My test program :
> I instanciate the NutchBean.
> I make a search.
> I call oNutchBean.closeSegments()
> Finally, I try to delete the index dir and i get :
>
> java.io.IOException: Unable to delete file:
> D:\Projets\application-data\recherche\indexations\indexes-
> 20060314-17h33.54\segments\20060314173333\parse_text\data
>        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:951)
>        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> :713)
>        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> :680)
>        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
>        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> :713)
>        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> :680)
>        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
>        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> :713)
>        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> :680)
>        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
>        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> :713)
>        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> :680)
>
> So, it is not ok yet.
>
>
>
>
> -----Message d'origine-----
> De : TDLN [mailto:[hidden email]]
> Envoyé : mardi 14 mars 2006 12:39
> À : [hidden email]
> Objet : Re: Problems
>
> I think so too, maybe as part of the upcoming 0.7.2 release.
>
> In the meantime, you can try to implement the fix suggested in this post
>
> http://www.mail-archive.com/nutch-user@.../msg03011.html
>
> If it works, for you, please let me know, I need the fix as well :)
>
> Rgrds, Thomas
>
>
>
> On 3/14/06, Laurent Michenaud <[hidden email]> wrote:
> >
> > It would be interesting to have a fix for 0.7
> >
> > -----Message d'origine-----
> > De : TDLN [mailto:[hidden email]] Envoyé : mardi 14 mars 2006
> > 12:32 À : [hidden email] Objet : Re: Problems
> >
> > Unfortunately, in the 0.7 release, the NutchBean does not clean up
> > properly after itself, so some SegementReaders and IndexReaders remain
> open.
> > I think this is fixed in the current code line. I had similar problems
> > in my app based on 0.7 - all that helped was killing the processes
> > blocking the index.
> >
> > Rgrds, Thomas
> >
> >
> > On 3/13/06, Laurent Michenaud <[hidden email]> wrote:
> > >
> > > Hi,
> > >
> > > I have integrated nutch in my application.
> > >
> > > In the admin part, i can launch the indexation.
> > > In the client part, i can launch searches.
> > >
> > > After a search,  i can't do an indexation again because the index
> > > files are used by the NutchBean.
> > >
> > > How can i do that please ?
> > >
> > > thx
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Problems

Jack.Tang
Hi

I hacked luence index dynamicly update one year ago(I used quartz as
scheduler). And hope the idea helps.

In indexing phase:
1. write index into directory
2. write one flag, say "done" when indexing is completed
3. try to delete the index some days ago(say two days ago).

In search phase:
1. scan the search.dir and try to find the lastest indices with "done" flag
2. search it.

I remembered Doug posted some great idea in 2005 year, pls try to
search the maillist.

Good luck!
/Jack

On 3/15/06, TDLN <[hidden email]> wrote:

> That's about where I got stuck as well. I didn't have time to look into this
> problem any further then. I think it's best to either check the trunk and
> see how it is solved there or just to move to 0.8 altogether.
>
> Rgrds, Thomas
>
>
> On 3/14/06, Laurent Michenaud <[hidden email]> wrote:
> >
> > Here where i am :
> >
> > *** In the NutchBean
> >
> > I have made the segments and localFileSystem
> > Variables as instance variables.
> >
> > private FetchedSegments segments ;
> > private LocalFileSystem localFileSystem ;
> >
> > In the init method :
> > localFileSystem = new LocalFileSystem();
> > segments = new FetchedSegments(localFileSystem,segmentsDir.toString());
> >
> > And i've created the method closeSegment :
> >
> > public void closeSegments() throws IOException {
> >        segments.closeSegments();
> >        localFileSystem.close();
> > }
> >
> > ** In the class FetchSegments
> >
> > I have added the method :
> > public void closeSegments() throws IOException {
> > Iterator iterSegments = segments.values().iterator();
> > while( iterSegments.hasNext()) {
> >          Segment oSegment = (Segment) iterSegments.next();
> >          oSegment.close();
> >        }
> > }
> >
> > ** In the subclass FetchSegments.Segment
> >
> > I have added the method :
> > public void close() throws IOException {
> >        if ( fetcher != null )
> >                fetcher.close();
> >        if ( content != null )
> >                content.close();
> >        if ( text != null )
> >                text.close();
> >        if ( parsedata != null )
> >                parsedata.close();
> > }
> >
> >
> >
> >
> > My test program :
> > I instanciate the NutchBean.
> > I make a search.
> > I call oNutchBean.closeSegments()
> > Finally, I try to delete the index dir and i get :
> >
> > java.io.IOException: Unable to delete file:
> > D:\Projets\application-data\recherche\indexations\indexes-
> > 20060314-17h33.54\segments\20060314173333\parse_text\data
> >        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:951)
> >        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> > :713)
> >        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> > :680)
> >        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
> >        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> > :713)
> >        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> > :680)
> >        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
> >        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> > :713)
> >        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> > :680)
> >        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:943)
> >        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java
> > :713)
> >        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java
> > :680)
> >
> > So, it is not ok yet.
> >
> >
> >
> >
> > -----Message d'origine-----
> > De : TDLN [mailto:[hidden email]]
> > Envoyé : mardi 14 mars 2006 12:39
> > À : [hidden email]
> > Objet : Re: Problems
> >
> > I think so too, maybe as part of the upcoming 0.7.2 release.
> >
> > In the meantime, you can try to implement the fix suggested in this post
> >
> > http://www.mail-archive.com/nutch-user@.../msg03011.html
> >
> > If it works, for you, please let me know, I need the fix as well :)
> >
> > Rgrds, Thomas
> >
> >
> >
> > On 3/14/06, Laurent Michenaud <[hidden email]> wrote:
> > >
> > > It would be interesting to have a fix for 0.7
> > >
> > > -----Message d'origine-----
> > > De : TDLN [mailto:[hidden email]] Envoyé : mardi 14 mars 2006
> > > 12:32 À : [hidden email] Objet : Re: Problems
> > >
> > > Unfortunately, in the 0.7 release, the NutchBean does not clean up
> > > properly after itself, so some SegementReaders and IndexReaders remain
> > open.
> > > I think this is fixed in the current code line. I had similar problems
> > > in my app based on 0.7 - all that helped was killing the processes
> > > blocking the index.
> > >
> > > Rgrds, Thomas
> > >
> > >
> > > On 3/13/06, Laurent Michenaud <[hidden email]> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I have integrated nutch in my application.
> > > >
> > > > In the admin part, i can launch the indexation.
> > > > In the client part, i can launch searches.
> > > >
> > > > After a search,  i can't do an indexation again because the index
> > > > files are used by the NutchBean.
> > > >
> > > > How can i do that please ?
> > > >
> > > > thx
> > > >
> > > >
> > >
> >
>
>


--
Keep Discovering ... ...
http://www.jroller.com/page/jmars