TokenStream#reset():boolean?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

TokenStream#reset():boolean?

Karl Wettin
How about depricating

 >  public void reset() throws IOException {}

and refactor it to allow making sure the stream was reset?

To avoid break backwards compatibillity with extentions of the  
TokenStream currently in trunk one could introduce something like this:

 >  /**
 >   * @param  requireConfirmedReset
 >   * @throws ResetException An IOException thrown if parameter  
requireReset
 >   *         was set true and the stream could not be reset.
 >   */
 >  public void reset(boolean requireConfirmedReset)
 >      throws IOException, ResetException {
 >    reset();
 >    if(requireConfirmedReset) throw new ResetException();
 >  }


and then depricate this new method at some strategic time to introduce  
an exception-less solution like:

 >  /** @return true if this stream was successfully reset. */
 >  public boolean reset() throws IOException { return false; }




      karl

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Michael McCandless-2

Karl,

I'm sort of confused by this proposal.  What is the driver here?  It  
seems like the overall goal is to have reset() return a boolean  
stating whether it was actually implemented by the subclass of  
TokenStream?

Mike

Karl Wettin wrote:

> How about depricating
>
> >  public void reset() throws IOException {}
>
> and refactor it to allow making sure the stream was reset?
>
> To avoid break backwards compatibillity with extentions of the  
> TokenStream currently in trunk one could introduce something like  
> this:
>
> >  /**
> >   * @param  requireConfirmedReset
> >   * @throws ResetException An IOException thrown if parameter  
> requireReset
> >   *         was set true and the stream could not be reset.
> >   */
> >  public void reset(boolean requireConfirmedReset)
> >      throws IOException, ResetException {
> >    reset();
> >    if(requireConfirmedReset) throw new ResetException();
> >  }
>
>
> and then depricate this new method at some strategic time to  
> introduce an exception-less solution like:
>
> >  /** @return true if this stream was successfully reset. */
> >  public boolean reset() throws IOException { return false; }
>
>
>
>
>     karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Karl Wettin
I just want to know if my token stream managed to reset or not.  
Especially that parts of composite streams.

           karl

4 jul 2008 kl. 12.13 skrev Michael McCandless:

>
> Karl,
>
> I'm sort of confused by this proposal.  What is the driver here?  It  
> seems like the overall goal is to have reset() return a boolean  
> stating whether it was actually implemented by the subclass of  
> TokenStream?
>
> Mike
>
> Karl Wettin wrote:
>
>> How about depricating
>>
>> >  public void reset() throws IOException {}
>>
>> and refactor it to allow making sure the stream was reset?
>>
>> To avoid break backwards compatibillity with extentions of the  
>> TokenStream currently in trunk one could introduce something like  
>> this:
>>
>> >  /**
>> >   * @param  requireConfirmedReset
>> >   * @throws ResetException An IOException thrown if parameter  
>> requireReset
>> >   *         was set true and the stream could not be reset.
>> >   */
>> >  public void reset(boolean requireConfirmedReset)
>> >      throws IOException, ResetException {
>> >    reset();
>> >    if(requireConfirmedReset) throw new ResetException();
>> >  }
>>
>>
>> and then depricate this new method at some strategic time to  
>> introduce an exception-less solution like:
>>
>> >  /** @return true if this stream was successfully reset. */
>> >  public boolean reset() throws IOException { return false; }
>>
>>
>>
>>
>>    karl
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Michael McCandless-2

But what would cause reset() to not actually work?

And what is a composite stream?

I'm just a little confused on the use case here I think...

Mike

Karl Wettin wrote:

> I just want to know if my token stream managed to reset or not.  
> Especially that parts of composite streams.
>
>          karl
>
> 4 jul 2008 kl. 12.13 skrev Michael McCandless:
>
>>
>> Karl,
>>
>> I'm sort of confused by this proposal.  What is the driver here?  
>> It seems like the overall goal is to have reset() return a boolean  
>> stating whether it was actually implemented by the subclass of  
>> TokenStream?
>>
>> Mike
>>
>> Karl Wettin wrote:
>>
>>> How about depricating
>>>
>>> >  public void reset() throws IOException {}
>>>
>>> and refactor it to allow making sure the stream was reset?
>>>
>>> To avoid break backwards compatibillity with extentions of the  
>>> TokenStream currently in trunk one could introduce something like  
>>> this:
>>>
>>> >  /**
>>> >   * @param  requireConfirmedReset
>>> >   * @throws ResetException An IOException thrown if parameter  
>>> requireReset
>>> >   *         was set true and the stream could not be reset.
>>> >   */
>>> >  public void reset(boolean requireConfirmedReset)
>>> >      throws IOException, ResetException {
>>> >    reset();
>>> >    if(requireConfirmedReset) throw new ResetException();
>>> >  }
>>>
>>>
>>> and then depricate this new method at some strategic time to  
>>> introduce an exception-less solution like:
>>>
>>> >  /** @return true if this stream was successfully reset. */
>>> >  public boolean reset() throws IOException { return false; }
>>>
>>>
>>>
>>>
>>>   karl
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Karl Wettin
4 jul 2008 kl. 21.02 skrev Michael McCandless:

>
> But what would cause reset() to not actually work?

Extentions of TokenStream does not have handle reset():

   /** Resets this stream to the beginning. This is an
    *  optional operation, so subclasses may or may not
    *  implement this method. Reset() is not needed for
    *  the standard indexing process. However, if the Tokens
    *  of a TokenStream are intended to be consumed more than
    *  once, it is necessary to implement reset().
    */
   public void reset() throws IOException {}

> And what is a composite stream?

I just came up with that name. As in the composite pattern.  A stream  
formed by one or more streams and perhaps some code. A TokenFilter  
would be the most simple one.

> I'm just a little confused on the use case here I think...

Here are a few things I think of:

Perhaps my composite stream that I iterated for pre processing reasons  
is really simple to reset but the input streams does not support reset.

If you add a stream to a field and that stream is exhausted at  
addDocument beacuse it did not support reset, you'll end up with an  
empty field without a warning.

It is a waste of  esources to reset all the parts of a composite  
stream if one of the parts failed to reset.


          karl


>
>
> Mike
>
> Karl Wettin wrote:
>
>> I just want to know if my token stream managed to reset or not.  
>> Especially that parts of composite streams.
>>
>>         karl
>>
>> 4 jul 2008 kl. 12.13 skrev Michael McCandless:
>>
>>>
>>> Karl,
>>>
>>> I'm sort of confused by this proposal.  What is the driver here?  
>>> It seems like the overall goal is to have reset() return a boolean  
>>> stating whether it was actually implemented by the subclass of  
>>> TokenStream?
>>>
>>> Mike
>>>
>>> Karl Wettin wrote:
>>>
>>>> How about depricating
>>>>
>>>> >  public void reset() throws IOException {}
>>>>
>>>> and refactor it to allow making sure the stream was reset?
>>>>
>>>> To avoid break backwards compatibillity with extentions of the  
>>>> TokenStream currently in trunk one could introduce something like  
>>>> this:
>>>>
>>>> >  /**
>>>> >   * @param  requireConfirmedReset
>>>> >   * @throws ResetException An IOException thrown if parameter  
>>>> requireReset
>>>> >   *         was set true and the stream could not be reset.
>>>> >   */
>>>> >  public void reset(boolean requireConfirmedReset)
>>>> >      throws IOException, ResetException {
>>>> >    reset();
>>>> >    if(requireConfirmedReset) throw new ResetException();
>>>> >  }
>>>>
>>>>
>>>> and then depricate this new method at some strategic time to  
>>>> introduce an exception-less solution like:
>>>>
>>>> >  /** @return true if this stream was successfully reset. */
>>>> >  public boolean reset() throws IOException { return false; }
>>>>
>>>>
>>>>
>>>>
>>>>  karl
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Michael McCandless-2

If we make this change (migrate to "boolean TokenStream.reset()"),  
what would IndexWriter do if it calls reset and false is returned?

Are you saying that you want to get to the point where we force all  
TokenStream subclasses to implement reset (and return true)?  Should  
we just eventually make reset() abstract (somehow in an incremental  
backwards compatible way, if there is one)?

Mike

Karl Wettin wrote:

> 4 jul 2008 kl. 21.02 skrev Michael McCandless:
>
>>
>> But what would cause reset() to not actually work?
>
> Extentions of TokenStream does not have handle reset():
>
>  /** Resets this stream to the beginning. This is an
>   *  optional operation, so subclasses may or may not
>   *  implement this method. Reset() is not needed for
>   *  the standard indexing process. However, if the Tokens
>   *  of a TokenStream are intended to be consumed more than
>   *  once, it is necessary to implement reset().
>   */
>  public void reset() throws IOException {}
>
>> And what is a composite stream?
>
> I just came up with that name. As in the composite pattern.  A  
> stream formed by one or more streams and perhaps some code. A  
> TokenFilter would be the most simple one.
>
>> I'm just a little confused on the use case here I think...
>
> Here are a few things I think of:
>
> Perhaps my composite stream that I iterated for pre processing  
> reasons is really simple to reset but the input streams does not  
> support reset.
>
> If you add a stream to a field and that stream is exhausted at  
> addDocument beacuse it did not support reset, you'll end up with an  
> empty field without a warning.
>
> It is a waste of  esources to reset all the parts of a composite  
> stream if one of the parts failed to reset.
>
>
>         karl
>
>
>>
>>
>> Mike
>>
>> Karl Wettin wrote:
>>
>>> I just want to know if my token stream managed to reset or not.  
>>> Especially that parts of composite streams.
>>>
>>>        karl
>>>
>>> 4 jul 2008 kl. 12.13 skrev Michael McCandless:
>>>
>>>>
>>>> Karl,
>>>>
>>>> I'm sort of confused by this proposal.  What is the driver here?  
>>>> It seems like the overall goal is to have reset() return a  
>>>> boolean stating whether it was actually implemented by the  
>>>> subclass of TokenStream?
>>>>
>>>> Mike
>>>>
>>>> Karl Wettin wrote:
>>>>
>>>>> How about depricating
>>>>>
>>>>> >  public void reset() throws IOException {}
>>>>>
>>>>> and refactor it to allow making sure the stream was reset?
>>>>>
>>>>> To avoid break backwards compatibillity with extentions of the  
>>>>> TokenStream currently in trunk one could introduce something  
>>>>> like this:
>>>>>
>>>>> >  /**
>>>>> >   * @param  requireConfirmedReset
>>>>> >   * @throws ResetException An IOException thrown if parameter  
>>>>> requireReset
>>>>> >   *         was set true and the stream could not be reset.
>>>>> >   */
>>>>> >  public void reset(boolean requireConfirmedReset)
>>>>> >      throws IOException, ResetException {
>>>>> >    reset();
>>>>> >    if(requireConfirmedReset) throw new ResetException();
>>>>> >  }
>>>>>
>>>>>
>>>>> and then depricate this new method at some strategic time to  
>>>>> introduce an exception-less solution like:
>>>>>
>>>>> >  /** @return true if this stream was successfully reset. */
>>>>> >  public boolean reset() throws IOException { return false; }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> karl
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Karl Wettin

7 jul 2008 kl. 13.04 skrev Michael McCandless:

>
> If we make this change (migrate to "boolean TokenStream.reset()"),  
> what would IndexWriter do if it calls reset and false is returned?

I don't think the writer ever should call reset(), it is the consumer  
who is passing a stream to the writer that needs to make sure they are  
passing a valid stream. It is the consumer that has to do something in  
case they have exhausted a stream that can not be resetted. I.e. the  
consumer will need to wrap it in a cache of some kind.

>
> Are you saying that you want to get to the point where we force all  
> TokenStream subclasses to implement reset (and return true)?  Should  
> we just eventually make reset() abstract (somehow in an incremental  
> backwards compatible way, if there is one)?

Lets say it would be a nice goal that all streams that could reset  
implemented it. But it would not be a vital part of what I'm really  
looking for.


         karl


>
>
> Mike
>
> Karl Wettin wrote:
>
>> 4 jul 2008 kl. 21.02 skrev Michael McCandless:
>>
>>>
>>> But what would cause reset() to not actually work?
>>
>> Extentions of TokenStream does not have handle reset():
>>
>> /** Resets this stream to the beginning. This is an
>>  *  optional operation, so subclasses may or may not
>>  *  implement this method. Reset() is not needed for
>>  *  the standard indexing process. However, if the Tokens
>>  *  of a TokenStream are intended to be consumed more than
>>  *  once, it is necessary to implement reset().
>>  */
>> public void reset() throws IOException {}
>>
>>> And what is a composite stream?
>>
>> I just came up with that name. As in the composite pattern.  A  
>> stream formed by one or more streams and perhaps some code. A  
>> TokenFilter would be the most simple one.
>>
>>> I'm just a little confused on the use case here I think...
>>
>> Here are a few things I think of:
>>
>> Perhaps my composite stream that I iterated for pre processing  
>> reasons is really simple to reset but the input streams does not  
>> support reset.
>>
>> If you add a stream to a field and that stream is exhausted at  
>> addDocument beacuse it did not support reset, you'll end up with an  
>> empty field without a warning.
>>
>> It is a waste of  esources to reset all the parts of a composite  
>> stream if one of the parts failed to reset.
>>
>>
>>        karl
>>
>>
>>>
>>>
>>> Mike
>>>
>>> Karl Wettin wrote:
>>>
>>>> I just want to know if my token stream managed to reset or not.  
>>>> Especially that parts of composite streams.
>>>>
>>>>       karl
>>>>
>>>> 4 jul 2008 kl. 12.13 skrev Michael McCandless:
>>>>
>>>>>
>>>>> Karl,
>>>>>
>>>>> I'm sort of confused by this proposal.  What is the driver  
>>>>> here?  It seems like the overall goal is to have reset() return  
>>>>> a boolean stating whether it was actually implemented by the  
>>>>> subclass of TokenStream?
>>>>>
>>>>> Mike
>>>>>
>>>>> Karl Wettin wrote:
>>>>>
>>>>>> How about depricating
>>>>>>
>>>>>> >  public void reset() throws IOException {}
>>>>>>
>>>>>> and refactor it to allow making sure the stream was reset?
>>>>>>
>>>>>> To avoid break backwards compatibillity with extentions of the  
>>>>>> TokenStream currently in trunk one could introduce something  
>>>>>> like this:
>>>>>>
>>>>>> >  /**
>>>>>> >   * @param  requireConfirmedReset
>>>>>> >   * @throws ResetException An IOException thrown if parameter  
>>>>>> requireReset
>>>>>> >   *         was set true and the stream could not be reset.
>>>>>> >   */
>>>>>> >  public void reset(boolean requireConfirmedReset)
>>>>>> >      throws IOException, ResetException {
>>>>>> >    reset();
>>>>>> >    if(requireConfirmedReset) throw new ResetException();
>>>>>> >  }
>>>>>>
>>>>>>
>>>>>> and then depricate this new method at some strategic time to  
>>>>>> introduce an exception-less solution like:
>>>>>>
>>>>>> >  /** @return true if this stream was successfully reset. */
>>>>>> >  public boolean reset() throws IOException { return false; }
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> karl
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TokenStream#reset():boolean?

Michael McCandless-2

Karl Wettin wrote:

>
> 7 jul 2008 kl. 13.04 skrev Michael McCandless:
>
>>
>> If we make this change (migrate to "boolean TokenStream.reset()"),  
>> what would IndexWriter do if it calls reset and false is returned?
>
> I don't think the writer ever should call reset(), it is the  
> consumer who is passing a stream to the writer that needs to make  
> sure they are passing a valid stream. It is the consumer that has to  
> do something in case they have exhausted a stream that can not be  
> resetted. I.e. the consumer will need to wrap it in a cache of some  
> kind.

Actually DocumentsWriter already calls reset() now before pulling  
tokens from the TokenStream, but, I agree it really shouldn't have to.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]