Quantcast

how to rebuild a index corrupted / async commit problem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

how to rebuild a index corrupted / async commit problem

Cristian Lorenzetto
hi , i m thinking about what you told me in previous message and how to
solve the corruption problem and the problem about commit operation
executed in async way.

I m thinking to create a simple transaction log in a file.
i use a long atomic sequence for a ordinable transaction id.

when i make a new operation
1) generate new incremental transaction id
2) save the operation abstract info in transaction log associated to id.
    2.a insert ,update with the a serialized version of the object to save
    2b delete the query serialized where apply delete
3) execute same operation in lucene adding before property transactionId
(executed in ram)

4) in async way commit is executed. After the commit the transaction log
until last transaction id is deleted.(i dont know how insert block after
commit , using near real time reader and SearcherManager) I might
 introduce a logic in the way a commit is done. The order is simlilar to a
queue so it follows the transactionId order. i Is there a example about
possibility to commit a specific set of uncommit operations?

5) i need the warrenty after a crud operation the data in available in
memory  in a possible imminent research so i think i might execute
flush/refreshReader after every CUD operations

if there is a failure transaction log will be not empty. But i can rexecute
operations not executed after restartup.
Maybe it could be usefull also for fixing a corruption but it is sure the
corrution dont touch also segments already commited completely in the past?
or maybe for a stable solution i might anyway save data in a secondary
repository ?



for your opinion this solution will be sufficient . It is a good solution
for you, i m forgetting some aspects?

PS Another interesting aspect maybe could be associate the segment
associated to a transaction. In this way if a segment is missing i can
apply again it without rebuild all the index from scratch.




> 2017-03-21 0:58 GMT+01:00 Michael McCandless <[hidden email]>:
>>
>>> You can use Lucene's CheckIndex tool with the -exorcise option but this
>>> is quite brutal: it simply drops any segment that has corruption it detects.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Mon, Mar 20, 2017 at 4:44 PM, Marco Reis <[hidden email]> wrote:
>>>
>>>> I'm afraid it's not possible to rebuild index. It's important to
>>>> maintain a
>>>> backup policy because of that.
>>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 5:12 PM Cristian Lorenzetto <
>>>> [hidden email]> wrote:
>>>>
>>>> > lucene can rebuild index using his internal info and how ? or in have
>>>> to
>>>> > reinsert all in other way?
>>>> >
>>>> --
>>>> Marco Reis
>>>> Software Architect
>>>> http://marcoreis.net
>>>> https://github.com/masreis
>>>> +55 61 9 81194620
>>>>
>>>
>>>
>>
>
Loading...