replication test problems

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

replication test problems

daremind
Hi, solr folks,

I followed the tutorial and everything worked well.

I then read the collection and distribtution twiki:
http://wiki.apache.org/solr/CollectionDistribution and tried to test
replication using the scripts.

What I did are the following:
- downloaded solr-nightly binary to two different boxes.
- ran the post script to create the indexes on one box, using it as the
master.
- ran snapshooter on the master; a snapshot file is created.
- ran rsync-enable and rsync-start at the master
- set up the master box config in the scripts.conf file on the slave box.
- ran snappuller-enable on the slave
- ran snappuller on the slave.

Here's a problem I got. It says there's no match for snapshot.* found on the
master box. This is wrong, there's one such file exists.

- I then ran snappuller specifically on the snap file that's on the master:
  ./bin/snappuller -n snapshot.20061023172655

This time it worked. and  running installer created the index successfully.

So there're two questions:

1. why the pattern match failed when running puller without params?

2. when I ran snappuller, it kept asking me for passwords for at least three
four time. Is this because it's using SSH? I have sudo permission on the
master box. This makes me think if it's a cron-job that calls up the
snappuller, how does the cron deal with this password prompts?



many thanks,

-Hui
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Chris Hostetter-3

: Here's a problem I got. It says there's no match for snapshot.* found on the
: master box. This is wrong, there's one such file exists.
:
: - I then ran snappuller specifically on the snap file that's on the master:
:   ./bin/snappuller -n snapshot.20061023172655
:
: This time it worked. and  running installer created the index successfully.

It's possible that there may be a subtle bug in the logic for determining
hte latest snapshot on your OS, the relevent section of of snappuller
is...

 # get directory name of latest snapshot if not specified on command line
 if [[ -z ${snap_name} ]]
 then
     snap_name=`ssh -o StrictHostKeyChecking=no ${master_host} "find ${master_data_dir} -name snapshot.* -print"|grep -v wip|sort -r|head -1`
 fi

...which OS are you running this on?

: 2. when I ran snappuller, it kept asking me for passwords for at least three
: four time. Is this because it's using SSH? I have sudo permission on the
: master box. This makes me think if it's a cron-job that calls up the
: snappuller, how does the cron deal with this password prompts?

to run in a cron, you would need to set up ssh keys for the uid the crons
will run as which have empty passphrases (so they can function without a
passphrase prompt).

Another posisble reason why the snappuller failed for you the first time,
is if you accidently misstyped your passphrase one of the times it asked
you (it's not snapppuller asking you this, snappller is just executing
ssh, and ssh is asking you directly) ... one of those times was the code
snippet i included above, which makes an ssh call just to get the name of
hte most recent snapshot.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

daremind
Hi, Hoss,

Thanks for the reply!

For #2, I think I just need to setup the passwordless SSH with empty
passphase. right?

For #1:

I'm using the following Enterprise version:

Linux version 2.4.21-37a6 (gcc version 3.2.3 20030502 (Red Hat Linux
3.2.3-47))

I tried to run the find command
find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.* -print

directly on my box and it gave the same result:
zsh: no matches found: snapshot.*

I'm not familiar with shell scripting, so any thoughts?



Thanks,

-Hui


On 10/30/06, Chris Hostetter <[hidden email]> wrote:

>
>
> : Here's a problem I got. It says there's no match for snapshot.* found on
> the
> : master box. This is wrong, there's one such file exists.
> :
> : - I then ran snappuller specifically on the snap file that's on the
> master:
> :   ./bin/snappuller -n snapshot.20061023172655
> :
> : This time it worked. and  running installer created the index
> successfully.
>
> It's possible that there may be a subtle bug in the logic for determining
> hte latest snapshot on your OS, the relevent section of of snappuller
> is...
>
> # get directory name of latest snapshot if not specified on command line
> if [[ -z ${snap_name} ]]
> then
>      snap_name=`ssh -o StrictHostKeyChecking=no ${master_host} "find
> ${master_data_dir} -name snapshot.* -print"|grep -v wip|sort -r|head -1`
> fi
>
> ...which OS are you running this on?
>
> : 2. when I ran snappuller, it kept asking me for passwords for at least
> three
> : four time. Is this because it's using SSH? I have sudo permission on the
> : master box. This makes me think if it's a cron-job that calls up the
> : snappuller, how does the cron deal with this password prompts?
>
> to run in a cron, you would need to set up ssh keys for the uid the crons
> will run as which have empty passphrases (so they can function without a
> passphrase prompt).
>
> Another posisble reason why the snappuller failed for you the first time,
> is if you accidently misstyped your passphrase one of the times it asked
> you (it's not snapppuller asking you this, snappller is just executing
> ssh, and ssh is asking you directly) ... one of those times was the code
> snippet i included above, which makes an ssh call just to get the name of
> hte most recent snapshot.
>
>
> -Hoss
>
>


--
Regards,

-Hui
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Bill Au
It looks like find running under zsh is the problem.  In this case it does
like the wildcard (*).
I don't really know zsh so I will have to spend some time to investigate.

Bill

On 10/30/06, Yu-Hui Jin <[hidden email]> wrote:

>
> Hi, Hoss,
>
> Thanks for the reply!
>
> For #2, I think I just need to setup the passwordless SSH with empty
> passphase. right?
>
> For #1:
>
> I'm using the following Enterprise version:
>
> Linux version 2.4.21-37a6 (gcc version 3.2.3 20030502 (Red Hat Linux
> 3.2.3-47))
>
> I tried to run the find command
> find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.*
> -print
>
> directly on my box and it gave the same result:
> zsh: no matches found: snapshot.*
>
> I'm not familiar with shell scripting, so any thoughts?
>
>
>
> Thanks,
>
> -Hui
>
>
> On 10/30/06, Chris Hostetter <[hidden email]> wrote:
> >
> >
> > : Here's a problem I got. It says there's no match for snapshot.* found
> on
> > the
> > : master box. This is wrong, there's one such file exists.
> > :
> > : - I then ran snappuller specifically on the snap file that's on the
> > master:
> > :   ./bin/snappuller -n snapshot.20061023172655
> > :
> > : This time it worked. and  running installer created the index
> > successfully.
> >
> > It's possible that there may be a subtle bug in the logic for
> determining
> > hte latest snapshot on your OS, the relevent section of of snappuller
> > is...
> >
> > # get directory name of latest snapshot if not specified on command line
> > if [[ -z ${snap_name} ]]
> > then
> >      snap_name=`ssh -o StrictHostKeyChecking=no ${master_host} "find
> > ${master_data_dir} -name snapshot.* -print"|grep -v wip|sort -r|head -1`
> > fi
> >
> > ...which OS are you running this on?
> >
> > : 2. when I ran snappuller, it kept asking me for passwords for at least
> > three
> > : four time. Is this because it's using SSH? I have sudo permission on
> the
> > : master box. This makes me think if it's a cron-job that calls up the
> > : snappuller, how does the cron deal with this password prompts?
> >
> > to run in a cron, you would need to set up ssh keys for the uid the
> crons
> > will run as which have empty passphrases (so they can function without a
> > passphrase prompt).
> >
> > Another posisble reason why the snappuller failed for you the first
> time,
> > is if you accidently misstyped your passphrase one of the times it asked
> > you (it's not snapppuller asking you this, snappller is just executing
> > ssh, and ssh is asking you directly) ... one of those times was the code
> > snippet i included above, which makes an ssh call just to get the name
> of
> > hte most recent snapshot.
> >
> >
> > -Hoss
> >
> >
>
>
> --
> Regards,
>
> -Hui
>
>
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Chris Hostetter-3
In reply to this post by daremind
: For #2, I think I just need to setup the passwordless SSH with empty
: passphase. right?

correct.

: I tried to run the find command
: find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.* -print
:
: directly on my box and it gave the same result:
: zsh: no matches found: snapshot.*

FYI: i don't think that's exactly a fair comparison, when you run that
command, your shell tries to evaluate the '*' before executing find,
but when it's run via ssh, that doesn't (usually) happen ... except it
sounds like Bill thinks the problem is that it does happen if your remote
shell is zsh.

could you by any chance try running snappuller as a user whose shell is sh
or bash (specificly: shows shell on the master is sh or bash) ?


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

daremind
I found that if I add single quotes to the pattern as below, it worked on
the command line of my zsh env, so I'll try add the quotes in the snappuller
script and test it again.

find /home/yjin/apps/solr-nightly/example/solr/data/ -name 'snapshot.*'
-print
/home/yjin/apps/solr-nightly/example/solr/data/snapshot.20061023172655



will keep you updated!

regards,
-Hui


On 10/31/06, Chris Hostetter <[hidden email]> wrote:

>
> : For #2, I think I just need to setup the passwordless SSH with empty
> : passphase. right?
>
> correct.
>
> : I tried to run the find command
> : find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.*
> -print
> :
> : directly on my box and it gave the same result:
> : zsh: no matches found: snapshot.*
>
> FYI: i don't think that's exactly a fair comparison, when you run that
> command, your shell tries to evaluate the '*' before executing find,
> but when it's run via ssh, that doesn't (usually) happen ... except it
> sounds like Bill thinks the problem is that it does happen if your remote
> shell is zsh.
>
> could you by any chance try running snappuller as a user whose shell is sh
> or bash (specificly: shows shell on the master is sh or bash) ?
>
>
> -Hoss
>
>


--
Regards,

-Hui
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Bill Au
In reply to this post by Chris Hostetter-3
It is definitely zsh related.  I got the same error when I run the find
command under zsh, but not with bash or ksh.

I also found this:
http://www.zsh.org/mla/users/2005/msg00295.html

Obviously that won't work for bash or ksh.

Bill


On 10/31/06, Chris Hostetter <[hidden email]> wrote:

>
> : For #2, I think I just need to setup the passwordless SSH with empty
> : passphase. right?
>
> correct.
>
> : I tried to run the find command
> : find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.*
> -print
> :
> : directly on my box and it gave the same result:
> : zsh: no matches found: snapshot.*
>
> FYI: i don't think that's exactly a fair comparison, when you run that
> command, your shell tries to evaluate the '*' before executing find,
> but when it's run via ssh, that doesn't (usually) happen ... except it
> sounds like Bill thinks the problem is that it does happen if your remote
> shell is zsh.
>
> could you by any chance try running snappuller as a user whose shell is sh
> or bash (specificly: shows shell on the master is sh or bash) ?
>
>
> -Hoss
>
>
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Chris Hostetter-3

: I also found this:
: http://www.zsh.org/mla/users/2005/msg00295.html
:
: Obviously that won't work for bash or ksh.

interesting ... i allways assumed it worked because the shell wasn't
evaluating the * when executed by ssh, i had no idea the the shell tries
to expand it and leaves it alone if there is no expansion.

Bill: what do you think about explicitly putting in the single quotes as
Hui suggested?  that should still work under bash and sh right?



-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Mike Klaas
On 10/31/06, Chris Hostetter <[hidden email]> wrote:

> Bill: what do you think about explicitly putting in the single quotes as
> Hui suggested?  that should still work under bash and sh right?

That should work in bash, at least.  Backslash-escaping is also an option.

The semantics of file globbing in bash are irritating.

-Mike
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Bill Au
I did some testing and blackslash-escaping also works:

find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.\*
-print

Hui, can you verify that?

I am already using single quote in the snappuller script to specify the find
command
to as an argument to ssh.  I could change that to double quote and then use
single quote for snapshot.*, or blackslash escape the *.

I am fine with either way.  Does anyone has any strong preference?
If not, I will just randomly choose one.

Bill

On 10/31/06, Mike Klaas <[hidden email]> wrote:

>
> On 10/31/06, Chris Hostetter <[hidden email]> wrote:
>
> > Bill: what do you think about explicitly putting in the single quotes as
> > Hui suggested?  that should still work under bash and sh right?
>
> That should work in bash, at least.  Backslash-escaping is also an option.
>
> The semantics of file globbing in bash are irritating.
>
> -Mike
>
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

daremind
Yap, Bill.

The backslash-escaping one works for my zsh as well.   And I'm sure you
checked it's working for other major shells.

So I would say backslash seems to be a good solution since you don't have to
worry about double-single quotes.

Thanks!


regards,
-Hui



On 11/1/06, Bill Au <[hidden email]> wrote:

>
> I did some testing and blackslash-escaping also works:
>
> find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.\*
> -print
>
> Hui, can you verify that?
>
> I am already using single quote in the snappuller script to specify the
> find
> command
> to as an argument to ssh.  I could change that to double quote and then
> use
> single quote for snapshot.*, or blackslash escape the *.
>
> I am fine with either way.  Does anyone has any strong preference?
> If not, I will just randomly choose one.
>
> Bill
>
> On 10/31/06, Mike Klaas <[hidden email]> wrote:
> >
> > On 10/31/06, Chris Hostetter <[hidden email]> wrote:
> >
> > > Bill: what do you think about explicitly putting in the single quotes
> as
> > > Hui suggested?  that should still work under bash and sh right?
> >
> > That should work in bash, at least.  Backslash-escaping is also an
> option.
> >
> > The semantics of file globbing in bash are irritating.
> >
> > -Mike
> >
>
>


--
Regards,

-Hui
Reply | Threaded
Open this post in threaded view
|

Re: replication test problems

Bill Au
I have created a bug to track this:

https://issues.apache.org/jira/browse/SOLR-63

I will attach a patch to the bug shortly.

Bill

On 11/1/06, Yu-Hui Jin <[hidden email]> wrote:

>
> Yap, Bill.
>
> The backslash-escaping one works for my zsh as well.   And I'm sure you
> checked it's working for other major shells.
>
> So I would say backslash seems to be a good solution since you don't have
> to
> worry about double-single quotes.
>
> Thanks!
>
>
> regards,
> -Hui
>
>
>
> On 11/1/06, Bill Au <[hidden email]> wrote:
> >
> > I did some testing and blackslash-escaping also works:
> >
> > find /home/yjin/apps/solr-nightly/example/solr/data/ -name snapshot.\*
> > -print
> >
> > Hui, can you verify that?
> >
> > I am already using single quote in the snappuller script to specify the
> > find
> > command
> > to as an argument to ssh.  I could change that to double quote and then
> > use
> > single quote for snapshot.*, or blackslash escape the *.
> >
> > I am fine with either way.  Does anyone has any strong preference?
> > If not, I will just randomly choose one.
> >
> > Bill
> >
> > On 10/31/06, Mike Klaas <[hidden email]> wrote:
> > >
> > > On 10/31/06, Chris Hostetter <[hidden email]> wrote:
> > >
> > > > Bill: what do you think about explicitly putting in the single
> quotes
> > as
> > > > Hui suggested?  that should still work under bash and sh right?
> > >
> > > That should work in bash, at least.  Backslash-escaping is also an
> > option.
> > >
> > > The semantics of file globbing in bash are irritating.
> > >
> > > -Mike
> > >
> >
> >
>
>
> --
> Regards,
>
> -Hui
>
>