dw_dev_training: Gearman on Dreamhack

anarres (

anarres) wrote in

dw_dev_training,
@ 2010-07-08 12:48 pm UTC

Entry tags:

dreamhacks, gearman

Gearman on Dreamhack

Has anyone got any hints on getting Gearman working on a Dreamhack? I keep getting this: '[Error: Unable to get gearman client. @ hack.dreamwidth.net]'

(Read 25 comments) - (Post a new comment)
(Flat) (Top-level comments only)

no subject

pauamma
2010-07-20 03:01 pm UTC (link)

Hmm. I'm getting outta my depth here, trying to get an expert to look at it. (I'll still tell you about any ideas I may have.) Also, Sophie mentioned that you configured the wrong Gearman port on your hack.dreamwidth.net dreamhack. You have

@GEARMAN_SERVERS = ( '127.0.0.1:8214' );

in etc/config-local.pl, but that's your web port. You should be using 17214 for Gearman on hack.dreamwidth.net.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 9 comments) (Show 9 comments)

the saga continues...

anarres
2010-07-23 08:32 pm UTC (link)

So, I've had another go at the debugging you suggested. Running

~/dw/bin/gearmand -p 17096 --debug=1

and then attempting to do something that uses gearman gives this error message:

Error: Can't locate object method "CMD_" via package "Gearman::Server::Client" at /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm line 490.

And there's nothing in the apache error log. The line it's complaining about in /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm is this:

my $cmd_name = "CMD_" . Gearman::Util::cmd_name($cmd);

It seems to be asking for a module /usr/local/share/perl/5.10.0/Gearman/Util.pm, which doesn't exist.

Maybe the problem is that as Alierak said it should be looking at the Gearman modules in ~/dw, not way off in the filesystem? But I tried reinstalling gearman with perl Makefile.PL PREFIX=$HOME/dw as Alierak suggested, but that doesn't seem to have worked.

But my way of installing was a little different from Alierak's. When I copied gearmand and Makefile.PL into ~/dw/bin and tried to install it complained that it couldn't find some other files, so instead I installed from ~/dw/cvs/gearman/server, I don't know if that makes a difference.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 8 comments) (Show 8 comments)

	Re: the saga continues... pauamma 2010-07-24 02:25 pm UTC (link)
	Looks like there's something wrong in the way Gearman components are installed, or perhaps conflicts between the ones installed in ~/dw/cgi-bin and the ones installed in /usr/local/share/perl/5.10.0/. I asked Afuna to check. (Reply to this) (Thread from start) (Parent)

Re: the saga continues...

pauamma
2010-07-26 07:36 pm UTC (link)

OK, I've had some success on dreamhack.afunalmatata.com making sure gearmand runs, by *not* using $LJHOME/bin/gearmand.

You may want to try the following commands to see what happens:

rm $LJHOME/bin/gearmand # get rid of possible troublemaker
gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096 # Actually run gearmand
cat $LJHOME/temp/gearmand.pid # Check 1
netstat -A inet -a -n -p | grep 17096 # Check 2
ps auxww | grep -i gearmand # Check 3

(Note: I haven't actually tried running Gearman workers yet - that's next on my list, but since it looks like you have problems with gearmand itself, you may want to try this method.)

For reference, I got the following output from these commands, and if it works for you, the output should be similar:

dh-pauamma@li-166:~/dw/cvs/dw-free$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17082
dh-pauamma@li-166:~/dw/cvs/dw-free$ cat $LJHOME/temp/gearmand.pid
6235
dh-pauamma@li-166:~/dw/cvs/dw-free$ netstat -A inet -a -n -p | grep 17082
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17082 0.0.0.0:* LISTEN 6235/perl
dh-pauamma@li-166:~/dw/cvs/dw-free$ ps auxww | grep -i gearman | less afuna

2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl

/usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1009 6242 0.0 0.1 7452 968 pts/7 SN+ 18:26 0:00 grep -i gearman

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 6 comments) (Show 6 comments)

Re: the saga continues...

anarres
2010-07-26 09:13 pm UTC (link)

My output is similar to yours. I get the same error when trying to make a credit card payment though.

dh-anarres@li-166:~$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096
dh-anarres@li-166:~$ cat $LJHOME/temp/gearmand.pid
6699
dh-anarres@li-166:~$ netstat -A inet -a -n -p | grep 17096
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17096 0.0.0.0:* LISTEN 6699/perl
tcp 0 0 127.0.0.1:54655 127.0.0.1:17096 ESTABLISHED 8304/perl
tcp 0 0 127.0.0.1:54654 127.0.0.1:17096 ESTABLISHED 10458/perl
tcp 0 0 127.0.0.1:54656 127.0.0.1:17096 ESTABLISHED 8329/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54654 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54655 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54656 ESTABLISHED 6699/perl
dh-anarres@li-166:~$ ps auxww | grep -i gearmand
afuna 2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1028 6699 0.0 1.5 46456 7892 ? SN 21:01 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres/dw/temp/gearmand.pid -p 17096
1028 6704 0.0 0.1 7448 972 pts/10 SN+ 21:03 0:00 grep -i gearmand

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 5 comments) (Show 5 comments)

Re: the saga continues...

pauamma
2010-07-29 08:53 pm UTC (link)

OK, just ran the following:

ps uww 6699 8304 8329 10458
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1028 8304 5.3 1.3 263712 7088 ? SN Jul19 769:30 perl shop-creditcard-charge
1028 8329 5.4 2.4 263716 12948 ? SN Jul19 781:20 perl shop_creditcard-charge.PROPER
1028 10458 0.0 0.6 263636 3328 ? SN Jul20 5:10 perl shop-creditcard-charge
1028 6699 90.4 0.5 46584 2632 ? RN Jul26 3879:34 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres /dw/temp/gearmand.pid -p 17096

and it looks like you have 3 CC gearman workers trying to connect to gearmand, and gearmand is eating oodles of CPU. That doesn't sound healthy. Kill all 4 processes and make sure they're dead, restart gearmand, and go through the checks again:

cat $LJHOME/temp/gearmand.pid
netstat -A inet -a -n -p | grep 17096
ps auxww | grep -i gearmand

Then, start apache, and start 1 (and only 1) worker process (either shop-creditcard-charge or shop_creditcard-charge.PROPER, depending on which you need), in verbose mode.

$LJHOME/bin/worker/shop-creditcard-charge -v

(or shop_creditcard-charge.PROPER).

Test it, and see if things work better. (You should be getting messages from the worker process. Paste them in your reply, and ^C to kill the worker when you're done.)

I haven't tried running the shop worker, but I've been able to test sysban gearman processing, and it worked without a hitch. My guess is that something (perhaps the three shop worker processes) was messing things up. If you still have problems, I'll walk you through a test using sysban, so we can make sure your basic gearman configuration works.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 4 comments) (Show 4 comments)

	Re: the saga continues... anarres 2010-07-30 12:13 pm UTC (link)
	This is probably a dumb question but, how do I kill processes? (Reply to this) (Thread from start) (Parent) (Thread) (Hide 3 comments) (Show 3 comments)

Re: the saga continues...

denise
2010-07-30 12:16 pm UTC (link)

Ha! One I finally know! (No such thing as a dumb question around here.)

Once you've identified the process ID ('ps -aux' will get you a list of the full set of processes running on the machine), "kill [processid]", then wait. It should then be gone when you list the processes again. If it isn't, you might need to wait a few minutes and issue the kill again; if that doesn't work, "kill -9 [pid]" will get rid of a really stuck process, but you should reserve that for the stubborn ones, since it forces an exit and doesn't do a graceful shutdown.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 2 comments) (Show 2 comments)

	Re: the saga continues... anarres 2010-07-30 03:31 pm UTC (link)
	Thanks, I think I killed it. Phew. (Reply to this) (Thread from start) (Parent) (Thread) (Hide 1 comment) (Show 1 comment)

	Re: the saga continues... denise 2010-07-30 03:34 pm UTC (link)
	You're welcome! I'm just excited that there was a question I knew the answer to. G (Reply to this) (Thread from start) (Parent)

(Read 25 comments) - (Post a new comment)
(Flat) (Top-level comments only)