Gearman on Dreamhack

[anarres]

Has anyone got any hints on getting Gearman working on a Dreamhack? I keep getting this: '[Error: Unable to get gearman client. @ hack.dreamwidth.net]'

Comments

[pauamma]

Are you trying on hack.dreamwidth.net, or on dreamhack.afunamatata.com? If the latter, you should configure it in config-local.pl http://dw-dev-training.dreamwidth.org/22144.html?thread=153984#cmt153984, using port 17096 instead of 17103.

Thus,

@GEARMAN_SERVERS = ( '127.0.0.1:17096' );

and

~/dw/bin/gearmand -p 17096 &

What happens then?

[anarres]

Ah, so my port number is... something completely different from what I thought it was, d'oh! Having done what you suggest I now get a different error on the BML page, 'Internal Failure We're sorry, but something failed internally on our end attempting to process your payment. Your credit card has probably not been charged.' And attempting to run a gearman worker script by hand gives this: 'First gearmand server in @LJ::GEARMAN_SERVERS (127.0.0.1:17096) isn't responding.'

So... this takes me one more rung up the ladder of debugging, I think..

[anarres]

And

~/dw/bin/gearmand -p 17096 &

gives this:

[1] 3330

[pauamma]

Hmm. I checked on both dreamhack.afunamatata.com and hack.dreamwidth.net and ps auxww doesn't show a gearmand running as dh-anarres, which means it might not start properly. What happens if you run

~/dw/bin/gearmand -p 17096

instead? (You may need to ^C to terminate it, if it doesn't terminate of its own accord.)

[anarres]

It hangs, and doesn't give me back a command prompt until I do ^C, but doesn't do anything else.

[pauamma]

OK, no error message.

Hmm. Something to try on dreamhack.afunamatata.com:

First, check the gearman client config:

grep -i gearman ~/dw/etc/config*

Then, clear your Apache logs:

> ~/apache/logs/error_log
> ~/apache/logs/access_log

Next, start Apache, then run the following command:

~/dw/bin/gearmand -p 17096 --debug=1

(It won't give you a shell prompt, but don't worry.)

Then, do something that should use Gearman, using your browser. This should give you a message about a client request. Reply with all output from your shell session (starting with the grep), the content of error_log (you can ^C out of gearmand at this point), and the output from your web browser. (Sanitize as needed.)

[anarres]

Basically, nothing happened, I'm not sure my dreamhack even tried to use Gearman. Is there something I could do that would definitely use Gearman, that doesn't involve making an electronic payment? The problem is that I've hacked the payments system in an attempt to get it to accept an unvalidated credit card. It isn't working, but that could either be because Gearman doesn't work, or just due to my bull-in-a-china-shop approach to trying to hack the payments system.

The output in the browser (http://www.anarres.dreamhack.afunamatata.com/shop/creditcard_wait?ordernum=39-qm3qp4zf4qnxw36x56m9&cctransid=24) is:

Internal Failure
We're sorry, but something failed internally on our end attempting to process your payment. Your credit card has probably not been charged.

Here's the output:

dh-anarres@li-166:~$ grep -i gearman ~/dw/etc/config*
/dreamhack/8096-anarres/dw/etc/config-local.pl: @GEARMAN_SERVERS = ( '127.0.0.1:17096' );
dh-anarres@li-166:~$ > ~/apache/logs/error_log
dh-anarres@li-166:~$ > ~/apache/logs/access_log
dh-anarres@li-166:~$ apachectl start
dh-anarres@li-166:~$ ~/dw/bin/gearmand -p 17096 --debug=1

dh-anarres@li-166:~$ nano ~/apache/logs/error_log

NOTE: Google::Checkout::* Perl modules were not found.
NOTE: Google::Checkout::* Perl modules were not found.
[Tue Jul 20 09:50:27 2010] [notice] Apache/2.2.9 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operati$
[Tue Jul 20 09:51:07 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

[pauamma]

Hmm. I'm getting outta my depth here, trying to get an expert to look at it. (I'll still tell you about any ideas I may have.) Also, Sophie mentioned that you configured the wrong Gearman port on your hack.dreamwidth.net dreamhack. You have

@GEARMAN_SERVERS = ( '127.0.0.1:8214' );

in etc/config-local.pl, but that's your web port. You should be using 17214 for Gearman on hack.dreamwidth.net.

the saga continues...

[anarres]

So, I've had another go at the debugging you suggested. Running

~/dw/bin/gearmand -p 17096 --debug=1

and then attempting to do something that uses gearman gives this error message:

Error: Can't locate object method "CMD_" via package "Gearman::Server::Client" at /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm line 490.

And there's nothing in the apache error log. The line it's complaining about in /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm is this:

my $cmd_name = "CMD_" . Gearman::Util::cmd_name($cmd);

It seems to be asking for a module /usr/local/share/perl/5.10.0/Gearman/Util.pm, which doesn't exist.

Maybe the problem is that as Alierak said it should be looking at the Gearman modules in ~/dw, not way off in the filesystem? But I tried reinstalling gearman with perl Makefile.PL PREFIX=$HOME/dw as Alierak suggested, but that doesn't seem to have worked.

But my way of installing was a little different from Alierak's. When I copied gearmand and Makefile.PL into ~/dw/bin and tried to install it complained that it couldn't find some other files, so instead I installed from ~/dw/cvs/gearman/server, I don't know if that makes a difference.

Re: the saga continues...

[pauamma]

Looks like there's something wrong in the way Gearman components are installed, or perhaps conflicts between the ones installed in ~/dw/cgi-bin and the ones installed in /usr/local/share/perl/5.10.0/. I asked Afuna to check.

Re: the saga continues...

[pauamma]

OK, I've had some success on dreamhack.afunalmatata.com making sure gearmand runs, by *not* using $LJHOME/bin/gearmand.

You may want to try the following commands to see what happens:

rm $LJHOME/bin/gearmand # get rid of possible troublemaker
gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096 # Actually run gearmand
cat $LJHOME/temp/gearmand.pid # Check 1
netstat -A inet -a -n -p | grep 17096 # Check 2
ps auxww | grep -i gearmand # Check 3

(Note: I haven't actually tried running Gearman workers yet - that's next on my list, but since it looks like you have problems with gearmand itself, you may want to try this method.)

For reference, I got the following output from these commands, and if it works for you, the output should be similar:

dh-pauamma@li-166:~/dw/cvs/dw-free$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17082
dh-pauamma@li-166:~/dw/cvs/dw-free$ cat $LJHOME/temp/gearmand.pid
6235
dh-pauamma@li-166:~/dw/cvs/dw-free$ netstat -A inet -a -n -p | grep 17082
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17082 0.0.0.0:* LISTEN 6235/perl
dh-pauamma@li-166:~/dw/cvs/dw-free$ ps auxww | grep -i gearman | less afuna

2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl

/usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1009 6242 0.0 0.1 7452 968 pts/7 SN+ 18:26 0:00 grep -i gearman

Re: the saga continues...

[anarres]

My output is similar to yours. I get the same error when trying to make a credit card payment though.

dh-anarres@li-166:~$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096
dh-anarres@li-166:~$ cat $LJHOME/temp/gearmand.pid
6699
dh-anarres@li-166:~$ netstat -A inet -a -n -p | grep 17096
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17096 0.0.0.0:* LISTEN 6699/perl
tcp 0 0 127.0.0.1:54655 127.0.0.1:17096 ESTABLISHED 8304/perl
tcp 0 0 127.0.0.1:54654 127.0.0.1:17096 ESTABLISHED 10458/perl
tcp 0 0 127.0.0.1:54656 127.0.0.1:17096 ESTABLISHED 8329/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54654 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54655 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54656 ESTABLISHED 6699/perl
dh-anarres@li-166:~$ ps auxww | grep -i gearmand
afuna 2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1028 6699 0.0 1.5 46456 7892 ? SN 21:01 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres/dw/temp/gearmand.pid -p 17096
1028 6704 0.0 0.1 7448 972 pts/10 SN+ 21:03 0:00 grep -i gearmand

Re: the saga continues...

[pauamma]

OK, just ran the following:

ps uww 6699 8304 8329 10458
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1028 8304 5.3 1.3 263712 7088 ? SN Jul19 769:30 perl shop-creditcard-charge
1028 8329 5.4 2.4 263716 12948 ? SN Jul19 781:20 perl shop_creditcard-charge.PROPER
1028 10458 0.0 0.6 263636 3328 ? SN Jul20 5:10 perl shop-creditcard-charge
1028 6699 90.4 0.5 46584 2632 ? RN Jul26 3879:34 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres /dw/temp/gearmand.pid -p 17096

and it looks like you have 3 CC gearman workers trying to connect to gearmand, and gearmand is eating oodles of CPU. That doesn't sound healthy. Kill all 4 processes and make sure they're dead, restart gearmand, and go through the checks again:

cat $LJHOME/temp/gearmand.pid
netstat -A inet -a -n -p | grep 17096
ps auxww | grep -i gearmand

Then, start apache, and start 1 (and only 1) worker process (either shop-creditcard-charge or shop_creditcard-charge.PROPER, depending on which you need), in verbose mode.

$LJHOME/bin/worker/shop-creditcard-charge -v

(or shop_creditcard-charge.PROPER).

Test it, and see if things work better. (You should be getting messages from the worker process. Paste them in your reply, and ^C to kill the worker when you're done.)

I haven't tried running the shop worker, but I've been able to test sysban gearman processing, and it worked without a hitch. My guess is that something (perhaps the three shop worker processes) was messing things up. If you still have problems, I'll walk you through a test using sysban, so we can make sure your basic gearman configuration works.

Re: the saga continues...

[anarres]

This is probably a dumb question but, how do I kill processes?

Re: the saga continues...

[denise]

Ha! One I finally know! (No such thing as a dumb question around here.)

Once you've identified the process ID ('ps -aux' will get you a list of the full set of processes running on the machine), "kill [processid]", then wait. It should then be gone when you list the processes again. If it isn't, you might need to wait a few minutes and issue the kill again; if that doesn't work, "kill -9 [pid]" will get rid of a really stuck process, but you should reserve that for the stubborn ones, since it forces an exit and doesn't do a graceful shutdown.

Re: the saga continues...

[anarres]

Thanks, I think I killed it. Phew.

Re: the saga continues...

[denise]

You're welcome! I'm just excited that there was a question I knew the answer to. *G*