anarres (
anarres) wrote in
dw_dev_training2010-07-08 12:48 pm
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
![[site community profile]](https://www.dreamwidth.org/img/comm_staff.png)
Entry tags:
Gearman on Dreamhack
Has anyone got any hints on getting Gearman working on a Dreamhack? I keep getting this: '[Error: Unable to get gearman client. @ hack.dreamwidth.net]'
no subject
@GEARMAN_SERVERS = ( '127.0.0.1:17103' );
no subject
no subject
no subject
no subject
$ perl Makefile.PL
$ make
$ make test
$ make install
But I got this:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR: Can't create '/usr/local/bin'
Do not have write permissions on '/usr/local/bin'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
so I had a look and there is a /usr/local/bin/gearmand, but clearly this isn't the *right* gearmand for it to look at, I want it to look at the one in ~/dw/bin instead..
no subject
no subject
no subject
perl Makefile.PL PREFIX=$HOME/dw
no subject
Thus, and What happens then?
no subject
So... this takes me one more rung up the ladder of debugging, I think..
no subject
~/dw/bin/gearmand -p 17096 &
gives this:
[1] 3330
no subject
no subject
no subject
Hmm. Something to try on dreamhack.afunamatata.com:
First, check the gearman client config:
grep -i gearman ~/dw/etc/config*
Then, clear your Apache logs:
> ~/apache/logs/error_log
> ~/apache/logs/access_log
Next, start Apache, then run the following command:
~/dw/bin/gearmand -p 17096 --debug=1
(It won't give you a shell prompt, but don't worry.)
Then, do something that should use Gearman, using your browser. This should give you a message about a client request. Reply with all output from your shell session (starting with the grep), the content of error_log (you can ^C out of gearmand at this point), and the output from your web browser. (Sanitize as needed.)
no subject
The output in the browser (http://www.anarres.dreamhack.afunamatata.com/shop/creditcard_wait?ordernum=39-qm3qp4zf4qnxw36x56m9&cctransid=24) is:
Internal Failure
We're sorry, but something failed internally on our end attempting to process your payment. Your credit card has probably not been charged.
Here's the output:
dh-anarres@li-166:~$ grep -i gearman ~/dw/etc/config*
/dreamhack/8096-anarres/dw/etc/config-local.pl: @GEARMAN_SERVERS = ( '127.0.0.1:17096' );
dh-anarres@li-166:~$ > ~/apache/logs/error_log
dh-anarres@li-166:~$ > ~/apache/logs/access_log
dh-anarres@li-166:~$ apachectl start
dh-anarres@li-166:~$ ~/dw/bin/gearmand -p 17096 --debug=1
dh-anarres@li-166:~$ nano ~/apache/logs/error_log
NOTE: Google::Checkout::* Perl modules were not found.
NOTE: Google::Checkout::* Perl modules were not found.
[Tue Jul 20 09:50:27 2010] [notice] Apache/2.2.9 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operati$
[Tue Jul 20 09:51:07 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
no subject
the saga continues...
~/dw/bin/gearmand -p 17096 --debug=1
and then attempting to do something that uses gearman gives this error message:
Error: Can't locate object method "CMD_" via package "Gearman::Server::Client" at /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm line 490.
And there's nothing in the apache error log. The line it's complaining about in /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm is this:
my $cmd_name = "CMD_" . Gearman::Util::cmd_name($cmd);
It seems to be asking for a module /usr/local/share/perl/5.10.0/Gearman/Util.pm, which doesn't exist.
Maybe the problem is that as Alierak said it should be looking at the Gearman modules in ~/dw, not way off in the filesystem? But I tried reinstalling gearman with perl Makefile.PL PREFIX=$HOME/dw as Alierak suggested, but that doesn't seem to have worked.
But my way of installing was a little different from Alierak's. When I copied gearmand and Makefile.PL into ~/dw/bin and tried to install it complained that it couldn't find some other files, so instead I installed from ~/dw/cvs/gearman/server, I don't know if that makes a difference.
Re: the saga continues...
Re: the saga continues...
You may want to try the following commands to see what happens:
(Note: I haven't actually tried running Gearman workers yet - that's next on my list, but since it looks like you have problems with gearmand itself, you may want to try this method.)
For reference, I got the following output from these commands, and if it works for you, the output should be similar:
dh-pauamma@li-166:~/dw/cvs/dw-free$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17082
dh-pauamma@li-166:~/dw/cvs/dw-free$ cat $LJHOME/temp/gearmand.pid
6235
dh-pauamma@li-166:~/dw/cvs/dw-free$ netstat -A inet -a -n -p | grep 17082
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17082 0.0.0.0:* LISTEN 6235/perl
dh-pauamma@li-166:~/dw/cvs/dw-free$ ps auxww | grep -i gearman | less afuna
2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl
/usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1009 6242 0.0 0.1 7452 968 pts/7 SN+ 18:26 0:00 grep -i gearman
Re: the saga continues...
dh-anarres@li-166:~$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096
dh-anarres@li-166:~$ cat $LJHOME/temp/gearmand.pid
6699
dh-anarres@li-166:~$ netstat -A inet -a -n -p | grep 17096
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17096 0.0.0.0:* LISTEN 6699/perl
tcp 0 0 127.0.0.1:54655 127.0.0.1:17096 ESTABLISHED 8304/perl
tcp 0 0 127.0.0.1:54654 127.0.0.1:17096 ESTABLISHED 10458/perl
tcp 0 0 127.0.0.1:54656 127.0.0.1:17096 ESTABLISHED 8329/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54654 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54655 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54656 ESTABLISHED 6699/perl
dh-anarres@li-166:~$ ps auxww | grep -i gearmand
afuna 2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1028 6699 0.0 1.5 46456 7892 ? SN 21:01 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres/dw/temp/gearmand.pid -p 17096
1028 6704 0.0 0.1 7448 972 pts/10 SN+ 21:03 0:00 grep -i gearmand
Re: the saga continues...
ps uww 6699 8304 8329 10458
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1028 8304 5.3 1.3 263712 7088 ? SN Jul19 769:30 perl shop-creditcard-charge
1028 8329 5.4 2.4 263716 12948 ? SN Jul19 781:20 perl shop_creditcard-charge.PROPER
1028 10458 0.0 0.6 263636 3328 ? SN Jul20 5:10 perl shop-creditcard-charge
1028 6699 90.4 0.5 46584 2632 ? RN Jul26 3879:34 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres /dw/temp/gearmand.pid -p 17096
and it looks like you have 3 CC gearman workers trying to connect to gearmand, and gearmand is eating oodles of CPU. That doesn't sound healthy. Kill all 4 processes and make sure they're dead, restart gearmand, and go through the checks again:
cat $LJHOME/temp/gearmand.pid
netstat -A inet -a -n -p | grep 17096
ps auxww | grep -i gearmand
Then, start apache, and start 1 (and only 1) worker process (either shop-creditcard-charge or shop_creditcard-charge.PROPER, depending on which you need), in verbose mode. (or shop_creditcard-charge.PROPER).
Test it, and see if things work better. (You should be getting messages from the worker process. Paste them in your reply, and ^C to kill the worker when you're done.)
I haven't tried running the shop worker, but I've been able to test sysban gearman processing, and it worked without a hitch. My guess is that something (perhaps the three shop worker processes) was messing things up. If you still have problems, I'll walk you through a test using sysban, so we can make sure your basic gearman configuration works.
Re: the saga continues...
Re: the saga continues...
Once you've identified the process ID ('ps -aux' will get you a list of the full set of processes running on the machine), "kill [processid]", then wait. It should then be gone when you list the processes again. If it isn't, you might need to wait a few minutes and issue the kill again; if that doesn't work, "kill -9 [pid]" will get rid of a really stuck process, but you should reserve that for the stubborn ones, since it forces an exit and doesn't do a graceful shutdown.
Re: the saga continues...
Re: the saga continues...