dw_dev_training: Gearman on Dreamhack

anarres (

anarres) wrote in

dw_dev_training,
@ 2010-07-08 12:48 pm UTC

Entry tags:

dreamhacks, gearman

Gearman on Dreamhack

Has anyone got any hints on getting Gearman working on a Dreamhack? I keep getting this: '[Error: Unable to get gearman client. @ hack.dreamwidth.net]'

(25 comments) - (Post a new comment)
(Flat) (Top-level comments only)

	no subject alierak 2010-07-08 01:30 pm UTC (link)
	Yeah, I have it on mine. I got gearmand from dw/cvs/gearman/server/ and put it in dw/bin/, then ran it with "~/dw/bin/gearmand -p 17103 &". You'll need a different port for yours; ask sophie. Then in config-local.pl, I have: @GEARMAN_SERVERS = ( '127.0.0.1:17103' ); (Reply to this) (Thread) (Hide 1 comment) (Show 1 comment)

no subject

alierak
2010-07-08 04:57 pm UTC (link)

I seem to recall needing another step during gearmand installation. It comes with some perl modules like Gearman::Server that (for me) ended up in ~/dw/share/perl/5.8.8/Gearman/ so I had to add that to PERL5LIB in ~/.bash_profile in order for gearmand to find it. I don't remember exactly what command I used to install gearmand but it probably involved 'perl Makefile.PL' in some way.

(Reply to this) (Thread from start) (Parent)

	no subject azurelunatic 2010-07-08 11:00 pm UTC (link)
	I'm linking to this both in case it has helpful hints, and so that if there's stuff missing from it, maybe it can get edited in: http://wiki.dwscoalition.org/notes/Setting_up_Gearman (Reply to this) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject alierak 2010-07-09 03:52 am UTC (link)
	As it stands, I guess that would be fine for setting it up on your own server, but not really on a shared dreamhack server where different users might need different versions, ports, etc. I had the same problem with MogileFS, of course. (Reply to this) (Thread from start) (Parent)

no subject

anarres
2010-07-09 12:39 pm UTC (link)

Thanks! I copied gearmand into ~/dw/bin and attempted to do all this:

$ perl Makefile.PL
$ make
$ make test
$ make install

But I got this:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
ERROR: Can't create '/usr/local/bin'
Do not have write permissions on '/usr/local/bin'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

so I had a look and there is a /usr/local/bin/gearmand, but clearly this isn't the *right* gearmand for it to look at, I want it to look at the one in ~/dw/bin instead..

(Reply to this) (Thread) (Hide 3 comments) (Show 3 comments)

	no subject anarres 2010-07-09 12:52 pm UTC (link)
	Hmm, the error message I get has changed, from 'Unable to get gearman client'to 'Unable to insert Gearman job.' (Reply to this) (Thread from start) (Parent) (Thread) (Hide 1 comment) (Show 1 comment)

	no subject anarres 2010-07-09 01:01 pm UTC (link)
	Oh hang on, I think that error might be due to the script I'm running and not due to Gearman itself. (Reply to this) (Thread from start) (Parent)

	no subject alierak 2010-07-09 01:27 pm UTC (link)
	Yeah, there was an option to tell it where to install, the default location being /usr/local/ - I think this is what I did: perl Makefile.PL PREFIX=$HOME/dw (Reply to this) (Thread from start) (Parent)

no subject

pauamma
2010-07-18 07:03 pm UTC (link)

Are you trying on hack.dreamwidth.net, or on dreamhack.afunamatata.com? If the latter, you should configure it in config-local.pl http://dw-dev-training.dreamwidth.org/22144.html?thread=153984#cmt153984, using port 17096 instead of 17103.

Thus,

@GEARMAN_SERVERS = ( '127.0.0.1:17096' );

and

~/dw/bin/gearmand -p 17096 &

What happens then?

(Reply to this) (Thread) (Hide 16 comments) (Show 16 comments)

no subject

anarres
2010-07-18 08:16 pm UTC (link)

Ah, so my port number is... something completely different from what I thought it was, d'oh! Having done what you suggest I now get a different error on the BML page, 'Internal Failure We're sorry, but something failed internally on our end attempting to process your payment. Your credit card has probably not been charged.' And attempting to run a gearman worker script by hand gives this: 'First gearmand server in @LJ::GEARMAN_SERVERS (127.0.0.1:17096) isn't responding.'

So... this takes me one more rung up the ladder of debugging, I think..

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 15 comments) (Show 15 comments)

	no subject anarres 2010-07-18 08:30 pm UTC (link)
	And ~/dw/bin/gearmand -p 17096 & gives this: [1] 3330 (Reply to this) (Thread from start) (Parent) (Thread) (Hide 14 comments) (Show 14 comments)

no subject

pauamma
2010-07-19 02:02 pm UTC (link)

Hmm. I checked on both dreamhack.afunamatata.com and hack.dreamwidth.net and ps auxww doesn't show a gearmand running as dh-anarres, which means it might not start properly. What happens if you run

~/dw/bin/gearmand -p 17096

instead? (You may need to ^C to terminate it, if it doesn't terminate of its own accord.)

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 13 comments) (Show 13 comments)

	no subject anarres 2010-07-19 05:57 pm UTC (link)
	It hangs, and doesn't give me back a command prompt until I do ^C, but doesn't do anything else. (Reply to this) (Thread from start) (Parent) (Thread) (Hide 12 comments) (Show 12 comments)

no subject

pauamma
2010-07-19 07:30 pm UTC (link)

OK, no error message.

Hmm. Something to try on dreamhack.afunamatata.com:

First, check the gearman client config:

grep -i gearman ~/dw/etc/config*

Then, clear your Apache logs:

> ~/apache/logs/error_log
> ~/apache/logs/access_log

Next, start Apache, then run the following command:

~/dw/bin/gearmand -p 17096 --debug=1

(It won't give you a shell prompt, but don't worry.)

Then, do something that should use Gearman, using your browser. This should give you a message about a client request. Reply with all output from your shell session (starting with the grep), the content of error_log (you can ^C out of gearmand at this point), and the output from your web browser. (Sanitize as needed.)

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 11 comments) (Show 11 comments)

no subject

anarres
2010-07-20 10:10 am UTC (link)

Basically, nothing happened, I'm not sure my dreamhack even tried to use Gearman. Is there something I could do that would definitely use Gearman, that doesn't involve making an electronic payment? The problem is that I've hacked the payments system in an attempt to get it to accept an unvalidated credit card. It isn't working, but that could either be because Gearman doesn't work, or just due to my bull-in-a-china-shop approach to trying to hack the payments system.

The output in the browser (http://www.anarres.dreamhack.afunamatata.com/shop/creditcard_wait?ordernum=39-qm3qp4zf4qnxw36x56m9&cctransid=24) is:

Internal Failure
We're sorry, but something failed internally on our end attempting to process your payment. Your credit card has probably not been charged.

Here's the output:

dh-anarres@li-166:~$ grep -i gearman ~/dw/etc/config*
/dreamhack/8096-anarres/dw/etc/config-local.pl: @GEARMAN_SERVERS = ( '127.0.0.1:17096' );
dh-anarres@li-166:~$ > ~/apache/logs/error_log
dh-anarres@li-166:~$ > ~/apache/logs/access_log
dh-anarres@li-166:~$ apachectl start
dh-anarres@li-166:~$ ~/dw/bin/gearmand -p 17096 --debug=1

dh-anarres@li-166:~$ nano ~/apache/logs/error_log

NOTE: Google::Checkout::* Perl modules were not found.
NOTE: Google::Checkout::* Perl modules were not found.
[Tue Jul 20 09:50:27 2010] [notice] Apache/2.2.9 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operati$
[Tue Jul 20 09:51:07 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 10 comments) (Show 10 comments)

no subject

pauamma
2010-07-20 03:01 pm UTC (link)

Hmm. I'm getting outta my depth here, trying to get an expert to look at it. (I'll still tell you about any ideas I may have.) Also, Sophie mentioned that you configured the wrong Gearman port on your hack.dreamwidth.net dreamhack. You have

@GEARMAN_SERVERS = ( '127.0.0.1:8214' );

in etc/config-local.pl, but that's your web port. You should be using 17214 for Gearman on hack.dreamwidth.net.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 9 comments) (Show 9 comments)

the saga continues...

anarres
2010-07-23 08:32 pm UTC (link)

So, I've had another go at the debugging you suggested. Running

~/dw/bin/gearmand -p 17096 --debug=1

and then attempting to do something that uses gearman gives this error message:

Error: Can't locate object method "CMD_" via package "Gearman::Server::Client" at /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm line 490.

And there's nothing in the apache error log. The line it's complaining about in /usr/local/share/perl/5.10.0/Gearman/Server/Client.pm is this:

my $cmd_name = "CMD_" . Gearman::Util::cmd_name($cmd);

It seems to be asking for a module /usr/local/share/perl/5.10.0/Gearman/Util.pm, which doesn't exist.

Maybe the problem is that as Alierak said it should be looking at the Gearman modules in ~/dw, not way off in the filesystem? But I tried reinstalling gearman with perl Makefile.PL PREFIX=$HOME/dw as Alierak suggested, but that doesn't seem to have worked.

But my way of installing was a little different from Alierak's. When I copied gearmand and Makefile.PL into ~/dw/bin and tried to install it complained that it couldn't find some other files, so instead I installed from ~/dw/cvs/gearman/server, I don't know if that makes a difference.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 8 comments) (Show 8 comments)

	Re: the saga continues... pauamma 2010-07-24 02:25 pm UTC (link)
	Looks like there's something wrong in the way Gearman components are installed, or perhaps conflicts between the ones installed in ~/dw/cgi-bin and the ones installed in /usr/local/share/perl/5.10.0/. I asked Afuna to check. (Reply to this) (Thread from start) (Parent)

Re: the saga continues...

pauamma
2010-07-26 07:36 pm UTC (link)

OK, I've had some success on dreamhack.afunalmatata.com making sure gearmand runs, by *not* using $LJHOME/bin/gearmand.

You may want to try the following commands to see what happens:

rm $LJHOME/bin/gearmand # get rid of possible troublemaker
gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096 # Actually run gearmand
cat $LJHOME/temp/gearmand.pid # Check 1
netstat -A inet -a -n -p | grep 17096 # Check 2
ps auxww | grep -i gearmand # Check 3

(Note: I haven't actually tried running Gearman workers yet - that's next on my list, but since it looks like you have problems with gearmand itself, you may want to try this method.)

For reference, I got the following output from these commands, and if it works for you, the output should be similar:

dh-pauamma@li-166:~/dw/cvs/dw-free$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17082
dh-pauamma@li-166:~/dw/cvs/dw-free$ cat $LJHOME/temp/gearmand.pid
6235
dh-pauamma@li-166:~/dw/cvs/dw-free$ netstat -A inet -a -n -p | grep 17082
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17082 0.0.0.0:* LISTEN 6235/perl
dh-pauamma@li-166:~/dw/cvs/dw-free$ ps auxww | grep -i gearman | less afuna

2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl

/usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1009 6242 0.0 0.1 7452 968 pts/7 SN+ 18:26 0:00 grep -i gearman

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 6 comments) (Show 6 comments)

Re: the saga continues...

anarres
2010-07-26 09:13 pm UTC (link)

My output is similar to yours. I get the same error when trying to make a credit card payment though.

dh-anarres@li-166:~$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096
dh-anarres@li-166:~$ cat $LJHOME/temp/gearmand.pid
6699
dh-anarres@li-166:~$ netstat -A inet -a -n -p | grep 17096
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17096 0.0.0.0:* LISTEN 6699/perl
tcp 0 0 127.0.0.1:54655 127.0.0.1:17096 ESTABLISHED 8304/perl
tcp 0 0 127.0.0.1:54654 127.0.0.1:17096 ESTABLISHED 10458/perl
tcp 0 0 127.0.0.1:54656 127.0.0.1:17096 ESTABLISHED 8329/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54654 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54655 ESTABLISHED 6699/perl
tcp 0 0 127.0.0.1:17096 127.0.0.1:54656 ESTABLISHED 6699/perl
dh-anarres@li-166:~$ ps auxww | grep -i gearmand
afuna 2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1028 6699 0.0 1.5 46456 7892 ? SN 21:01 0:00 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres/dw/temp/gearmand.pid -p 17096
1028 6704 0.0 0.1 7448 972 pts/10 SN+ 21:03 0:00 grep -i gearmand

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 5 comments) (Show 5 comments)

Re: the saga continues...

pauamma
2010-07-29 08:53 pm UTC (link)

OK, just ran the following:

ps uww 6699 8304 8329 10458
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1028 8304 5.3 1.3 263712 7088 ? SN Jul19 769:30 perl shop-creditcard-charge
1028 8329 5.4 2.4 263716 12948 ? SN Jul19 781:20 perl shop_creditcard-charge.PROPER
1028 10458 0.0 0.6 263636 3328 ? SN Jul20 5:10 perl shop-creditcard-charge
1028 6699 90.4 0.5 46584 2632 ? RN Jul26 3879:34 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres /dw/temp/gearmand.pid -p 17096

and it looks like you have 3 CC gearman workers trying to connect to gearmand, and gearmand is eating oodles of CPU. That doesn't sound healthy. Kill all 4 processes and make sure they're dead, restart gearmand, and go through the checks again:

cat $LJHOME/temp/gearmand.pid
netstat -A inet -a -n -p | grep 17096
ps auxww | grep -i gearmand

Then, start apache, and start 1 (and only 1) worker process (either shop-creditcard-charge or shop_creditcard-charge.PROPER, depending on which you need), in verbose mode.

$LJHOME/bin/worker/shop-creditcard-charge -v

(or shop_creditcard-charge.PROPER).

Test it, and see if things work better. (You should be getting messages from the worker process. Paste them in your reply, and ^C to kill the worker when you're done.)

I haven't tried running the shop worker, but I've been able to test sysban gearman processing, and it worked without a hitch. My guess is that something (perhaps the three shop worker processes) was messing things up. If you still have problems, I'll walk you through a test using sysban, so we can make sure your basic gearman configuration works.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 4 comments) (Show 4 comments)

	Re: the saga continues... anarres 2010-07-30 12:13 pm UTC (link)
	This is probably a dumb question but, how do I kill processes? (Reply to this) (Thread from start) (Parent) (Thread) (Hide 3 comments) (Show 3 comments)

Re: the saga continues...

denise
2010-07-30 12:16 pm UTC (link)

Ha! One I finally know! (No such thing as a dumb question around here.)

Once you've identified the process ID ('ps -aux' will get you a list of the full set of processes running on the machine), "kill [processid]", then wait. It should then be gone when you list the processes again. If it isn't, you might need to wait a few minutes and issue the kill again; if that doesn't work, "kill -9 [pid]" will get rid of a really stuck process, but you should reserve that for the stubborn ones, since it forces an exit and doesn't do a graceful shutdown.

(Reply to this) (Thread from start) (Parent) (Thread) (Hide 2 comments) (Show 2 comments)

	Re: the saga continues... anarres 2010-07-30 03:31 pm UTC (link)
	Thanks, I think I killed it. Phew. (Reply to this) (Thread from start) (Parent) (Thread) (Hide 1 comment) (Show 1 comment)

	Re: the saga continues... denise 2010-07-30 03:34 pm UTC (link)
	You're welcome! I'm just excited that there was a question I knew the answer to. G (Reply to this) (Thread from start) (Parent)

(25 comments) - (Post a new comment)
(Flat) (Top-level comments only)