anarres: (Default)
anarres ([personal profile] anarres) wrote in [site community profile] dw_dev_training2010-07-08 12:48 pm
Entry tags:

Gearman on Dreamhack

Has anyone got any hints on getting Gearman working on a Dreamhack? I keep getting this: '[Error: Unable to get gearman client. @ hack.dreamwidth.net]'
alierak: (Default)

[personal profile] alierak 2010-07-08 01:30 pm (UTC)(link)
Yeah, I have it on mine. I got gearmand from dw/cvs/gearman/server/ and put it in dw/bin/, then ran it with "~/dw/bin/gearmand -p 17103 &". You'll need a different port for yours; ask [personal profile] sophie. Then in config-local.pl, I have:

@GEARMAN_SERVERS = ( '127.0.0.1:17103' );
alierak: (Default)

[personal profile] alierak 2010-07-08 04:57 pm (UTC)(link)
I seem to recall needing another step during gearmand installation. It comes with some perl modules like Gearman::Server that (for me) ended up in ~/dw/share/perl/5.8.8/Gearman/ so I had to add that to PERL5LIB in ~/.bash_profile in order for gearmand to find it. I don't remember exactly what command I used to install gearmand but it probably involved 'perl Makefile.PL' in some way.
azurelunatic: Vivid pink Alaskan wild rose. (Default)

[personal profile] azurelunatic 2010-07-08 11:00 pm (UTC)(link)
I'm linking to this both in case it has helpful hints, and so that if there's stuff missing from it, maybe it can get edited in: http://wiki.dwscoalition.org/notes/Setting_up_Gearman
alierak: (Default)

[personal profile] alierak 2010-07-09 03:52 am (UTC)(link)
As it stands, I guess that would be fine for setting it up on your own server, but not really on a shared dreamhack server where different users might need different versions, ports, etc. I had the same problem with MogileFS, of course.
alierak: (Default)

[personal profile] alierak 2010-07-09 01:27 pm (UTC)(link)
Yeah, there was an option to tell it where to install, the default location being /usr/local/ - I think this is what I did:

perl Makefile.PL PREFIX=$HOME/dw
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2010-07-18 07:03 pm (UTC)(link)
Are you trying on hack.dreamwidth.net, or on dreamhack.afunamatata.com? If the latter, you should configure it in config-local.pl http://dw-dev-training.dreamwidth.org/22144.html?thread=153984#cmt153984, using port 17096 instead of 17103.

Thus,
@GEARMAN_SERVERS = ( '127.0.0.1:17096' );
and
~/dw/bin/gearmand -p 17096 &
What happens then?
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2010-07-19 02:02 pm (UTC)(link)
Hmm. I checked on both dreamhack.afunamatata.com and hack.dreamwidth.net and ps auxww doesn't show a gearmand running as dh-anarres, which means it might not start properly. What happens if you run
~/dw/bin/gearmand -p 17096
instead? (You may need to ^C to terminate it, if it doesn't terminate of its own accord.)
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2010-07-19 07:30 pm (UTC)(link)
OK, no error message.

Hmm. Something to try on dreamhack.afunamatata.com:

First, check the gearman client config:

grep -i gearman ~/dw/etc/config*

Then, clear your Apache logs:

> ~/apache/logs/error_log
> ~/apache/logs/access_log

Next, start Apache, then run the following command:

~/dw/bin/gearmand -p 17096 --debug=1

(It won't give you a shell prompt, but don't worry.)

Then, do something that should use Gearman, using your browser. This should give you a message about a client request. Reply with all output from your shell session (starting with the grep), the content of error_log (you can ^C out of gearmand at this point), and the output from your web browser. (Sanitize as needed.)
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

[personal profile] pauamma 2010-07-20 03:01 pm (UTC)(link)
Hmm. I'm getting outta my depth here, trying to get an expert to look at it. (I'll still tell you about any ideas I may have.) Also, Sophie mentioned that you configured the wrong Gearman port on your hack.dreamwidth.net dreamhack. You have
@GEARMAN_SERVERS = ( '127.0.0.1:8214' );
in etc/config-local.pl, but that's your web port. You should be using 17214 for Gearman on hack.dreamwidth.net.
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

Re: the saga continues...

[personal profile] pauamma 2010-07-24 02:25 pm (UTC)(link)
Looks like there's something wrong in the way Gearman components are installed, or perhaps conflicts between the ones installed in ~/dw/cgi-bin and the ones installed in /usr/local/share/perl/5.10.0/. I asked Afuna to check.
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

Re: the saga continues...

[personal profile] pauamma 2010-07-26 07:36 pm (UTC)(link)
OK, I've had some success on dreamhack.afunalmatata.com making sure gearmand runs, by *not* using $LJHOME/bin/gearmand.

You may want to try the following commands to see what happens:
rm $LJHOME/bin/gearmand # get rid of possible troublemaker
gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17096 # Actually run gearmand
cat $LJHOME/temp/gearmand.pid # Check 1
netstat -A inet -a -n -p | grep 17096 # Check 2
ps auxww | grep -i gearmand # Check 3


(Note: I haven't actually tried running Gearman workers yet - that's next on my list, but since it looks like you have problems with gearmand itself, you may want to try this method.)

For reference, I got the following output from these commands, and if it works for you, the output should be similar:

dh-pauamma@li-166:~/dw/cvs/dw-free$ gearmand -d --pidfile=$LJHOME/temp/gearmand.pid -p 17082
dh-pauamma@li-166:~/dw/cvs/dw-free$ cat $LJHOME/temp/gearmand.pid
6235
dh-pauamma@li-166:~/dw/cvs/dw-free$ netstat -A inet -a -n -p | grep 17082
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:17082 0.0.0.0:* LISTEN 6235/perl
dh-pauamma@li-166:~/dw/cvs/dw-free$ ps auxww | grep -i gearman | less afuna

2953 0.0 0.1 46420 704 pts/4 S Jul18 0:00 /usr/bin/perl /usr/local/bin/gearmand
1009 6235 0.0 1.4 46560 7792 ? SN 18:24 0:00 /usr/bin/perl

/usr/local/bin/gearmand -d --pidfile=/dreamhack/8082-pauamma/dw/temp/gearmand.pid -p 17082
1009 6242 0.0 0.1 7452 968 pts/7 SN+ 18:26 0:00 grep -i gearman
pauamma: Cartooney crab wearing hot pink and acid green facemask holding drink with straw (Default)

Re: the saga continues...

[personal profile] pauamma 2010-07-29 08:53 pm (UTC)(link)
OK, just ran the following:

ps uww 6699 8304 8329 10458
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1028 8304 5.3 1.3 263712 7088 ? SN Jul19 769:30 perl shop-creditcard-charge
1028 8329 5.4 2.4 263716 12948 ? SN Jul19 781:20 perl shop_creditcard-charge.PROPER
1028 10458 0.0 0.6 263636 3328 ? SN Jul20 5:10 perl shop-creditcard-charge
1028 6699 90.4 0.5 46584 2632 ? RN Jul26 3879:34 /usr/bin/perl /usr/local/bin/gearmand -d --pidfile=/dreamhack/8096-anarres /dw/temp/gearmand.pid -p 17096

and it looks like you have 3 CC gearman workers trying to connect to gearmand, and gearmand is eating oodles of CPU. That doesn't sound healthy. Kill all 4 processes and make sure they're dead, restart gearmand, and go through the checks again:

cat $LJHOME/temp/gearmand.pid
netstat -A inet -a -n -p | grep 17096
ps auxww | grep -i gearmand

Then, start apache, and start 1 (and only 1) worker process (either shop-creditcard-charge or shop_creditcard-charge.PROPER, depending on which you need), in verbose mode.
$LJHOME/bin/worker/shop-creditcard-charge -v
(or shop_creditcard-charge.PROPER).

Test it, and see if things work better. (You should be getting messages from the worker process. Paste them in your reply, and ^C to kill the worker when you're done.)

I haven't tried running the shop worker, but I've been able to test sysban gearman processing, and it worked without a hitch. My guess is that something (perhaps the three shop worker processes) was messing things up. If you still have problems, I'll walk you through a test using sysban, so we can make sure your basic gearman configuration works.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

Re: the saga continues...

[staff profile] denise 2010-07-30 12:16 pm (UTC)(link)
Ha! One I finally know! (No such thing as a dumb question around here.)

Once you've identified the process ID ('ps -aux' will get you a list of the full set of processes running on the machine), "kill [processid]", then wait. It should then be gone when you list the processes again. If it isn't, you might need to wait a few minutes and issue the kill again; if that doesn't work, "kill -9 [pid]" will get rid of a really stuck process, but you should reserve that for the stubborn ones, since it forces an exit and doesn't do a graceful shutdown.
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)

Re: the saga continues...

[staff profile] denise 2010-07-30 03:34 pm (UTC)(link)
You're welcome! I'm just excited that there was a question I knew the answer to. *G*