Questions about the statistics system

[anarres]

Hi all! I'm getting ready to start my GSoC project on Usage and Business Statistics, and I'm just starting to realize how much stuff I don't know! I hope this is the right place to post some questions about the statistics system:

1. How can I use the stats system modules in a perl script? I would like to write a simple script like this as a way of learning about how the stats system works:

--------------

use DW::StatStore;
use DW::StatData;
use DW::StatData::AccountsByType;

my $accounts_by_type = DW::StatData::AccountsByType->load_latest( DW::StatStore->get( "accounts" ) );

# MORE CODE...

--------------

but if I do this within my Dreamhack perl complains that it can't find the modules I asked it to use (even though exactly the same code works fine within /stats/site.bml!) I've tried adding an absolute path to the modules, but it turns out that these depend on a bunch of LJ modules which I can't find.

2. How can I get my Dreamhack to display /admin/stats.bml? (This is only displayed to users with admin priviledges).

3. What are 'crumbs'? I've seen this word in a couple of places, and I've no idea what it is.

4. Just to check, is it the case that if I'm editing a BML page, I should convert it to Template Toolkit?

5. /stats.bml uses the old legacy stats system - is it going to be replaced, or left as it is? If it's going to stay, should I add graphs to it (i.e. display the info on this page in graph form - apart from Age Distribution, which already has a graph)?.

Comments

[kareila]

1. A page run in web context inherits the LJ environment from the web server. For a standalone script, you need something like this at the head of the file:

require "$ENV{'LJHOME'}/cgi-bin/ljlib.pl";

(I copied that from one of the scripts in /bin)

2. You should be able to grant yourself admin privileges on your dev box by visiting the /admin page and looking at the options available.

3. I've never used crumbs, but my understanding is that LJ implemented them to track how users moved between different areas of the site. I don't know if DW is tracking them or not.

4. You don't have to, but in general, we are moving in that direction, so conversion would be a good thing. (I personally don't understand TT yet.)

5. I think that's a question for pauamma, but you can see the work that's been done so far in bug 124.

Hope this helps!

Correction re #1

[pauamma]

Actually, the preferred way is to use

use lib "$ENV{LJHOME}/cgi-bin";

, then use the modules as you normaly would. See

Re: Correction re #1

[anarres]

I still can't quite get it to work... I tried to run this:

---
use lib "$ENV{LJHOME}/cgi-bin";
use StatData;
use StatStore;

my $accounts_by_type = DW::StatData::AccountsByType->load_latest( DW::StatStore->get( "accounts" ) );
---

and got the error: Undefined subroutine &LJ::MemCache::get called at /dreamhack/home/8214-anarres/dw/cgi-bin/LJ/Typemap.pm line 180.

Re: Correction re #1

[pauamma]

Hmm. Stripped of comments, bin/worker/stats-collection starts with:

#!/usr/bin/perl
package DW::Worker::StatsCollection;

use strict;
use warnings;

use lib "$ENV{LJHOME}/cgi-bin";
use YAML;
use DW::StatStore;

use base 'LJ::Worker::Manual';

and it works fine for me. What do you have in the LJHOME and HOME environment variables? ie, what does the following output when typed from the shell prompt in your dreamhack:

echo HOME $HOME LJHOME $LJHOME

(it looks like it can't find some of the modules, and this is the most likely reason).

Oh, and just in case, is the use StatData; a pasto for use DW::StatData; ? (and ditto for StatStore)

Re: Correction re #1

[anarres]

Hmm... the command echo HOME $HOME LJHOME $LJHOME returns:

HOME /dreamhack/home/8214-anarres LJHOME /dreamhack/home/8214-anarres/dw

which I think is probably right? But I've just noticed that my /stats/site.bml just shows a blank page - but it was working before, after you showed me how to turn on stats collection. (I've checked that etc/stats-collectin.conf and bin/worker/stats-collection have the lines suggested in http://wiki.dwscoalition.org/notes/Stats_Design). So I'm worried that somewhere along the way I must have done something to make it stop working :-P

(Looks for a lazy solution) I need to bring my dreamhack up to date anyway, and if I caused the problem by fiddling with something somewhere, maybe updating my dreamhack would solve that?

Ah, and sorry, I had left out the DW:: in DW::StatData, and ditto for StatStore, above.

Re: Correction re #1

[anarres]

So I got rid of the error message above by doing this:

use LJ::MemCache;

I'm now getting a different error message: it's complaining that it can't call the method "get" on an undefined value. I think this is real progress!

Re: Correction re #1

[pauamma]

To cure the symptom, you should call LJ::MemCache::init() soon after use LJ::MemCache; But I think there's an underlying problem, because I don't think you need to use LJ::MemCache directly, so I'd like to look at that instead. Can you post your code somewhere (eg, pastebin site) and link?

Re: Correction re #1

[anarres]

here it is:

http://pastebin.com/pTC16zfj

Using LJ:::MemCache

[pauamma]

Executive summary: instead of

use LJ::MemCache;

, use

require "ljlib.pl";

Then you don't need to call LJ::MemCache::init (ljlib.pl does it for you) and everything should work fine without any error message.

Background discussion: http://dw-dev.dreamwidth.org/57740.html.

Re: Using LJ:::MemCache

[anarres]

Thanks!!

Re: Correction re #1

[pauamma]

HOME /dreamhack/home/8214-anarres LJHOME /dreamhack/home/8214-anarres/dw

which I think is probably right?

Yeah, they look good.

But I've just noticed that my /stats/site.bml just shows a blank page - but it was working before, after you showed me how to turn on stats collection. (I've checked that etc/stats-collectin.conf and bin/worker/stats-collection have the lines suggested in http://wiki.dwscoalition.org/notes/Stats_Design). So I'm worried that somewhere along the way I must have done something to make it stop working :-P

How long ago did you last run the stats-collection worker? Typically, the stats page will be empty (just the header and footer, no actual stats) if you didn't run the worker within the last 24 hours. If the stats page is *completely* blank (no header or footer either), or if you did run the stats worker in the last 24 hours, you probably have another problem, like a syntax error somewhere, in which case looking at the error log for your Apache should provide more info.

I need to bring my dreamhack up to date anyway, and if I caused the problem by fiddling with something somewhere, maybe updating my dreamhack would solve that?

I don't think that's the cause of your problem (see above), but that's a good thing to do anyway.

Re: Correction re #1

[anarres]

This is really odd - it was completely blank before (not even header and footer displaying) and now it looks the same as dreamwidth.org/stats/site - i.e. it gives the message "(Sorry those statistics aren't available right now)" for the three different types of stats, but I guess that's what it's meant to do at the moment. I did update my dreamhack so maybe that's what did it. One less thing to try to fix, I supose.

Re: Correction re #1

[pauamma]

Interesting. The message (repeated for all stat groups) is what you get if you didn't run the collection worker recently enough. (I was wrong earlier about he effect of that - brainfart)

[pauamma]

Precision re #4: what kareila said, but if you're thinking of the stats display pages (admin and public) I'm working on converting them, and should have a revised patch uploaded soon.

Re #5: the plan is that eventually, those stats it provides that the new system doesn't provide will get converted and merged in, at which point graphs for them would be nice. But that's low-priority.

[yvi]

3. Dreamwidth doesn't use crumbs, so no need to worry about them.

[cesy]

I can't answer your questions, but yay for statistics!