dw_dev_training | Question thread #3

Hi!

So I'm working on Bug 2886, and I've reached the point (with lots of help!) where I MIGHT have a working patch. And now I need to test it.

And I'm kind of stuck, to be honest.

How do I go about designing a testing plan for a bug like this? What kind of use cases do I need to take into account?

Thanks, guys!

Well, you've got 3 fields for each link, so it's a matter of testing how it works depending on what's filled:

- all 3 fields
- text + URL
- title + URL
- text + title
- text only
- title only
- URL only

Also, what happens if you put a URL in the title field? Text in the URL field? What happens if people use the URL to a picture? HTML in the title or text field?

Remember, when testing, you need to test both what you implemented, and what already existed, to make sure you didn't break it ;)

Good luck!

A few thoughts off the top of my head, everyone feel free to expand, elaborate, add test cases...
- Links without a title (should still be possible to specify them, and they should still display correctly)
- Links with an empty title
- Links with a not-empty title that's shorter than the maximum allowed
- Links with a not-empty title that's exactly the maximum allowed
- Links with a not-empty title that's 1 byte longer than the maximum allowed
- Links with a not-empty title longer than that
- Links with a not-empty title with a Unicode character that straddles the maximum allowed (eg, if the maximum allowed is 40 bytes, what happens if a 2-byte Unicode character takes positions 40 and 41?)
- Links with HTML markup characters in the title (eg, innocuous-looking text"></a><script src="http://haxxor.example.net/pwned.js" /><a title=")
- Links with entities in the title, but no markup (like I &hearts; titles)
- Links with a single or double quote in the title

Finally, it shouldn't break any existing automated tests (which isn't to say that all tests in our test suite need to pass, but any test that passed should still pass, and any test that failed should fail in roughly the same way). Note that the test suite can be overwhelming, so you may want to leave that for a reviewer/committer to check.

Edited 2011-11-21 18:08 (UTC)

Still working on my script to post Pinboard bookmarks to Dreamwidth. (And much thanks to both you and

foxfirefey for your answers last time. I completely forgot to get back at you, my bad entirely. I'll try to be better this time :) )

I'm stuck on, I think, a fairly easy question, that I nevertheless can't find an answer to.

To use the Dreamwidth API (and the Pinboard one), I need the user's password. My script is PHP, and I tell people installing it to do so outside of the public folders, to reduce problems, but even though, I'm not particularly happy at having to store the password, not even encrypted, in the script. Everything I can find about storing passwords securely is about sites that ask you to login, but I do the opposite: I send the password outside. (Unless I'm a dunce and there is a totally different solution I didn't think of.)

I need to store the script on a web server, because the point is for it to run automatically without human input.

Anyone has the beginning of a solution for me?

Ideally, the directory the password (and script) is in should be outside the Apache document tree (which may be what you mean by "outside the public folders). If that's not possible, you might be able to handle it without a gaping hole by having the password file in a separate directory that also contains a .htaccess that disables directory indexing and that denies HTTP access to all files by requiring authentication and giving an empty list of authentication methods, Thus, the password data will only be readable by locally-running scripts and won't leak out if someone tries to wget the password file by its URL. (This fails if an attacker manages to get their own code running on the server(*), but there's little you can do to protect from that no matter which approach you use.)

(*) If your script lives on a shared server, which is common for PHP, you should make sure that the server operator enforces separation between customers, eg by having a separate Unix account for each virtual domain's directory tree or something along those lines.

outside the Apache document tree (which may be what you mean by "outside the public folders)
That's indeed what I meant.

you should make sure that the server operator enforces separation between customers
Mine does, and so does the host I recommend, but I'll have to add this to the requirements, then.

Thanks!

also, I should add, we have plans to implement OAuth, which will let you authorize an external service to have access to your account, etc. Someday!

Would it work to store the login cookies, instead of the passwords?

Would it work to store the login cookies, instead of the passwords?
I have no idea! XD I'm using the script as a learning process actually, and I'm making it up as I go along, probably reinventing the wheel a couple of times.

I'm going to keep your suggestion in mind for when I have time to tackle the documentation. Thanks!

Reinventing the wheel occasionally leads to advances in wheel technology, so go for it. *g*

If you can bribe someone to help me with the user facing bits... >_> ( also uh, there should totally be a bug for OAuth stuffsies )

huh, I thought there was a bug for it. if not, open one!

And hey, if you open the bug, make it needs-design and lay out what the user-facing bits need to do/be/say/etc.

I could use a project to work on, and I know something about OAuth - I've interacted with Twitter's OAuth implementation as a dev, and implemented the client-side bits of it from scratch with no library (which I did so that I knew how it worked; for anything other than local tools, I'd obviously use a library), so... I'm willing to help!

[edit: Which is to say, I implemented it without an OAuth-specific library. I used libraries for URI escaping, URI parsing, HMAC-SHA1 encoding and base64 encoding.]

Edited 2011-11-21 20:24 (UTC)

I am only newly poking at the DW code and am new to most of this. I don't have enough understanding yet to ask interesting questions. But here is an uninteresting question I have about something!

In cgi-bin/bml/scheme/global.look, this is line 171:
GRIN=>{S}<grin>

Does that actually serve a purpose somewhere? It seems like all <?GRIN?> would do if used in a BML document is cause <grin> to print, which then wouldn't show up to the user, but wouldn't do anything else either. Is this just a silly programming easter egg of some kind, or does this actually do something I'm not seeing? (I also did a rgrep and couldn't find <?GRIN?> occurring anywhere.) eta: Oh, wait. I'm realizing now it'd make <grin> print, not <grin>. So the user would indeed see <grin>, I guess?

Edited 2011-11-22 14:21 (UTC)

Yes, they would indeed see <grin>. Also, the .look files for the legacy (pre-TT) skins have that unpleasant wrinkle that they define in all-uppercase stuff that's then referenced in all-lowercase elsewhere. (I got burned by that one when trying to clean them up early on in DW's life, IIRC.) In this case, though, it lr ooks like it's really not used, UPPERCASE, lowercase, or mixEd CAsE. (And it was already present in the initial version of that file Brad checked in on LJ, so it's really old.

Hum. Well if it's super old maybe long ago in LJ code it did serve a purpose, but now doesn't? Like idk maybe Brad had <grin> print out on some pages? That seems kind of silly. ;p

Also, it is very useful to know that about the case! I think I first searched for it in all-uppercase and then did a search with that "-i" non-case-sensitive option also just to be sure. :)

I would guess it predates LJ and is derived from something like IRC or Usenet slang. It would have been common at one time for people to type <grin> or <G> in a non-HTML chat environment, and they probably tried to do the same thing in early LJ comments. Unfortunately this theory is completely non-Googleable, so you'd probably have to find some ancient crusty netizens to ask about it.

That makes sense! Hmmm, although I wonder how one could use that from a user's end. Maybe earlier LJ comments didn't parse HTML in entries/comments like they do now? (Heh, I just tested to see if posting or commenting <?GRIN?> now would do anything, but ...nope, alas.) XD

Another question! I am poking at MoodThemeChooser.pm (and other things) in an attempt to change out places where tables were used for layout purposes, to use CSS positioning instead. In MoodThemeChooser.pm there is this variable $moodtheme_extra which takes its value somehow from the hook mood_theme_extra_content? And then another variable $show_special which is set based on that? (I am still just trying to figure out Perl, lol.) ANYWAY. I cannot figure out where mood_theme_extra_content comes from or what it is supposed to signify. I read the wiki page on hooks and the old LJ documentation on them but I'm still confused. And I did an rgrep search but couldn't find mood_theme_extra_content anywhere else. It's making me scratch my head, b/c I can't figure out how I should change the CSS version of the layout in relation to this element.

As far as I can tell, with the table, what it changes is that if $moodtheme_extra is false, then all the example "mood theme" images/descriptions (which are contained in <td> elements) are in ONE row. If $moodtheme_extra is true, then after every 2nd image it inserts a new <tr>. But not knowing what the heck mood_theme_extra_content is, I don't know why it's doing that or whether I should also do that. I wasn't sure also if maybe this is related to the width of the table, b/c I just told my CSS positioning to size the width of that element automatically, but ...if mood_theme_extra_content is related to mood themes that are extra super wide or something, maybe that is why it happened? (I'm positioning those elements with float:left so I could just add in a clear:left probably to the same effect.) (eta: or set a maximum width to the containing element to eliminate the need for this workaround?)

Anyway, confusion. I figure also for testing purposes it's important to know what that hook means so I can test the two different cases. On that end, also, I guess I have another question. How do I make a user account in my dreamhack a "paid account" so I can test paid features? Maybe there is an obvious answer to this somewhere but I couldn't find it on the wiki.

Edited 2011-12-03 20:16 (UTC)

For the paid account: go to /admin/priv while logged into the system account or someone with admin:*, give yourself the 'payments' priv, then go to /admin/pay and enter the username of the account you want to upgrade. the subsequent bits should be self-explanatory *G*

meanwhile, without looking into it too closely, i'm going to guess that mood_theme_extra_content is almost certainly some ljcom hook (the non-free lj code), and therefore we don't have it -- most (though not all) of the hooks defined in the code are. To look for where in the code it's referenced, you'd do:

rgrep -i mood_theme_extra_content $LJHOME/cvs/dw-free/* | more
rgrep -i mood_theme_extra_content $LJHOME/cvs/dw-nonfree/* | more

Since neither returns any results except the bit in the widget, it's pretty certain that it was a ljcom-specific hook (and now i'm curious; i looked it up (don't worry, you aren't supposed to know where to find it, I just happened to) and it looks like it has to do with sponsored mood themes, which we don't have. (There are a lot of hooks scattered around the code that LJ used to pull in ljcom-specific things like that -- some of them we've repurposed for other dw-nonfree things that are useful, some of them we just ignore.) So you can safely ignore it.

Yeah, I did
rgrep -i mood_theme_extra_content * > whatevertempfilename
while in the parent directory of my dreamhack, and those were the only places where it showed up.

What categorizes "ljcom" things? Are they all related to sponsored things, or are they code bits that are not open source or something? I wasn't really positive why the Mercurial repositories have both the "dw-free" and "dw-nonfree" branches (but the live directory doesn't?).

Since DW has no sponsored mood themes, in working on editing that module to use CSS positioning should I also edit it to not take that hook+corresponding variables into account? Rather than having lines of code that serve no purpose / do nothing? Or comment them out or something?

Also there were some hooks I was unfamiliar with also when I was poking at S2PropGroup.pm (which I've taken a break from to work on others instead). I don't remember them off the top of my head, but, it looks like that thing you linked to might be a good place to look up some of them. If there are some that don't appear to be registered anywhere in the DW build that I have, but I find them on that site, is it usually the case that they are ljcom things that are not in use? Should I double check any time I find one like that before assuming that it doesn't serve a purpose?

P.S. Thanks, it was causing me immense aggravation not being able to find that thing anywhere, ha.

Edited 2011-12-03 22:50 (UTC)

It's ... complicated! Basically, any time there's anything that makes assumptions about account structure, or references specific communities on LJ that might not exist on other sites, or assumes things that might not exist elsewhere, it's generally done with a hook. (For a good example of where hooks are used on DW for those reasons, check out htdocs/community/index.bml in dw-free and cgi-bin/DW/Hooks/Community.pm in dw-nonfree: "oh crap! I need to use DW-specific comms in the code!")

In general, if you see a hook being called in the code but you can't find any place where the hook is actually defined, it's almost certainly a ljcom hook. You can check for it in the ljcom code, but you can't copy or use that code.

(ljcom is the LJ equivalent of dw-nonfree: it is source-available (for comparison) but not Open Source and other sites can't use it. livejournal is the LJ equivalent of dw-free, which is freely usable anywhere. When we forked from LJ, we were only legally permitted to use things in the livejournal repository, not livejournal + ljcom the way livejournal.com uses, which is one reason why there are features on livejournal.com that we don't have.)

In this case, yeah, I'd just go ahead and remove that hook call in the mood theme widget if you're working there, since there's no reason to have it other than for something we don't use. If someone needs a hook there again in the future, they can make a new one.

Thanks for the explanation! It is especially useful b/c I had read the wiki page on hooks as well as the old LJ documentation on hooks, and I'd looked in places like cgi-bin/DW/Hooks to try to find where some of them were defined and was really frustrated when I couldn't find them anywhere. (lol, I did rgrep first and then when I couldn't find them was like, "BUT THE WIKI SAYS THAT HOOKS ARE DEFINED SOMEWHERE, THEY HAVE TO BE DEFINED SOMEWHERE!!!! AHHHHHHHHHHHHHH" *commence useless manual searching of file structure even tho I already confirmed it wasn't there with my rgrep search*)

I am curious: why is some of the DW code open source but the stuff in dw-nonfree isn't?

The stuff that's in dw-nonfree is basically the stuff that either a) we cannot legally redistribute (stuff where the licensing says that we can use it, we just can't redistribute it, and putting it in dw-free for other people to use is redistributing) or b) stuff that is part of our unique branding (the Tropospherical Red/Purple site schemes, our logos, etc). There's also some stuff in there that refers to DW-specific setup: the suggestions generator (posts to a specific DW comm that might not exist on other sites), some places on the site where we refer to DW specific comms (like the hook I mentioned earlier), some bits of our account structure (although most of the paid vs premium paid division is very "baked in" to the code, we do make at least a token effort to assume that other sites might have other account structures) -- stuff like that.

Thanks again for more explanation! :)

Anytime! No question too big, no question too small.

I'm baaaaaaaaaaaack?

Thanks to

denise's response to my question I was able to figure out the hooks in MoodThemeChooser.pm and now I think I have a version that correctly uses CSS positioning. Only then I tried to figure out how to generate a patch and got totally lost. I tried to follow the instructions on the wiki for maintenance/syncing my CVS installation/whatever first. Okay. But from what I can gather ...the instructions on patching on most of those pages assume I was doing my edits in the CVS directory and generating patches as I went along? Now I'm just lost and frustrated. One of the "getting started" pages on the wiki said that you could do your editing in either the live directory or in the CVS directories and that it had something to do with personal preference for workflow? So I did it in the live directory but now can't figure out how to get THAT to sync with my CVS directory. The problem is that I did change files other than MoodThemeChooser.pm, so when I generated a "compare everything" patch it had a lot of other changes I don't want to include in this patch. But I don't want to JUST declare MoodThemeChooser.pm b/c I also changed the files governing the CSS rules for that page (in both htdocs and ssldocs). So I guess I could generate three separate patches, one for each of those files, but that seems annoying, and then if I had more than three files could get old really soon.

I backed up the other files in my live directory that I changed anyway so that's why I wanted to just sync/override the whole thing, then I could copy my modified MoodThemeChooser.pm & related files back in there to only generate the patch based on those. Also though when I did the patch comparing everything in my live directory with the CVS one, there was a lot of weird stuff in there I definitely didn't change, like, config files related to ...time and things like that? (I don't have the patch file in front of me right now).

Uh basically I'm just confused and frustrated and wondering whether it's best to wipe my entire installation, reset it from scratch and then try to figure out this other method of doing my edits in the CVS directory instead? (And then I use patches to make those edits apply to the live directory so I can see them when I go to the site or something? Is that how that works?)

Fear not, you don't have to delete everything! It is possible to sync between live and cvs fairly easily. These instructions will walk you through setting up an alias to do it so you don't have to run the whole command every time.

(Also, if you wind up with changes in a patch that you don't want (other changed files, etc) you can just delete those changes. Make sure you get everything from the part that starts "diff -r" to the beginning of the next "diff -r", or else there will be problems applying the patch).

1) First, do this: http://wiki.dwscoalition.org/notes/Dreamhack_getting_started#Protect_your_configs_from_updates (that will prevent your config changes from being included in every patch ever).

2) Second, while logged into your DH account, cd ~/bin and pico -w sc. In the window that results, paste:

Hit ctrl-X, say yes when it asks you to save, and once you're back at the command line type chmod +x sc. Then, pico -w sl and paste:

Ctrl-X, save, and type chmod +x sl.

Once you've done that, you'll be able to type sl to sync live files to cvs, and sc to sync cvs files to live. I, personally, do all my changes in live, test them on the site, then "sc" to move them from live->cvs to generate the patch.

The parts of the instructions that assume you're generating patches as you go have to do with the fact that unless you want to muddle around with learning mercurial queues -- which I never have -- you will want to revert out your changes from the cvs directory after every patch you generate, or else each subsequent patch is going to contain all the other patches you did and the system will bitch at you when you try to upgrade to the newest code. (The command to revert changes is 'hg revert --all', and you need to be in ~/dw/cvs/dw-free or ~/dw/cvs/dw-nonfree when you run it.)

So, my own personal workflow goes:
1) make changes in live, test, get it working, yadda
2) "sl" to sync live files to cvs
3) "hg diff > ~/bugXXXX.patch" to generate a patchfile in my home directory
4) open the patch file, confirm it's got everything I need and doesn't have anything I don't need, upload to bugzilla, etc
5) "hg revert --all" in ~/dw/cvs/dw-free to revert the changes I just made in cvs
6) "sc" to sync cvs files (the unchanged versions) back to live and reset to the pre-patch state

This way doesn't take advantage of a lot of the things you can do with Mercurial, but given the complexity of the bugs I'm working on (ie, not much) it's just easier for me to do it that way, personally! If you want to learn Mercurial someone else would have to help out. *g*

Well I had to tweak some things but I think I eventually sort of made this work. :) So thanks, that was helpful. One weird thing -- I noticed there's no ssldocs directory inside dw-free? At least I don't seem to have one. But there are two (identical) copies of the .css file I changed, one in htdocs/stc/widgets and the other in ssldocs/stc/widgets. But since my dw-free directory contains no ssldocs directory, the patchfile is only showing the differences in the htdocs version.

That's okay! ssldocs is an automagically-generated thing to allow for certain things to be served securely without errors or warnings. Just ignore it. :)

Okay, well, I attempted to upload what I came up with on bugzilla, so we'll see how that goes. ^__^

To elaborate on what denise said: to decide what needs to be copied where updating live from cvs/, cvsreport.pl uses $LJHOME/cvs/multicvs.conf(*) (and multicvs-local.conf and multicvs-private.conf if they exist). If you'll look at that file, you'll see a bunch of lines like:

dw-free/htdocs/shop/ ssldocs/shop/

which instruct cvsreport.pl to copy from the first directory to the second directory.

(*) That file itself is copied from $LJHOME/dw-free/cvs/multicvs.conf each time you run cvsreport.pl -s, but the new content isn't used to decide what to copy. That's why you occasionally need to run it twice in a row.

I have one! How do you get the new update page on one's hack?

Edit $LJHOME/etc/config-local.pl, look for %LJ::BETA_FEATURES

Add this:

        "updatepage" => {
            start_time  => 0,
            end_time    => "Inf",
        },

(Inf == infinity ;-))

Oh I did that, I think and it doesn't seem to work. :/
This is what I have:

$ENABLE_BETA_TOOLS = 1;

and then:

%LJ::BETA_FEATURES = (
"journaljquery" => {
start_time => 0,
end_time => "Inf",
},
"updatepage" => {
start_time => 0,
end_time => "Inf",
},
);

What am I doing wrong?

Hmm, it's still not showing up on /betafeatures? That is, there's just journaljquery there?

Can you make sure that there there are no other places where the config is defined?

grep -r BETA_FEATURES $LJHOME/etc

No, it's not and journaljquery isn't there either.

I've done that, thanks, it's just the one file you mentioned:

/dreamhack/home/8174-ninetydegrees/dw/etc/config-local.pl

Dumb question but this is always what gets me: did you stop and restart your 'hack after you edited the configs?

Yep, I have a script which does everything for me but I've just used the stop and start commands to make sure and still no beta features.

Edited (duh.) 2011-12-09 20:30 (UTC)

Hmm. Can you try:

perl -c /dreamhack/home/8174-ninetydegrees/dw/etc/config-local.pl

That will check the syntax of the configuration file; hopefully that gives us a cule.

I get:

/dreamhack/home/8174-ninetydegrees/dw/etc/config-local.pl syntax OK

I do have copies of config-local.pl in /dw/cvs/local/etc and /dw/cvs/dw-nonfree/etc but they have the code I mentioned earlier too.

Hmmm. Okay, try this:

perl /dreamhack/home/8174-ninetydegrees/dw/etc/config-local.pl

(which actually runs the file -- should be no harm done!)

I get:

Can't locate object method "account" via package "type" (perhaps you forgot to load "type"?) at /dreamhack/home/8174-ninetydegrees/dw/etc/config-local.pl line 99.

Line 99 is where the shop configuration starts and I don't think I've ever changed it but here's what it says:

%SHOP = (
key => [ $USD, months, account type, cost in points ],
prem6 => [ 20, 6, 'premium', 200 ],
prem12 => [ 40, 12, 'premium', 400 ],
paid1 => [ 3, 1, 'paid', 30 ],
paid2 => [ 5, 2, 'paid', 50 ],
paid6 => [ 13, 6, 'paid', 130 ],
paid12 => [ 25, 12, 'paid', 250 ],
seed => [ 200, 99, 'seed', 2000 ],
points => [],
);

Aha, okay. Comment out the "key" line; looks like that wasn't technically a syntax error, but it was causing the configuration file to not be executed fully.

We really should have a bug open for this; I vaguely recall we used to have one but we couldn't figure out how to trigger the problem (heh), and I can't seem to find it now...

Anyway, comment that out and things should work okay!

Ha! I figured it out thanks to the backup file as I got your comment! Thanks for all the help!

Excellent! Have fun :)

You first need to update the code on your dreamhack the usual way(*). Then, substitute your dreamhack's domain name (probably something like www.ninetydegrees.hack.dreamwidth.net) for www.dreamwidth.org in the "sign up for beta" URL. (See dw_beta or dw_news for the exact URL.) This should let you sign up for it on your dreamhack.

(*) If you did already, you may want to update it again anyway. There's been some substantial changes in the past week, IIRC.

You mean http://www.ninetydegrees.hack.dreamwidth.net/betafeatures right? It says there are no features currently available for beta testing. :/

Hmm. Then it looks like Afuna's guess was (not exactly surprisingly) closer than mine.

Hi!

If one were interested in getting involved in bugfixes and development, but one's only code experience is outdated (HTML 3.0 and Java a few versions ago), which languages would you lovely people suggest one brushes up on?

For front-end development:
- HTML4, CSS3, and the Template Toolkit templating language
- Javascript, the jQuery library, and the jQueryUI library

For backend development:
- SQL

For both:
- Perl

Also, feel free to pop up in #dreamwidth-dev if you need to discuss specifics, depending on which bug(s) you're looking at.

HELLO AND WELCOME, we are kind of ridiculously full of the welcome wagon

In case you have not had ALL the links thrown at you yet:

* Dreamhacks, our hosted developer environments, because you don't want to try to install the code yourself
* The Wiki, especially Development and Getting Started
* The #dreamwidth and #dreamwidth-dev irc channels, on irc.freenode.net
* The Mercurial web frontend if you want to just read through the code instead of checking it out
* BUGZILLAAAAAAA, especially the unassigned effort-minor bugs

Sweet. Thanks for the links! This community is awesome :)

Question thread #3

Testing

Re: Testing

Re: Testing

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject