Dreamwidth Development Training

Updating the Database via changes to en.dat

Sun, 13 May 2012 15:17:20 GMT

Me again :/ Still on bug 2152.

I wanted to change some of the labels in the settings for display. I found these in en.dat, changed them, updated the database and everything was working fine. But, all of a sudden, any new changes I make in en.dat don't appear to be getting picked up - this includes resetting everything using patch -R and updating the database* - I'm still getting my new labels turning up rather than the old ones.

What might I have done? (I did have a play with the test suites following some discussion recently and denise (I think) mentioned these can corrupt the database in some way).

I'm using a dreamhack, in case that's relevant.

*

stop-apache

$LJHOME/bin/upgrading/update-db.pl -r -p --innodb
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$LJHOME/bin/upgrading/texttool.pl load

start-apache

comments

Including changes to config files in patches

Mon, 07 May 2012 18:49:27 GMT

Still working on Bug 2152. The bug involves allowing people to have multiple stickies and the number they can have depends upon their account type. Adding this feature is going to involve a new config setting where someone sets the numbers of stickies allowable for each account type. However I've done the whole protect your configs from updates thing which means the new config settings aren't showing up in the patch files I create.

Should I just document the extra settings needed in the files? especially since the actual numbers associated with each account type aren't particularly important to the code and should probably be decided upon by the powers that be separately from the actual patch? or should I include some default values in the patch and, if so, what is the best way to go about getting these picked up by cvsreport.pl -d?

comments

Working on a DW Bug: Step by Step

Sat, 05 May 2012 14:30:59 GMT

So, when I woke up this morning, in #dreamwidth, there was a discussion of a bug in auto-expanding lj-cuts in a particular entry. It looked interesting enough that I figured I would poke at it, since there was a theory it was a Javascript bug, and Javascript is a language I speak. (The actual bug was that clicking on a particular lj-cut expansion would expand to a different lj-cut text, now filed as Bug 4435.)

First, I didn't even know how to expand LJ-cuts inline. So I started from pretty much nothing here. :) But in the end, I think I found the bug, and attached a patch to Zilla, so I figured I'd write up my process.

Start: (6:46) Okay, clicking on the cut gives the wrong thing. Open up my browser's web console, and check the HTTP request that it's sending is right -- it specifies cutid=1, but is given one from a much later cut. (6:48) To me, this suggests that the problem is server side (not Javascript), so I have to dive into code I can't just load in my browser. I experiment a bit more to ensure that this appears to be the case, and determine I can't see any evidence that this is client side. (6:58).

I Googled 'dreamwidth code' (which didn't get me very far), then 'dreamwidth mercurial', which got me to the hg repo (even though it was the 5th result) first. After realizing I couldn't figure out how to check out the code from there, I went back to the first hit for that search, the opensource page, and made my way through to the wiki and from there to Getting Started and from there to Scratch Installation and from there to Downloading the code from which I grabbed bootstrap.pl, and from there to the one line I wanted: 'hg -q clone http://hg.dwscoalition.org/dw-free'. (Yes, if I knew mercurial better, I probably could have done this without following any of those steps ... but I didn't. In fact, I didn't even have mercurial installed at the time. So I also installed that.)

So, it's now 7:04, and after grep -r-ing for rpc_cuttag, I find cgi-bin/DW/Controller/RPC/CutExpander.pm, and realize that all the work in the cut expander is done by LJ::CleanHTML. My memory of working on LJ almost a decade ago was that this particular chunk of code was a hairy, nasty, mess -- as is almost any code that parses HTML -- so I was unsurprised to find that little has changed ;)

After looking briefly at CleanHTML (07:11), I decided that this was almost certainly a side effect of the particular entry text -- simple tests showed that this worked, but this entry was clearly failing. I checked briefly whether any community mods for that community might be in #dreamwidth -- the answer to which was no. (07:12).

At 07:15 I summarize my progress so far to the channel:

Javascript is doing fine. IT's asking for the right cut. The server side is broken.
The server side RPC mechanism is likely doing the right thing; there's no smarts there.
Once it gets past that, it goes into the CleanHTML code, which in addition to being the gnarliest perl code I've ever not touched, definitely behaves very differently when in "looking for cut mode"
than it does otherwise, so it's not super-surprising that this fails.

At 07:19, I basically declare defeat: without the source text for the bug, it will be very difficult to debug, but I continue to read the CleanHTML source anyway, trying to understand it slightly better. It's clear that lj-cut handling isn't trivial. I find a couple interesting things in the code, but nothing that calls to me as the source of the bug.

( this gets long )
Total time: ~2 hours, start to finish, with a lot of stumbling around to figure out how things were supposed to work, and I'm still not particularly sure that I did things right, but it does look like I actually fixed the problem... the question is really just whether it also made anything else worse :)

comments

DW object-oriented programming explained (Part 3) - using classes

Tue, 01 May 2012 22:16:31 GMT

It's been a while since I posted anything in this series of entries, and I apologise for that! Today, though, I'll be explaining how to actually use classes in Perl and the Dreamwidth code and constructing new objects.

( How do you use classes? )

That's a lot to take in, so I'll leave off for now! I hope I've explained things well, but I'm sure that there are bits that are going to be incomprehensible. If you need me to explain anything further, please ask in the comments - I'll be happy to try to explain it.

comments

Can a journal entry id be 0?

Tue, 01 May 2012 18:37:47 GMT

For Bug 2152 I've found I need to create a subroutine that checks whether a form element refers to a valid entry in the user's journal separately from the various subroutines for actually saving it as a sticky.

If a journal entry id can never be zero then this subroutine could return the id number if it is valid and 0 otherwise. In that case the subroutine can replace duplicate code in at least two places, but obviously if an entry can be 0 then the subroutine had better just return 0 or 1.

comments

Test Plan Walkthrough: Bug 2154

Sun, 25 Mar 2012 13:02:24 GMT

As well as uploading a patch file for bug 2154, I uploaded the notes I had made for the test plan for the patch in case they were of use in the review process. denise asked if I could write a post explaining how I came up with the tests for this comm, since it was something people often asked about.

( Generating Tests for Bug 2154 )

comments

New Bugzilla section: Dreamwidth wiki work

Sat, 24 Mar 2012 08:59:41 GMT

The Dreamwidth wiki isn't well-suited to keeping track of changes needed to the wiki itself and actions/feedback/progress on them, but Dreamzilla is ideal for those, and it now has a section dedicated to them. When searching for bugs or filing a new one, select "Project Documentation" as the product and "Wiki documentation" as the component.

This being Dreamwidth, there is substantial overlap between the technical and the social/cultural, but some restraint is advised. As an example, documenting what can make me lose The Game is probably handled outside Bugzilla.

comments

Question thread #6

Sat, 24 Mar 2012 07:49:45 GMT

It's time for another question thread!

The rules:

- You may ask any dev-related question you have in a comment. (It doesn't even need to be about Dreamwidth, although if it involves a language/library/framework/database Dreamwidth doesn't use, you will probably get answers pointing that out and suggesting a better place to ask.)
- You may also answer any question, using the guidelines given in To Answer, Or Not To Answer.

comments

Design Discussion: A Process Question really

Tue, 13 Mar 2012 13:57:23 GMT

I've picked a second bug to work on (2154) and I've realised there are basically two approaches to fixing it, one involves leaving the current interface as it is and building some (probably simple but potentially ugly) smarts into the code and the other involves getting a bit more info from the user up front.

So this is partly a question about which approach is preferable, but more generally a question about the appropriate place to have this kind of discussion. Should I use the comments in bugzilla, for instance? Similarly were I to pick the route that involved fiddling with the user interface, should screen shots of the potential new interface go into bugzilla for sign off (or review or ???) before a patch file is generated?

comments

Bug walkthrough: 2886

Mon, 12 Mar 2012 21:18:36 GMT

Hi!

I wrote a bug walkthrough, which I have hidden behind a cut.

( Long walkthrough is LONG. )

comments

Bugzilla: Getting Started Question

Sun, 04 Mar 2012 19:30:39 GMT

I'm a really, really new developer (fired up my dreamhack for the first time about an hour ago) and I've been working through the documentation and poking through the code. I've found an unclaimed bug in a babydev bug list which looks like a nice place to start.

My question is should I claim the bug on bugzilla now, or wait until I'm reasonably confident I've found all the relevant bits of code and have a plan for how I'm going to go about fixing it?

comments

DW object-oriented programming explained (Part 2)

Fri, 17 Feb 2012 23:27:28 GMT

Welcome to the second part of the series on object-oriented programming - or OO - as it applies to the Dreamwidth codebase. :)

If you haven't already read the first part, you'll want to do that before reading this part. I also realise that I never got around to explaining what 'methods' are in the first post, so I'm going to do that right now before delving into the main part of this post:

( What are methods? )

As with the last post, if you have any questions on this, feel free to let me know in the comments!

So, with that explanation of methods out of the way, it's time to move onto our next topic - how it applies to the DW codebase.

I'm going to do this as a few posts, each dealing with their own topic, because I've got a fair amount to say about them. I'm still not entirely sure how many there'll be, but I'm writing them one at a time so there may be some time (a few days to a week) between each one.

A couple of things to note before I begin:

This post may require some basic knowledge of Perl and/or programming in general. Not much, I promise! (Things such as what a 'string' is, etc.) But all the same, if anybody finds themselves confused by anything I write, feel free to ask for clarification in the comments. I won't bite!

Secondly, if you're used to OO from another language, you'll find some things about Perl's implementation of OO to be strange and baffling. That's because Perl wasn't actually designed with OO in mind; OO support came later, and to be honest, it shows. Still, it's what we use, so I hope I can at least help with understanding it.^(**)

^(**) There is a version of Perl in the works which does a much better job of not only OO but a lot of other things - Perl 6 - but at the cost of revamping a lot of the language such that you probably wouldn't be able to use it without spending some time making sure your code conformed to it. For this series, therefore, I'll be concentrating on Perl 5, which is what most Perl developers - including DW and LJ - use.

With all that said, let's move onto our first topic!

( What is an 'object' in Perl? )

That's about it for this post. There's a lot of stuff here so feel free to ask questions if there's anything you don't understand! My next post will probably talk about how you can create and use an object, as well as some example of existing classes in the codebase.

comments

template toolkit

Tue, 14 Feb 2012 21:29:11 GMT

How does it work?

Does anybody have good TT resources to throw at me? I'm having trouble wrapping my brain around it.

comments

DW object-oriented programming explained (Part 1)

Mon, 13 Feb 2012 21:11:48 GMT

The Dreamwidth codebase uses object-oriented programming a lot, or "OO" as it's known. But what exactly is OO? In this post I'm going to explain the concepts of OO. I plan this to be the first in a series; in later posts I'll explain how OO programming relates to the DW codebase, some gotchas that might arise, and other such things.

For now though, an introduction to the basic concepts of OO!

( So, what is OO exactly? )

And that, in a nutshell, is what OO is all about. In the next post on this subject - which may be in a few days or a week, I don't know yet - I'll be talking about how this applies to the Dreamwidth codebase. :)

If anybody has any questions so far about what I've said in this post, please feel free to comment!

comments

Question thread #5

Thu, 09 Feb 2012 20:04:58 GMT

Strip tab characters from multiple text files and replace them with spaces (or something else)

Fri, 03 Feb 2012 20:09:30 GMT

I switch between Gedit, Notepad++, and vim fairly often depending on what I'm doing and whose computer I'm on. Sometimes I end up with tab characters where I really wanted four spaces, mainly when I'm using vim and I haven't figured out how to get vim to not do this. Gedit and Notepad++ have settings to use spaces instead of tabs, so there's no issue there.

Either I don't notice the tab characters until after I've put lots of them in the file I'm editing, or I'm editing a file from someone else whose editor uses tab characters for indentation. I know its not a big deal to some people, but tab indentation mixed with space indentation is a huge pet peeve of mine.

Thus, a perl script was born:

#!/usr/bin/perl -w

use File::Copy;

# This script replaces tab characters in a file with four spaces, or whatever else you want

$num_args = $#ARGV + 1;
$num_warnings = 0;
$tab_replace = "    ";  # Change this to whatever you want in place of tabs

if ( $num_args == 0 ) {
    print "usage: strip-tabs.pl file1 [file2, file3...]\n";
    exit;
}

# Have files to parse...
for my $f ( 0 .. ($num_args - 1) ) {
    if ( open(INPUTFILE, "<$ARGV[$f]") ) {
       if ( open(OUTPUTFILE, ">$ARGV[$f]~") ) {
           # Start parsing the file
           while ( my $line = <INPUTFILE> ) {
               $line =~ s/\t/$tab_replace/g;
               print OUTPUTFILE $line;
           }
           # Copy over original file here
           close(OUTPUTFILE);
           close(INPUTFILE);
           if ( !move("$ARGV[$f]~", $ARGV[$f]) ) {
               print "Could not write output to file $ARGV[$f]: $!\n";
               $num_warnings += 1;
           }
       }
       else {
           close(INPUTFILE);
           print "Could not create output file for $ARGV[$f]: $!\n";
           $num_warnings += 1;
       }
    }
    else {
        print "Could not open $ARGV[$f] for reading: $!\n";
        $num_warnings += 1;
    }
}
die "$num_warnings warnings encountered during file operation." unless $num_warnings == 0;

View Gist (strip-tabs.pl)

Feel free to gank away if you find it useful!

comments

Question thread #4

Wed, 04 Jan 2012 17:11:20 GMT

Delayed Zilla mail?

Sat, 31 Dec 2011 07:18:35 GMT

I've been getting mail from http://bugs.dwscoalition.org/ late for a few days. Like 24 hours later. Does anyone else have the same problem?

Edit: mystery solved.

comments

Hooks

Sat, 03 Dec 2011 22:41:48 GMT

This came up in the previous questions post, and it's confusing enough that I thought I'd bring it up to a top-level post: hooks, and what to do if you see something in there that uses one but you can't figure out where the hook is.

LiveJournal uses a lot of hooks to pull ljcom (their equivalent of dw-nonfree) data into livejournal (their equivalent of dw-free) without hardcoding ljcom-specific things into livejournal. You'll see them called like this:

LJ::Hooks::run_hook( "hook_name" );

(sometimes with some other parameters added, or as a variable assignment). Some of them are also used in dreamwidth code: sometimes the hooks that were already defined in the code were useful for our purposes, or we wanted to do the same kind of thing with dw-nonfree code that was being done with ljcom code -- you'll find those in cgi-bin/DW/Hooks. I'd say probably about 90% of the hooks that are tied into the code aren't actually defined in either dw-free or dw-nonfree, though, because we just haven't needed them yet.

So, if you see a hook defined in the code, but you grep the code and nothing turns up for the hook name except the line that calls it -- it's probably an old ljcom hook that we haven't started using yet. You can look at ljcom's cgi-bin/LJ/Hooks directory, if you really are curious about what the hook is meant to do, but since ljcom is not free software, you can't and shouldn't copy the hook directly.

comments

Question thread #3

Mon, 21 Nov 2011 16:33:46 GMT

Console commands not working

Fri, 18 Nov 2011 14:26:23 GMT

I've just begin the process of setting up my Dreamhack config according to the wiki. I've gotten to the point of updating my code to the latest version (as described in Dev Maintenance) and creating several test accounts.

No matter which account I'm logged into or what privs that account has, I cannot run console commands at all. Not even commands available to all accounts like ban-set. Every command gives an error message "There is no such command '<command-name>'." At this point, I haven't messed with any of the existing code.

How do I fix this?

Edit: Underscores! I was using dashes instead of underscores in the console commands and my brain didn't catch that. Solved. :D

comments

Profiles: Connect: questions on character limits and Gizmo

Sat, 29 Oct 2011 12:34:25 GMT

/htdocs/manage/profile/index.bml seems to have arbitrary character limits for some of the external usernames you can have in the 'Connect' section of the profile.

Skype, Delicious and Twitter have 40 but usernames can't be over 32, 32 and 15 characters respectively.

Last.fm has 255 but it's actually 15 for usernames and 60 for groups.

Is there a reason for this?

Also, Gizmo is now Gizmo5, which is no longer supported. Should it be kept anyway?

comments

Stepping up code review, especially for newer coders!

Fri, 28 Oct 2011 01:06:42 GMT

Hello, folks!

For those who don't know me, my name's kj (yes, that's what I go by in off-line life, too -- blame my mom!) and I do a bit of code review here and there. Well, did a bit. Recently I've been encouraged to do more of it, and I'm certainly up to it.

One of the things I really like about Dreamwidth is the welcoming nature of the project. Both old coders new to the project and new coders in general are welcomed with open arms, and frankly, there's just not enough of that these days. Being a new coder in general is especially hard because it's hard to tell if you're going in the right direction, and waiting a long time for someone to look over your work, test it, give feedback, and put it up for commit can really slow one down when you're learning. Uncertainty is possibly the most difficult part of learning.

One way I'd like to help with that is to pay extra-special attention to the needs of newer coders and other people (who are, like me) new to the project. If you'd like me to review your code, feel free to ask; via e-mail, in IRC (#dreamwidth-dev on Freenode), or by Cc:ing me specifically on the bug, and I'll get to it as soon as I'm able. I can't make any promises on wait time (I've got a day job, a fiancée, a teenager, and a dog!), but I'll do my best to make it happen as soon as I can.

comments

Question thread #2

Thu, 20 Oct 2011 14:04:37 GMT

New page about the care and feeding of workers

Fri, 07 Oct 2011 18:57:26 GMT

http://wiki.dwscoalition.org/notes/Workers, linked from http://wiki.dwscoalition.org/notes/TheSchwartz_Setup#On_your_own_install (end of section). Should that link be more prominent?

comments