Planet phpBB

So, in less than 3 days ep.io will shut down. It was a great service and I want to use this opportunity to say thank you to Andrew and Ben for it and all the hard work they put into it.

I think it was December 2010 when I first heard about it through some chatter and later on at DjangoCon.eu in Amsterdam. At first I thought this might be interesting but I wasn't really into using cloud services for this kind of stuff. But then I got the chance to give it a try through a beta invite and some performance constraints on one of my servers. Since then PyGRAZ.org has been running on it and except for the understandable start-up delay on the free plan it was a blast! Basically, ep.io and the workflow when using it set kind of the mark for me on how I want sites to be deployed. It was just so easy :-)

So now that ep.io is gone, Heroku and Gordor seem to be these places where probably most folks will move to. Well, that or just hosting sites by themselves again, as I did with PyGRAZ.org again (with some Fabric-scripting to make deployments as painless as on ep.io).

Bye, ep.io. You will be missed :-(

(The ep.io logo was screenshot'd from the ep.io website)

"Bye-bye ep.io" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

For the last couple of weeks I've been playing around with tmux as kind of an alternative/addon/... to iTerm2 and my terminal-focused work-style in general. One things that drove me nuts again and again is that -- for some reason -- I could never get the split-window command to respect my current working directory. I tried the first two solutions mentioned on the ArchLinux-wiki but the second simply did clash too much with how I'm using split-window and the first solution didn't work at all (for some reason PWD didn't propagate correctly and so default-path always defaulted back to the cwd of tmux' parent shell).

With all other options gone I started to look into the third solution. This one involved extracting the PWD from the pane's shell from the outside via /proc. But since I'm working on OSX, this doesn't work quite like this.

The main differences here are that I can't use /proc to look for the PWD but have to rely on lsof and that ps doesn't have a forest-rendering. So the resulting script looks like this:

#!/usr/bin/env ruby
=begin
Helper script for tmux' split-window command that retains the original PWD
This script is based on https://wiki.archlinux.org/index.php/Tmux#.2Fproc_method
=end
SHELL = 'reattach-to-user-namespace -l zsh'

session_id, window_id, pane_id = `tmux display-message -p "#S:#I:#P"`.chomp.split(/:/)
tty = nil
in_sessions = false
in_right_session = false
in_right_window = false
in_right_pane = false

# First step is to find our tty
`tmux server-info`.each_line do |line|
    break if line.start_with? 'Terminals'
    if line.start_with? 'Sessions: '
        in_sessions = true
        next
    end
    if in_sessions and line =~ /^\s\d+: #{session_id}: \d+ window..*[flags=.*]/
        in_right_session = true
        next
    end   
    if in_right_session and line =~ /^\s{3}#{window_id}:/
        in_right_window = true
        next
    end
    if in_right_window and line =~ /^\s*#{pane_id}: \/dev\/(\S+) /
        tty = $1
        break
    end
end

raise "No matching tty found" if tty.nil?

# Now let's find the process with this tty
pid = `ps -t #{tty}`.split($/)[1].split(' ')[0]

# ... and now its PWD
pwd = `lsof -a -d cwd -p #{pid} -F n`.split($/)[1][1..-1]
args = ARGV * " "

# Let's put it all together by calling split-window with the shell and the path
`tmux split-window #{args} 'cd #{pwd}; #{SHELL}'`

Thankfully, getting access to a process' PWD is pretty straight forward with lsof -d cwd and ps has a nice shortcut for filtering for a specific tty. And yes, I know that the tty-scanning is really rocky up there and this is probably the one part of this script that will evolve over time. The solution for the missing "--forest" support is also not perfect yet, but so far it works just fine for me :-)

So, once you've put this script somewhere in your path (for me that ~/.local/bin/split-in-cwd.rb) and made it executable, all that is left to do, is to add it to your split-window bindings:

unbind s
bind s run '~/.local/bin/split-in-cwd.rb -v'
unbind v
bind v run '~/.local/bin/split-in-cwd.rb -h'

"tmux, split-window and pwd" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

When you're working on your own on a project with JavaScript you usually know how you like your code formatted, structured and organized. But once you are part of a team, be it on a spare-time project or during office hours, there needs to be consensus about this not only for you but also for the rest of the team. Only very few things are a bigger waste of time then ending up with badly formatted or structured code because everyone was adhering to their own personal coding style.

Nicholas C. Zakas with his new book "Maintainable JavaScript" tries to prevent this by comparing multiple coding style-guides and describing common JavaScript coding practices as well as how to automate things like testing against these best practices and he does a pretty good job at that. I bought this book as in the "Early release" version, so some of the points I make down below might not be applicable anymore.

The first part compare different style-guides and does some cherry-picking here and there. The result is very informative and luckily comes pretty close to the style I personally prefer. Also, the author does mention JSHint and JSLint right from the get-go. This first part might perhaps seem a bit basic for most readers but the comparison between the JQuery coding guidelines and those of other projects make it quite an informative read.

The second part describes common coding practices like avoiding global variables, loose coupling, exception handling and so on. Personally, this was basically the part of the book I was expecting when reading the title and I was not disappointed. There was some very good stuff in there like how to do exception throwing and handling in a cross-browser compatible manner. Also the drawbacks and advantages of browser-detection vs. feature-detection where described very clearly.

For some of the topics discussed in this part there are already tons of solutions in the popular JS frameworks like JQuery, Dojo and YUI available. Where appropriate the author includes examples from those libraries.

One thing I would have preferred reading more of here was the whole modularization of JavaScript development. The author goes into that when explaining how to get rid of global variables but this and how to clearly structure a larger project would have been a killer-chapter for me. There is also no mention of the whole CommonJS-approach on modularization which goes hand-in-hand with a total lack of server-side discussions. There is a short chapter on AMD and also RequireJS gets mentioned but more would have been greatly appreciated.

Perhaps the third part could have lost some pages to put those into part two ;-) This third and last part of the book deals with automating what was taught before and making it ready for prime-time via deployment. The tool of choice here is Ant, which receives IMO a bit too much focus here, but given that quite a lot of tools around the JS ecosystem are Java-based this makes sense. That said, it kind of made me skip most of this part since I have already worked with most of the tools mentioned in this chapter before.

"Maintainable JavaScript" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

• Configure your SSH server to accept private key logins. In /etc/ssh/sshd_config make sure that there's a line that says PubkeyAuthentication yes (and that there is no # infront of it). If you change this file, you need to restart the sshd service. If you're not sure, stop and ask somebody here before you break it.
  
• On your local machine (not the server), create yourself a pair of keys with ssh-keygen -t rsa (you can use other options than rsa, but I'm keeping it simple). Do not specify a password. Save the keys in the locations prompted.
  
• Open the contents of the id_rsa.pub file that you just created (it's one very long line of text), and copy the contents into the end of the file $HOME/.ssh/authorized_keys on the server machine. Create the file if it doesn't exist.
  

I found it interesting to read that phpBB is participating in the Google Summer of Code this year. One of the features that they hope to get out of the process is a post revision history… something that I started designing years ago, and will be releasing into production on my largest board within the next few weeks! Great minds thinking alike and all of that…

Turns out that in this case, phpBB2 was probably easier to work with than phpBB3. With phpBB2 the post text is already separated from most of the other post meta data (such as poster, poster IP address, and so on) which made it very easy to design a way to track post revisions. In phpBB3 they combined the phpbb_posts and phpbb_posts_text tables into a single table. In order to track post revisions in that case, the design would call for splitting the text out into a separate table in order to track the post text revisions without having to duplicate all of the other post meta data.

In any case, I have finished the coding and testing for everything and it’s functional. What’s left is going back and adding in some security checks to make sure people can’t do things they’re not supposed to do…

If you ever need to keep an image's center centered on a website no matter what viewport size your browser has, there is a neat little trick I stumbled upon on StackOverflow provided by Amber Yust. The goal is that the image is cropped not only on the right but also on the left side to keep the center of the image also in the center of the screen. The appoach mentioned by Amber is to position the image absolutely with an offset of 50% from the left and then re-adjust this offset using a negative margin of minus half the image's actual width.

#container {
    position: relative;
}
#container img#demoimage {
    display: block;
    margin: auto;
    position: absolute;
    left: 50%;
    margin-left: -600px;
/* If 600px is half of your image's width */
}

This has one caveat, though: You have to know the dimensions of the image in advance or determine them via JavaScript and adjust the negative margin after the fact.

(function(img) {
    // This will get called, once the image has actually been
    // loaded from the server
    function updater() {
        img.style.position = "absolute";
        img.style.left = "50%";
        img.style.marginLeft = "" + -img.width / 2 + "px";
        console.log(img.style.marginLeft);
    }
    img.addEventListener('load', updater, false);
})(document.getElementById("demoimage"));

If JavaScript is disabled, the margin:auto up there will at least center the image if there is enough room for it and prevent a slight jumping if JavaScript is enabled and the repositioning takes place.

For IE you will have to add some extra code to ensure your event is fired even then the image is already available within the browser's cache, but that's mostly it.

Another approach to this issue, which I usually prefer, is to have the image itself be situated in the background of the controller and positoned in its center. The upside of this solution is, that you don't need to position the image using JavaScript if you know its height to make room for it in the container. A downside here is, that without JavaScript this won't work if there is an imagemap on top of the image (which was the case and the original reason why I was looking for a different solution).

"Keeping image centers in the center" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

Everytime I travel I take some time to also check for local retailers of messenger bags et al. It kind of started more than 10 years ago with trip to Hamburg where I picked up my first Crumpler bag and it continues to this day with my visit to the NYC Chrome store about a month ago.

This time I left the shop (on the second visit) with a brand new Chrome Buran laptop messenger bag which should replace the Crumpler Spanky Jones for me. The problems with the Spany Jones is, that besides the laptop and some cables, you don't get all that much into it. The whole flying zipper concepts gets in the way here.

I want something new

The only other bag I could seriously use a replacement for this was that first Crumpler bag I bought in Hamburg (the Sticky Date) which is massive, though and not really something I'd want to carry with me every day on my way to work.

The Buran

That's where the Buran comes in. It is slightly larger than the Spanky Jones (45.72 x 31.75 x 15.25 cm vs 41.00 x 31.00 x 19.50 cm) but it has so much room for everything I want to have with me. I guess, the comparison is not really fair, though, since they both have difference target audiences. While the Crumpler bag is mostly a general purpose bag it is quite obvious that the Chrome bag is mostly for bike commuters with the tight shoulder bag and the region on the shoulder pad where you can attach utility bags etc.

The Chrome Buran

Right now I have among other things my 15" MBP, a large A4 notebook, a moleskin-style small notebook, various pens and fountain pens, a 0.5l water bottle, a book, a document map, some duct-tape ... The Spany Jones would only hold the laptop, the cables, the moleskin and the pens comfortably.

The bag is mostly divided into 3 parts: The main compartment which also holds a separate laptop sleeve, a utility compartment with things like pen holders, and the front of the bag which has two pockets with about the dimensions of a Kindle.

Tons of space

Regarding wearing comfort, unlike most Crumpler bags, the Buran doesn't have a padded back but the padding of the laptop sleeve is enough to not make it unfortable when you have a laptop with you. In general it took me quite some time to adjust the back to a way I truly enjoy wearing it, but once you're there, it is awesome. Part of the comfort is also the belt buckle. At first I thought of this more a toy or even just a design element, but it is incredibly useful. After a couple of days I never took the bag off without using the buckle. You just grab the briefcase handle, hit the button and off it goes!

The magic buckle

The position of the support strap for when riding your bike takes a bit of getting used to but it keeps the bag in place very well. And unlike other solutions this one even keeps the hard parts away from you by placing the shoulder strap between it and your body.

When it comes to weather resistance and resilience I have high hopes there. The material looks and feels very strong and there is quite a lot of trampolin in there, so water shouldn't be a problem.

"Chrome Buran" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

Alternate title for this post was “The Butterfly Effect.” You have have heard about that concept, where a butterfly flapping its wings in New York causes storms in Tokyo or something along those lines. I had a similar moment this weekend when I discovered that a clock being set ahead by one hour resulted in iTunes deleting songs after they were synced to my iPod. It’s a long story, so I’ll start at the beginning.

NetGear Duo Backup Process

My last post on this blog was about how I used a shell script on my NetGear Duo to automate the backup process for my various phpBB2 boards and other web sites. As part of that post I wrote:

To make sure I didn’t miss out on backup files any more I also added a line at the end of the process to email me the results of the backup script. Now each morning I can check my smartphone and confirm that the backup process ran correctly the night before.

For a while this backup process ran just fine. The notification came in to my phone just as expected. Then at some point I noticed that the back up process which was supposed to run at 2AM was instead running at 3AM. I did not worry about it too much. I just assumed that there was something going on with the cron schedule and I would look at it later when I had time. The important issue (downloading the back up files) was being handled, albeit at a different time than expected.

Ancient Virus

Remember nimda? It was a virus that came out back in 2001. I guess the formal classification is that it was really a worm, not a virus, but many folks use the terms interchangeably even though they’re not the same. I had some friends with some older computer hardware that they were trying to retire, but they still had some data on the older machines that they were trying to retrieve. The problem is that the files they needed to move were too big for a thumb drive, this particular computer had no cd-writer (it was from 2000, after all), and they weren’t comfortable removing the hard drive and moving it to a new machine. So they brought it over to my house.

I first tried removing the hard drive and putting it into an external drive chassis to read the files. Interestingly enough, the drive would not get show up. I wasn’t sure why, but that wasn’t the problem I was trying to solve. I noticed that the original computer case had a network port, so I decided to put the hard drive back in the box and connect it to my network. We had already established that the machine would boot up so I knew that would work.

As soon as we did that, my own computer started going crazy, making all sorts of beeps and noises! Turns out that this ancient machine was infected by the Nimda worm, and as soon as it got connected to my network it started to spread. Funny, being attacked by a ten-year old worm! I say “funny” because my AVG installation seemed to be doing its job and properly sending all of the infected files to the quarantine. I guess if you’re in the anti-virus (anti-worm) market, you had better make sure that you can capture anything that’s ten years old just as efficiently as newer stuff.

Ultimately we were able to recover the files for my friends, and I learned my lesson. The next time they brought over a computer I set up an isolated network between their machines rather than attach them to my network.

Interesting footnote: my iTunes library is stored on my NetGear device, both for redundancy and for file sharing purposes. Because it’s shared, that particular volume is open to the network. Because it’s open to the network, I think Nimda dropped a file into every single folder, which I then had to clean up. More on this later.

iTunes Difficulties

Around the same time iTunes released a new version (10.6) that was supposed to handle album artwork better. Now normally I am not a fan of jumping on the latest update. In fact I have my “automatic update” turned off for iTunes just to avoid experiencing a broken system when something that I have is already working. But in this case I had experienced a number of issues with managing my iTunes art, so I decided to install the update. It does, in fact, handle album artwork much better so I was pleased with the update.

A few days later I made a trip to a local used book / compact disc store and came home with a stack of about 20 cds or so. I ripped them using iTunes, reset the artist from “First Last” format to “Last, First” as I like to do, and downloaded album art from my favorite album art site Album Art Exchange. These are all steps that I normally would do. While performing these various steps I was also listening to some of the new music that I had purchased. (”New” is a relative term, as I recall on this particular trip I found a number of “classic rock” artists from the 80’s so it was a nice blast of nostalgia.) Once I had performed all of my updates I synced iTunes to my iPod.

Here’s where it gets weird.

I went back to play some of the songs, and they were gone! The files were physically missing from the hard drive.

Investigation Part I

Remember that earlier I mentioned that my iTunes library is stored on a public network share, and that it had been attacked by Nimda, right? That was the first area I investigated. I first did a full scan of the entire system with AVG. I did a search for the file “droppings” that Nimda had left behind, and when I found them I deleted them or removed them from the AVG quarantine. I wondered if the worm was renaming some of my audio files rather than creating these as new files, but the only songs that were deleted were new ones. Nothing that had been ripped months (or years) ago was being deleted, it was only the new files.

I downloaded some specific scanners designed to find and remove Nimda and confirmed that I was infection free. Now what?

Investigation Part II

I re-ripped the music from cds, and carefully repeated the process. It seemed that once the songs were on the iPod, as long as I recovered them somehow iTunes was fine. It did not do any further deleting. I wasn’t sure what was going on, but at least the problem was something I could recover from. Next I ran a controlled test. Here’s what I did, and what I observed.

1. I ripped three new cds (new meaning ones that I had not ripped from my collection yet)
2. I navigated to the folder and confirmed that all songs had been ripped. I then zipped up the contents of each folder.
3. I made some minor edits to the data once the cds were ripped. As mentioned above, I prefer artist names to be “Last, First” rather than “First Last” so I generally update the artist and the album artist to reflect this. I also have a group tagging strategy, so every song has at least one entry in the grouping field. All of this worked fine.
4. I sync’d to my iPod. But this time I kept a Windows Explorer session open on one of the new folders. Gotcha… I watched some (but not all) of the files disappear.
5. I then checked my iPod. The files were there on the iPod, they just were no longer on my hard drive. Of the 3 new albums, here are the specific details.

   Album 1 – 11 tracks ripped, track 1, 3, 5, 6, 8, and 10 missing after sync
   Album 2 – 9 tracks ripped, track 1 missing after sync
   Album 3 – 11 tracks ripped, all 11 tracks deleted during sync process

   Seemingly a random sequence of events. I am not sure what to make of this right now.

6. To recover I unzipped the archived files I made for each album, and verified that iTunes could see the data (no more ! next to the song).
7. I sync’d to my iPod again. Now remember that the songs already exist on my iPod, they were just deleted from my hard drive. So there’s nothing to update. After the sync (the second time) the songs are still there on the hard drive, still on the iPod, and no issues.

What About Other Data?

At this point I had eliminated the Nimda worm from consideration. I went through some other applications and carefully created specific files on various folders throughout my network disk device. I opened / edited / closed and did a number of other operations. Nothing I did deleted files in other areas of the network disk device, so at this point I eliminated that from consideration. It really seemed to be the iTunes 10.6 update that was the culprit here, but how to fix it?

10.6.1 came out a few weeks later, and I immediately updated. It did not fix my problem.

Investigation Part III – Isolation

I decided to try a different iTunes installation. I repeated this experiment with the same cds on another computer. This computer also has iTunes 10.6 but is only used to sync to an iPhone belonging to my wife. There were no songs on this computer prior to me adding these test albums, so the library is very small. The library on my computer contains over 40,000 songs. Nothing went wrong on my wife’s computer, which is also running Windows XP. The primary difference is that on the second computer the iTunes library is stored locally rather than on the network device.

But I have already done what I can to eliminate the network device as the culprit.

I’m going in circles.

On my last test, back on my computer, I ripped a single cd and did not do any of my normal steps. I ripped and then moved immediately to sync to my iPod. The entire album – all 21 tracks – was deleted. Even the folder was removed!

Butterfly Wings

Yesterday (Sunday) I checked my email for the back up process. It ran, but all of the files were stamped August instead of April, and the year was 1992! I opened a shell session on the network disk device and confirmed that yes, it thought the date was 1992. I opened the administrator tool and confirmed that the date there was also showing as 1992. The device was set up to sync to a time server hosted by NetGear, so I turned that off and reset to the proper time manually. I confirmed that the new date took, and then went back and reset to the NetGear time service. It was back in 1992 again. A quick check of the NetGear forums found several other posts about the same thing, some of which suggested moving to a different time service.

Now it gets interesting. When I moved to a public time service (off of the private NetGear hosted service) not only did I get the correct time, but the hour difference mentioned at the very beginning of this post was also corrected!

All of a sudden a light went off. Could the one-hour time difference be impacting my iTunes?

Problem Resolved

In my earlier tests, when I ripped and immediately synced a single album, it meant that the entire set of songs was processed within an hour. I wondered if these songs were being deleted because iTunes was confused. According to my network disk device, these files would have been ripped one hour into the future! I know iTunes tracks a ton of data, including last played date, last skipped date, and of course the ripped date. Since the ripped date and the file create date on the operating system were out of sync by an hour, I can only assume that iTunes was doing something weird and ultimately deleting the files. The reason the deleting was more random when I ripped a bunch of songs was because (I guessed) the overall process took longer than one hour. If I worked quickly, more songs got deleted. If I took my time, fewer songs were deleted.

To test this concept I deleted the most recent album from iTunes and synced my iPod to clear everything up. I then ripped and synced the same album and none of the tracks got dropped. Everything was processed just fine. Earlier using these same steps all 21 tracks had been deleted.

Ultimately it appears that the Nimda worm was a red herring. It had nothing to do with the issue. The network drives were also functioning perfectly from a mechanical perspective. The network was also fine. I was not having a problem with any other application, only iTunes.

It was a combination of the network time service being wrong in conjunction with some weird program bug in iTunes that was causing random files to disappear. Once I reset my time using a public time service (the same one used for my Windows system) iTunes functioned fine.

That’s the butterfly effect. A clock is wrong by one hour and random music tracks are deleted as they are synced to my iPod.

That was my fun for the month. How was yours?

Creating components using CRXDE and CRXDE Lite can be frustrating. Both are buggy and are different environments than what you are use to working with. You can use Maven to create a package that it can then install the package in CQ5.

If you have ever extracted a package then you’ll see that there are two folders within it, jcr_root and META-INF. Both of these files are important and they determine how the Maven project should be setup. META-INF holds the filters (paths) that the package contains. jcr_root has all of your files for components, templates, etc. I typically put these files in src/main/content in the Maven project so my code will have that path.

First thing you need to do is in pom.xml under <build> add the following code so that it knows to include your files in the bundle. This also excludes any file vault (VLT) files.

<resources>
    <!-- filter meta information to get some properties into the files -->
    <resource>
        <directory>${basedir}/src/main/content/META-INF</directory>
        <targetPath>META-INF</targetPath>
        <filtering>true</filtering>
    </resource>
    <!-- exclude .vlt control files and tests -->
    <resource>
        <directory>${basedir}/src/main/content/jcr_root</directory>
        <excludes>
            <exclude>**/.vlt</exclude>
            <exclude>**/.vltignore</exclude>
            <exclude>**/.DS_Store</exclude>
        </excludes>
        <targetPath>jcr_root</targetPath>
    </resource>
</resources

Next you need to setup a profile that uses Groovy to install the package to CQ5 using POST.

<profile>
    <id>installUI</id>
    <activation>
        <property>
            <name>installUI</name>
        </property>
    </activation>
    <properties>
        <hostname>http://localhost</hostname>
        <hostport>4502</hostport>
        <publishhostname>http://localhost</publishhostname>
        <publishhostport>4503</publishhostport>
        <packageName>${artifactId}-${version}</packageName>
        <cquser>admin</cquser>
        <cqpassword>admin</cqpassword>
        <showUpload>true</showUpload>
        <showInstall>true</showInstall>
        <projectDir>${basedir}</projectDir>
    </properties>
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.gmaven</groupId>
                <artifactId>gmaven-plugin</artifactId>
                <version>1.2</version>
                <dependencies>
                    <dependency>
                        <groupId>org.codehaus.gmaven.runtime</groupId>
                        <artifactId>gmaven-runtime-1.7</artifactId>
                        <version>1.2</version>
                    </dependency>
                    <dependency>
                        <groupId>commons-httpclient</groupId>
                        <artifactId>commons-httpclient</artifactId>
                        <version>3.1</version>
                    </dependency>
                    <dependency>
                        <groupId>commons-logging</groupId>
                        <artifactId>commons-logging</artifactId>
                        <version>1.1.1</version>
                        <type>jar</type>
                        <scope>compile</scope>
                    </dependency>
                </dependencies>
                <executions>
                    <execution>
                        <phase>install</phase>
                        <goals>
                            <goal>execute</goal>
                        </goals>
                        <configuration>
                            <defaults>
                                <cqservers>localhost:4502</cqservers>
                                <cquser>admin</cquser>
                                <cqpassword>admin</cqpassword>
                                <packageName>${artifactId}-${version}</packageName>
                                <projectDir>${basedir}</projectDir>
                            </defaults>
                            <source>
                                import org.apache.commons.httpclient.*
                                import org.apache.commons.httpclient.methods.*
                                import org.apache.commons.httpclient.auth.*
                                import org.apache.commons.httpclient.methods.multipart.*

                                //method declaration for execute of PostMethod
                                def executePost(httpclient, httppost, outputResponse){
                                    try {
                                        httpclient.executeMethod(httppost);
                                        if (outputResponse){
                                            println httppost.responseBodyAsString
                                        }
                                    } catch (Exception e) {
                                        println("exception: " + e)
                                    } finally {
                                        httppost.releaseConnection()
                                    }
                                }

                                def uploadAndInstall(hostPort, packageName, projectRoot, httpclient){
                                    //upload package
                                    def httppost = new PostMethod('http://' + hostPort + '/crx/packmgr/service.jsp')
                                    def file = new File(projectRoot + '/target/' + packageName + '.jar')

                                    println("Installing package" +packageName+" to " + hostPort)

                                    if(file.exists()){

                                        def parts = [new FilePart("file", file)] as Part[]
                                        httppost.setRequestEntity(new MultipartRequestEntity(parts,
                                        httppost.getParams()))

                                        executePost(httpclient, httppost, true)

                                        //install package
                                        def installURL = 'http://' + hostPort +
                                        '/crx/packmgr/service/.json/etc/packages/' + packageName + '.zip'
                                        println('INASTALL: ' + installURL)
                                        httppost = new PostMethod(installURL)
                                        httppost.addParameter("cmd", "install")

                                        executePost(httpclient, httppost, true)
                                    } else {
                                        println("File does not exist for " + packageName + ". Not Deploying.")
                                    }
                                }

                                def user = project.properties['cquser']
                                def pass = project.properties['cqpassword']
                                def packageName = project.properties['packageName']
                                def projectRoot = project.properties['projectDir']

                                // set up the client
                                def httpclient = new HttpClient()
                                def defaultcreds = new UsernamePasswordCredentials(user, pass)
                                httpclient.getState().setCredentials(AuthScope.ANY, defaultcreds)
                                httpclient.getParams().setAuthenticationPreemptive(true)

                                println "CQSERVERS:" + project.properties['cqservers']

                                project.properties['cqservers'].tokenize(',').each {
                                    uploadAndInstall(it, packageName, projectRoot, httpclient)
                                }

                            </source>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <!-- Compile Java classes for OSGi bundle -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</profile>

When building the project make sure you select the installUi profile or if you are using the command line add the argument -DinstallUI and it will automatically install the package into CQ5.

I have created a sample project that contains everything you need that you can download here: cq5-sample-ui.zip

For my job I do a lot of CQ5 work. Often this involves creating OSGI bundles. To install an OSGI bundle you either have to put the bundle in /apps/myproject/install or manually install it in the Felix Web Console. Both of these methods get old after doing it multiple times.

Sling has a Maven plugin that simplifies this process by installing the bundle for you when you build the project. Simply add the following code to your pom.xml file for your Maven project and then select the install-osgi profile when building.

<profile>
	<id>install-osgi</id>
	<activation>
		<property>
			<name>install-osgi</name>
		</property>
	</activation>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.sling</groupId>
				<artifactId>maven-sling-plugin</artifactId>
				<version>2.0.4-incubator</version>
				<executions>
					<execution>
						<id>install-osgi</id>
						<goals>
							<goal>install</goal>
						</goals>
						<configuration>
							<slingUrl>http://localhost:4502/system/console/install</slingUrl>
							<user>admin</user>
							<password>admin</password>
							<skip>false</skip>
							<bundleStartLevel>20</bundleStartLevel>
							<bundleStart>true</bundleStart>
							<refreshPackages>true</refreshPackages>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</profile>

Or if you are using command line to build the project, add -Dinstall-osgi to the arguments. You can create as many profiles as you’d like for the different environments you may be working with (author, publish).

Some years back I purchased a couple of NetGear Duo RAID disk enclosures. Each enclosure was loaded up with dual 1TB disk drives from Western Digital. Everything written to one drive was mirrored to the second drive. This helped protect against a single disk failure.

I also had a very old (PIII processor!) computer that was running linux. At one point I was using it as a development environment, but it had long since been retired from that duty. Why was it still around? I had created mount points for each of the RAID arrays on this linux box, and it was responsible for going to my web server (which hosts this blog, among other things) and downloading the nightly database backup files. (My web server runs a hot backup at 1AM each morning using the mysqldump command.) This linux box also had a script (running at 2AM) that would ftp to my web server, retrieve all of the database dumps, download them, add a date stamp to the file name, and then copy the resulting files out to the RAID array. The entire process was automatic, transparent, and for a long time was quite robust.

Until a few weeks ago.

At various points I would go out to the disk array and check to see that the backup files were properly being retrieved. The last time I checked, it seemed that I was not getting any backup files… at least not for the past several weeks. Attempting to log in to the development / backup box via ssh proved to be unsuccessful. After rebooting the box, though, everything came up and seemed to work. I was able to run the backup script without errors.

The next morning, however, there was no backup file from the prior night, and the box was once again locked up. To be honest, it wasn’t worth fixing. It was noisy, a power hog, and the only thing it was doing at this point was downloading the nightly backup files. The NetGear Duo was also running a variation of linux, so why not try that? I installed an add-on that enabled ssh access to the device and went to town. After a couple of different attempts I eventually got a cron job set up that would connect to my web server and do all of the same things my old linux box used to do, with the added advantage that the Duo was already on as a file server anyway so it was now taking on an additional task.

Life was good, and backup files began to show up on a regular basis once again. To make sure I didn’t miss out on backup files any more I also added a line at the end of the process to email me the results of the backup script. Now each morning I can check my smartphone and confirm that the backup process ran correctly the night before.

Time To Upgrade

But never one to leave well enough alone I decided to try upgrading one of the Duo devices to a newer model, the Ultra 2 Plus. This new device had a lot more memory, dual network ports, and a multi-core CPU. Everything should be faster. I also upgraded to the “black” series of Western Digital disk drives as I had experienced intermittent problems with using the “green” series inside the RAID enclosures. Everything came in last week and was installed with only a little fuss. I got my security configuration reset so that I can use the scp command (secure copy) to download the files from my web server. The last hurdle was that the new Ultra 2 Plus device did not have a mail command installed! After some research and posting on the ReadyNas forums (which by the way run phpBB) I got the answer: I have to use the sendmail command instead. This command is not quite as friendly as the mail command, but I got it to work.

Database Backup Script

My web server already has a cron job scheduled at 1AM that uses the mysqldump command to create a database dump of every database that I want to back up. Here is the following cron job which is scheduled on the Ultra 2 Plus. It’s responsible for connecting to the server, downloading the files, adding a date stamp, and then copying the files out to the designated sub-directory on the RAID. Finally it creates the mail file and sends it out. The list of databases to back up is included in a text file so that I don’t have maintain that script.

#!/bin/bash

# first establish local path location
cd /w_drive/incoming/

# next get updated copy of dbnames.txt
scp username@example.com:db_backups/dbnames.txt .

for dbname in `cat dbnames.txt`; do

#       Go get database backup file
        echo "Processing $dbname "
        scp username@example.com:db_backups/$dbname.sql.gz .

#       Establish some local variables
        filename="_`date +%Y-%m-%d`.sql.gz"
        filename="$dbname$filename"
        outputpath="/w_drive/$dbname/$filename"

        echo "Moving $dbname.sql.gz to $outputpath"

#       Rename local copy of db backup file to include date
        mv $dbname.sql.gz $filename

#       Then move dated backup file to proper output path
        mv $filename $outputpath
        chmod 644 $outputpath

done

# Build mail formatted file
echo "date: `date +%Y-%m-%d`" > mail.txt
echo "to: email@example.com" >> mail.txt
echo "subject: Backup" >> mail.txt
echo "from: email@example.com" >> mail.txt
cat /w_drive/incoming/getbackup.out | grep -v "tty" >> mail.txt

# Send the email
/usr/sbin/sendmail email@example.com < /w_drive/incoming/mail.txt

As an additional bonus, my office is a lot quieter without the extra server running. I don't know that it will make a noticeable difference in my power bill, but it will certainly help at least a little bit.

I've head iOS devices for a really long time. My first one I got during PyCON UK 2008 in Birmingham with a 2nd gen iPod touch, kind of as a mobile movie player. Over the years I grew quite fond of the aesthetics and the general look and feel of the iPhone 3GS and later on the iPad 2. But somehow, over the years, I also grew somehow tired of it. I couldn't help myself feeling a bit underwhelmed by the waste the primary "screen" of the OS represented. Instead of useful information you're always greeted by your favorite apps.

I don't want to take part in that whole Android vs. iOS fanboy war so I'll just leave it at that: I definitely see the usefulness of a widget concept when it comes to screen-estate on small devices.

And when I saw Ice Cream Sandwich in November in Berlin during the #GDDDE I definitely wanted to give it a try. My previous cell-phone plan had expired and had been overpriced to begin with. So, I had gotten a contract without minimum length a month before #GDDDE just to keep all my options open, and in December I finally decided on getting the Android 4.0 flagship handset by Samsung, the Galaxy Nexus.

Probably the first thing I did was install all the apps (if available) that I had used on iOS before: Evernote, Spotify, Scotty and a couple more. Then came the first important step: Getting podcasts.

In case I start to babble again: At the end of the article is a tl;dr section ;-)

Podcasts on ICS

With iOS podcasts are straight forward. iTunes can handle them for you and it was all I needed since 2008. In fact, podcasts are for me the most important thing of the morning after reading the news, so getting a good experience on my new phone is crucial.

For Android and years ago I had heard of Google Listen, so this was the first app I tried. Not bad, but kind of completely outdated. I definitely looks like the experiment it actually is. So after looking around a little bit I stumbled upon Pocket Casts. Both of them handle not only the playback of the weekly shows but also all other aspects of subscription management. Sure, I could have used Pocket Casts also on my iPhone but here I was more or less forced to jump into the water on undetermined temperature... and it was perfect ;-)

The experience

So my primary use-cases are all covered: E-mail (even better thanks to GMail integration), music, podcasts and note-taking. As for the rest of Android and ICS, I got used to the different UI pretty quickly. Within perhaps a day I was as quickly with my Galaxy Nexus finding my way from e-mails to news-posts and back as I was using my iPhone. Is iOS more consistent? Perhaps, but more in the positioning of buttons than the actual look and feel of apps. After Game Center Apple fanboys should probably think twice before dissing Android apps ;-)

Another big plus for me is the whole integration of online contacts with the classic contact management. I admit I'm a pretty heavy Google+ user so I really like that all my contacts there are already synchronized. Same goes for Twitter (I haven't installed the Facebook app yet simply because I don't use Facebook that often and try to not use it at all except for logging into Spotify).

And since I'm a Google+ user I also use GMail quite heavily, which, naturally, integrates extremely nicely with the rest of the system. In general the whole system seems to offer enough hooks for developers to integrate their apps into the core aspects of Android. For example, the browser offers a "share" button which has Evernote and todo.txt hooked into it. Mobile Safari didn't allow me to send content straight to Evernote which had become extremely annoying to me over the last couple of months.

And the bad parts

I guess, over the last couple of paragraphs I've, despite my original plan, started to read like some Android fanboy and partially that is also true. But, naturally, iOS and iPhones haven't come to be where they are right now by being a bad choice. The whole integration between iTunes and the players and apps on all iOS devices is something Google can't provide right now. Amazon perhaps in the future.

Another integration issue is the one of hardware. With my iPhone I could simply go into a store and buy a pair of headphones labelled as "works with the iPhone" and things like pausing playback, increasing and decreasing the volume worked right away. Not so much with the Galaxy Nexus. But at least after an update Pocket Casts can use my earphones' pause button :-)

But this is pretty much all the negative I can write about my experience with Android so far. I'm pretty sure there are more dark corners to be discovered but for now I'm extremely happy I dared the switch :-) I'm probably also closer than ever to wanting to develop for a mobile device thanks to the more open developer infrastructure. But more on this if I really find the time to dive into Dalvik ;-)

TL;DR

• I'm positively surprised by how Android feels with ICS
• All the apps I need are there and sometimes even more stable than on iOS (e.g. Spotify).
• Don't use Google Listen for podcasts but instead invest in something like Pocket Casts if podcasts are important to you.
• Pause buttons on headphones seem to work, volume buttons do not.
• Contact integration with Google accounts and Twitter is really handy.
• Sharing integration not only with Google services but also things like Evernote work out of the box (after installing Evernote)

"Getting to know Android" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

This is my second attempt at my 29 gallon tank. I didn’t really know what I was doing with my first attempt. I saw some plants at Petco and thought it would be cool to try it. It failed miserably since I didn’t know anything and was trying to learn as I went. I’ve learned that there is lots of work and learning that needs to be done in order to have a planted tank. I need to be patience and learn as much as I can before jumping in with both feet.

I’ve decided to give it a shot again. I’m attempting to do a high tech setup on a budget.

The Details

Substrate: I’m am using a 50/50 mix of flourite and black gravel. It is about 3 inches deep.
Lighting: I have three 8.5 inch clamp lights from Home Depot with 27 watt (or was it 23? I don’t remember) 6500k spiral CFL blubs in them. I write another post explaining my light in more detail.
Filter: I have a SunSun HW-302 canister filter that I plan on using after I fill the tank with water.
CO2: I have a 20# CO2 tank with an Aquatek Regulator and bubble counter. Along with the filter this isn’t being used at the moment since the air contains plenty of CO2 so I don’t need to inject it into the tank.
Fertilizers: I’m still doing research on these but I plan on adding fertilizers once I add water to the tank. I’ll post more about this when I have this figured out.
Plants: I’m going to stick to mostly easy plants and set myself up for success. I’ll throw in a couple harder plants once I get things figured out. I plan to have a heavily planted tank.
Fish: I want some schooling fish for sure. I haven’t thought about this a whole lot since the plants have been my main focus.

Last night I was about to release a new version of django-flatblocks, when I ran into a weird problem:

$ python setup.py check -r                                                                                                                                                                           [develop] 13:58+0100
running check
Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    zip_safe = False,
  File ".../distutils/core.py", line 152, in setup
    dist.run_commands()
  File ".../distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File ".../distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File ".../distutils/command/check.py", line 69, in run
    self.check_restructuredtext()
  File ".../distutils/command/check.py", line 111, in check_restructuredtext
    for warning in self._check_rst_data(data):
  File ".../distutils/command/check.py", line 138, in _check_rst_data
    parser.parse(data, document)
  File ".../docutils/parsers/rst/__init__.py", line 157, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)

    .....

  File ".../docutils/parsers/rst/states.py", line 991, in implicit_inline
    return [nodes.Text(unescape(text), rawsource=unescape(text, 1))]
  File ".../docutils/nodes.py", line 331, in __new__
    return reprunicode.__new__(cls, data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26: ordinal not in range(128)
</module>

The cause of this issue was a bit unexcected: A string in the release notes of a version I had already released about a year ago. So what changed? Probably the Python version but also most definitely the version of docutils I had installed in order to preview the release notes before every release.

So I started downgrading from 0.8.1 all the way to 0.5.x to finally make the check command not die on me anymore.

There is also an issue in Python's issue tracker which has a patch attached (and committed) that solves this issue. So if you don't want to downgrade docutils in your virtualenv, this is probably the way to go :-)

"When setup.py check fails" was written by Horst Gutmann and is licensed for reuse under Creative Commons Attribution-Share Alike 3.0 Austria License.

I have posted a topic on phpbb.com letting folks know I’m not going to renew my phpBB3 MODs-related domains. If you’re interested, hop on over to phpbb.com and send me a PM to let me know.

First come, first served.

Way back in 2009 phpbb.com was hacked via an exposure in the phplist mailing list software. (Just to be very clear, there was no exploit found in the phpBB code used to run the site, only this supplementary program used to run the email notification system.) One of the results of the hack of phpbb.com was that several thousand user passwords were exposed and posted on the Internet. They were exposed, frankly, because they were too simple, and that’s not phpBB’s fault.

Pet Names Don’t Make Good Passwords

People often struggle with remembering passwords. Yes, there are programs that can help you with that, but are they really that much more secure than writing things down on a piece of paper? Here’s one take from one of my favorite web comic authors:

The bottom line is that a password is supposed to protect an account from unauthorized access. It’s not supposed to prevent authorized access, but for infrequently used resources it can do that as well.

Password Storage

One indicator of concern to me is whether a web site can send you your existing password when you forget it. That means they’re likely using an encryption process rather than a hashing algorithm to store passwords. And that means anyone with access to the decryption key can read every password in the system. phpBB2 uses a hashing algorithm which means as a board owner / administrator I can safely say to any of my users that I cannot tell them what their password is. I can assign them a new one, and of course there is a self-service option available to phpBB2 users that will do just that, but I cannot tell them what their existing password is. In a default phpBB2 installation a user password is hashed and stored as a 32 character string. Similar passwords will generate very dissimilar hash strings, so there is no way for someone to easily guess what a password might be simply based on the results of the hash. For example, here are a few similar input values and the resulting MD5 hash outputs:

test     098f6bcd4621d373cade4e832627b4f6
Test     0cbc6611f5540bd0809a388dc95a615b
tset     751ec45015a704a39dc403001c963e97
test1    5a105e8b9d40e1329780d62ea2265d8a

Despite the similarity of the input values they all have very different hash values. That means that similar hash values are going to have very different input values as well. Also, because a hash is generated by a one-way function there is no way to recreate the input value based on the hash. (There are some cases where two different input values could generate the same output hash; that’s allowed.)

If that’s the case, then how can I tell if a user has entered the correct password when they log in? It’s actually really simple. When a user logs in, phpBB takes the password provided on the form, hashes it, and then compares the resulting hash value to the hash stored in the database. As long as the input values are the same (entering “test” as my password will always generate 098f6bcd4621d373cade4e832627b4f6 as the hash value) then the password matches and the user is logged in.

How, then, were hackers able to decipher the passwords stored in the phpBB.com database if there is no way to reverse a hash process? They didn’t. Instead they used a lookup table (also called a rainbow table) to match up known hash values with their source value. Imagine taking a dictionary and running every word in it through the hashing process. When you’re done, you have a list of hashes and their source. By matching password hash values against the list of known entries the hackers were able to figure out what a number of phpBB.com user passwords were.

To help defeat this sort of process, phpBB3 now uses a salted hash which provides even more security. I imagine someone has back-ported this to phpBB2 as a MOD but I have not had time to look for one yet. But the fact is if a secure password is used (a combination of words and numbers, or as the cartoon above suggests several words run together) the odds of the password appearing in a rainbow table are extremely slim, and even the simple hashing algorithm used in phpBB2 is essentially safe. Even changing the case of one letter helps, as shown in these two hash examples from earlier:

test     098f6bcd4621d373cade4e832627b4f6
Test     0cbc6611f5540bd0809a388dc95a615b

Changing the “t” to a “T” changes the hash. It might be even better to change something other than the first letter, for example changing the “e” to an “E” as in tEst. That’s a very easy word to remember and it’s not likely to show up on any rainbow table, unless a hacker wants to run every possible combination of upper and lower case letters for every possible dictionary word. Ultimately it would be better to use a combination of letters (mixed case), numbers, and even symbols where allowed.

So how important is it that phpBB3 uses a salted algorithm for passwords? If people were smart about what passwords they use then it really would not matter so much. But as shown in some of the related links at the end of this post, people are not very smart about the passwords they pick. In that case, the salting process is quite beneficial.

Social Engineering

Which brings me to my final issue for this post: people can be fond of certain passwords and often use (reuse) them on more than one site. If you use the same password for a phpBB board that you use for your banking system, then change it. Change it now. For one thing, most phpBB board owners do not use a secure protocol (such as HTTPS rather than HTTP) for their sites. That means the password is exposed during the transmission of the form data. Ultimately no matter how a password is stored or how complex the password might be, if people can be tricked into giving up their passwords by phishing emails, web site plugins, and other techniques, that’s a problem. If a hacker manages to grab the password for one site, they may then be able to use it on other sites. I have made a practice of using a unique password on every service I sign up for in order to prevent this for a very long time. But once again, xkcd.com says it better:

Related Links

• phpBB Password Analysis
• 500 Worst Passwords

Animations from an installation at L'Eclaireur in Paris.

L'Eclaireur Sévigné has a unique installation with 147 screens dotted around like picture frames. Early in 2011 I was commissioned to create some original videos for the space, which have been playing for the last six months.

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/41558647?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34656394?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34657460?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34657542?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34657573?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34657651?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34657710?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34658252?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34658271?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34658286?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34659879?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>

<iframe allowfullscreen="1" frameborder="0" height="329" mozallowfullscreen="1" src="http://player.vimeo.com/video/34659899?portrait=0&amp;loop=1" webkitallowfullscreen="1" width="585"></iframe>