Notes for setting up a new Mac

• Install xcode
• Install brew
• Move /usr/local/bin higher up in /etc/paths
• Install git
• Config user and email
• Add SSH key to github
• Clone linux files repository
• Install macvim
• brew install macvim --override-system-vim
• Install the newer version of iterm2

Os X vim/git configuration

I spent a while today finally ironing out some small annoyances I've had for a while with my OsX development setup. I had some problem setting the EDITOR variable for git. Even though it was set to vim and which vim showed me /usr/local/bin/vim, whenever I would run git commit the standard OsX version of vim would open up. I ended up just setting EDITOR to the absolute path.

Next up was finally figuring out a way to get a simple way to press esc when using vim. Earlier I had resorted to using ctrl-[ but I was getting really fed up with it now. I found two nice additions to the Os X preferences called KeyRemap4MacBook and PCKeyboardHack. What I wanted to do was change my left control to be esc and change my caps lock to be left control. I have a Japanese Macbook Air where the keyboard looks like this: http://4.bp.blogspot.com/-a7eg4DXrq9Y/TfWtph3- ovI/AAAAAAAAKfs/c-IVbHNOGPg/s1600/Japanese-MacBook-Air-Keyboard-Layout.jpg I had some trouble initially since my mapping from caps lock to control would end up being translated to esc as well, because I had remapped control to that. In the end I managed to hack my way around the problem by adding a custom mapping for esc to control. And then map caps lock to esc instead... Confusing, no?

This is the curse of wanting to eliminate all the small annoyances you have while programming. Now back to trying to set up a portfolio page on github pages.

Compiling Vim on Ubuntu Natty

The latest version of vim 7.3 included in Ubuntu Natty has a bug (Ruby-related I think) that causes the awesome command-t plugin to stop working. To get around this you will have to compile vim yourself to get to a higher patch level. I've done this a couple of times recently but always forget which dependencies need to be installed and which configure flags to enable and which make targets to run if you change stuff. So I thought I'd try to document my findings in case I have to do it again and to try to figure out exactly which packages need to be installed. On some occasions I've ended installing hundreds of megs of dependencies just to be on the safe side.

1) First you need to clone the vim repository using mercurial, this step is easy and is described on the official vim.org site: http://www.vim.org/mercurial.php

2) Install dependencies: sudo apt-get build-dep vim This pulls in about 554 MB of stuff at the time of writing and I'm not sure if all of it is needed but it at least is quick to type. Running configure at this point, gawk was still missing.

2) Next you need to configure the build you create. I've found the following to be smallest subset of required flags: ./configure --with-features=huge --enable-rubyinterp

3) then make and make install

Deploying a Django app with Apache and mod_wsgi

At work recently I've been deploying a small Django app. It's taken quite a long time. Most of the time things have been moving forward nicely but every now and then there has been some misleading detail in some instruction which has made me stand in place for a long time.

In a nutshell (on Ubuntu)

• Apache needs the mod_wsgi module


• Use the WSGIScriptAlias directive to point to the location of your project's .wsgi file


• Point to your settings in the .wsgi file

This procedure is described in How to use Django with Apache and mod_wsgi

Complications

1. 
   I used virtualenv to prevent the installed modules from conflicting with other modules. This meant I had to follow the Virtual Environments section of mod_wsgi's wiki. You don't have to follow these instructions precisely. The instructions are aimed at preventing system-wide modules from shadowing/overriding modules in your virtualenv. Simply using site.addsitedir with the virtualenv's site packages directory worked for me.
   



2. 
   NOTE! The instructions in Django's docs also neglected to mention that you have to append two paths to your sys.path. You have to append both your project's path as well as the project's parent's path. If you don't add the parent you can't specify os.environ['DJANGO_SETTINGS_MODULE'] prefixed with your project's name. If you don't add the project itself you can't import the apps inside it.
   



3. 
   The server that was going to run the web app only had Python 2.4 installed. I wanted to make sure that my development environment would be reasonably close to the production environment. Installing an older version of python was very easy. All you had to do was use make altinstall instead of make install. However, now you also need to compile the mod_wsgi module manually. You use the --with-python=/usr/local/bin/python2.4 configure flag to specify which version to compile for. Then when you make the module it ends up in a hidden .libs folder. Move it from there to Apache's modules folder. I chose to rename it to wsgi_2.4.so or something so it wouldn't overwrite the module installed by apt-get earlier. Check some .load file from your /etc/apache2/mods-available/ for reference. Then you can add your own wsgi_2.4.load and wsgi_2.4.conf files to mods-available and use a2enmod to enable it. Now since you are using your own vintage Python version you have to additionally use the WSGIPythonHome /usr/local/ (or something, my mind is a bit unsure of this detail at the moment) directive to specify the prefix to your Python installation.
   



4. 
   Finally if you are serving static content (css, images, js etc.) you should be doing that straight with your web server instead of through Django. Django has a very nice app called django.contrib.staticfiles which through adding STATIC_URL to your settings.py file, adding django.core.context_processors.static to TEMPLATE_CONTEXT_PROCESSORS and prefixing your static urls with {{ STATIC_URL }} allows you to easily serve files with runserver. Then for production use you will want to set up the STATIC_ROOT variable in settings.py, put application-specific media in a static folder and run manage.py collectstatic to have all your media files transferred into the STATIC_ROOT. Now you can serve the files from this location by using the Alias directive in Apache for example.
   

I felt way more l33t when I had to set up the 2.4 version of Python manually than by simply installing everything through apt-get like a n00b.

Google Nexus S

Earlier this week I finally decided to upgrade my old Nokia E51 to an Android phone. Iphones are cool and all, but as a programmer who would like to make some apps of my own, it would be too much to have to both get a Mac and learn objective C as well as iCocoa-whatever-it's-called. You can even code Python on the android!

I think I bought the E51 around 2007-2008. It was good enough for some basic web surfing and worked excellent as a mobile broadband device on train trips at least. However, for a long time I've felt inferior to all the smart phone owners.

I'll get the minor problems I've had out of the way first, so I can end on a positive note. The wlan at the office has been very weak. It has seemed to work fine in my wlan at home though. I was able to download some of my Spotify playlists without problems. Also Angry Birds crashed during start-up and after that there was some bug that shifted the whole set of levels one step down, so I had to reset the game's data.

Apart from these problems, I've been very satisfied with the Nexus S so far. The jump in web browsing convenience from the E51 is huge. The UI is very smooth. One of the reviews I read before getting the phone mentioned something about slight delays in the UI, but I haven't noticed anything. Setting up e-mail was incredibly easy. I guess it helps that I use gmail both for my personal e-mails and for my work e-mail. Now I get nice little notifications every time I get an e-mail to either of them.

There are a lot of nice touches too. I like the effect when the screen is turned off which looks a bit like an older TV screen being turned off. There were some pretty nice blip and blop message notification sounds to choose from.

There are still many areas I haven't explored that much yet. I have only tried taking two pictures so far. Both of which seemed perfectly fine. I haven't tried tracking that much with the GPS yet. There are still a lot of settings to explore. Today, for example, I figured out that it was possible to easily switch between languages while typing an SMS. Another thing which is a bit unclear is how to tell which apps can show a widget with extra information on the home screen.

I've only downloaded free apps so far, so I don't know if you can buy apps without a credit card in Finland yet. Not that it matters that much to me. I have to say I'm a bit overwhelmed with all the crap apps in the app market though. I would much rather have an app market which is free from Coder Joe's MyFirstAndroidApp which is either a crappy clone of some other game (like Angry Birds, Bejeweled or Tetris) or some lame memory game with naughty pictures. Google, please give us a way to filter these apps out!

Well I guess that's about it. This was just supposed to be a brain dump about my initial opinion of my new awesome phone.

PS. The image entering functionality on blogspot (blogger?) is awful. I tried putting the cursor where I wanted the image to go. Press "Add Image" and both of my uploaded images go to the top of the article. No way to drag them in the Compose view. So you have to edit the hellish html to get the image where you want...

O'Reilly Disaster Relief 50% Special Off E-books

Today O'Reilly had a great 50% sale on all e-books with all revenues going to the Japanese Red Cross for disaster relief after the earth quake + tsunami from about 10 days ago.

I bought the following books for a rainy/sunny day:

High Performance MySQL, Second Edition

I personally prefer Postgres. I think MySQL feels a bit flimsy for some reason I can't articulate. However MySQL is still very widely used and I felt that whatever lessons I learn from this book probably applies to other SQL databases.

Canvas Pocket Reference

I coded a simple clickomania code with a friend at work which uses canvas for drawing so I was interested in learning more about using canvas.

Programming Android

I don't even own a smart phone :( But I'm still interested in learning about programming for Android. I'll probably end up owning one sooner or later. Just not in a hurry to get one.

TCP/IP Network Administration, Third Edition

This is probably a lot of stuff that I already learned in school many years ago. But I feel like I still have some gaps in my understanding. Gaps that don't usually affect my every day work, but gaps that I still want to fill, well just because.

Anyway, this was a great move by O'Reilly. They deserve a lot of credit for putting this special together. The total paid amount is at $70k at the moment. Let's hope it goes even higher and that Japan finally gets the power plant situation under control.

"The web" has been replaced

I'm working on my own Rails application during evenings at the moment just wanted to point out something I already had noticed a few months ago: most of the old technologies that made up "the web" a while ago have now been isolated through a new layer of "languages" (I hesitate to call html and css languages).

JavaScript can now be written using coffeescript. CSS can be written using sass/scss/less. HTML can be written using haml. Now there might have already been replacement technologies for all these areas a long time (don't know too much about alternative browser languages), but now it's really feeling like these technologies are good enough to be used in real projects.

My Rails application is going to attempt to use all of them. While I think haml is nice and clean and it's great to not have to type all those annoying angular braces, I'm even more excited about coffeescript and scss. As most people who take the time to learn JavaScript notice, it actually has some really good parts but it's bogged down by some pretty last-millennium syntax and some really awkward design and implementation mistakes. On the other hand, I think coffeescript has the nicest syntax of all languages at the moment.

I haven't really used the CSS replacements that much yet. Layout and design aren't my strongest areas. However in nearly every web application I've worked on so far, the CSS has been very messy. I'm hoping that scss might help alleviate this problem.

Why PHP sucks #1

Welcome to my series of posts of why PHP is the worst language I've ever had to work with. Today's topic is error handling. Or perhaps PHP's lack of it. One of PHP's philosophies seems to be that the programmer should not be troubled by any nasty errors. It's better to attempt to do something even if that is wrong. Failing fast with a clean error message and a stack trace is not one of PHP's virtues.

PHP seems to (I'm just guessing here) have only had errors (with different codes for warnings, notices etc.) before. Now it also has exceptions. This is pretty annoying because you end up having to write your error logging code. This involves setting up your own custom error handler which rethrows the errors as exceptions. Then you can add your own custom exception handler which can obtain a stacktrace from the exception and print it out inside <pre> tags so that you can make some sense of it.

Then you figure out that PHP treats "fatal errors" differently. It bypasses your error_handler and just shuts down your application instead. Then you figure out that you can register a shutdown function and try to output a stacktrace manually from there. Unfortunately by the time this function is called PHP has already torn down most of the stuff you were using and the only thing in your stacktrace is gonna be your own shutdown function.

So you end up installing the xdebug extension and by enabling it you can finally get some output when your program. Except it's a big mess of lines where you end up just having to go through all the mentioned lines to see if you can figure out what's causing your problems. Then after a while you notice that you can echo a single opening <pre> tag at the beginning of your script while debugging your errors and suddenly all the output from xdebug is quite readable. But the fact that you have to go through all of this crap shows why PHP sucks.

Programming books I read in 2010

I read quite a few programming-related books in 2010. Here are the ones I can remember:

Tehokas Java EE-sovellustuotanto

Perhaps the most important book I read all year. Helped me in getting my current job. Taught me the basics of Maven, Hibernate, Wicket, Spring and so on. All of which were needed in the first project I was working in.

Javascript: the Good Parts

Very nice book. I read this for the Javascript study group we've started at my workplace. Recommended to read more than once if you really want to learn Javascript.

Effective Java

Very good Java book.

Refactoring to Patterns

I just skimmed through the descriptions of the patterns and skipped most of the longwinded refactoring instructions. If I remember correctly the smaller refactorings of Martin Fowler's were more immediately useful.

Patterns of Enterprise Application Architecture

While the content of this book was different from what I expected based on the "enterprise application" of the title, it was still a very good book. It taught me more about how Hibernate works. And I've referred to this book a few times when refactoring a project at work.

The Pragmatic Programmer

I had already read this one before. I just read through it again because it was such a long time since I last read it and because it's such a short and easy read.

Facts and Fallacies of Software Engineering

Sort of interesting. The abstraction level was quite high. I was aware of many of the ideas in the book already.

Pro Git

Online book about git. In 2010 I greatly improved my understanding of Git. This book was one of the best sources for learning about it.

CSS Mastery: Advanced Web Standards Solutions

HTML and CSS is one of the biggest pain points of making web applications. One of the things I remember the best from this book was how often the chapters had to have some extra workaround crap instructions for IE and what a pain in the ass it is to make rounded corners.

Rails Recipes

This one was written for Rails 1, so it was a bit out of date. At the moment, I can't really think of any useful recipes I learned from this.

Git version control best practices

Keep commits small, they cost nearly nothing to make in git

• Allows you to easily revert a faulty commit without reverting working code at the same time


• You can easily get an idea of what happened to the project by simply reading the commit message titles in a history browser


• No more of those "committed some stuff" commits

Code new features in separate branches, they cost nearly nothing to make in git

• Allows you to easily return to the latest working state in the master branch when your client wants you to fix a bug discovered on your project's deployment day


• You can just leave your branch to "collect dust on the shelf", if it turns out that the feature was too time-costly to implement, and continue working on it at any point later

• Even the "minimal" tasks usually require more commits than you originally thought

Use no-fastforward when merging

• Because the merged branch groups commits its easier to see which commits belong to the same feature when observing the source history


• All the commits that make up the feature can be reverted in one step if the branch was found to be faulty

Use add -p to split up modifications into separate commits

• Related to keeping commits small

Corollary to "Code new features in separate branches"

• Don't develop directly in the master branch

Use a remote branch if you want to collaborate with other developers on a feature

• If a feature is too big for you to tackle alone, you should push it to your "main repository". The other developers can then pull it and contribute to it and when the large feature is done it can be merged into the master branch

Getting used to Java's Date and Calendar classes

Today I managed to complete the createBooking method of my BookingController. At least it runs without any errors. But the page that shows the week's bookings does not reflect the added booking. I spent quite a lot of time debugging my Calendar-related code. For a while I forgot that its MONTH field's index is zero-based, which caused me a lot of confusion. I was again tempted to start using Joda Time but I would like to refrain from adding too many dependencies. I would also have needed to add joda-time-hibernate and find/create property editors for setting up the Joda Time objects. Sure, this might have been something that would have taken like 15 minutes and worked perfectly, but it could also have led to unwanted trouble.

Anyway, when I was trying to see how far my BookingController was to working, I was doing some really painful manual testing. First I restarted Jetty, then I refreshed my main page, then I had to create a new user, then I went back to the main page, then I clicked the booking time I wanted, then I clicked submit on the form and then finally I would see an exception or error. Not very agile at all. The easiest way to start looking at what's wrong would be to run a separate hsqldb process which I could connect a DB client to and see if my requests have made any changes to the DB. But a better way would be to create a test case that calls my controller instead and then verifies that the expected changes have happened to the database. It would be even better to extract some stuff from my controller so I can test the url routing and form parsing separate from the database-related activities.

Sorting out how to use Spring Security again

It had been a while since I read about Spring Security. Now that I was trying to create a new booking I had to retrieve the currently logged in user to be able to add the new booking to that user's bookings. So I thought I'd just slap a <global-method-security secured-annotation="enabled" /> in my application-security.xml and then add a @Secured("ROLE_USER") on my method that was mapped to ("/") and Spring should pop up a login form for me when accessing that URL. But unfortunately it didn't work that easily. The page showed up just like before, with no login form in sight. So I went back to reading the Spring Security manual and looking for some posts on Spring's forums and after a while I noticed that other people had solved similar problems by putting the global-method-security element in the same xml file as the one where they set up their Controller beans. Since I use the component-scan element to automatically instantiate my @Controller-annotated classes in my servletname-servlet.xml file, I thought I'd just move the global-method-security element to that file. At first I got a long spew of exceptions and call traces, but that was just related to some incorrectly configured xml namespaces. When I sorted that out Spring Security finally worked.

The exercise also straightened out one misconception I had about spring. I thought the DispatcherServlet's servletname-servlet.xml file created a new application context that was separate from the WebApplicationContext which Spring's context loader sets up. But then I found this in the Spring manual "In the web MVC framework, each DispatcherServlet has its own WebApplicationContext, which inherits all the beans already defined in the root WebApplicationContext." However I still don't understand why the DispatcherServlet context didn't inherit the global-method-security element from my WebApplicationContext which contained my security beans.

Registering custom property editors with Spring MVC

Recently I've been able to concentrate much more on more normal coding tasks rather than just figuring out how to configure my application. I got to the point where I wanted to receive data entered and use that to automatically fill in property values for a business object. Fortunately Spring does have support for this. You just add a parameter of the type of your desired object and Spring will setup the properties based on their names from the encoded form values.

Spring has built-in support for conversion from Strings to a large variety of types. But if your business objects has a property of an unsupported type you need to register your own custom property editor that converts a String to that unsupported type. It took me a few tries to get it right but I seem to have gotten my custom editor set up properly now.

The mistakes I made were the following:

First I was trying to set up a CustomEditorConfigurer bean in my application context and use its customEditors property to map my class to an editor. Then I noticed that the customEditors property was deprecated in Spring 3. So I changed my setup to use a PropertyEditorRegistrar instead to register my custom property editor. This still didn't work. Then I figured out that if you just add these as beans in the application context they are probably only available for converting Strings inside that application context. If you want your property editors to be available for binding web requests to objects you need to use Spring's Controller's initBinder method to register your editor with a binder. And if you are using annotation-based controller you additionally need to use the @InitBinder annotation on that method.

First screen shot

Finally started working more on the actual construction of views and models instead of just figuring out all the infrastructure. I have now added a table to the laundry booking system. The data for the model (times and dates) are automatically generated to show the current week. Tomorrow I'll work more on fixing the links to forms for creating actual bookings. The first version is going to be quite Web 1.0 so there will be page reloads for most actions. So I can spend all my time on working with Java EE instead of Javascript. However when I get far enough I plan to add some Javascript and try out jQuery.

Finally got my whole Java EE stack completed

Ok just a quick update. Adding spring declarative transaction did the trick. I was now able to add a new user account through the a web form -> Spring Controller -> EntityManager and it shows up in my user list for testing purposes. I'm still gonna finish reading the Spring Transaction chapter and then I think I can finally start working on my domain model and then move on to the Controllers and views.

By the way. Annotation-based Spring MVC seems pretty good. There is a great deal of flexibility both in terms of what types of input your controller methods can take and what output they can return. Spring basically reads your thoughts and figures out what you wanted to do. Well not really, but it's very flexible. It almost feels a bit un-Java-like. Spring MVC with annotations basically fulfills the requirements that made me search for a rest-framework without the rest parts that I don't really care too much about when making a human-facing web application (such as content negotiation regarding different input and output formats, like json and xml.)

Settled on Spring MVC

Ok it wasn't as bad as I briefly anticipated to set up a url redirection scheme for getting nice URLs with Spring MVC. I found this nice answer on stackoverflow that actually linked to a sample MVC project in spring's repository that shows a really easy to use example of how to use the Tuckey url rewriting filter.

Firsty you simply map all your Spring MVC classes under a path like /controllers. Then you set up explicit redirects to all your subdirectories that contain static files like /javascript, /css and so on and anything that doesn't match that gets mapped to a url where the path starts with /controllers. After doing this I almost (but not quite yet) have figured out all the parts to be used in a simple web app. I was able to get my Spring-managed EntityManager injected and managed to get it to run some queries. But I still have to read the Spring manual chapter on transactions because I still don't feel like I know that well how they work yet. I could just follow a tutorial and just use the <tx:annotation-driven> element and annotate my classes and methods with @Transactional and everything would probably work. But like with all other technologies I've been learning for Java EE recently, I don't like using stuff without understanding how they work.

Frustration with java web frameworks

I've been delayed in doing any actual work on my first java EE web app because of web framework problems. I gave up on trying to get a definite answer on how @PersistenceContext works and thought I'd just try to see if I could get Jersey Resource backed by an EntityManager to work and output some hello world stuff. After correcting a few configuration mistakes I got some weird exceptions. I found some mailing list discussion that indicated that I was mixing different versions of spring modules. I was a bit surprised because I remember picking the 3.0.X version on all my modules. However it turned out that my jersey-spring integration library depended on a module called spring which was at version 2.5.6. Unfortunately Jersey doesn't have an IRC channel so I wasn't really able to verify if this was causing the problem.

So I did what I was trying to avoid doing. I tried switching to Spring MVC. I had understood it had support for "restful" URLs. After a while though I noticed that you are not supposed to use /* as the url-pattern for Spring's DispatcherServlet because the views are resolved through the DispatcherServlet. This means that if you try to redirect to /WEB-INF/jsp/myresource_show.jsp for example, DispatcherServlet will try to match that URL to a Controller. Since you probably haven't defined this as the mapping for a Controller you will get a No mapping found error. This also means that the DispatcherServlet will attempt to map all static files (images, css, javascript) to Controllers. This is probably because Spring isn't that different from Struts as I had hoped for, and it wants you to suffix actions with something silly like ".do". But I don't want to do that. Another solution is putting your resources in a special resources subdir and putting that as your url-pattern for DispatcherServlet. Alternatively you should be able to do create some sort of elaborate url-rewriting scheme. Uuugh. Why couldn't just jersey-spring play nicely?

Integrating a Spring application context with a web application

I think I finally figured it out last night. Most of the tutorials and documentation on using Spring's DI container seemed to only show to programmatically create an ApplicationContext by instantiating a subclass of it in a main method. So I was trying to figure out where you were supposed to do this in a web application. I had forgotten that you could use a ContextLoaderListener that would create an ApplicationContext (known as the root app context) from the xml files listed in an init-param in the web app context. But even when I rediscovered that, I had trouble figuring out if my Jersey Controllers were supposed to manually get beans from this root app context somehow.

So I spent a while browsing through various tutorials and the Spring documentation, and asked a lot of questions in the #spring IRC channel. At one point it eventually clicked and I figured out that most web frameworks need to provide some way of integrating with Spring, as opposed to this being completely standardized from Spring's side. The thing that made me figure this out was that I looked at how Spring MVC integrates with Spring beans and how Jersey integrates with Spring beans.

- Spring MVC has a DispatcherServlet that retrieves beans which extend Controller from an application context on the classpath that is named servletname-servlet.xml
- Jersey retrieves classes annotated with @Component from the root web app context
- Struts uses a Spring plugin that defines the application context file where the required Struts Actions are wired

And when I realized this, I noticed why I had been going about it the wrong way earlier when I was trying to figure out how to get the root web app context to manually get my beans from there. Although I also learned about Spring's WebApplicationContextUtils which would allow me to do that if needed.

Much progress with hibernate and JNDI with Jetty

Today I've been reading a lot about Hibernate and setting up JNDI objects in Jetty. I was trying to find out more about how to solve the problem of loading the same sample data set into an hsqld database every time I start Jetty.

At first I had planned to create a servletContextListener that would use DbUnit to import the data when the context is initialized. But since I am using Hibernate's hbm2ddl.auto option for automatically creating the database tables, I wanted to make sure the tables had been created by the time the servletContextListener is supposed to import the data into them.

The first step was to figure out how worked in servlet containers and specifically how to configure Jetty to create them. I set up an hsqld.DataSource and stored it in the InitialContext so it can be retrieved using JNDI. I also noticed you can get objects from JNDI Contexts in Spring's DI container.

Then I moved on to finding out what causes Hibernate to create the tables. First Persistence.createEntityManager(nameOfPersistenceContext) is called, which in turn causes a Hibernate SessionFactory to be created. This automatically exports schema DDL to the database when hbm2ddl.auto is true. Well I took a long break here and noticed that I have to continue with this tomorrow. I have to see if you are supposed to create some webapp-global objects in the servlet container itself or if Spring has some way to create beans that would be global for the servlet container.

Filling a database with some test data

I think I've got a decent understanding of Spring security now and would like to test that everything works. I need to figure out a way to put up a new database when running Jetty and put some rows into a user table. I am using hsqldb for the few small persistence tests I've written so far. I thought I would also use hsqldb for manually testing my application when it runs in Jetty. The Java EE-book I've read mentioned that you can use DbUnit to import data into a database from an XML file. But I think the main problem at the moment is that I have an inadequate understanding of hsqldb, so I'll have to start by learning more about that.