Terabyte Labs

On October 22nd, 2009, Nationwide, perhaps the only bank in the UK to support it, disabled OFX reports and with it support for hundreds of different financial tools including Microsoft Money, Wesabe, Moneydance and even my own OFX client. According to their support page, this was due to the burden put on their customer services in fielding questions about Microsoft Money and the lack of in-kind support from Microsoft who stopped selling Money earlier this year.

What Nationwide don’t seem to realise is that it costs them nothing to quietly leave OFX support enabled for use by the hundreds of other programs out there that are not Microsoft Money. If customer service calls were costing too much then they should end the customer support rather than the technical support. Why punish all of us who were happily using the Nationwide OFX service and putting no burden on their support staff?

If someone can recommend a decent UK bank that supports the OFX protocol to access transaction data, I would seriously consider moving my money away from Nationwide. They may believe they are saving money by disabling OFX support but from the comments I’ve been receiving, I’d wager they’ll lose even more if all of us Wesabe, Moneydance and Money users were to go elsewhere.

Random | 2 Comments »

Money Management

Knowing how much we spend on what is a problem everybody faces and people have come up with many different solutions based on their needs. Personally, I’ve found I’m too lazy to maintain an Excel spreadsheet and I don’t trust myself with having no system at all.

I first started looking at money management software when I first heard of mint.com. It looks like a great service but unfortunately doesn’t support UK banks yet. Next I looked into desktop software that might offer a similar set of features. There are plenty out there such as MS Money, Moneydance, Quicken, GNU Cash etc. Unfortunately all of these lacked some essential feature (such as auto-categorisation), were too difficult to use or too expensive.

So I started looking into writing my own application with the features I wanted. This would have a line chart showing my overall balance over time, a pie chart showing the breakdown of my expenses into categorisation. And of course transactions would be loaded and automatically cateogrised with the minimum level of fuss. On this point I wrote an application that automatically logs into my online bank account and downloads my statement in csv format. While I eventually got this working (not a trivial task) I realised it was too clunky and fragile as it’s likely that my bank will change their online interface at some point in the future.

Next I looked into downloading transactions from my bank via the OFX protocol. This is the protocol that lots of banks use to allow software the ability to download transaction information and also potentially execute transfers. It was not easy to find the information I needed to connect to my bank (Nationwide) via OFX as the bank’s details are not officially published. Microsoft use these details when connecting to Nationwide with MS Money. I had to do quite a bit of digging until I found a link on a newsgroup to a file hosted by Microsoft containing the necessary details for connecting to Nationwide’s OFX server. Armed with this I posted the details to a thread on the Moneydance forums. That was 7 months ago and Moneydance have still done nothing about using my information to add support for Nationwide despite a lot of requests from other forum users.

I moved on from the downloading transactions part of my application to the application itself. I figured it would still be useful even if I had to occasionally download transactions manually. Here’s what I came up with:

I even came up with a decent categoriser dialog:

With the application itself shaping up I went back to the transaction download problem. This time I found a nice snippet of Python code for connecting to a server via OFX. I translated this to a Java application that could download a list of my accounts from Nationwide, then download a list of transactions within a certain date period from a given account. This program is available for download here.

Now that I had the ability to download OFX statements from my bank, I looked into the other management services I initially dismissed because I could only import data via OFX. Wesabe was one of these so I tried creating a new account but lo and behold, they now suddenly support Nationwide for automatic uploads. This means they do the whole OFX download themselves automatically. This was great as now I don’t need any manual process at all to see my transactions nicely split into categories. The funny thing was, support was added the same day that I finished making the OFX downloader and this was thanks to my initial investigative work into the Nationwide OFX server details. Someone had picked up on my post in the Moneydance thread, given the details to Wesabe, who within just 9 days integrated support for Nationwide into their application. So great work guys, you well exceeded my expectations and I’m glad I could help other fellow Nationwide users.

Now that little saga is over, onto the real issue of the software itself. The great thing about Wesabe is that I can view and manage my transactions from anywhere. This means I can log on while at work and enter what I spent in cash for lunch. The bad thing about Wesabe is that it doesn’t give a breakdown your expenses as a pie chart and it doesn’t present your overall balance as a line chart.

To rectify this I’ve chosen to abandon my Money Monitor program and try to go in a slightly different direction. I’ve always wanted to learn .NET programming so my next project will be to create a ASP.NET application that connects to Wesabe via the Wesabe API. Downloads transaction data and displays it all in nice pretty graphs. Stay tuned for updates!

Code, Life | 1 Comment »

Looking back at my post after the first iPhone was announced, it’s quite suprising how many of the predictions about the second iPhone were actually realised. Of course when I wrote it, it wasn’t just a matter of guess what would be improved but also a wish list of things I’d have liked to see before actually buying one.

I said the iPhone 2.0 would:

be released 1.5 years after the first – Well that seemed about right.

have 3G – Ok I predicted this would be in the European version. Obviously Apple had more trouble than most phone makers in getting us this feature.

be smaller – Ok so the 3G is actually slightly thicker but without the metal back manages to be 2g lighter. It would still be nice if it could fit in your pocket though.

have all the same features – A slight underestimation here. This was based on my reasoning that Apple would be quickly bringing out a version that fixes all the problems with the old one rather than spending money on new features. Of course the price was the big thing that Apple admitted put a lot of people off – especially outside the US. I also reasoned that they’d want to leave some features for the next generation of their phone rather than do they very best they possibly could right away.

have longer battery life – One thing Apple came through on. Somehow they managed to squeeze more life out of it despite the addition of 3G which is a known battery killer. I suspect that extra width went mostly into a bigger battery.

have more memory – Well we didn’t have to wait for the 3G version for this. They doubled the initial max capacity from 8GB to 16GB back in February this year.

a slightly better camera (maybe with an led flash) – Clearly I underestimated the importance to most people of a decent camera on their phone. It would be nice to have but clearly people are not planning to use it replace their digital point and shoots.

have more custom iPhone internet applications – This was a big focus during the annoucement of the 3G. The SDK is looking very interesting and I’m looking forward to seeing what people will come up with with the new hardware.

have reasonable prices for unlimited mobile internet access – I still haven’t seen as much response from phone networks about this point as I’d expected. The unlimited data plans are still pretty dear (about £7.50 / month if I’m not mistaken). Still we’re moving in the right direction.

get an overall price reduction – Apple claims to have made big improvements here by dropping the cost of the initial phone but are actually increasing the overall cost by increasing the data plan costs.

And the biggest improvement that I didn’t expect at all (and I don’t think anyone else did either) is of course A-GPS. I told myself I’d get the phone as soon as they added GPS and now that they have, well, I’m still not going to get one. The monthly plan prices are still way too much (3 times what I’m currently paying). The good thing about these announcements is that a lot more people I know will be getting one now which will give me a chance to play around with it and see what I think. Roll on July 11th!

Tech | No Comments »

There are plenty of websites and blogs discussing which techniques and methodologies we should use as software developers. When should and shouldn’t we use the Singleton pattern? Should try to future-proof code against possible changes in requirements or should we make sure it only fulfils our immediate needs? Should we write self-documenting code or short and quick to type with explanations left in comments? Which web framework is best, Tapestry or Wicket?

Asking these kinds of questions is a good way to learn about new technologies or techniques you have have not been familiar with and can give you a lot of insight into other parts of the software development world. You can apply some of these pieces of knowledge to broader parts of your job. For example, learning that perhaps using a Singleton pattern was not such a great idea you may decide to review the other parts of your application where you chose a common design pattern over a possible simpler design.

But there is a downside to learning about these new technologies and techniques and that is realising that even if you have a better solution, you may well not have the power to implement it. If you have a well developed web application based on JSF, it’s probably too late to consider using something like Ruby on Rails when you realise it’s better suited to that technology. An established database application based on SQLJ technology is going to be practically impossible to migrate to a much better technology such as Hibernate.

But this is a well known problem in software. You make your choice of technologies at the start of the project. If you need to switch half way through that’s too bad because you’ll just have to pay some high paid consultants to walk you through the process.

Another well known problem is that of technical debt. You write some quick hack to implement a feature or fix a bug as quickly as you can knowing that it could be done better if you just had more time. When you have that time, you tell yourself, you will come back to that problem and implement it correctly. Only that time doesn’t come. And even if it does, well your hacked together solution has been working fine on production for a few weeks now and it hasn’t caused any problems so why bother changing it and risk breaking the same part of the system again?

There are many proposed solutions to prevent the build up of technical debt such as unit testing, code reviews, proper architecture and design meetings, scheduled code maintenance time etc. The problem is there are not many solutions for how to pay off technical debt once you’ve incurred it. In the financial analogy it’s not so hard because you know how often and how much you need to pay back. With technical debt it’s very difficult to measure and it’s even more difficult to pay back.

We come across bad code all the time. “Why didn’t they create a list of objects instead of using a big array” we ask. We could change this code ourselves but it’s too late. The array is used throughout almost all parts of the application. Any change to it now would introduce a big risk or breaking something, or even worse subtly change the behaviour such that it goes unnoticed until it goes live. There are many factors that can make paying back technical debt very difficult. Maybe if we examined them in detail we might find ways to start tackling the task. The worst thing however, is coming up with a great solution to a problem in an application and simply not being able to implement it due to the risk of breaking the entire system.

Code | No Comments »

Today we heard about an interesting development in P2P file sharing technology from Harvard University researchers and a bunch of geeks in Amsterdam from a company called Tribler. Potentially these guys can come up with what I’d consider the next generation of file sharing technology. Previously we’ve had programs such as eMule and Kazaa which were able to download from multiple sources at once. Then bit torrent came along and showed that it was just as easy to download your file from others even if they haven’t received the complete file. Theoretically, bit-torrent uses close to the most efficient method of transfer possible for getting a file from one box to x number of other boxes.

But bit-torrent is not ideal. The problem lies in the fact that 90% of us have asymmetric connections as for some reason most ISPs think they can get away with giving us 1:16 upload/download ratios. If this is the average ratio within a swarm it means that for a client to receive a file at their maximum possible download speed they need at least 16 people to download 16 different parts of the file from. Or, in other words, for each file you download, you need to spend 16 times the time it took to download uploading it back into the swarm.

The problem with bit-torrent is that it doesn’t reach anywhere near its maximum potential because people are not willing to spend that much time with their PCs running if they can get away with just leeching a file. Private torrent trackers are a good way of solving this problem because everyone has a ratio that they have to maintain. The tracker enforces that you upload at least 70% of what you download. This means you can get some awesome speeds with some popular files. The problem is that the system is centralised and there are only so many users and torrents that it can keep track of and so rare or unpopular files do not get the same benefits.

Tribler appear to be making a lot of effort to defeat the problems of public and private torrents. They are trying to decentralise the user score system. A user will be given points by other users depending on how much they upload. The problem they need to solve however is how many points people give for very popular files and for very rare ones. More points for popular files will mean more and more of the very few people with gigabit connections will be able to max out their download speed. More points for rare files will mean faster transfers for rare files but slower transfers for those with high-speed connections.

The goal Tribler have at the moment is to develop a fast and efficient file transfer process but geared specifically towards streaming video. The reason for this of course is that they can then sell the technology for millions to TV companies or video distribution services such as YouTube. But TV and video on demand is quite a specific use case. Files must be transfered head first rather than in random chunks; most files are unlikely to remain popular after a few weeks and most of the time people will be wanting the same relatively small set of video files. Tribler will have to tailor their system to be most efficient for this particular kind of usage.

Ideally we’d have a completely universal file transfer system which would work equally well with streaming video as with distributing rare random files. Perhaps it might even be possible for each user to tailor how they want their bandwidth to be used based on what services they desire. That way people with similar interests would automatically form their own mini swarm of files. In any case I can see a very interesting future in the field of file sharing. We’ve only just begun to scratch the surface of what can be done with our increasingly speedy broadband connections.

Tech | No Comments »

I’ve recently finished reading a book called The Undercover Economist by Tim Harford. The book attempts to uncover the hidden truths of every day economic systems (though in fact it is more of a global economics 101) and is a very interesting read. It seems that a great many of the world’s situations can be explained by very basic economic principles such as scarcity or the principle of asymmetric information. For example Harford explains why it is impossible to find a good second hand car because the customer has no idea whether the car dealer is being honest about his claims of the quality of the cars he is selling. The fact that both parties don’t have all the information detriments both of them because the customer can’t tell which cars are good or bad and the dealer can’t convince his customer which cars are genuinely good value. In economic theory, markets work most efficiently when everyone knows everything there is to know about the entire market.

This got me thinking to an area where information is very thin on the ground: salaries. Why don’t people tell each other how much they earn? Why is it such a taboo subject? According to the theory if everyone knew how much their colleagues earned they’d be in a much better position to know what kind of salary they should be getting. Employers wouldn’t be able to offer unreasonably low salaries to people just because they don’t know how much they should be getting. Salaries should work on a system of supply and demand like every other market. If C# developers were paid twice as much as Java developers then people would train to fit the job that best suited their abilities and desired salary. At the same time the competition for jobs would drive down over-inflated salaries. Eventually the outcome – as with all free markets – would be that everyone was paid just enough to to be happy but not too much that someone else would offer to do the same job at a lower price.

Without knowing how much your colleagues are earning it is impossible to know how valuable the work you are doing is to the company. If a company wants people to work at their best they could offer benefits such as a performance related bonus. A lot of companies seem to have implemented bonus schemes without realising that unless employees know how much bonus they are getting or how much of the bonus is based on performance the bonus has no real effect at all. Also unless we know how large a bonus our colleagues are getting we have no method of relating performance to renumeration. The same reasoning explains why communism doesn’t work. If extra effort can only be seen as a benefit to others, no one will have a reason to try to perform any better than anyone else.

Perhaps a completely transparent salary might then encourage too much competition. People would threaten leave if they felt they were treated unfairly. In a work place it’s very difficult for anyone to have an objective opinion of their own or other people’s value as an employee. Perhaps a completely free market of workers would reduce the overall quality of work due to poor workers demanding the same salary as their coworkers or good workers continually switching from one company to another. Either way, all I know is that I’m not getting enough…

Life, Work | No Comments »

As I promised in my ‘To-Do’ list a while ago, I’ve added some of my old projects that I’ve been meaning to include on this site for a while now. It’s unlikely that many people will find them useful (except perhaps my Download Meter Utility) but I like to keep them here for prosperity more than anything else. I spent a lot of time on some of those projects so it would be a shame if they were to disappear without having before seen the light of the web.

I’m also slowly beginning to fulfill my goal of developing a proper C++ application. At the moment I’ve successfully set up the MS Visual C++ 2005 IDE and compiler as well as the wxWidgets GUI library. Hopefully I’ll even be able to make cross platform applications now that I have the tools set up. The first task I will attempt will be the Project Euler compendium. I already have the design set out in my head and could probably code it up in Java within a day. In C++ though I’d expect it to take a few weeks for me to get the basics hammered out.

Code | No Comments »

I’m in the process of trying to finish of Shyguy and I always feel it’s good to note down what remaining tasks you have at this stage of development. I also find it’s good to publish them to instill some degree of commitment to actually attempt the tasks.

• Add bonus timer
• Add loading screen
• Add special effects for bonuses
• Add instructions
• Add desktop shortcuts option
• Alter highscore table to use a database

That’s all I can think of right now. I will probably have a couple more things to add as I progress. Right, now back to work…

Code | No Comments »

I thought I’d make a list of some projects and tasks that I’d like to complete sometime in the future. It seems that right now I don’t have much more free time than I need to shop, cook, eat and sleep but if I write something down here hopefully it will give me a better idea of how to organise my time.

Website

• Create a new front page of the site and move the blog to the sub page. done
• Add more projects and content to the site. done
• Integrate my photo gallery to the site and upload all my digital photos. in progress
• Add a personal section including a CV.

Projects

• Finish Shyguy’s Cave of Death. done
• Finish the Word Dissociation Game.
• Start on a new Slick based project (maybe a multi-player rpg of some kind).
• Learn C++ by creating a Project Euler solution compendium. in progress

Other

• Sort out my finances.
• Save enough money to buy a PC.
• Tidy my room.

I’m sure there are more but I think that’s more than enough to get me started.

Code, Life, Random | No Comments »

Recently we’ve been hearing about new classes of stimulant drugs such as Modafinil which supposedly promote wakefulness rather than hyperactivity or euphoria. These kinds of drugs appear to be very effective at reducing sleepiness and improving concentration and memory without affecting sleep or having any side effects from withdrawal.

Modafinil and other drugs in its class are sold to help sufferers of Narcolepsy or ADHD but are widely used by people with no such conditions as a way of simply enhancing their general brain functions. Millions of us could benefit from holding better concentration while at work or at school. Or remembering things such as timetables or doing mental arithmetic.

There are few other non-medicinal self enhancement drugs that offer so much potential benefit to such a wide range of people. Caffeine seems to be the nation’s current brain fuel of choice but most people who take it regularly end up simply using it to abate the withdrawal symptoms of the caffeine taken the day before. The question therefore becomes, should people take drugs simply to better themselves? Should people be allowed to use them while taking or revising for exams? Should people be allowed to use them to gain advantages over colleagues at work? Why shouldn’t people be able to function at a higher level than they would naturally? After realising how much better they can be, why would anyone want to live a drug free life?

Personally I find it very difficult to concentrate throughout the day unless I’ve had at least 8 hours’ sleep. My job requires a high level of concentration and logical processing for which I need the highest level of alertness I can get. I would love to try a Modafinil some day just to see how much more code I can churn out in a day. What I’m not so sure about is do I want to need to take the drug every day in order to maintain the same productivity at work? Even if there’s no danger of physical or psychological dependence would I develop a functional dependence such that the drug effectively becomes a medicine rather than a self enhancement?

My theory is that if Modafinil came in a hot drink form, it would so conveniently replace coffee and tea that people would reject any attempts to limit its use by the government. Also it would probably need to be bitter in taste enough to discourage children from becoming dependent on it. Hmm I guess this is how consumption of caffeine and alcohol has worked for centuries. Isn’t it convenient to have traditional beverages to mask our widespread drug taking customs?

Life, Random, Work | 2 Comments »

Since its beginning, Google Maps has always been the best web mapping program out there. Everyone was amazed when Google introduced draggable maps that actually filled your screen rather than some tiny portion of your browser that was enclosed by ads. Then shortly after they wowed us again by introducing Satellite images. Then a great thing occurred on the 19th March 2005 when Great Britain was added to the Google World. Of course since then Google have done a fantastic job of improving the parts of the world covered, the road data in more countries, the improved driving directions, better local business listings, traffic info etc. But one thing that I’ve always thought was missing ever since I started to use Google Maps was being able to save customised data to your maps. The most obvious example is storing a collection of saved locations. Other programs have been able to do this long before Google Maps introduced it a few months ago. Finally, more recently we’ve seen the addition of stored routes along with detailed HTML descriptions and also the ability to share your custom mapping information publicly. This makes a welcome improvement as a lot more information can now be given by allowing people to write about their local area and share the information via a map. As an example, I’ve created a map showing the route I take to cycle to work.

But I believe there are still many improvements I’d like to see from Google maps. One thing is the ability to see the length or a route. At least that way I could actually see how fast I cycle when I go to work. It would also be nice to be able to list all the businesses on a particular street and see where they are. That way you could go browse the high street shops without having to walk along it.

The profusion of 3rd party programs based on the Google Maps API demonstrates the capabilities that the Google Maps technology has. I hope we haven’t seen the last of the new developments to this great program.

Tech | No Comments »

I’m normally pretty good with taking care of my stuff. In fact I don’t think I lost anything more valuable than a pen until a few months ago. I guess it started with potentially my most valuable possession of all – my wallet. I was returning on the train slightly dazed and weary and must have not made the effort to place my wallet back into my pocket after taking it out to put my train ticket in it. I got off the train without my wallet and didn’t realise until that evening. I canceled my debit card and requested a replacement the following morning. Fortunately no one had tried to charge anything on the card before then. I’m incredibly bad at getting things done (the number of things I’m putting off by writing this is testament to that) so I didn’t bother to replace my driver’s license, young person’s railcard or the wallet itself. The only thing I replaced was the oyster card which is pretty essential if you’re traveling around London all the time. About two weeks later I got a letter saying that my wallet was at Berko station. I picked it up and the only thing missing from it was the oyster card.

Next I was on a bus and left a carrier bag containing the lease for my house and my empty lunch container. Again nothing valuable lost.

Then I managed to leave my bag at the pub (my nice back rucksack this time) containing my trainers and a hat. Rang the pub later that night and a very unhelpful woman wasn’t even able to tell me whether they had it or not. A few days later at work the people I was with told they had it so again fortunately nothing valuable lost.

Next I managed to lose my rear bike light after using it exactly one time. Guess that fell out my coat pocket.

Next exactly one week after losing my bag at the pub I lost my bag at the pub again. This time I was well aware that I might lose but still left it unattended by the stairs at the pub and it had disappeared by the time I wanted to leave. This time the bag contained a hat, a book (a damn good one at that) and my front bike light which I’d used exactly zero times.

I really don’t know what’s changed in the past few months. I’m obviously not the kind of person to trust with keeping things safe but somehow this flaw of mine hasn’t been a problem at all in the past. I guess up till now I’ve just been incredibly lucky to find things where I expect them to be.

I’ve now got so many things to replace and it is going to be a pain to get them back. This isn’t just from losing things but replacing broken things or new things required for my house. On top of that I’ve been spending way too much money recently. I have to keep within the budget of my salary which isn’t easy when half of it is already taken up by rent. Anyway that’s for another post. Before then I still need to break down my outgoings to find out exactly what I’m spending on what.

PS: This rant may seem rather trivial given that I haven’t really lost anything of great value. Yesterday I learnt that the British teacher who was found dead in a bath tub in a Japanese apartment was actually the girlfriend of one of the editors of Bit-Tech (a site I visit often) and for some reason decreasing the degree of separation between yourself and someone else increases the empathy you feel for that person. I don’t expect anyone to feel any empathy for my losing my bag but still, at least it gives me something to blog about.

Life, Random | No Comments »

Hello and as my past self has already predicted I’m writing from my new room in south London. I would describe the place a bit but since I’m the only one reading this suffice to say that it’s pink and it’s quite nice. Bad things about it are the constant noise of traffic, the lack of curtains, the lack of decent shower in the bathroom and the dodgy wall socket which sparks and makes strange noises. Hopefully though I can get most of those sorted at some point.

Not much else to report at the moment. The job is going ok still mostly staring confusedly at code all day. I predict that my next post will be either existential or ranting in nature.

Life, Work | No Comments »

Well I’m finally employed. About time too. The company’s called Markit and they sell pricing information of financial instruments called derivatives. I’ll let you know when I’ve figured out what that means. I’ve been working a week now and not really go up to much. It seems 90% of the time I’m just staring confusedly at the company’s systems amazed at how much I don’t know. Still I’ve been assigned a few simple tasks so far and been able to do some useful work already which is good.

Next thing is finding a place to live. Somebody told me today that life is full of compromises. I think that’s defintely true when it comes to finding somewhere to live. It’s a constraint satisfaction problem between size, location, quality, area, housemates and price. All the places I’ve seen so far have been severly wanting in at least two of those categories.

You may have noticed the vast gaps in time between my blog posts. Suprisingly this isn’t because I’ve got nothing to blog about but I just seem to have no time to make posts. Extrapolating my current post frequency I expect I will be making my next post from my nice comfortable new room somewhere in south London. Til then…

Work | No Comments »

Yesterday I failed another interview. This time with Bloomberg. Hopefully I will soon be looking back on this from the wasted hours of my new job and wonder how I might have imagined what I would be doing at the then present time. Finding a job just takes time and persistence.

Anyway, these are some of the questions I remember which I was faced with almost as soon as I sat down:

1. Given an integer, how could you find out whether it was odd or even?
2. What properties does a Java Object object have?
3. Write a function that returns the factorial of a given integer.
4. Tell me what is wrong with this C code? (The code has a dangling pointer error in it).
5. Given an 8bit integer, how would you count the number of binary “ones”?
6. Given 8 coins, one of which is heavier than the others, how do you find the heavy coin using only two weighs of a old fashioned balance?
7. Describe a binary tree. What can it be used for? How long does it take to search? Why does it take this long?
8. Describe a hash table.
9. Given a list of words, how would you find which words contained an anagram of themselves within that list?
10. Given a list of words, how would you find the 25 most common words in that list?

I struggled answering questions 2, 4, 5, 9 and 10. I guess that’s why they didn’t want me. Still it was definitely interesting. I’ll have to research more about these questions and see if I can come up with some decent solutions.

Work | No Comments »