Mac@Work : The Ultimate Business Machine

The
Ultimate
Business Machine

Technology, business
and innovation.

And, not least, about
the Mac.

Weblog • Archive • Cutedge

by: Bernard Teo

Fri 08 Sep 2006

Business, Computing, and the Mac

Category : Technology/numberTest.txt

I realised I hadn't written anything in three weeks. But I'm only now re-surfacing, having spent all that time inside the bowels of Luca, pulling out all references to doubles and floating point numbers, and replacing them with Cocoa's NSDecimalNumber class.

Is it worth it? Consider the following code snippet :

- (void)test:(id)sender { NSDecimalNumber *decNo = [NSDecimalNumber decimalNumberWithString:@"0.0001"]; NSDecimalNumber *total = [NSDecimalNumber zero]; int i; for ( i = 0 ; i < 10000 ; i++ ) { total = [total decimalNumberByAdding: decNo]; } NSLog(@"total, as NSDecimalNumber = %@", total); double numberAsDouble = 0.0001; double totalAsDouble = 0.0; for ( i = 0 ; i < 10000 ; i++ ) { totalAsDouble = totalAsDouble + numberAsDouble; } NSLog(@"total, as double = %1.24f", totalAsDouble); }

So, we've got two numbers. One expressing 0.0001 as an NSDecimalNumber and the other as a floating point number. We loop around and add them up 10000 times. These are the results we get :

[Luca 1397] total, as NSDecimalNumber = 1 [Luca 1397] total, as double = 0.999999999999906186154419

With the NSDecimalNumber, we get the result we expected, 1. The floating point number, on the other hand, came very close to 1 but then, not quite.

That's the problem we face working with floating point numbers. They really do float away from what they are meant to be.

Let's see what havoc this causes in an accounting application. Let's say we're adding a long list of debits and credits. The two columns often won't balance, if we don't apply a mathematical function to round the figures. But then, when do you do the rounding? After the addition, or on each line item? And to how many decimal places? Two, if you're working with US dollars, none if we're working with the Japanese yen.

The code easily becomes very complicated - unecessarily so because if the computer knew how to do its math, we wouldn't have to round at all.

So that's what NSDecimalNumber does - it lets you work with greater precision so you get the result you want without having to go through all these contortions.

That doesn't mean you eliminate rounding completely. You still have to apply a rounding function when you multiply, say, a monetary figure by an exchange rate, and you have to decide how many places you want to round it to (probably to the number of decimal places used by the currency you base your accounting system on). And you still have to work out how to handle dividing 1 dollar between 3 people, like who gets to keep the last cent. But at least we've cleared up, say 80% of the clutter in our code.

So, imprecise calculation is one problem we faced in business computing that we can solve with that new data structure, called the NSDecimalNumber in Objective-C, and BigDecimal in Java.

Then there's the problem of storage. Does the database system handle exact-number mathematics? Luckily, MySQL does, with its Decimal data type (since MySQL 5.0.3). So shuttling the data between Luca and MySQL is sweet - things go in and out the way you want.

Not so with SQLite3. You get all this muck (in red, below) because SQLite still works only with floats and doubles and it can't represent a number like 513.58 exactly :

The problem comes when you read the data back. (Again, just when do you round the data, and to how many decimal places?) I'll have to attack this problem next.

But let's move on. I keep mentioning that we need to figure out how many decimal places to round, say, an exchange rate calculation to, because it depends on the context.

In the picture below, I show a Luca payment voucher where three currencies are involved. The original invoice was billed in Singapore dollars, and is being paid with Malaysian dollars. The base currency (the local currency used by our fictitious example company) is in US dollars and that's what we use to book the bank charges and the currency exchange gain under.

In this case, all three currencies are denoted to two decimal places. But if we get paid in Yen, we'd like, as Mac users, to see no decimal places at all when we're entering the Yen value. And when we print out that voucher we'd like to see the Yen symbol being printed, too.

Fortunately, Cocoa has a user-interface class called the NSNumberFormatter class, that I have built a custom sub-class for, and that I can use inside Interface Builder, as if it were one of the supplied palette objects :

And the great thing about being able to use your own custom number formatter class is that it opens up a lot of possibilities. Suddenly, you're able to format numbers the way the French or the Swedes or the Indians want, and since you can get hold of your formatter object in your code during run-time, you can do a lot of currency formatting on the fly, or let the users customise it themselves.

If the user decides to format the Yen to no decimal places, you will be able to tell from the "Max fraction digits" parameter, above, and that's what you use to control the rounding behaviour. Plus you can set the rounding behaviour to say, the traditional way (which has a bias towards rounding up), or "bankers' rounding" where numbers are rounded down as often, statistically, as they are rounded up. And you can gather all these customisations in one logical place, so it's possible to avoid hard-coding anything in your application, and thereby making it all truly multi-currency.

Cocoa is a wonderful development environment. Sometimes, like when I managed to get this working in Interface Builder, I was so excited and I wonder why don't more people see it? This is so great. Like, with the Spell Out Style, above, you can do cheque printing. And, though I haven't done it yet, you should be able to print it with fillers like asterisks between the currency symbol and the money value.

Now, if all these are not doing things to support a business, then I wonder what is. And if the Mac is not a business machine, then what is?

It took me three hours, from knowing nothing about Interface Builder palettes, to actually creating my own number formatter and getting it working in my own nib files. And I actually did this while parked near some people noisily playing mahjong in some relatives' house. I say this because I'm constantly amazed at how productive this development environment is. And yet how little people know about it in the mainstream.

I was attending this talk by the CEO of Sybase, John Chen, yesterday (it was a very interesting talk - he shared his experience doing business in China - lots of interesting people pass through Singapore - I ought to be attending more of these). Anyway, I was talking to someone who found out that I'm working on the Mac, and he says, oh that's only a niche market, right? for the arty-farty types, right? Right.

Read The Long Tail - lots of interesting things are happening at the niches. The niches, as a whole, are going to get bigger than what used to pass for the mainstream. We're going to eat their lunch, all these IT types.

Posted at 7:05AM UTC | permalink

Fri 18 Aug 2006

BigDecimals

Category : Technology/BigDecimals.txt

This is very good tutorial, showing how BigDecimals are used in Java code :

"Make cents with BigDecimal - Write Java programs to calculate and format currency".

We can map all these calls to Objective-C in an equivalent way.

Also, read this :

"The Seven Habits of Highly Dysfunctional Design - In your face ! BigDecimal, BigInteger and BigMistake".

Posted at 3:45AM UTC | permalink

Of Real Numbers, Floats, and Decimals

Category : Technology/decimals.txt

My last post about problems handling floating point numbers for financial calculations brought me a couple of responses.

I make an assumption in Luca that all currencies work with two decimal places, and I round them wherever I can and store them to two decimal places. This way my accounts (should) always balance since, for every voucher, I make sure that the debits and credits are rounded to two decimal places and compared to be equal before I allow them to be saved.

But what if I'm working with a currency that doesn't use two decimal places - they use three, or four, or five, or ... none?

Guy Harris says - "Yes, the Japanese yen has no decimal places at all."

That's just the motivation I need to go and dig further because the assumption I used wouldn't work if I want Luca to be used just about anywhere possible.

Luca allows the base currency (the currency you use to base your accounting on and produce the financial statements) to be different from the billling currency and the settlement currency. So what if we get paid in Yen? Rounding the yen to two decimal places would make no sense at all.

So what to do?

I got one other mail from Quintin May that pointed me towards a whole new world of possibilities :

"I read your post regarding floating point and rounding in Luca. It has been my experience when working with financial data to NEVER use floating point numbers. In Java, the BigDecimal type is preferred for this type of data representation. I'm not fluent in Objective-C, so I don't know if it has an equivalent data type. This article goes into excruciating detail on floating point arithmetic: http://docs.sun.com/source/806-3568/ncg_goldberg.html. Try a Google search for "objective-c" "bigdecimal".

And that's what I did.

Objective-C seems to have an equivalent class - the NSDecimalNumber. It has methods for adding, subtracting, multiplying, dividing, and comparing two objects of the same NSDecimalNumber type, plus conversion to formatted strings for display, with the correct separators and currency symbol, depending on locale.

This looks like just what I need.

Better still, new to MySQL 5.0, there's an equivalent exact-number data type called Decimal.

So if all these pan out, a good strategy to take is : don't round, store all financial values as Decimal data types in MySQL, and load them into NSDecimalNumber data objects in Objective-C. Display them in windows and views as strings formatted according the appropriate locale, set according to whether the displayed value is in the billing, settlement or base currency. Do not add them like you would double or floating point numbers. Use the supplied methods, for addition, multiplication, and especially for comparison.

So, this looks like a very neat idea. But will it all work out? And what about SQLite? It doesn't have an equivalent data type. There's only one way to find out - Just Do It !

I've often asked myself why I bother writing a weblog or why people would bother coming over to read it. Well, this is one benefit. I get responses. When I'm stuck, they point me to new, fortuitously more productive directions. And I hope you, dear reader, learn something useful also when I do.

Posted at 2:00AM UTC | permalink

Wed 16 Aug 2006

Do the math ... or when is 156.14 != 156.14 ?

Category : Technology/maths.txt

This is a Cash Disbursement voucher in Luca :

I have a rule that says you can't save a voucher unless the debits and credits balance. So, here we have $156.14 being paid out of a checking account, recorded against an equal $156.14 in expenses. Yet the Save button refuses to be enabled. What is going on here?

Looking a bit closer at the objects :

NSLog(@"vouAmount = %f", [vouAmount doubleValue]);
NSLog(@"ttlAmount = %f", [ttlAmount doubleValue]);

we get :

Luca[1046] vouAmount = 156.140000
Luca[1046] ttlAmount = 156.140000

It's as we would expect but it remains puzzling, so we zoom even closer :

NSLog(@"vouAmount = %1.24f", [vouAmount doubleValue]);
NSLog(@"ttlAmount = %1.24f", [ttlAmount doubleValue]);

we get :

Luca[1046] vouAmount = 156.140000000000014779288904
Luca[1046] ttlAmount = 156.139999999999986357579473

The vouAmount is taken straight off the object. The total column has a rounding function applied to it, but still it's just one line item, for simplicity's sake. So both items are off the expected figure of 156.14. They never balance.

This happens only at certain values, not all the time. It happens because "real numbers" in digital computations are only a very close approximation to the mathematical concept of real numbers. The difference is infinitesimally small (my favourite expression in engineering school), but it's there all the same. It's what throws us off.

This reminds me of that paradox of Zeno, of a foot race between Achilles and a tortoise. The tortoise is given a headstart, but picture Brad Pitt as Achilles, huffing and puffing, always covering half the distance to the tortoise, always gaining, but never quite reaching, because the half-distance keeps getting smaller, and smaller, and smaller, but never disappears. It's become infinitesimally small.

The solution is to round both numbers before comparing them, even the one entered exactly as 156.14.. But we have to take care to do the rounding at lots of places, not just when the value is entered. For example, you do rounding when one value is multiplied by another. But what happens when you divide a dollar by three. You lose that last infinitesimally small fraction of a cent because computers work with discrete values, not a continuum.

So, that's why building such systems, mundane though they seem, is hard. It's a craft. It's precision work. But it's not the kind that will win Apple Design Awards at WWDC.

I'm fixing this in a Luca 2.2.2 release. But I have the rounding done all over the place. The art is to find a way to control this in just one spot in the application. What happens when I find a user whose currency system works with other than two decimal points? Is there such a thing?

Posted at 7:59AM UTC | permalink

Mon 14 Aug 2006

6000 Customers

Category : Commentary/6000Customers.txt

I think we've crossed 6000 paid customers.

I just read that the guys at Parallels sold 100,000 copies of their virtualisation software at $80 per copy. So 6000 pales into insignificance when compared with that.

But I know it'll mean something to me if I do get to cross the 10,000 mark. Building a business one customer at a time.

Is this too slow? Slow is when you're waiting for time to pass in Eternity.

"To see a world in a grain of sand,
And a heaven in a wild flower,
Hold infinity in the palm of your hand,
And eternity in an hour."

Now, who said that?

I'm reading this book "Art and Physics - Parallel Visions in Space, Time & Light" by Leonard Shlain. If you've ever done physics in school and know enough to tell a Monet from a Manet, this may make interesting reading (in spite of the pretty bad reviews it's gotten at Amazon, but one man's meat is another man's poison and so on ...)

Posted at 4:44PM UTC | permalink

DNS Enabler in German

Category : Technology/DNSEnablerDeutsch.txt

Matthias Schmidt at Admilon contributed this German localisation of DNS Enabler 2.1, including the new Bonjour panel.

Thanks, Matthias.

Posted at 1:09PM UTC | permalink

Luca 2.2.1

Category : Technology/Luca2dot2dot1.txt

Over the weekend I've fixed some (rather embarrassing) user interface bugs with the last release of Luca, like buttons and text not sticking to their corners when windows get resized, or the default financial period not getting saved to preferences when the save-to-defaults button was pressed.

Notifications also work better now, so when something happens in one window, e.g., when a voucher gets deleted, the totals showing on other windows will all get updated automatically.

Also, I had some problems mapping date fields between SQLite and MySQL, especially handling NULL data types. These are handled better now.

Things To Do

1. I need to improve the speed in which data is imported/exported between SQLite and MySQL. This is currently slow and still not robust enough to handle, e.g., three years worth of data without hanging.

2. I need to put in a button to let the user choose whether to drop a database before exporting the data. Currently it assumes you want to overwrite the destination database and tables in MySQL.

Here is someone I know who's using Luca with MySQL now - Jan Steinman at VeggieVanGogh.

3. Need to handle the "last one cent" problem better, e.g., when a dollar is split three ways and you put them back again, you get 99 cents. Which account should get the last one cent so that the debits and credits always balance, no matter how often you go back and edit or rollback the data?

This problem shows up when you're apportioning amounts to cost centres or when you're doing currency exchange gain/loss calculations. I don't think I've ever totally solved it while we were doing custom applications - once in a while, our user would find that she can't save the voucher because the accounts are out by one cent. In a custom application, we're on hand to guide the user to correct this manually. But not when it's going to be a "shrink-wrapped" application. So I've got to pin this down once and for all.

4. Tax handling on line items rather than on the consolidated total.

5. After all these I think I can move on to the invoicing module, a payroll module, or a time-billing module? Actually I have pieces of quite a lot of these in 4th Dimension code.

I'm like a PC guy who just loves Macs. I do these PC things for PC-using companies but I want to use a Mac. I'm stuck between art and science, engineering and design, the PC and the Mac.

Luca is a work-in-progress - for the Mac as The Ultimate Business Machine. Do keep the suggestions coming.

Posted at 12:54PM UTC | permalink

Sat 12 Aug 2006

Luca 2.2 with MySQL

Category : Technology/Luca2dot2MySQL.txt

I've released Luca 2.2 with support for storing the accounting data in a MySQL database. I've also written the import/export code to shuttle the data between SQLite and MySQL, and back.

I realised then that I've got the basis of the code to do a "universal" SQL data converter, to shuttle data between any kind of SQL database. If I have time, I'll take a look at PostgreSQL or Oracle. (Now, look how richly endowed we all are on the Mac. Who would have thought?)

Luca is getting more stable now and working faster. There are probably a few more bugs lurking inside the system but I thought I'd better get this out as it's quite usable now, and I've made some improvements to the workflow and made several bug fixes. I hope that by the next release, I can be sure that Luca will be thoroughly reliable.

I'm going to stress-test this release next. I've already seen where it's still breaking. But I've worked hard to get to this point and release it because I want this idea to take hold in people's minds - that you can really do a lot on the Mac.

With this release, people who want to integrate an accounting system to the rest of the custom, say, PHP-based workflow that they've built can now do it. The database is open and you can flow your data onto it or get data out of it in ways that I cannot even anticipate now.

And you can get at your accounting data even when you're half-way round the world. I've yet to put in the concurrency-handling code. But we've done this years ago while it was still a 4th Dimension-cum-Oracle-based system (though that would never have worked outside a local area network) and so I know we can do it.

I'll get to the point one day when all these things - MailServe, WebMon, DNS Enabler, Luca, etc - will all come together and make sense as one seamless whole. I have an idea what that is but I can't yet explain, even to myself, what that means. I'm learning as I go.

They say (actually it was Steve Jobs) that you shouldn't start a company just because you want to start a company. You start a company because nobody else believed in an idea and the only way you can get it out of your system (and keep from going crazy) is to go out and build it on your own.

Posted at 4:15AM UTC | permalink

Wed 09 Aug 2006

DNS Enabler 2.1

Category : Technology/DNSEnabler2dot1.txt

I've released DNS Enabler 2.1.

It includes a Bonjour panel to allow DNS Enabler to be used as a configurator to enable wide-area discovery of Bonjour devices :

More information about the types of Bonjour services you can currently configure, and what you need to enter into the individual columns in the Bonjour panel, can be found at : http://www.dns-sd.org/ServiceTypes.html (DNS SRV (RFC 2782) Service Types).

What DNS Enabler saves the user is the need to know the specific syntax to set up SRV and TXT records at the DNS server. All he needs to know is the Bonjour service type, e.g., _http._tcp for publicising web pages (that may be served even from local private networks - you can use the Port number field to publicise a different port other than 80 and port-map that incoming request to a specific local machine) and the domain that will serve that request. The service name is a label that will show up in the Bonjour menu in, say, Safari. And the TXT column contains the path to that specific web page. (The TXT column stores different things for different services. See the cited Bonjour reference).

Stuart Cheshire may be showing this panel at work sometime in his Bonjour talk at WWDC on Thursday, 10th August, at 9.00 am (PST).

PS : I've also added a button to populate the Forwarders field with the list of DNS servers that the user has set up in his Network Preferences. I've left the interface somewhat cluttered at the moment because I'm planning to add a couple more features in the next two weeks, and I'll clean up the interface then.

Posted at 6:28AM UTC | permalink

Sun 30 Jul 2006

Upcoming Updates

Category : Technology/NewUpdatesEndJuly.txt

I'm working on a couple of things. I'll be releasing a version of Luca Accounting soon that will work with MySQL. I just need to write the script that will allow the initial built-in SQLite data to be exported to MySQL.

If there's anybody who can't wait and want to try this out now, just let me know. It's still missing the concurrency handling that is needed for multi-user access (that's the next step), but this is going to make it possible for anyone to access his accounting data from anywhere in the world (if he so chooses), as well as make the database open to all sorts of custom integration.

The second thing I'm working on is this :

This is experimental. It's an extension to DNS Enabler that will allow the user to configure the DNS server so that it'll also broadcast Bonjour services (that the user has happened to have set up) so that they can be discovered, browsed, and accessed from the wide-area network (rather than on just the local network).

I may also add a couple more new features to the next DNS Enabler release that people have asked for (e.g., the ability to set the DNS server so that it doesn't do recursive lookups, and the ability to restrict the range of IP addresses that you allow lookups to be made from).

So, lots of things to do.

Posted at 5:06AM UTC | permalink

Thu 20 Jul 2006

Working around Workaround Bonjour

Category : Technology/WorkingAroundWorkaround.txt

I've updated both DNS Enabler (to 2.0.9) and MailServe (to 2.1.6) to handle the bug introduced by the OS X 10.4.7 update (that "Workaround Bonjour" thing) whereby the system would stall for about 60 seconds whenever a call is made to /bin/launchctl to start a launchdaemon (like BIND, Fetchmail, etc).

I've changed DNS Enabler so that the Restart DNS button avoids calling launchctl, so that the only time you actually see the stall is when you first start the DNS service (be patient - it'll clear - and the DNS Server will still work correctly), and not when you stop the service, and especially not when you restart the service to make changes to your DNS data, which is pretty much what we do most of the time.

I've done the same to the Restart Fetchmail button in MailServe, so that you don't get held up when you need to change Fetchmail parameters.

It's important to note that this happens only on 10.4.7, and that the DNS and Fetchmail services (as well as Postfix, POP, IMAP, etc, i.e, anything started or enabled using launchctl) are all still launched correctly.

Up to now, I've ascertained that this stall only happens when an application makes a call to the "/bin/launchctl load" comamnd (well, sometimes on launchctl unload, depending on the sequence of the calls you make). I believe this is probably an unfortunate side effect of the security-related changes Apple made to patch up the launchdaemon mechanism (launchd) in the 10.4.7 update.

I've just installed a second broadband line to my home so that I can now do all the testing that I had wished I could do without interrupting service on my live server. I happened to use DNSUpdate and also Lingon just now and both of them stalled. Looked up the Console - and it's our friend, Workaround Bonjour. When is Apple going to fix this?

As another aside - as I was posting this on the live server (from this second broadband line), I noticed that things had suddenly become rather sluggish on the live server. Guess what? - it's all the people coming over, via MacUpdate and Version Tracker, for DNS Enabler. So that's how my server feels like from outside. You never know when you're inside your own network.

Posted at 3:56PM UTC | permalink

Sun 16 Jul 2006

MailServe 2.1.5 Update

Category : Technology/MailServe2dot1dot5.txt

I've updated MailServe to 2.1.5, to make improvements to the way MailServe triggers Fetchmail.

It was while working on Fetchmail that I realised Apple broke launchctl on 10.4.7, probably as a side effect of the security changes they were making to launchd.

I've added a way to specify a global value for the Polling Interval (default 60 seconds) and the Time-out Interval (default 45 seconds but can be set to 0 for no time-out).

Also the quotes around the * in the .fetchmailrc file (when Fetchmail is used in multi-drop mode) interfered with the multi-drop operations. This has been corrected in this release. (Thanks to Fergus McMenemie for showing me there was a problem.)

The French localisation for MailServe has been updated by Joselyne Rochaud and Corentin Cras-Méneur, who also took the opportunity to update the French localisations for DNS Enabler, WebMon and Postfix Enabler.
Thanks, Josy and Corentin.

Posted at 6:38AM UTC | permalink