Mac@Work : The Ultimate Business Machine

The
Ultimate
Business Machine

Technology, business
and innovation.

And, not least, about
the Mac.

Weblog • Archive • Cutedge

by: Bernard Teo

Sun 11 Nov 2007

From Postfix Enabler to MailServe for Leopard

Category : Technology/FromPFEToMSL.txt

While re-reading the previous post about testing MailServe for the Leopard 10.5.1 Developer Build, besides spotting a couple of grammatical errors (a weblog is performance art - mistakes are part of the art), I realised I forgot one huge chunk of the testing process - outgoing smtp, i.e., the process that sends mail out the server.

How can I ever forget that? This was the other bug I spotted on Leopard and it was only corrected at the very last developer release.

I couldn't get outgoing smtp authentication working on Leopard for a very long time. This is the process that authenticates your server with another mail server that you are trying to use as a Smart Host, so you can relay mail through it and not have your mail (coming, as it is, from a dynamic IP address) flagged as spam.

I needed to use this feature myself, so I set to debug it doggedly. Then I found it was due to a couple of files missing on Leopard and reported that as a bug and, thankfully, that was fixed by Apple in time for the final release.

So I test it now. But first, check that the domain name works by hitting the web server. Always check that you can actually hit the server, either via the web browser or via the command line by pinging it, before you move your mind onto the mail server. I can't emphasise that enough.

I have an image I hold in my head of problem solving as a series of concentric circles. Every step you take must shrink the number of possibilties, the number of possible errors, that you have to consider. Otherwise you're going backwards. With every step you take, the possibilities must converge, so you get to the point, eventually, where you're able to decide that it's probably this, or that, but no other. So the decision as to which step to take, among the many, is very important. Take the step that reduces the number of possible outcomes. I've found that when people come to me to help them debug their systems, most of the time I'm actually helping them devise a problem solving strategy. It's nothing more than that.

I first test that I can send mail out without using a Smart Host. See? Don't complicate things. Be patient. Take the step where the outcome tells you something definite - that you have a working smtp server that knows how to send mail to another mail server (even if your mail gets rejected eventually due to its contents, or due to the prevailing anti-spam rules at that particular receiving mail server). If you didn't even get this to work, there is no point testing against a smart host, with all the attendant complications with the authentication parameters.

Even if you're having to use the Smart Host feature because your ISP is blocking port 25, you can check that your mail server is actually able to send mail out by using a test setup on your local network, using local private IP addresses instead of domain names.

So if you're able to send mail the default way, next, make the server go through a smart host. If you know an smtp server that'll allow you to use it as a smart host without authentication, so much the better, test against it.

That worked, so I test against a server that does require my server to authenticate against it. And then I test it with SSL.

Because I have two broadband lines coming into my home, I do all my tests against my own live server (cutedgesystems.com) on which I can set all sorts of conditions to act like any smart host would.

So, everything seems to be working on 10.5.1. But of course, I have to test it all again when the "real" 10.5.1 comes out for everyone.

So, I'm thinking. It really is a disadvantage calling my application Postfix Enabler, when it does so many other things beside enabling Postfix. It's like, I have the temerity to charge for something that others offer for free. But are they? Offering the same things, I mean. But like my auntie always tells me, life is too short for one to be resentful. It's a beautiful day out there. I have a date with my wife :-)

Posted at 5:44AM UTC | permalink

Fri 09 Nov 2007

Leopard 10.5.1 - just testing

Category : Technology/JustTesting.txt

I've downloaded the developer build of Leopard 10.5.1 for testing. This is what I have to go through to test each release of OS X.

First, I've started a test machine on my other broadband line, running Leopard 10.5.1. This will host a domain that I always use for testing - lifeassets.com (a domain that will eventually be used by my wife, who is a financial adviser).

I'm so glad I snagged that domain name before someone else did because it sounded so right - Life Assets. It's about financial independence, health, a balanced life, family, happiness - all assets in life's balance sheet. There are loads of ideas we can build around this core concept.

So, the first thing to test is that the domain name works - that it will lead people looking for the lifeassets.com server back to my test machine (which, incidentally, is an Intel-based MacBook running OS X Leopard 10.5.1). I use DNSUpdate to keep my public IP address sync'ed with the domain name, even though I'm on a broadband line, where the IP address changes periodically.

I have a Unix shell script written by my friend, Hai Hwee, that does this, too, and I plan to merge this into MailServe for Leopard so people running mail servers on dynamic IP addresses will have one less piece of software to worry about.

I keep looking for things to eliminate - buttons, fields, tab panels, whetever - e.g., in MailServe for Leopard, I've combined the Start/Stop/Restart buttons into one button. There used to be one set for each function - Postfix, POP, IMAP, Fetchmail - so, that's a lot of buttons eliminated.

To test that the domain name works, I've started up Web Sharing in the Sharing Preferences for the server machine and I try hitting lifeassets.com using a browser on my other broadband line. The Apache test page comes up, so I know I'm set and I can move on to concentrate on my mail server.

I'm amazed so many people skip this step. When they can't hit their mail server, they may already have the mail services, like SMTP, POP and IMAP, running correctly, but they simply can't reach their server because they hadn't managed to get the domain name-IP address mapping set up correctly. So they thresh about, solving the wrong problems in an ever-widening circle.

You need to check that you can reach the server via its domain name, both from outside and also from inside your local network if you've situated the server behind a router (because you may have one of those routers that don't know how to route outgoing packets back to a local machine that has been port-mapped to a public IP address). You need to do this so that you can be sure that any problem that arise after that step would be solely due to the introduction of the mail server.

So, if the domain name-IP address mapping works, I launch MailServe for Leopard and start up Postfix and the POP and IMAP services.

With SMTP authentication turned off and the server set to relay mail for machines on the same network (the default setting), I try to send mail from a client machine on the local network to, say, my .Mac account.

I look into my .Mac mail and, true enough, the message arrives and I reply to it. With the mail client set to retrieve from the lifeassets.com POP server, I can see the reply coming in, signalling that outgoing SMTP and POP work on my Leopard server. I turn off POP and create an IMAP account, and I can see the message in the IMAP Inbox. Then I create an IMAP folder, and move the message into it, and all is well on the IMAP front.

So ports 25, 110, and 143 are all working. What about SSL?

I create a test cert using the MailServe interface and turn on SSL modes for POP and IMAP and repeat the process described earlier. If all goes well, I can conclude that ports 993 and 995 are working properly. And the cert creation process, too.

Next, onto SMTP Authentication. For that, I move my mail client onto the other broadband line. Now that I'm not on the same network as my server, I'll need to authenticate with it to send mail through.

But first, I need to test that I can't send mail through it without authentication. You wouldn't want your server to relay mail for all and sundry on the Internet.

So, I send mail and it gets stuck in the Outbox and that's good.

I set MailServe for Leopard to relay mail for clients who authenticate and I choose the simpler OS X built-in accounts method as the authentication mechanism.

I change my mail client setting to send the authentication parameters to the server and try sending the stuck message again. This time it goes through and I'm smug. Lovely, isn't it.

Then I try to do the authentication via SASLDB. This was where the smile was wiped from my face for two whole months. Stuck while I try to solve it. Until I found it was an Apple bug.

Now, I try to send a new message and ... it doesn't go through. The worry comes back. But the I remember. Of course, it doesn't go through. I hadn't changed the authentication parameters on the mail client to use CRAM-MD5. I make the change and, swoosh, the mail goes through. Phew! I never stop worrying aboout this - that Apple will break it with each software update.

Now I test SSL all over, and POP and IMAP all over again, for the mail client connecting over the remote network. It all gets to be boring, until something doesn't work and then I'll take boring, anytime.

What else, do I need to test? Oh yes - "Require SSL" - for all three protocols. If you don't use SSL, you can't connect. Period. MailServe users wanted this, so MailServe users get this.

Also, alternate SMTP ports - MailServe has the ability to open up more ports, e.g., 2525 or 52525, for mail clients. Of course, we have to test SSL and non-SSL modes all over again for these ports.

And the ability to receive mail for additional secondary domains. There's also the Virtual Alias Domains variant, where mail for the same user in different domains go to different mailboxes.

What else? The log buttons - the Postfix and Fetchmail log. Even a simple thing like this could freeze the first release version of MailServe for Leopard.

Which reminds me, we've got Fetchmail, too. How can I forget? Such pain, so many more permutations. Fetchmail accessing POP, IMAP mailboxes, with or without SSL, keep, no-keep, polling intervals, time-out intervals, multi-drop mode.

At this point, if you're not tired reading this, you're a masochist, just like me.

I can go on and on.

In MailServe for Leopard, I have a new mode for configuring the mail server - as an admin user logged in using a non-admin account. So that creates another cycle for testing.

Then I always have to remember to test against an installation without Xcode loaded - in case I've inadvertently used a Unix feature that's only available if Xcode is installed. Of course, a lot of Mac users don't know Xcode from the Da Vinci Code.

So, all that testing. It really is a lot of work.

Posted at 3:35PM UTC | permalink

Wed 07 Nov 2007

Maven for Leopard

Category : Technology/MavenForLeopard.txt

The current version of Maven already works on Leopard. I've just tested it on Leopard. So, Luca and Maven, for the moment, now work on both Tiger and Leopard.

But I'm planning new versions of both applications that will take advantage of unique Leopard features. So it's possible that future versions of Luca and Maven will only run on Leopard.

But I'll get back to them in late December when I've cleared a few things I plan to do on MailServe, DNS Enabler and WebMon. It has been an exhausting last two weeks, and I hope to take a couple of days break to rest and to think. And then, I'll start work again.

I've been asked, why do I want to work on Maven when there are already competing solutions like CocoaMySQL, etc. The only answer I have is that I build Maven for myself, so that I can build Luca faster.

The important thing about Maven is not (at least, not yet) the CocoaMySQL-type GUI. It's the database access frameworks that I've embedded inside it and which do the heavy-lifting of talking to the databases in a consistent programmer-productivity-enhancing way, and which are the exact same ones I use in Luca. Maybe because I've spent so much of my working life in PC/Windows-dominated IT/MIS environments, I have a different view of the requirements for data-access and data-mining tools than, perhaps, most other Mac developers. I'm trying to marry the two - to build tools that won't be scoffed at by IT/MIS folks; if possible, to build tools that could make their jaws drop in areas where they appreciate - e.g., raw power and performance and the ability to express difficult concepts to get at the data - and yet still have the elegance we come to expect as Mac users. For that, we need minute control over the database access layer.

Another question. Why do I persist in calling it Maven, when there's already an Apache project by that name? In time, the name may change. But for now, Maven captures the essence of what I want the user to be when he or she is using this tool to navigate, access, and understand the database.

For now, all this is just a dream. But now and then, I get a glimpse of how to achieve it.

Posted at 1:58PM UTC | permalink

Wed 31 Oct 2007

Luca for Leopard

Category : Technology/Luca2dot6.txt

I've updated Luca to work with Leopard. It's in the 2.6 version.

Posted at 11:05PM UTC | permalink

Sat 27 Oct 2007

MailServe, WebMon and DNS Enabler for Leopard

Category : Technology/ForLeopard.txt

I think I'm ready to release them all now.

I've merged Postfix Enabler into MailServe so I'll just have that single product to support for Leopard.

So, MailServe for Leopard is at http://cutedgesystems.com/software/MailServeForLeopard/

DNS Enabler for Leopard is at http://cutedgesystems.com/software/DNSEnablerForLeopard/

and

WebMon for Leopard is at http://cutedgesystems.com/software/WebMonForLeopard/

Thanks for waiting, all those who've been coming over here to check for their progress. I'll continue to work on them and on their documentation, but I'd better not hold you up any longer.

I've done as much testing as I could on them, with my friend Hai Hwee's help, who's now camped out at our home - we've tested it on Intel , PPC, admin user, non-admin user, SASLDB, smart host, SSL, no SSL, Fetchmail, you name it, she's tested it. But if anything can go wrong, it will, especially when real users get their hands on it. So I'll just keep the announcements to these pages, to keep the workload manageable, and let's see how well these versions hold up.

I hadn't had much sleep these last two or three days. So I'm all ready to crash. I hope the apps do hold up, This is really living on the edge.

Posted at 6:53PM UTC | permalink

Morning has broken

Category : Commentary/MorningHasBroken.txt

The sun's up and I've readied all three applications - MailServe, DNS Enabler and WebMon - for testing. Found a bug with DNS Enabler for the Bonjour setup panel and managed to fix that.

I've created new web pages for DNS Enabler for Leopard, and WebMon for Leopard. Only one more to go - the documentation for MailServe for Leopard.

I don't think I'm ever going to see the AppleStore delivery for Leopard this weekend. There were no apologies, and no phone calls for status updates. What's the chance the delivery guys will work over the weekend?

I think I'll drop by a physical store when they open in a few hours and get a copy. If the testing goes well, I might be able to do a release by the end of today, the 27th, my time, in about 16 hours.

Posted at 12:30AM UTC | permalink

Fri 26 Oct 2007

WebMon for Leopard

Category : Commentary/WebMonForLeopard.txt

Okay, I'm done with a version of WebMon for Leopard, as well.

It's been a long - sometimes discouraging - journey, getting all these applications to work again on Leopard because I often feel like I've been going over the same ground again and again. It has been so much work just to recover the same set of features.

But the ground has changed, underneath. There's much that we can now build on. I think I read somewhere Steve Jobs saying, this - Leopard - is the foundation for a decade of improvements and we will go on from here, polishing and improving the applications.

There's a lot I am looking forward to building, from here on.

With our son having begun his own journey through the (rather frightening) Singapore school system (my God, do we all think our kids are geniuses? I feel for the kids), a side-effect is that my wife and I getting more time to delve into our respective work. I love this work, so it's been a welcome change, and it's the kind of work where, beyond a certain stage, beyond a certain level of mastery, you feel like you can just do magic with your fingers, like a Druid. So, there's going to be more to come.

I'm now moving on to setting up the web pages for these new Leopard-ready versions. With any luck, I can get them done before the sun comes up again. Then I'll go to a shop and buy a retail copy of Leopard to test out. It's been a miscalculation, buying from the online store. So much for guaranteed delivery on the 26th.

I'm sustained by coffee and this amazing gamelan music by a (Japanese?) group called Jalan Jalan (Malay for going for a walk)- from the Bali and Bali Dua albums.

Posted at 6:22PM UTC | permalink

Leopard ships but where's my copy?

Category : Commentary/LeopardShipsBut.txt

Question : Just looked at your blog and saw that there is a new version of your software but I am not sure where I can download it and buy it. Could you please provide a link or let us know if they will be available later today or tomorrow?

Answer : I haven't received my final version of Leopard yet to test it against. It was supposed to be, latest, an hour ago. I ordered from the online store for guaranteed delivery on the 26th October but it's now 9.30 pm and there's no sign of the delivery man. It's been quite a let-down - that's Apple at its worst. No call. Nothing, even though I called AppleStore to confirm that it'll be delivered on time this morning. I will put up the MailServe and DNS Enabler web pages in a day or two after I've managed to make a final test against the shipping version of Leopard. Believe me, you'll be glad I did.

Posted at 1:48PM UTC | permalink

Leopard Ships

Category : Commentary/LeopardShips.txt

Any minute now, my copy of Leopard should be arriving at the door. I've just gotten MailServe and DNS Enabler for Leopard done, but I hadn't had much time to test the recent changes that I'd made to the applications to keep pace with the last developer release. And I can't quite believe there aren't going to be any further surprises with the final Leopard release.

I've still some way to go yet with WebMon and I have still to get all the web page documentation updated.

Plus, I've just sold my house and bought a shophouse that I'm planning to move to for the coming year. So it's going to make for a hectic final quarter of the year.

This is how MailServe for Leopard looks like :

and DNS Enabler for Leopard :

Got to get back to work now.

Posted at 12:05PM UTC | permalink

Fri 19 Oct 2007

Getting ready for Leopard

Category : Technology/GettingReadyLeopard.txt

I've been getting lots of mail about the status of the projects, with Leopard coming in only seven days. And I don't have time to answer them all because I'm deep into this thing. I've got MailServe almost ready. Then there's DNS Enabler. And I may hold off on WebMon for a few more days after Leopard ships, to see if I've hit anything that I hadn't foreseen, with the first two apps.

I can see one already - firewall. I don't have a firewall application, yet, that I can offer people, and Leopard's firewall panel looks strangely less usable than Tiger's - it takes us more than a few steps back - how is an ordinary user going to open ports 25, 110, 143, etc in Leopard? I'll leave this aside for the moment because I don't have time to solve it.

So this is what I have up to now :

MailServe/Postfix Enabler.

I'll probably be merging these two into just MailServe. Apple chose this latest developer release of 3 weeks ago to make a change as to how Postfix starts up. The launchd plist we used in Tiger doesn't work anymore. I managed to find a solution but, by having to do that, and having gone so far, I found that it opened up a way to put together a few things that I've been experimenting with over the last two years - like :

1) how to allow MailServe/Postfix Enabler to be configured from a non-admin account (a lot of requests for that), and

2) how to avoid using sudo so that it doesn't open up a security hole during the time MailServe/Postfix Enabler is running

3) how to configure Postfix and make changes to your system without touching a single one of the original system files, so that the system is left in its original pristine state if you choose to de-install any of my applications - also hope this will make it easier to get the mail server running for people who've messed up their original system files because I don't even need to look into these system files now

4) and how to give visual indications that the relevant mail-related ports are opened and working without getting people to do a telnet something or other on the terminal,

5) plus a few more that I forgot

Oh, yes, how to make sure that the passwords you store in MailServe's Smart Host list is not so easily seen in the clear in the app's preferences plist. I think I've also almost solved that.

And how to get Fetchmail to start up more reliably than the hack I was using on Tiger. I've solved this as a benefit of solving the original problem, posed by having to find a way to start up Postfix on Leopard.

And so I've put this lot in, only in the last three weeks, because somehow it has all clicked, and I'm furiously trying to get everything tested and ready for the new MailServe to be released also on the 26th.

I was watching this Google video - achieving your childhood dreams - and about how brickwalls are meant to stop "those other people". So, I couldn't bash through the problem that I had - it was too hard. But I could climb over the wall - to scale it - and in doing so, I saw connections that I hadn't seen before. And then it all clicked. Now, I love being able to switch to a non-admin account and being able to continue to configure Postfix.

DNS Enabler.

I made a version that will work with the changes that Apple has wrought on Leopard. What I had planned to do was to release this version that I am already using on the 26th, so that people who need to get their DNS running on Leopard, using DNS Enabler, can have something to work on immediately, and then I'll put in (and release) the rest of the things I'm working on in the following weeks.

I expect a lot of support load the first few weeks Leopard ships and so I'm wary of changing too many things at once since I've had so few weeks for testing them.

MailServe already has a boatload of changes to take advantage of the things I can now do on Leopard but these things might break - if they don't, then all the new techniques that I'm applying on MailServe will also go into DNS Enabler.

But, then, I've still some days yet. so I might still be able to get these changes done and hope the gods smile on me without any mishaps. (If they break, please be patient. This is really living on the edge.)

WebMon.

I've got a WebMon that will now work with the new Apache2 on Leopard. But I think I've got one more major thing to fix. I may not be ready on the 26th with WebMon.

Pricing

I plan to merge MailServe and Postfix Enabler into just MailServe and price it, like DNS Enabler and WebMon, at 15 USD. I won't be doing upgrades (because it costs significantly more to put in the backend administrative systems to handle all those upgrade permutations). I thought, if I charge upgrade-like pricing each time, it'll allow me to simplify my business and spend more time working on building these applications. I've got more planned - LDAP Enabler, Dovecot (if I get permission to bundle it), anti-spam stuff, iPhone! (I've got all the pieces to do a CRM-like system - contacts, accounting, email, address-book integration, and so on...)

PS : Luca is already Leopard-ready though I'm looking forward to improving its interface. Apple has bundled a nice new Xcode with Leopard. Whatever gives the developers more power will show up eventually in the quality of the applications. I think that's the one thing that won't be readily apparent to an end user - all those invisible improvements that Apple has made to the OS foundations with Leopard - it has made what was already a very productive platform even more conducive to the development of great apps. If we can get past the teething problems with Leopard in the first few weeks, I think we can all start to smile. This is a helluva of a good platform to build your business on, by far.

Posted at 1:38AM UTC | permalink

Sun 23 Sep 2007

Eye candy, rolling stone, and the man on the mountain

Category : Technology/sisyphus.txt

I'm now working on the latest Leopard release (9A559, 21st September). It's looking good, the first version I can stand to look at - or work on - for any stretch of time. The previous releases were all plain butt ugly.

Leopard's looking like a Mac again. Something people could like using even more than Tiger. Give me eye candy anytime - when you think we were that close to disaster :-)

But I can't even start up Postfix now.

I feel like Sisyphus rolling the boulder up the mountain. When you finally get it to the top, you know you're going have to do it all over again, come tomorrow.

Albert Camus wrote a book - "The Myth of Sisyphus" - "a lucid invitation to live, and to create, in the very midst of the desert".

Posted at 6:51AM UTC | permalink

Wed 19 Sep 2007

Leopard Updates for September

Category : Technology/LeopardUpdatesSept.txt

I've finally managed to get SASL authentication working again with Postfix on Leopard. After tearing my hair out over the last few months, I used up one of my Apple Developer "support incident" credits to get help. And it turns out that it was a bug after all. I received a patch from Apple and everything's well again.

I've got to steer clear of breaching my NDA, but here's a review of how well (or badly) I'm shaping up to support Leopard :

Fortunately, Luca and Maven (including my database access frameworks for MySQL, SQLite and PostgreSQL) already work on Leopard and I can put those aside now for a while.

But there's a slew of changes - all good - in the underpinnings of the OS (e.g., updated versions of all the good stuff, like Postfix, OpenLDAP, BIND, Apache, etc.) such that MailServe/Postfix Enabler, WebMon and DNS Enabler got all whacked out of shape.

But that's eventually for the good of the end-user and I think I've finally managed to recover all the things that used to work, i.e., every single feature working on Tiger will continue to work on Leopard, though things have changed underneath.

So, that was the hard stuff.

Yet, that's not to say, knowing Apple, that things will continue to work right on to the final release. I'll only believe it when I test everything again with the shrink-wrapped version of the OS.

Plus, my experience with releasing my applications when Tiger first shipped was that lots of unforeseen issues are going to crop up. Things that can break will break. With luck, the support that people are going to need may be, maybe, just short of horrendous.

So, while I have a glimmer in my eye about the things I'm looking forward to building with this nice new OS (not so nice, the new Finder, Desktop and the Dock, which are pretty gaudy and you can see the screen shots yourself on Apple's site), I'll be a bit defensive this time with the versions that'll be released when Leopard ships, and introduce the new stuff gradually when people are nicely bedded in.

Posted at 4:51AM UTC | permalink