System Design Pattern #1

I'm starting to see common ways to design complex systems, so I figured I might as well start a series of blog posts which will be posted whenever I find something new.

The first design pattern, though it might be fairly obvious, is addressing the problem of: "Well, we can't have this system fail -- ever. Never-ever-ever FAIL, or you lose your job." That problem is fairly common. Your OS is a good example of a never-die system. Web servers are another one.

The way to solve the problem is to add a layer of indirection between the core and the services. The bottom layer will be the core system that will, to the best of your ability, NEVER fail. The top layer will be any mashup of services. Why does this design work so well? Because you can make changes to your extended services very easily while guaranteeing that any service failure will never affect the core system. What's the other way to do this? Combining the services with the core system. That's just a recipe for disaster.

I'm amazed at how hard it is to see the design. At the bughouse server, we have 30,000 LOC of C, and everybody was still trying to add features to the server! The simplest design, staring right at them in the face, was to farm the work out to automated bots. The additional power of such a design was that you could write a bot in any language you wanted. That's when you know you have a real winning design -- when the interface is so damn clear, you don't even need to specify the language you need to write the program in.

We had a similar problem at Nutanix regarding a automated test system that needed to, ahem, "NEVER FAIL." I won't talk too much about it, but the last time we discussed the system, we were still talking about extending the whole thing with yet more Python. It's just so clear now -- you need to define an interface for easily adding services!

Anyway, this concludes my rant for today. That was a very enlightening write-up. I think I might do more of these as the design patterns come in.

Relection on life

You know what's really interesting? You only really know what you want to do in life when you start hating what you are doing right now.

Expedient way to move around Linux CLI

Hey all,

I have a solution for a common problem. When you're SSH'd into a server, it's a pain in the ass to cd around. Well, I wrote a small bash script that should make this much quicker.

It operates on the habit of bookmarking. If you're at a directory that you want to bookmark, you do this:

aguo@unix1:~/Documents/10605$ j add

Type an alias for the current working directory, or ^C to quit:

ml

and "j ml" -> ~/Documents/10605, etc.

To install, just save my .jrc file as "~/.jrc", and then ". .jrc" in your ".bashrc".

https://github.com/mallochine/jarvis2

--Alex Guo

Cygwin -- Installing curl.exe

Hey all,

In this post, I'll be writing about a problem that took me a couple hours to resolve, especially since there was no other post that ran into the same problem.

So, this is the problem -- curl doesn't work:

$ curl
/usr/bin/curl.exe: error while loading shared libraries: cygcurl-4.dll: cannot open shared object file: No such file or directory

But I tried everything -- installing curl via setup.exe, apt-cyg install curl, etc. The source of the problem was that I was using the wrong terminal. I had a 64-bit machine, and I was using the 32-bit version of Cygwin. While working, it was incompatible in very important ways.

Even if you tried installing curl via setup.exe, you'd still have the wrong version of curl:

$ ls /bin | grep curl
curl.exe
cygcurl-3.dll

So you need to use the 64-bit Cygwin terminal, which will give you this output:

$ ls /bin | grep curl
curl.exe
cygcurl-4.dll

Designing an online program to read from a stream

Consider the following scenario: you have a program that reads data from stdin.

Simple enough, right? But it's very easy to end up with a bad design, simply because, well, where do you start? What does this program really look like?

The idea is to have a Worker object handle each line, which will pass the line to a Parser object. The Worker will ask the Parser if the data is ready to be collected, and if it is, then the Worker will collect the data and do something with it.

This is the rough program skeleton, in Java pseudo-code:

	class SomeParser {
	parse(line) {
	// Do some parsing work.
	}
	dataReady() {
	// Easy way for others to figure out if the parser has data that's ready to be collected.
	}
	}
	class SomeWorker {
	handle(line) {
	SomeParser.parse(line);
	// For why dataReady() make good design, see Note 1.
	if (SomeParser.dataReady()) {
	data = SomeParser.getData();
	// Do some stuff with data.
	}
	}
	finish() {
	// Write this only if you want to print something out once
	// your program is done reading from the stream.
	}
	// other helper functions here...
	}
	int main() {
	while stdin.hasNextLine():
	SomeWorker.handle( stdin.nextLine() );
	// Only if you need to output additional data at the end of the stream.
	SomeWorker.finish();
	}

view raw gistfile1.java hosted with ❤ by GitHub

That's the main outline. Everything can be easily built on top of that.

Notes:

1. dataReady() is very good design, because it allows the parser to get the data in blocks of 2 or 3 or k at a time. If you were to handle the data immediately after each line was parsed, then you could only handle a block of 1!

Thanks to:

1. Anthony Guo for reading drafts of this post.