So.. I've been quite busy lately. I took a position back in September to come on as the IT Director for a company in Mesa, which means my commute times are about 3.5 hours a day. Because of that, my free time got cut down severely.
Lately, I've been working hard on getting everything running sanely and safely in their environment, which is quite a task. I am in charge of 6 developers and 2 sysadmins, and there is more work than any of us can sanely do.
So code-wise, what have I been doing? Well, I wrote a CDN server..
The basics of it are pretty simple:
I needed a server architecture that can host static data in a way that allows for simplistic transfer and synchronization across multiple sites and hosts.
Here's how I did it:
Front-End: NGINX, with Pound as the load balancer. Using Heartbeat to fail over front-end tasks between systems.
Back-End: Python-Django for APIs, file controlling, registration, logging, and testing. Handled by the Supervisord/Gunicorn.
Controller - Register/Remove Server
Controller - Accept File
Controller - Status Checks
Slave - Accept File
Slave - Status Checks
Layout: 2 Controllers. One with 1 floating IP a piece. 4 Clients - Each serving static files via NGINX. 1 Database Server - Percona MySQL 5.5. Used mostly for event logging.
Load testing: Breaking point is well over 10,000 requests per second. Being as the front-end only leverages nginx, which is quite a bit faster than Apache.
Back-End: While I was testing and "loading" data, it took approximately 30 minutes to process and propogate 26,000 files in a little over 30 minutes, at 100% accuracy. I'll take it.
Basic Idea: User uploads a file. File gets shot to the controller. Controller uses the url pathing information to figure out where the file needs to live on disk. Saves the file, creates a checksum, and logs the information to the DB. Then, controller pushes, in a staged manner, to all the slaves. If a slave isn't behaving, it gets removed from rotation and shut down. Easy Peasy. Warm-up time for the servers takes about 10 minutes from bare-metal. 3/4 of that is transferring the files.
Things I'm working on outside of this project:
Using a structured distributed hash-based system to store files across the environment similar to hdfs. Instead of storing everything on every server as I do now, I want to make sure the files are on at least 51% of the servers, and to make sure that requests that are routed to the server that the file isn't on get re-routed to the right ones. Basically, I'm scrapping the idea that I used in the work project, starting over, and open-sourcing it. The code will still be python-based, but I'll be using webapp2 instead of django, for the sake of speed. I'll also be doing host caching and want to change from a push-based model to a pull model using a distributed mesh to propogate information on new files similar in idea to zookeeper or doozer.
I know I am standing on the shoulders of giants with this project, but I want to see if I can find a way to do it better and faster, with less overhead. The open-sourcing of the project, which I will be doing on my own time, is just the icing on the cake.
Of course, I need to manage doing this some time outside of work, while running interference for 6 developers and 2 sysadmins.
That's okay. The idea's in my head. That's a good place to start.
Considering enacting a new policy that I won't accept a python web application if it won't run on a raspberry pi.
Anyone who's known me longer than a few days is very much aware of the fact that I'm not a terribly big fan of Windows. Because of this, and the sadistic nature of most companies I've worked at in the past, I almost ALWAYS end up maintaining the windows projects. Why? Because from what I've seen in the past, they're almost always the most screwed up projects, and I was the new guy. So what do you do when you get that bad project? Code Review! Figure out how the bad project works. From there, operationalize it.
You've done your code review, right? Most of these questions should be pretty easily answered. I can't code .NET, but even I can read it halfway decently.
Here's some questions you can ask yourself:
The big thing you need to remember is that the more times you do it, the easier it gets. Fill this stuff out as early as humanly possible. Make sure to keep it updated as the project evolves. The better your documentation backbone, the easier it is to maintain. These are things that both you, as a systems engineer, and your IT team will appreciate in the long run.
Oh, and companies: Please stop giving your bad code off the bat. You're just setting them up to fail.
Next post I'll give more information on breaking down each part of the runbook, starting with the application overview.
But I know this is true...
I wrote an application that checks world of warcraft realm status. It was actually pretty fun. I got 3 apps submitted in total, but that was my most fun one. Of course, I submitted them to the windows store, but none of them were actually accepted. Too many bugs.
So.. how did I do this?
First off, I grabbed a copy of the vm image:
Load it up in your favorite virtual machine and install visual studio express:
Finally, get the non-minified version of jquery from here:
Start a new project, add jquery to it, and start coding away.
Why did I pick world of warcraft to be my first one? I don't know, because I could. I'm a nerd like that..
Anyhow, you can check out my code doing the following:
git clone https://repo.antitux.us/antitux/js-tastetest.git
Play around with it and let me know what you think. I know it's broken and barely works. It builds, it tests, and it runs. That's all I care about.
Oh, one other thing: The CSS that they use is actually pretty darn clean. I like it a lot.