Database guru Andy Leonard shares some insight into Twitter's woes.
the bottleneck is [likely] occurring in the database. It's those pesky queries. And the users causing them to be executed. It's <insert-your-favorite-excuse-here> - everyone and everything, except the designers and architects who built a non-scaling solution.
Second - although I could be wrong about this - I would wager good money that Twitter doesn't need a DBA. I've seen / heard / experienced this before. They have a talented team of application developers who have built large applications in the past and never once paid a database professional. Look at the money they've saved! </sarcasm>
Cost-cutting leads many PHB's to assume that developers can build out databases and stored procedures that are production ready.
That can work, to a point.
Developers can create database schemas and stored procedures, but they rarely have the skill or patience to produce optimized data structures or queries.
In my time, I've seen some scary things that developers have put together.
For example, one commercially available survey software that I won't name here, had code in it that retrieved of tens of thousands of rows from the database only to filter them out in a for/each loop in C#.
Needless to say, that ended up being a big bottleneck in terms of CPU use and network bandwidth.
The sales rep swore up and down about the scalability of the solution, when, in reality, it didn't take much traffic to bring the data center to its knees.
The moral of the story: trust your developers to write code, not manage huge amounts of data.
Depending on your developers, you may want to be careful about how much to trust them with the data.
Experienced DBAs can be a real pain to work with and that's what makes them great.
They demand referential integrity and schemas and won't let you make one little change to the schema to simplify what code you have to write.
Experienced database professionals slow down project development. We get in the way. We muck about with stress tests and bulky architectures and referential integrity and schemas and the like. Who needs us? We're an unnecessary expense.
Or are we?
As I've learned, DBA's just don't just enforce data integrity rules to make developers' lives more difficult, they do it to ensure that the database will be fast and reliable.
Yes, Virginia, you do need a DBA.
In case you think you don't, let me ask you three questions:
Where does your application retrieve data from?
Where does your application store data to?
Where does your application's transactions take place?
A malfunction in your database impacts everything else in your application.
Just ask Twitter.