Build? Buy? Both?
Here are some ideas that might help you decide.
The vendor sales team: Please give us your money
Every 3rd party vendor has a bunch of really nice people with friendly smiles and company issued button down shirts. While they likely have great technical expertise and are easy to get along with, make no mistake about what their job is: to get you to spend money. It actually seems like a pretty neat job, you just sometimes have to take statements these folks make with a bit of skepticism.
The typical initial presentation this group will give includes an overview of all the cool features their product provides, an architectural layout of a basic installation, and a slide showing an explosion of company logos representing their install base. While this is a nice starting point, these first conversations rarely go into enough depth to allow you to make an informed decision. It's a good idea to get a round of these from all the vendors you are considering before going deeper with any one of them. For open source or build from scratch alternatives, create a presentation that goes to the same level of depth so that all possibilities are given equal consideration.
How do you go deeper in a way that will let you make a decision?
Building assessment criteria: Spreadsheets abound
Start by listing out the requirements that each group that will be involved in running the end solution has. Business constituents will likely have a list of features, IT staff might enumerate specific technologies they may or may not want, support could want to know about logging capabilities, and management may worry about the financial stability of the vendors being considered. Think about the different lifecycles of the various parties involved and total implementation costs (machine acquisition and any post-sales consulting) as well.
Inevitably, your criteria will get placed into some kind of spreadsheet scorecard that can be used to objectively measure the different options against one another. Each criterion gets a row where it is scored with some numerical range and weighted relative to it's overall importance. The decision makers in your company fill out scores after each detailed vendor presentation and whoever scores the highest wins.
While this approach is a good one to take it's important to remember that it is a guide, not an absolute measure. A spreadsheet like this can help synthesize large amounts of information for easy viewing but that doesn't mean that the weightings or scoring are perfect or that non-quantifiable criteria might lean your decision on way over another. Be sure to bring along a healthy dose of common sense when it comes time to make a decision.
With criteria in hand, how do you go about getting enough detail from the vendor to score it?
Use Cases and Being a Little Mean
If you simply hand over your criteria scoring spreadsheet to the vendors and say, "Can your product do this?" the answer will either be "Yes" or "It's in our next release." A better approach is to look at your criteria and come up with step by step use cases that represent as many as possible. Then ask the vendors to demonstrate how each scenario can be accomplished using their product. You still have to look out for the "It's in our next release" answer, but you'll get a deeper look at the product.
Give the vendor very little time between when they see your use cases and they demonstrate how each is fulfilled with their offering. This prevents some behind the scenes magic or that "special beta version" from appearing during the demo that just happens to meet your exact need but isn't in the full release yet. Shrinking that window of opportunity puts extra pressure on the vendor sales staff to come up with a demonstration, but you get a truer picture of the out-of-the-box capabilities.
- Remember that those nice vendor sales folks are trying to sell you something
- Think about different lifecycles and total deployment cost when creating your assessment criteria
- A scoring spreadsheet is nice, but not a complete substitute for common sense
- Build use cases from the assessment criteria and give vendors very little time to implement a demonstration to get an accurate reading on what their product can do