Picture Credit: spatiallyadjusted.com
Today there was a debate among the #clouderati about whether OpenStack should standardize their APIs around AWS APIs. Even though two years back I had an opinion that standardizing around AWS API will be good because of interoperability advantages, I have since changed my position and I now feel that it is too early to standardize the APIs. The back and forth debate on Twitter brought in enough emotions, marketing pushes, etc.. I thought I will do a quick post to list out my arguments against standardization (keep in mind I am not arguing against standardizing on AWS API but rather I am arguing that we don’t standardize now against any API). Here are some of the reasons why I am against standardization.
- We are still very early in the cloud services maturity. Any efforts to standardize at this point will stymie innovation. Using Simon Wardley’s (one of the biggest proponent of standardization around AWS APIs) own technology lifecycle model, it is my argument that we are still either in the innovation part or early custom development part of the curve for cloud infrastructure. We can worry about standardization when we get to the products stage of the lifecycle. There is no need to short-circuit this natural evolution and productize everything around AWS APIs. Update: Simon is arguing that my application of his lifecycle model is not right. He says it has to be applied for infrastructure as a whole and see cloud as the utility phase. He may be right in the application of the lifecycle model but I still feel that it is too early to standardize the cloud infrastructure because it is a very tiny piece of the entire infrastructure market.
- AWS APIs are proprietary and Amazon hasn’t open licensed them. Even though we could reverse engineer the API and take advantage of DMCA safe harbor provisions, it is still not worth the risk when we take into account the fact that we are still in the early stages of cloud infrastructure technology. I will be suspicious of Amazon’s future intentions in spite of them having a large ecosystem. Just ask some of their ecosystem players how they feel after being poached by Amazon.
- Even though AWS has wide reach and many want compatibility with AWS API, there are third party tools that ensure interoperability. One of the criticisms against the use of third party tools is that it works on lowest common set of features. I do agree with this argument. However, the pain levels among the customers hasn’t reached a point where there is large scale demand for standardization. Whenever the pain levels becomes intolerable, there will be enough market pressure calling for standardization with all the players sitting across the table and working things out (much like the election year payroll tax cut in US politics). Till that point, any efforts to standardize the APIs is premature.
- If reliance on third party tools is the case of compromising on lowest common set of features, premature standardization on AWS API is a case of stymying innovation around what AWS APIs can allow.
- One of the arguments put forward for standardization around AWS API is the guestimate putting AWS revenues at 1 Billion. Even though it is a big number if we just consider the cloud segment, it is peanuts when we consider the entire IT. One cannot standardize the future of IT based on such small marketshare.
- If PaaS is going to be the future face of IT, why bother about standardization at the infrastructure now stymying any potential innovation up the stack?
Anyhow, I always tweak my models based on the real world data, as opposed to economists approach of wanting real world data to fit their models, and I am ok with changing my opinion (didn’t I do it already on this topic?) if there are enough evidence to demand an early standardization. Till I see any need for it, I will keep arguing against any standardization around AWS APIs NOW.
Krish – good points. Although I embrace many of your concepts, I’d like you to consider the following:
1. Some of the AWS API’s are fairly stable (as measured by their own documentation and History Log) and are better fits for the standardization process. No need to do all at once.
2. Each API isn’t an ‘all or nothing’ proposition. Imagine ‘AWS Native Edition’ (driven by Amazon) and an ‘AWS extended edition’ that would accomodate community requirements.
3. The legal considerations around the API have scared some buyers & resellers away from AWS. This could be fixed once and for all.
4. The PaaS guys (myself included) who write software to an IaaS layer hate writing multiple bindings across each cloud. Waste of time and money.
The argument of ‘should they or shouldn’t they’ is irrelevant. The only question is, which companies will be participating in taking it to a standards body. And those discussions are already underway…
Jeff
Fair points. re: your point 3, I haven’t seen any willingness from Amazon in encouraging any standardization process. I have my concerns on this.
re: your point 4, it is not always necessary to write bindings for individual infrastructure platforms. We could use the available libraries too.
I’ve watched this unfold over the last two years[1] and must say I find “the place we are in today” (re: debates around ‘standardizing’ the API) both sad and fully predictable.
As I mentioned in a comment to Mark’s post some time ago [2], this is, IMO, the wrong conversation to have. Certainly ten years of WS-* has provided enough evidence that a single static set of “function-style” APIs is not only unworkable in a distributed network that spans both space (the planet) and time (many years of required inter-operability); this attempt at a static API is also inadequate.
Efforts should be focused on designing an “Open Stack” media type; one that exposes all the functionality available today *and* allows for supporting new funcitonality as it comes online; including allowing vendors to add support for “custom extensions” w/o upsetting the rest of the community or breaking existing implementations.
I would hope a decade of wandering through the weeds, attempting to mimic the design of local network APIs and forcing on the Internet would have convinced us all that there *could* be another way. But alas, it seems the folks with the power to “make this happen” are unable to make the next step; to move on to building a system that actually *works* at planet scale.
Instead, this is devolving into a strong-arm match to see “who’s API wins.”
bummer.
[1] https://twitter.com/#!/mamund/status/26403488642
[2] http://www.markshuttleworth.com/archives/765#comment-368536
Agree with your views Krishnan. I am assuming that by standardization you are indicating that other vendors/projects like OpenStack adopt AWS APIs (and not some standard body proposing these as reference).
While this may have some short-term advantages, I don’t think this approach works in the long term. We are at a point where the innovation on the public clouds is starting to happen with more and more vendors announcing public clouds. Given this, if we go with adopting AWS APIs, how does one go about adding new functionality to that ? Should we wait for Amazon to add that functionality. If not, what prevents each vendor taking a different tangent for each additional functionality not offered by AWS today. And what would happen to the “AWS Standards” then?
As long as the underlying models are similar, other open source projects like DeltaCloud can fill in the API syntax gap. When we talk about standards, let us get the models (or as Mike said, Media Types) right first.
Krish,
My feeling is Amazon doesn’t need to be part of the AWS standardization process, they’re going to do what they’re going to do. Continuing to wait for them to belly-up to the table is silly. The important activity is standardizing the extensions and over-rides. In regard to your comment that we can use 3rd party libraries as a bridge to multiple clouds, that’s great in concept but hasn’t worked for us in real life. The libraries remain immature and fall apart when you hit the complex scenarios.
Mike – I’m confused on what you’re proposing. I understand your concerns around the AWS SOAP/WSDL API… but are you suggesting that the AWS Query style won’t work for your needs?
Thanks,
Jeff
Jeff:
Thanks for the question.
What I am saying is that, by adopting a message-based approach to expressing the API (instead of using URIs and object-based payloads), we can use features of AWS or any other interface in a compatible way; allowing both consumers and producers to optimize the interactions over time.
The _details_ of AWS API are not of much concern to me; the notion that the community is attempting to decided on *just one* static interface is what I see as the problem.
I would encourage everyone to have a look at the fog ruby cloud service library http://fog.io. They have a nice way of standardizing the core functionality of the various clouds under a similar API but allow extensions to support the extra facilities of clouds. They support cloud formation for amazon AWS. Chef use Fog in there integration with cloud services.
Most “big” cloud vendors and government purchasers aren’t going to touch Amazon’s APIs with a ten-foot pole; not only are the IPR issues huge (MSFT can still hear “triple damages”), but what version of the APIs would you base things on? And, of course, any changes by Amazon would nullify the advantage of basing your standard on them.
Amazon could change that by submitting their APIs under clear terms to another organisation, but no sign of that yet.
All of that said, Krish you’re absolutely right that the day is young and we still need to sort out what “cloud” really is before we lock it down in stone. There needs to be convergence, but you can’t hurry that along.
Mike, making things format-centric would indeed be helpful, but at the end of the day, you do need to get the bits onto the wire, so *some* amount of API is necessary.