Why is RCP, EMF, p2, databinding... so hard to learn?

There are some interesting discussions on my recent blog entry on "How to explain EMF?". The debate is between those who find it hard to learn EMF and those who have a hard time seeing the difficulty. I think there is a general pattern behind this debate. The same arguments could be made for almost any non trivial technology/framework inside and outside eclipse (EMF, RCP, p2, databinding, [your "favorite" hard to learn technology],...).

Why do many people really suffer form the steep learning curve and others master the same technology easily?

I think we have to step back for a moment from any concrete technology and we should to look at how the technology gets created and how learning and applying of the technology works.

The creators of the technology/framework have a deep knowledge in the particular domain the framework solves. They have created several software systems where they saw some reoccurring patterns. They realize by putting the common parts into a framework they can dramatically increase their productivity, because the work is then reduced to assemble the pieces and write some "glue code" (in some sense they create a kind of domain specific language for that problem). As long as the creator(s) of the abstraction are the only users of the system, there is no real problem and they can show dramatic increase of productivity, because they can focus on the business relevant part of the system.

Now comes the next phase, where others adopt the technology. Those early adopters are usually people with a similar experience background as the original authors of the framework. They have typically written such systems by hand or created their own framework and they see the value of the technology. For them, learning the system is almost effortless because they understand the underlying problems and the solution provided by the cool technology. Good candidates of successful adopters are typically consultants using the technology in multiple projects. They report impressive increase of productivity. Let's call them expert users.

Now another category of users, the normal users wants to use the technology. They see the impressive results created by the expert users. The expert users tell them how effortless this is and how quickly you can create solutions.This is where the problem starts! The expert users understood the problem before they started, they understand the solution in depth and they reapply the technology to many projects. The normal users typically have a single project where they want to apply the technology. They might have a vague understanding that the new technology fits to their problem. The "documentation" build from expert users for expert users make no sense, because there is a lot of background knowledge and experience missing. Normal users have to put a lot of effort into learning and understanding a complicated system just to apply it once.

If the normal user manages to use the technology to solve his problem, his solution is often "suboptimal", because he might be happy to have found just one solution. The expert would see that it is by far not the best possible solution, because he has a much deeper understanding of the solution space. OTOH, the normal user might have a much better understanding of his specific problem domain and the expert might propose a bad solution, because his limited understanding of the problem domain.

Let me give you an analogy: Some weeks ago, my mother told me that she has a serious problem, because her old typewriter broke. She explained me how they do their tax-declaration: my father hand-writes tables with income and expenses and my mother types them with the typewriter. Well, they have a computer (which they use to read email and browsing). The solution is clear: a spreadsheet is all the technology they need to optimize the process. Me "the expert", could just tell them that they can be much more productive by using this "new and cool" technology. All, they'd have to do is to enter the numbers directly into a spreadsheet and with a few more mouse-clicks and a bit of excel programming he would even get some cool graphics and analysis.... Do you think I did that? No way. They are in their mid 70ies, they have done it "their way" all their life. How steep would their learning curve be? Does that mean they are stupid? My father is extremely good with his hand written tables, he uses colors and creates cool graphs by hand. He has an extremely good understanding of the money flow. He even knows exactly how much money they spend on me and my brothers from our birth to now. I wish I would have 10% of his knowledge on where my money flows.

In this example we see very well the difference between "expert users" and "normal users" of the technology. This technology would work really well for me and would not work at all for my parents. And even if I would try to teach them, I would have to use a very different language to explain it to them than I would use it explaining it to you. I would have to explain them what a spreadsheet is and so on. Then I could ask them if they prefer open-office or Microsoft office. Because they have absolutely no idea, even this simple this choice would stress them a lot.... Once they would start with an empty spreadsheet, there would be so much choices and decisions to make that they would be totally paralyzed. Would examples or documentation help? Well somehow, if an example does exactly what they want to do, but then it is almost the solution. Does that mean that spreadsheets are complicated technology? Well, it depends on your background. I think it would be more efficient for me to help them doing their tax declaration and prepare a spreadsheet and every time they do it, sit with them and help them. But I think the way they do it is OK for them, all I need to do is to fix my mothers typewriter.... If they were young and they had a lot of numbers for their tax declaration, it would make sense for them to learn the technology....

OK, I stressed this example enough... What does this mean for the "complicated" technologies around eclipse?

• There is a big danger that a great technology gets a "bad reputation", just because "normal users" want to apply it and they find it the technology incredibly hard to learn and to apply.
  
• As expert user, you can cause a lot of damage it you tell some unexperienced users to use a technology without guiding them. And in some cases it might be better to tell them not to use the technology if they are not "ready". For you it might be the right choice, for them it would cause disasters at all levels.
  
• Learning "complicated" technologies might not be worth the effort if you want to apply it once.
  
• Ask yourself if it is more efficient to hire an expert than to learn the technology. Especially if you want to apply the technology only once in a relatively simple way. In this case the expert can be 10-100 times more efficient than you.
  
• If you are a normal user and you still want to learn the technology, be prepared that is takes time and it is a painful process. In order to make educated decisions you have to become at least "Competent" (see the Dreyfus model of skill acquisition) or needs "Conscious competence" (see Four stages of competence).
  
• Maybe expert user and normal user should team up to write documentation and create example. For experts it is often very difficult to understand the problems of normal users and for normal users it is impossible to come up with good examples and documentation.
  

This post is a bit longer than my normal posts and I kept is as draft for a while but I decided to post it. It is a bit like with complicated technology: you can't explain it in two sentences ;-)....