Software Engineering and Agile

Agile Invasion

One of the cool thing in being MSE/MSIT-SE graduates is that people suddenly start asking you a lot of questions regarding Software Engineering. It is unfortunate that I don’t always posses an authoritative answer, but these great questions make me realize how people perceive SE and expand my horizon in general.

A friend asked: ‘What about this Agile thing? How can we ensure quality of work doing Agile instead of Software Engineering?’. After I did my puzzled face for a couple of seconds, I realized his concerned is about increasing popular Agile practices taking over the traditional software development methodologies. He wondered about how could we ensure quality without having proper documents and thinking about every move up front. Explaining Agile manifesto and some of the practices to him, I came to realize an important fact that…

A Shift of Energy

“Agile is not to replace Traditional Software Engineering, it is another way to do Software Engineering” — I replied

Think about it for a second. Look at all these Agile practices. It almost seems like Agile is proposing a total different way for software development. My argument is that the core values is still there, it’s just that the underlying activity is different. Here’s what I think: Agile is for born for coders: those people who love to write, craft and nurture codes rather than the grand plan and beautiful wall design. It is deliberately made such that the coders would enjoy doing SE practices by shifting their coding energy towards doing all these good practices. That is why it is gear towards executable artifacts rather than static documents that serve contractual and archiving purpose. For managerial aspects, Agile empowers the engineers to plan and estimate on a need-basis rather than view it from 10,000 ft strategic view.

Here are few examples that I can think of:

• Requirements: Specifications >> User Story
• Planning: All-Up-Front(tm) >> Iterative Planning
• Tracking: Earned Value >> Burndown Chart
• Estimation: Wideband Delphi >> Planning Poker
• Design: Design Documents >> JavaDoc
• Quality Assurance: Manual Test Case >> Automated Unit / Integration Test  / Test Driven Development
• Integration: Build Instructions >> Build Script
• Deployment: Deploy Instructions >> Continuous Deployment

To mock the Agile Manifesto: “While we value practices on the left, we think that the practices on the right deliver the same core value, but a whole lot more fun to do.” — Honest Coders

The Hidden Message

Imagine that you just get this young hotshot who just graduated and ready to code on all cylinders. Would he prefer “Write a deployment instruction” or “Write an automated deploy script”? What would you prefer?

Since the practices are geared toward, well, geeks. The key to success is to make sure that your team enjoy coding and implicitly loves to work with technologies. It is my wish that everybody coming in this business do share this same passion, but that is the assumption that has been proven wrong a few times before. Since the energy shift from papers and proof-reading into the more automated, more code-oriented, we need people who is passionate about these thing in the team to make it enjoyable.

As your team gets better at these Agile practices, they will soon forget that they are doing the boring SE stuff and should “… [reach] hyper-productivity by developing software as if playing a game” (of coding and collaborative meeting) [ref].

An Agile approach to Life Interruptions

A cost to stay ‘in’

Shig, One of my best MSE graduates pal, has this interesting idea of stop reading RSS and Facebook at all for a month. His idea is to get more time for himself to pursue what he would like to work on. Along with that idea, he also cited a few researches which point out that the younger generation brains are hard-wired to skip contents as there are too many interrupts in our life.

Right now I am at about 30,000 ft. in the air, deprived of my usual constant stream of contents. There is this big book in my hand that I am almost finished off. I can say that I read the whole book on the air, knowing that I won’t be doing my usual news updating, friends connecting stuff while cruising over beautiful US west coast. In matter of fact, I couldn’t quite remembered the last book that I was being able to complete. This is why I can’t help but agree with Shig about the time we gave up in pursuing leads from the next coolest thing we found in RSS, just to satisfy our unending curiosity.With that said ….

Embrace Changes

… I don’t totally agree with cutting ourselves off the grid just to make a time for yourself. Since when that the information dictates our action but not the other way around? How do we know that what is coming is not going to be something we are going to value? And .. to sound like an Agile Evangelist: ‘Changes are inevitable, instead of ignoring it, embrace it and manage it’.

Instead of completely ignoring all these wonderful technologies, I’d say SCRUM it! Make it such that our life is a little project in itself. It is okay to have change requests for yourself, as long as you consciously deal with them and did not give up your sprint goal just to tackle these changes. So here’s what I’m gonna do:

• Set my personal goal for the week. My own little sprint goal.
• See how much in a week I would like to be doing something (24×7 – work – sleep)
• Come up with ‘tasks’ for the week, which will include
  • Blogging time
  • Tasks related to my weeks goal: ex. Finished chapter one of this book, learn to use python to extract stocks info, moving to Seattle etc.
  • Use my morning time to check emails, RSS feeds and Facebook news of the day.
    • Stop push notifications for all above. I will check them myself in the morning.
  • If there’s a topic I would like to spend more time in, put it in ‘reading/cool stuff’
  • Check out cool stuff in the ‘checking out cool stuff time’ or free time.
  • At the beginning of the week, see if the cool stuffs I picked up in the week earlier changes anything about what I want to do.

Re-purpose an old technique

No matter how you look at it, this is the classical solution that Iterative Development tries to solve. Makes you focus only on a handful of goals while managing emerging stuff as changes that will be considered. Avoiding deciding what to do while doing. I recalled Marsha of the SEI mentioned that she used TSP to coordinate people in writing books, so why can’t we get a little Agile into our life?

Will tell you more how it goes in a few weeks. Any cool thoughts? See you in the comment!

อะไรก็ไม่รู้

ลองมาคิดๆดูว่า เพลงแต่งงานก็น่าจะมีเพลงรักที่น่ารักๆ อยู่บ้างนะ ก็เลยนึกถึงเพลง อะไรก็ไม่รู้ ของ โน้ตอุดมมาทันทีเลย เพลงนี้คอร์ดง่ายๆ 4 คอร์ด กับเมโลดี้ก็ไม่ค่อยมีอะไรโดดเด่น เป็นไม่กี่เพลงที่ชอบเนื้อหาเพลงไปด้วย แถวนี้ก็ไม่มีคนมาร้องให้ เลยต้องร้องเข้าเอง (เราเตือนคุณแล้วนะ)

Prototype Songs – Release 1

An update for my friends Aliya and Abin. I promised to send them something for a long time, but haven’t got a chance to. These are the songs I have quickly come up with. They are by no means finished and polished. Just a quick update So that you guys know what’s coming (Release early, release often!). Hopefully there aren’t too many bugs in the final release :p

A Whole New World

Can you feel the love tonight

Love me tender

Loving You

When You Say Nothing at All

Bonus: มั๊ง (Thai song)

Software Engineering >> Processes

One of the misconception that I overheard so many times is that software engineering is all about “boring process stuff only managers care about”. As I explore deeper into SE, I see much much more than just management (process) aspect of it. In general, I could cite a few definitions of SE from textbooks, but here’s my own version of it:

All those stuff that gears us towards writing software that delivers on time with at least defects as possible

A few of the “stuff” I am talking about are:

• Software Development Management and Processes – Know how close you are to completion. Increase predictability of project time. Examples:
  • Planning, Tracking and Oversight – Know where you are in the project
  • Risk Management – Know your odds of success
• Requirements Engineering and Modelling: Understands what your customer needs ( More often enable them to understand what they need!), decrease functional defects. Examples:
  • Formal Modelling - Mathematically infers properties for your software (We guarantee this to be deadlock free!)
  • Prototypes – A mock up replica of finished system
• Architectural Design – Satisfies the quality attributes of the system to build ( decrease defects, big ones )
• Quality Assurance – The most misunderstood about in the group. A lot of fun stuff emerges from this. Examples:
  • Languages Constructs – Surprising to some, but data types was born to prevent you from mixing your variables, is it not?
  • Static Analysis – Using a program to verify correctness of other programs. Very hot research topic
  • Peer Review/Pair Programming – Friends helps friends eliminate silly mistakes
  • Automated Testing
  • Continuous Integration – Guaranteed that all the builds works

It is sad oftentimes people think Software Engineering is reserved only for those who just finished their Bachelor in CS, and does not know coding enough to dig deeper. Truth is that folks who study SE handle themselves pretty well in, and love, coding. I believe that you need to code for a while before you can appreciate all good practices SE has to offer. SE won’t help you find new algorithms, but helps you decides which one to choose based on what is needed ( is Big O always the most important thing?). SE won’t help you learn new languages, but let you decides which one makes more sense to use ( Does readability always the most important thing?). I do believe that the worst code in the world is the perfect code for the wrong thing. Because you already wasted time, with no return. And I strongly believe all achievements in SE would prevent (or trying very hard to prevent) you from doing so.

Bottom line: I think SE is all about being effective. Folks who love constant personal/team improvements would definitely love SE. We will do whatever it takes to make great software — one that actually ships on time. Be it the process, be  it the tools, be it the organization culture to change, be it the new languages. We constantly pursue the art of making the right software, and making development fun in the process.

Condition-Consequence Risk Statements

I remembered somebody told me that good software managers are good risk managers. I tend to agree with that specific statement. The dramatic amount of delays we faced usually results from us overlooking some tasks (checklists and estimation techniques help in this one) or us unable to understand our project risks and deal with them accordingly (such as your vendor going out of business..).

Risk Statements

Okay, great! Lets get some risks on the table, then.

1. What if my company decides that they are not going to fund this project any more?
2. What if Microsoft drop all supports for .NET platform?
3. What if the PC I have got a virus and I have to send them to repair for 3 days?
4. What if my project manager is hit by a train?
5. What if the planet is hit by an asteroid, wiping out all major continents, humanity ceases to exist and all that we know as a civilization is gone?

I could go on and on (I am very good at these things) but I can imagine people eyebrows raised more and more from (1) to (5). What is it that makes (1) more likely to be happening than (5)? Can you point out which ones are actually useful?

In matter of fact, none of them has any value for risk management at all. They are not based on a factual context of imminent threats. In risk management, we do not seek to understand and mitigate every possibility of the universe. We just need to know what events are likely to happen, justified by some knowledge of current status, and plan to deal with them accordingly.

Condition-Consequence Risk Statements

Let me try this again.

1. The company memo indicates a transformation into an IT consultant, this project might not be funded anymore.
2. Microsoft begins to phase out their platform supports prior to 2007, we might have to rework the UI module as it relies solely on .NET.
3. The company firewalls is currently disabled, We might have to face with outage from a virus.
4. My project managers has got to cross this unguarded and poorly lit train tracks every night, he might get hit by a train.
5. (okay… probably let this one go)

Now it seems to hold more water, doesn’t it. What actually happens here is that you put factual elements (the underlined part) into risk statements. Essentially what you said is: “Based on this fact, I believe this is going to happen”. The underlined part should be hard, cold, non-controversial fact. The part that follows are your predictions based on the stated facts. Assessing their impacts and probability is another day’s story.

And remember to keep it short. Less than 30 words (just like the example) is more than enough to capture your concerns. It always looks redicoulous to me when people tried to elicit their risks using huge pile of forms, resulting in one phone book of risks that nobody cares about. This is bad. This is actually even worse than no risks management, because at that point you already wasted time on collecting them.

While this kind of risk statements seems overly simple by traditional standards, the method is actually in used and published by NASA (Page 6) and its Goddard Space Flight Center (Page 8, signed May 2009). Yes, you are looking at the agency that had sent people to the moon, landed rovers on mars and shot satellites into deep space. I am sure that they have got a lot of stuff that could go very very wrong. Yet they seems to live in peace with their risks that is managed based on facts rather than series of pessimistic opinions. Unless your projects goal is to explore the Pegasus galaxy, this should be sufficient for your needs.

A Question to Agile Evangelist

The reason why I bring this up is I have not seen Agile projects manage risks before (at least not explicitly), and the topic is not so popular among Agile community. They might discard this idea quickly probably because of the bulkiness of traditional risk documentation methods.  Given this lighter-weight risk statements, it might begins to make sense for Agile teams to put 2-3 of these statements up with their iteration plan. This should helps the whole team understands what are the things to watch out for, not just managements. Then we could revise it every iteration to be up-to-date. Any thoughts? See you in the comments.

ถ้าหาก

ก่อนน้องเดฟจะไปเรียนต่อเอก SE เราก็ต้องใช้น้องเดฟให้คุ้มซะหน่อย ฮ่าๆ

มีคนบอกว่่าเล่นเปียโนแล้วหน้าเอ๋อ คราวนี้ลองยิ้มให้กล้องแล้วนะ

อันนี้ลองเร่งเสียงขึ้นมานิดนึงกว่าอันที่แปะไว้ในเฟสบุ๊ค แต่ยังไงเปียโนก็ยังเบาอยู่ดี -*- คราวหน้าคงต้องเอากล้องไปไว้ใกล้ๆ หน่อยละ

Continuous Integration in 15 Minutes with Hudson

My motto: If something goes wrong, I want to know it now. Continuous Integration is always one of the most useful Agile practice I adore. It remove the phrase “But it was working on my machine .. yesterday!” out of the equation by putting a central environment that the build must run on. Most people might be more familiar with Cruisecontrol, but the lack of snappy setup and configuration for newbies is one of the big downsides it has. I was just introduced to Hudson recently and really likes the simplicity it offers. I recommend this to teams that start using CI, because it is really easy to set up.

CI Basics

First, let me talk real quick about CI, to summarize its benefits and allow us to see how Hudson comes to play with that. Continuous Integrations in the wild usually follow these steps.

1. One of the Developers in the team checks in code to a code repository (SVN, Mercurial, etc).
2. Hudson sees that there’s a new code and checks out the code in its own workspace
3. Hudson run the build script (usually Ant, or the increasingly popular Maven) that is checked in the code. The build script usually follow these steps ( I will surely blog more about this later ).
   1. Checking
   2. DB
   3. Generating code
   4. Compile
   5. Run automated test
   6. Run code analyzer tool (Static Analysis, Code Metrics , Test Coverage)
   7. Deploy the compiled code on test server
4. Hudson gather build results and compile a report (compile pass/failed, number of test passed/failed)
5. Hudson publish a report on the website, and send out (emails/rss/twitter) to the developer that breaks the build
6. The developer has to take responsibility to fix the error and commit in the fix

Installing Hudson

1. Hudson is distributed a form of (.war) file. Download the latest one from their website.
2. Prepare any Java Servlet container. Tomcat, would do. For windows, I suggest installing it as a service.
3. Use Tomcat Manager to upload and deploy hudson.war
4. Goto http://localhost:8080/hudson
5. On the left menu, goto “Manage Hudson”, then “Configure System”. Enter email server information

Configuring the build

1. Each build project is called jobs. There’s a link to create one once you browse to hudson, click on it.
2. Enter the job name. Choose “Build a free-style software project“.
3. In Source code Management section, choose the one you use. Provide the path to your project source repository.
4. In Build Triggers, select how often we want Hudson to grab the source code. In regular standalone projects, we usually do it every time a developer checks in code (“Poll SCM”). The parameter is a schedule in cron format. Every fifteen minutes ( */15 * * * *) should suffice for most projects. Note that this does not mean that we will build every 15 minutes, it means that Hudson will be checking for changes in the SVN every 15 minutes.
5. In build section, select add build step. Choose “Invoke Ant” and provide the target name. If the build.xml is not at the root of the repository, choose “Advanced…” and provide the ant file path.
6. In Post build actions, select Email notification and enter a list of everybody in your team. This will make Hudson send out an email to everybody when there’s a build broken.

Establishing Fix-the-Build Culture

This is probably the most important thing you need to do, even more important than the tool itself. This is when you get team members together. Introduce their new neighbor, the build server. Then establishing the team rule of “You break the build, you fix the build”. This does not mean that the junior member will be entirely on their own when problems arise. But he needs to be responsible for grabbing the right people when something happens and everybody is accountable for their commits.

Making such rule would ensure that you would have the near-shippable quality product every day. Well… something that compiles and automatically tested at least .

คนข้างล่าง

เพลงนี้อัดกับน้องเดฟไว้ตั้งนานแล้วล่ะ ชอบมากเลย น้องเดฟก็เสียงดีมากๆ เลยด้วย โพสไว้ตรงนี้แหละ เผื่อวันหลังมาค้นดูจะได้หาง่ายๆ นะ

OpenUP

OpenUP is what you get when you try to fit Agile practices in to the Unified process framework. The author(s) claim that the resulting process is an Agile one, a statement which might raise some eyebrows when looked closely at the specific practices prescribed. This paper serve as an introduction to OpenUP and provide rationales why OpenUP is considered Agile according to the Agile manifesto.

Using OpenUP

We can start with reading about OpenUP from the official site. While this could be a little overwhelming at first, following these step should help us make sense out of OpenUP a little faster.

1. The obvious starting point is the Introduction Page. There are two levels of planning in OpenUP. First, there’s a project plan that roughly define iterations and milestones to accomplish for all phases. Then we move on to iteration plan (comparable to SCRUM’s sprint) which pretty much defines what is to deliver in each increment. Iteration plan consist of multiple work items (tasks) to be completed.
2. Read about all practices in general. There are two types of practices in OpenUP: Management and Technical. Majority of these should be very familiar to those who use Agile in their everyday’s life.
3. Skim description of roles
4. Open up a section about work products ( artifacts ). See the example of project plan and iteration plan under “Project Management” section and take a quick look at them. Note that each phase are subdivided into iterations with goals as an acceptance criteria for that iteration.
5. Read about activities in each phase. These are obscurely placed under the “Delivery Processes” section. Click on the name of each phase to open it up. Then click on the (Phase name)[1..n] picture to see what’s inside.
6. Now we should be able to see an overall view of each phase. We are likely to refer back to this when we are doing the project/iteration planning. Click on the activity name (eg. Initiate Project) to see the step details, roles involved and work products expected.

Differences from XP/SCRUM

In principle, OpenUP looks like any Agile processes with a little more detailed practices defined. However, those who exposed to XP/SCRUM flavor of Agile (likely to be majority of us) would quickly see these differences.

• Phases (Inception, Elaboration, Construction, Transition) are defined – This obviously comes from an influence of Unified Process.
• Roles and responsibilities are defined
• Work products that are expected as a milestone document (Project plan, risk list, architecture notebook, etc.)
• Explicit risk management
• Encourage early architectural design
• Project plan with estimated dates for completion

My take on OpenUP

• I really like the idea of having a project plan. As much as we hate it, deadlines are part of life. The first thing you need to answers to management is the one million dollars question (sometimes literally!) of “when it’s going to be done?”. This might be challenging to answer with typical SCRUM backlog as things are being added all the time, so we really have control of only what’s in each iteration but not the overall plan for the project.  Having the project plan laid out a big picture of what’s going to be done, help us track if the project is really going to be finished in time and help management make the right decision dealing with big changes. This is what (conceptually) happened when I used OpenUP:
  
  [After Construction 2 product review]
  Customer: Yeah, I really like what you guys did for the graph here. This reminds me that we need to do a major overhaul to the input screen before this one. We also need some kind of batch processing for all these incoming data as well. As I look at it, I don’t think this program would make sense without these two things.
  PM: That might take a while to do. What do you guys think?
  Dev #1: My ballpark — 10 person days.
  Dev #2: Yeah sounds about right.
  PM: (pulls out the project plan) Let’s see, we are at the end of construction 2. We have construction 3, 4 and 5 to go. What if we move the profile page planned in Construction 4 to ‘nice to have’?
  Customer: No, I need that, too. I think we can push the form attachment later, though.
  PM: (move things in project plan). So, I’m putting in the two new things right here in construction 3, pushing everything down. Move attachment feature to nice to have for Construction 5. How does this sounds?
  Customer: Sounds good to me.

  As we can see, the PM still embrace changes. He just clearly presents what are the impact of changes to the deadline for the customer, so that they can make intelligent decision together.

• Architectural design early in the project. We can debate about this idea vs. the YAGNI principle influenced by XP. My opinion is that you should get the big obvious thing right because cost of changes is likely grow rapidly for these monsters. Fundamental changes to the system are not so easy to fix: “Can you do it like what you did having the central hub, but make it decentralized and secured?” — Good luck refactoring that! Might need another project just to make this kind of change. Early architectural decisions also help you identify the rigid parts from moving parts and help you contain those moving parts where chaos can safely happened.
• OpenUP assumes that some magic hands are doing the infrastructure work for you (how nice!). Truth is not a lot of people would have that luxery of having the project folder, templates, build servers set up. Even if we have it, somebody need to contact those support guys anyway. The developer role are pretty free during inception phase and would be ideal to take this job.
• Having roles give team members purposes. These tend to make meetings more effective as we consider different perspectives coming out from different roles (think six thinking hats…).
  
  Customer: Hey, what do you guys think if we need to track a record of each sales?
  Analyst: Is that only for the first time they pay or for all recurring payments as well? How do we track credit cards as well as purchase orders?
  Architect: We need to interface to those tracking vendors via web service. I’m afraid this might impact on the performance that we guarantee to have when doing the payment.
  PM: We might not be able to get to the UI changes we planned for the last iteration.
  Dev: Might be a nightmare to refactor all those billing logics!