Software Engineering

Archive for the ‘Software Engineering’ Category

Resharper by JetBrains

October 21st, 2008 by Sameer | No Comments | Filed in Software Engineering

What can I say.. I am impressed.. Very impressed. Try it out now – Resharper. Commercial license is only $349.00
What did I achieve? Few small things at the moment, improved “extract this into a method” functionality, suggesting when to use “const” for strings, site-wide analysis for problems, convert method to static. Very cool..
Here’s a list of features

Add parameters easily with Resharper

I was able to take a piece of code inside the function, say AccountsReceivable.GetReferenceNumber(“abc001″) and convert that to a parameter of my helper function just by highlighting it and selecting “introduce parameter”. It then went ahead and updated all references to this function to pass the AccountsReceivable.GetReferenceNumber(“abc001″) as if it was a variable

Create variables to replace common values

Here’s another case. Imagine you have some reference to something like dr["status"].ToString(), you can “Introduce Parameter” and it will ask you if you want to update just that one line, or update ALL references inside that function wtih the new variable name… pretty neat!

Also have you encountered the situation where some variable is declared near the top of a function and then its used somewhere in the middle… and you are stuck wondering whether you can get rid of it or you can modify the code without breaking it ? Well Resharper will allow you to join the declaration with the intiialization, so its re-partnered to its vaue

Easily worth the $350 price tag.

You will get the most bang for your buck in the first month or so anyway, use it to clean up and spiffy your app, and then try not to let it descend back into chaos.

Automated .NET testing with WatiN

October 20th, 2008 by Sameer | No Comments | Filed in .NET articles, Software Engineering

This software is super cool. WatiN – Automated Web Application Testing!

Here’s how you can use it. Download it. Download the test recorder (it’s not perfect, but its pretty good). Create a trivial test case so you know it’s working. Then build on that test case. It will output code for NUnit, MBUnit, or even VS2005Test

Load up your VS IDE, then execute it from Visual Studio 2005 or 2008 Test Runner. It will give you the results right there! Super cool or what??

With this framework I was able to create some pretty simple test SUPER fast. – Login test, check for correct page loaded. Then I tested some shopping cart logic (adding items, removing items, clearing cart.)

In conclusion you can effectively create tests with WatiN. Give it a try. It’s compatible with practically every testing framework (including Fitnesse) that you probably use.

Follow The Leader

July 17th, 2008 by Sameer | 1 Comment | Filed in Software Engineering, Work Related

In order to succeed as a team, in any sort of team, you have to follow this basic principle, which has been applied and maybe is accepted universally.

The principle is simple.

Appoint a Leader
Leader makes council with the team
Leader makes a decision
Team supports leader in his or her decision

Its that simple. In this way, an organization, a team, a family, or a company can move forward. Every decision that you need to make, is done in this manner. The team will work together for the best solution, but in the end the leader needs to make a decision. Once that leader makes a decision, the team needs to move forward WITH the leader.

This means, the leader doesn’t necessarily dictate, but he or she has gathered input from the team and made a decision. Then they will have to choose a solution and go with it. If the team continues to argue and fight over the decision, progress will be slow. I believe this applies in families too. There has to be a decision maker in the family, for example that is appointed for financial decisions, and then having a discussion or gathering input from the family is great, but in the end one person has to make the decision, and the family needs to be supportive, even if they don’t all agree with it. However, this applies in normal circumstances and there are caveats. There might be some cases where it would be unethical for team members to support a plan if its morally wrong or it goes against everything inside them and they feel it’s a plan headed for disaster.

It pains me that time and time again I see this happen – the leader makes a decision and the team continues to question him at every step of the way – “Why are we doing this”, “Why are we doing this”, and “why are we doing this” instead of understanding that they were appointed for this role (they might be your boss for example) and they are ultimately responsible for the decision, you need to do your job and support them. I have seen some cases where I questioned my manager because I didn’t see the wisdom in the decision he made, but in the end it turned out he was right and it was just my foresight wasn’t as far as his… So I learned my lesson, be patient, and be a team player.

As a team member, sometimes I have to swallow my feelings and say, “okay I don’t think this is the smartest decision and my idea is actually better, but I will go with you on this”.

From a developers perspective, you can think of this as requirements. Your boss gives you the requirements, you implement it. How you implement is up to you, but you won’t be able to proceed if you start arguing about the requirements. Requirements are requirements, once they are agreed upon, please continue.

So in normal circumstances, if you want to succeed, get a leader, and help him with his decisions, but in the end.. Respect the decision he makes and go with it. (He or she, that is).

The best team isn’t necessarily the one with the best players, it’s the one that plays together the best.

How to Waste Millions Of Dollars With Outsourcing (or Make Millions)

July 10th, 2008 by Sameer | No Comments | Filed in Software Engineering, Work Related

To management, the idea of outsourcing sounds very sexy…. The idea of producing the same content (code, or what not) at 1/2 or 1/3^rd the cost is almost a wet dream for management, if I may be so bold. Even though it sounds great in theory, it’s actually a very tricky function to master. Here are some things I have learned with my outsourcing experience.

Keep in mind I am not discussing the outsourcing style of passing requirements and getting the end product complete. I am discussing the style of hiring outsiders and working with them on a daily basis.

You have to start by looking at what is the purpose of outsourcing. Is it to save money? Or is it to improve quality? Or is it so that your team can focus on other things? Get this straight first before going any further. My points below are in the context that you are a software company (or at least do some software development) and you are considering outsourcing to save money and cut costs.

From friends, I know that some very popular companies outsource, such as E-Trade Canada, Accenture, and recently the new online T.V. web site Hulu which outsourced its development to China.

Before you start outsourcing, have your process solid – i.e. have regular scrum, know how much code you are generating each week, and so on. It’s very important that you have some idea of costs for developing software for your local team. If you have no idea, you won’t be able to see if you are really saving money or wasting it.
Know velocities of individual team members so you can measure cost effectiveness of your outsourced work.
Build your estimation skills. Read Joel’s article on estimation and his second article on estimation (which is really a promo for his bug tracking software but still worth reading) and Steve McConnell’s book on Software Estimation (highly recommended, very easy to understand).
Get smart/able/competent guys. This can make or break your outsourcing project. If you are going to get developers that need baby sitting, then hire a baby sitter on their end to clean up their code, otherwise you are going to waste your valuable resources fixing and re-fixing and re-fixing their code. In this case you might not be actually saving money.
Review their code. Someone on your side is going to have to review their code to make sure that they aren’t purposely obfuscating it in order to secure their jobs in the future. I have seen a Flash application that was built in this manner, the team overseas purposely messed up the code in such a way that it would be difficult for others to continue where they left off.
Learn from those who have done it before. If not, you are going to mess up big time, in many ways. Might be worth getting a consultant who has been successful with such projects. Another idea is to find someone who has connections "back home", and go there to see how some of the shops work.
Turnover is really high in India/Bangladesh/ and so on. This is because jobs start at really low salaries (like $200 a month) and go upwards to like $3000 a month (comparable to working in Canada or USA). You will need to find a way to solve this problem. Somehow you will need to get them to commit that guys will not drop like flies. This is so important because there is always an upfront cost to learn an application, and it becomes more as the complexity and lines of code increase.
Consider a cross-cultural learning program, you send some people there for a while, they come over here for a while. A lot of big companies do this. It’s almost a must.
For the team overseas, its important to spend your valuable time together in the beginning to ask lots of questions and understand the requirements as much as possible, in case there is a task that you run into questions, then leave it and work on something else.

Hamid, Axosoft CEO claims that Outsourcing is for Dummies. I think this isn’t true in all cases, as I have been able to apply outsourcing successfully on some small projects. However, it all depends on the case, and for building complicated software with a (geographically) fragmented team, you may just end up proving his point.

Updating Your Web.Config From Each Build with CruiseControl

June 17th, 2008 by Sameer | No Comments | Filed in .NET articles, Software Engineering

Let’s say you have CruiseControl.NET all set up nice and hunky dory. Now you love the fact that it labels your each build, and you want to somehow show that from your project. Here is how you can get your CruiseControl.NET project build number into your Web.Config. This concept can be applied to update any file.

It’s super easy. You will need to add an exec block to your ccnet.config

<exec>
<!-- Auto increment web.config build number -->
<executable>E:\Userdata\CruiseControl\Tools\MergeWebConfigValues.exe (my custom tool)</executable>
<buildArgs>"E:\Program Files (x86)\CruiseControl.NET\server\Main.state" e:\userdata\cruisecontrol\Dev_Main\Web.config</buildArgs>
<baseDirectory>e:\userdata\cruisecontrol\Dev_Main</baseDirectory>
<buildTimeoutSeconds>15</buildTimeoutSeconds>
</exec>

Now all you need to do is write the MergeWebConfigValues.exe

What does it do?
1. It reads the CCNET state file (provided by argument 1) and grabs the last label from there. (The state file is an XML file)
2. It then goes and writes to your Web.Config and updates the build number in there. (or writes to any specified file you like)

Lastly, update your code to read this value by adding some version or about page to read your Web.Config

You now instantly know what version your code is.

Unit Testing on Your Database

May 22nd, 2008 by Sameer | 2 Comments | Filed in .NET articles, Software Engineering

When unit testing on your database, you will run into a common problem.

Rolling back.

So you want to do some unit tests, and then you want to reset your database back to the nice squeaky clean version that doesn’t have half failed unit tests.

So how can you do this?

There are many ways to achieve this.

The best way I found (requires Win XP SP2 or Windows Server 2003) is to use Roy’s Unit Testing Rollback Attribute. Simply inherit his class, add a "DataRollback" attribute, and you are good to go. Using some complicated Interception logic and Enterprise Services (COM+) it rolls back all the database work that was done. It’s super easy to implement. Here is some sample code that shows you just how easy it is. You just have to download XtUnit (an extension to NUnit) to do this. (Full source code available)

using System;
using System.Data;
using System.Collections.Generic;
using System.Text;
using TeamAgile.ApplicationBlocks.Interception.UnitTestExtensions;
using NUnit.Framework;
using System.Data.SqlClient;
using Microsoft.ApplicationBlocks.Data;

namespace DBTest
{
    ///<summary>
    /// Test roll back functionality.
    ///</summary>
    [TestFixture]
    public class RollbackTest : ExtensibleFixture
    {
        [Test, DataRollBack]
        [Category("Database")]
        public void TestInsert()
        {
            //this method will be performed inside a COM+ transaction
            //this requires windows XP SP2 or better
            //Windows Server 2003 works as well.

            string strCnn = "your_conn_string";
            Guid random = Guid.NewGuid();
            string sqlI = string.Format(@"insert into log4net (message,date,thread,level,logger) values ('{0}',getdate(),'{1}','Debug','Test')", random.ToString(), System.Threading.Thread.CurrentThread.GetHashCode());
            SqlHelper.ExecuteNonQuery(strCnn, CommandType.Text, sqlI);

            string sqlS = string.Format(@"select count(*) from log4net where message='{0}'", random.ToString());
            int rowcount = (int)SqlHelper.ExecuteScalar(strCnn, CommandType.Text, sqlS);
            Assert.That(rowcount > 0, "Cannot find {0}", random.ToString());
        }
    }
}

This test passes successfully. What does that mean? The insert and select worked perfectly fine. After that, I did a query and found the database to be clean. So the rollback worked too.

There are also other ways to skin this cat (i.e. to achieve this goal). One is to use Spring Framework and extend their Unit Testing class (AbstractTransactionalSpringContextTests), and they will handle rolling back everything. Here is an article on this topic, but unfortunately I was not able to make it work for me due to some odd reason. Here’s hoping you have better luck. If you don’t have any transactions, and your code is wired to use Spring, it’s still also very easy, you just need to call TransactionManager.Rollback.

You can also try to achieve this using Nested Transactions if you have existing transactions implemented via Spring. But then you have to set up checkpoints and stuff like that.

I have some other ideas on how to achieve this that I will post later, God willing.

Unit test your life!

April 29th, 2008 by Sameer | No Comments | Filed in Software Engineering, Work Related

If you are not unit testing your code, chances are you are not unit testing in your life.

If you aren’t unit testing, START now! At the very least, do some “manual” unit testing in your code. How can you do this? Well, try running your code on a very basic case. Then try a bit more complicated case. Then another, then another. If you are smart, you are saving these cases using a testing framework like NUnit. If not, well at least you can have some confidence when your manager comes that you tried it comprehensively and that it’s not going to crash on you while you are showing it to him, or even worse, in a demo to the team or to your big boss.

I recently ran into some problems in my life which I managed to solve amazingly well by doing “unit testing”…

First problem, my DVD burner was going awfully slow. I had some complex and messy setup including an external IDE card, two burners, two hard drives, and all I know is that at some point in time something went wrong and it started going really slow. What I don’t know, is how it happened.

Second problem, I was doing some video encoding/rendering, and for some reason it was doing something bizarre and the application VirtualDub kept looping over and over and would never end encoding the file. Again, I don’t know what happened.

How did I solve these problems?

UNIT TESTING!!

For the first problem of the DVD drive. I removed everything from my PC and set up a very basic system which included 1 HD, 1 Burner, etc.. Then when I found this wasn’t working, a quick check online and I resolved the issue which was incorrect DMA settings. It was trying to send all the data through the CPU (PIO mode) instead of directly to the burners, which was causing a massive slowdown. Once this worked, I quickly put together my system again, and checked each case (HD on same IDE channel as Burner, on separate IDE channel, and so on).

With the encoding problem, again, I was very confused, but by unit testing the situation, I was able to resolve it. How did I do that? I tried encoding on a different machine, reinstalled the software, etc, etc, and it was still having problems.

And finally when I started from scratch, I removed the batch encoding, I removed the DiVX processing, and so on, and then made each test pass. Once the test passed, I added another level of complexity, until finally I figured out that VirtualDub was looping infinitely because I had the “segment AVI file” option enabled. I don’t know why this was the problem, but by unit testing, I was able to resolve it.

Lesson to learn? Unit testing (if you can call it that) can really help you solve such issues. Start from the base case, and slowly work back towards what you need. After each case, write down the results.

Microsoft felt strongly enough about Unit Testing that Visual Studio 2008 has built in unit testing (Wahoo!). As well, it integrates nicely even with NUnit or MbUnit (Don’t ask me how, though).

Five Points To Improve Your Estimation

March 28th, 2008 by Sameer | 2 Comments | Filed in Software Engineering, Work Related

Software Estimation is tricky business. You are often confronted with complicated technical (or even non-technical) work and asked “how long will it take?” on the spot. You are given fuzzy requirements (or no requirements) with ambiguous definitions and have to work with a code base that is best described as “chaos”. How can you give an accurate estimate with such a difficult environment?

I recently read an excellent book called “Software Estimation: Demystifying The Black Art"

Here are some amazing points that you can use.

1) We should not be pressured to reduce our estimates. Stick to your estimates – whatever they are. If management wants to reduce them, that’s fine, but stick to whatever you put in the first place. Managers trying to reduce estimates is a very silly thing to do, because by looking at the track record of our software industry, you will find that we consistently underestimate, not overestimate! So if you are pressuring your developers to reduce the estimate, you are asking for trouble. Better thing to do would be to see if the estimate is well founded, based on task level estimates, or is the estimate based on unjustifiable assumptions, etc.

2) Underestimating is MORE expensive than overestimating. So if you cannot estimate accurately, lean towards overestimating. Underestimating has an exponential additional cost, whereas overestimating has a potential of a linear extra cost due to Parkinson’s law. This is because underestimating results in lots of other problems such as having extra meetings to justify why you are behind schedule, having extra meetings to decide which features to cut, added stress on developers trying to meet tight deadlines, code having less quality in order to ‘just finish it’ so that developers can go home, having customers get angry at not-ready releases, having other teams who are waiting for the code to slip their schedules too, etc.. If you overestimate, then Parkinson’s law can kick in and developers will ‘fill in the empty time’, or as they say ‘work expands to fill the available time’

3) Task level estimates are best done by the developer who will be doing the work. A task level estimate is defined as a low level estimate for a particular item of work. For example, for building a workflow engine, a task level estimate is “2 hours to add the save button that will commit the changes to the database”.

4) With small teams (say under 5 people), estimates are most accurate when done bottom-up (i.e. from the developers). What this means is your complicated formulas for estimation are not so helpful in such environments.

5) There are many other ways to estimate such as ‘estimation by analogy’ (i.e. by looking at a similar project that was built), and so on, but I won’t explain those right now. The trick in general is to try to quantify as much as possible and leave subjectivity out of the picture. So if you can find out how similar your project is in terms of number of lines of codes, number of features, etc, you will have a better estimate. As well, don’t be fooled into thinking that more “estimation knobs” will give you a better result than less “estimation knobs”. I define an estimation knob as some sort of criteria you are using to estimate, such as “size of team”, “team programmer skills as a percentile of the industry”, “how many burgers they had for lunch”, etc.. It looks like it would give you a better result but in reality the estimate becomes more and more subjective as you add more “knobs”

As well, developers are more likely to try to meet their estimates if they are the ones who gave it, rather than if its dictated to them from above.

Lastly, estimation is a skill we can learn and improve over time. You will find that over the months, your developers will get better at estimating based on past experience.

In summary, stick to giving your developers clear requirements, lock them down, and then get them to break down the requirements into task items, and then estimate for those task items.

How Random is your Random??

December 5th, 2007 by Sameer | 1 Comment | Filed in .NET articles, Software Engineering

How random is your random?

Computers are very deterministic. What that means is that you put something in, you get something out. In order to get computers to perform "randomness", it is very difficult.

Why is this important to understand? Because we want to write our code properly, if we depend on the random function for some security purpose, such as for generating passwords, we are actually putting security holes in our application without realizing it.

In .NET Using RNGCryptoServiceProvider would give you much better random results than just a Random.Next()

http://msdn2.microsoft.com/en-us/library/system.security.cryptography.rngcryptoserviceprovider.aspx

However, in order to truly randomize your number, you would have to do something like use data from customer mouse movements, or something wierd like that. Alternatively you can use a hardware random number generator such as the one Intel created that uses thermal noise to generate real random numbers

To realize just how complicated this really is, lets look at this quote from the Pokerstars web site on how they shuffle the cards in their software:

SHUFFLE

"Anyone who considers arithmetic methods of producing random digits is, of course, in a state of sin." – John von Neumann, 1951

We understand that a use of a fair and unpredictable shuffle algorithm is critical to our software. To ensure this and avoid major problems described in [2], we are using two independent sources of truly random data:

* user input, including summary of mouse movements and events timing, collected from client software
* true hardware random number generator developed by Intel [3], which uses thermal noise as an entropy source

Each of these sources itself generates enough entropy to ensure a fair and unpredictable shuffle.
Shuffle Highlights:

* A deck of 52 cards can be shuffled in 52! ways. 52! is about 2^225 (to be precise, 80,658,175,170,943,878,571,660,636,856,404,000,000,000,000,000 ways). We use 249 random bits from both entropy sources (user input and thermal noise) to achieve an even and unpredictable statistical distribution.
* Furthermore, we apply conservative rules to enforce the required degree of randomness; for instance, if user input does not generate required amount of entropy, we do not start the next hand until we obtain the required amount of entropy from Intel RNG.
* We use the SHA-1 cryptographic hash algorithm to mix the entropy gathered from both sources to provide an extra level of security
* We also maintain a SHA-1-based pseudo-random generator to provide even more security and protection from user data attacks
* To convert random bit stream to random numbers within a required range without bias, we use a simple and reliable algorithm. For example, if we need a random number in the range 0-25:
o we take 5 random bits and convert them to a random number 0-31
o if this number is greater than 25 we just discard all 5 bits and repeat the process
* This method is not affected by biases related to modulus operation for generation of random numbers that are not 2n, n = 1,2,..
* To perform an actual shuffle, we use another simple and reliable algorithm:
o first we draw a random card from the original deck (1 of 52) and place it in a new deck – now original deck contains 51 cards and the new deck contains 1 card
o then we draw another random card from the original deck (1 of 51) and place it on top of the new deck – now original deck contains 50 cards and the new deck contains 2 cards
o we repeat the process until all cards have moved from the original deck to the new deck
* This algorithm does not suffer from "Bad Distribution Of Shuffles" described in [2]

PokerStars shuffle verified by Cigital and BMM International

PokerStars submitted extensive information about the PokerStars random number generator (RNG) to two independent organizations. We asked these two trusted resources to perform an in-depth analysis of the randomness of the output of the RNG, and its implementation in the shuffling of the cards on PokerStars.

Both independent companies were given full access to the source code and confirmed the randomness and security of our shuffle. Visit Online Poker Random Number Generator for more details.

[2] "How We Learned to Cheat at Online Poker: A Study in Software Security" – http://itmanagement.earthweb.com/entdev/article.php/616221
[3] "The Intel Random Number Generator" – http://www.cryptography.com/resources/whitepapers/IntelRNG.pdf"

Here is an article about how to shuffle a deck of cards: http://www.codinghorror.com/blog/archives/001008.html?r=31644 and in one of the links it explains a big security hole in their random number generation and how it could have been used to leverage thousands of dollars from players.

Here is a snippet of how to get Cryographically safe random numbers:

This will fill in the 8 bytes with a crytographically strong sequence of random values.

byte[] salt = new byte[8];
RNGCryptoServiceProvider rng = new RNGCryptoServiceProvider();
rng.GetBytes(salt);

Get A Build Process Now!

October 31st, 2007 by Sameer | No Comments | Filed in Software Engineering, Visual SourceSafe

This article will tell you how to get a build process. It uses CruiseControl.NET to automate the build. You can use any other tool you like including a .BAT file, but CruiseControl will do just fine. It’s very easy to set up, it uses an XML configuration file and does most of the difficult job like connecting to your source control, applying labels, and getting your files to build for you.

1. Install CruiseControl.NET

This software is just great. Its totally free. Really it is a sophisticated batch file (.BAT file). It allows you to set up a build process fairly easily. It allows you to execute arbitrary processes and check the return codes and then report the results to your developers (or managers, or what not).

For example, you set it up to monitor your source control system (say SourceSafe or even a local folder in your file system) every 30 minutes, and then perform a build on it if there is changes.

It will report who made the last set of changes, what they did (check in, check out, delete, etc) by grabbing that information from your source control and also display the comment from the user that checked it in. It also labels it in the source control as UNVERIFIED (build process failed), or by incrementing the build number (again, this is configurable).

2. Install CCTray (its a little icon that sits in your tray and interfaces to the cruise control server to notify you when the build(s) are broken)

If the build does not compile (according to the procedure you set it up to run such as MsBuild, or Visual Studio, or executing some process) it will turn red and the team will all know that the build is broken and must be fixed. One person can volunteer to fix this build. Included with CruiseControl. Or you can just use the web dashboard instead:

3. Perform more advanced setup on it.

You can configure it to do just about anything. Our CruiseControl does a nightly build, runs NDOC (Alpha version, discontinued, you can use SandCastle instead if you like) to get the latest documentation, and also we have a separate project that runs some database unit tests against 5 or 6 QA databases. The NDoc step is a bit tricky if you are running a “Web Site Project” because by default the “Web Site Project” does not create XML documentation files when compiled and requires modifications to your Web.Config. Getting NUnit to run requires some advanced setup, because NUnit requires you to have a copy of the Web.Config in the local folder where the tests are run. You can also run a bunch of fun stuff like Simian (Similarity Analysis, Duplicate Line counter), Fitnesses (another testing framework), NCover (Unit test coverage), etc.. It can also send out emails on each successful build if you like (technically it can do cartwheels if you have an exe file that you can make it call to do the cartwheels

)

4. Involve QA and the rest of the team to start using the build numbers and relying on CruiseControl to verify the site compiles. Also you can start to add more unit tests to your code and strengthening the quality of your product, and then using NCover to verify how much of your site is tested. Not to mention you can also stick FxCop in there! FxCop will analyse your code according to Microsoft Best Standards and create a report for you on bad code that is not according to their naming conventions, or bad code that is not using SQL Parameters (and could have SQL Injection vulnerabilities)

More to come soon. In summary – CruiseControl makes a world of difference. Get it now.

As Jeff Atwood wrote, "The F5 key is not a build process"

Archive for the ‘Software Engineering’ Category

Resharper by JetBrains

Related Reading:

Automated .NET testing with WatiN

Related Reading:

Updating Your Web.Config From Each Build with CruiseControl

Related Reading:

Unit Testing on Your Database

Related Reading:

How Random is your Random??

Related Reading:

Get A Build Process Now!

Related Reading:

Bookmarks

Meta

Author

Pages

Categories

Archives