Error Guessing in Software Testing

Error-guessing in software testing can find number of faults that systematic techniques may be fail to attend. Test cases are derived from experience of where defects have taken place in the past or software tester has acumen as to where defects can take place in the future.

Error-guessing should be used as a ‘mopping-up’ technique or as a supplement to systematic techniques, but not as the first choice approach.

Some persons are genius in detecting defects thanks to intuition and experience. There is no procedure for a greatly intuitive and ad hoc process.

Main ideas:

  • make a list of likely defects or error-prone situations -> test cases
  • identify test cases associated with assumptions that may have been made when writing the specifications

See an example: “Binary Sort”

  • only one entry is in table
  • table size is power of 2
  • table size is (power of 2) +/- 1

See another example: “Sorting subroutine”

  • input list is empty
  • single entry in input list
  • all entries are of the equal value
  • entries have already been sorted

A special form:

Data-structure based testing Look at data structures used and consider cases related to the structures being used.

See an example: “Linked list”

  • zero components
  • only one component
  • one less than max quantity of components
  • max quantity of components

Requirement Verification

As a tester if you are involved in the requirement phase you need to know , what attributes you should be looking for in the requirements? How can you ensure quality of the requirements? Following checklist gives you information about the attributes which should be tested for every requirement. These are just a subset of attributes that as a tester you will be interested in.

a. Complete: Every requirement should be complete and give complete information on what is expected. For example, provide a shopping cart to the website is not a complete requirement. Complete requirement could be stated as “Provide shopping cart to the registered user who can pay using credit card and are within the EU region”

b. Correct: This is self explanatory, it is expected that every requirement specified in the requirements document will be correct. This can be ensured by having stake holders/client representatives in the team.

c. Precise, Unambiguous and clear:
Each item in the requirement should be clear and have single interpretation, for every member in the team; the meaning of each item should be understood and specifications should be easy to read. For example, response time of the application should be good. Now good can have different meaning for different people. Requirement should be stated as “With an anticipated load of 1000 simultaneous users, system should respond with in 4 seconds.”

d. Consistent: Requirement should not conflict with each other. For example if one requirement states that “Additional 2% charges should be applied for the people shopping out side US” and another says “Options to buy/sell should be disabled for the people who are not in US”, these two requirements are conflicting. Either one of them is wrong or not clear, and you should seek clarification from the stake holders or clients representatives.

e. Relevant:
Each item is pertinent to the problem and its solution. For example, if we are dealing with the problem of providing web interface to the existing retail operation, information related to staffing may or may not be pertinent to the problem we are solving. You should hunt for clarification on why this item is present in the requirements documents.

f. Testable: Every requirement should be testable, it should be possible to determine whether the stated requirement has been satisfied or not. For example, requirements like application navigation should be very good is very difficult to test. Testable requirement should be that “The user should be able to navigate from one page to any other page on the site with maximum three clicks”.

g. Feasible:
It should be possible to implement requirement with the available techniques, tools and resources within the specified cost and schedule constraints.

h. Free of Unwarranted Design Detail:
The requirements specifications are a statement of the requirements that must be satisfied by the problem solution. Requirements document, ideally should not implementation and design details.

i. Manageable: The requirements should be expressed in such a way that each item can be changed without excessive impact on other items. Even if there is any impact, it should be minimum and and it should be possible to identify dependencies in the requirements for impact analysis.

GUI Testing Checklist

Purpose of this GUI Testing Checklist is to help you understand how your application can be tested according to the known and understood standards for GUI. This checklist can give some guidance to the development and QE, both the teams. Development team can make sure that during the development they follow guidelines related to the compliance, aesthetics, navigation etc. but onus of testing GUI is on the QE team and as a tester it is your responsibility to validate your product against GUI standards followed by your organization.

This GUI test checklist can ensure that all the GUI components are thoroughly tested. In the first part of this checklist, we will cover Windows compliance standard and some test ideas for field specific tests.

Windows Compliance Standards

These compliance standards are followed by almost all the windows based application. Any variance from these standards can result into inconvenience to the user. This compliance must be followed for every application. These compliances can be categorized according to following criteria

  1. Compliance for each application
    1. Application should be started by double clicking on the icon.
    2. Loading message should have information about application name, version number, icon etc.
    3. Main window of application should have same caption as the icon in the program manager.
    4. Closing of the application should result in “Are you sure?” message.
    5. Behaviour for starting application more than once must be specified.
    6. Try to start application while it is loading
    7. On every application, if application is busy it should show hour glass or some other mechanism to notify user that it is processing.
    8. Normally F1 button is used for help. If your product has help integrated, it should come by pressing F1 button.
    9. Minimize and restoring functionality should work properly
    Continue reading

Performance & Security Testing Checklist

1.1 LOAD
1.1.1 Many users requesting a certain page at the same time or using the site simultaneously
1.1.2 Increase the number of users and keep the data constant
1.1.3 Does the home page load quickly? within 8 seconds
1.1.4 Is load time appropriate to content, even on a slow dial-in connection?
1.1.5 Can the site sustain long periods of usage by multiple users?1.1.6 Can the site sustain long periods of continuous usage by 1 user?
1.1.7 Is page loading performance acceptable over modems of different speeds?
1.1.8 Does the system meet its goals for response time, throughput, and availability?
1.1.9 Have you defined standards for response time (i.e. all screens should paint within 10 seconds)?
1.1.10 Does the system operate in the same way across different computer and network configurations, platforms and environments, with different mixes of other applications?
Continue reading

Web Application UI Checklist

Testing user interface for web application is slightly different from testing user interface of traditional applications. Irrespective of the web application there are certain things which should be tested for every web application. Following checklist will give some information on items that should be tested to ensure quality of the user interface of your web application.

1.1 COLORS

1.1.1 Are hyperlink colors standard?
1.1.2 Are the field backgrounds the correct color?
1.1.3 Are the field prompts the correct color?
1.1.4 Are the screen and field colors adjusted correctly for non-editable mode?

1.1.5 Does the site use (approximately) standard link colors?
1.1.6 Are all the buttons are in standard format and size?
1.1.7 Is the general screen background the correct color?
1.1.8 Is the page background (color) distraction free?

1.2 CONTENT

1.2.1 All fonts to be the same
1.2.2 Are all the screen prompts specified in the correct screen font?
1.2.3 Does content remain if you need to go back to a previous page, or if you move forward to another new page?
1.2.4 Is all text properly aligned?
1.2.5 Is the text in all fields specified in the correct screen font?
1.2.6 Is all the heading are left aligned
1.2.7 Does the first letter of the second word appears in lowercase? Eg:

1.3 IMAGES

1.3.1 Are all graphics properly aligned?
1.3.2 Are graphics being used the most efficient use of file size?
1.3.3 Are graphics optimized for quick downloads?
1.3.4 Assure that command buttons are all of similar size and shape, and same font & font size.
1.3.5 Banner style & size & display exact same as existing windows
1.3.6 Does text wrap properly around pictures/graphics?
1.3.7 Is it visually consistent even without graphics?

1.4 INSTRUCTIONS

1.4.1 Is all the error message text spelt correctly on this screen?
1.4.2 Is all the micro-help text(i.e tool tip) spelt correctly on this screen?
1.4.3 Microhelp text(i.e tool tip) for every enabled field & button
1.4.4 Progress messages on load of tabbed(active screens) screens

1.5 NAVIGATION

1.5.1 Are all disabled fields avoided in the TAB sequence?
1.5.2 Are all read-only fields avoided in the TAB sequence?
1.5.3 Can all screens accessible via buttons on this screen be accessed correctly?
1.5.4 Does a scrollbar appear if required?
1.5.5 Does the Tab Order specified on the screen go in sequence from Top Left to bottom right? This is the default unless otherwise specified.
1.5.6 Is there a link to home on every single page?
1.5.7 On open of tab focus will be on first editable field
1.5.8 When an error message occurs does the focus return to the field in error when the user cancels it?

1.6 USABILITY

1.6.1 Are all the field prompts spelt correctly?
1.6.2 Are fonts too large or too small to read?
1.6.3 Are names in command button & option box names are not abbreviations.
1.6.4 Assure that option boxes, option buttons, and command buttons are logically grouped together in clearly demarcated areas “Group Box”
1.6.5 Can the typical user run the system without frustration?
1.6.6 Do pages print legibly without cutting off text?
1.6.7 Does the site convey a clear sense of its intended audience?
1.6.8 Does the site have a consistent, clearly recognizable “look-&-feel”?
1.6.9 Does User cab Login Member Area with both UserName/Email ID ?
1.6.9 Does the site look good on 640 x 480, 600×800 etc.?
1.6.10 Does the system provide or facilitate customer service? i.e. responsive, helpful, accurate?
1.6.11 Is all terminology understandable for all of the site’s intended users?

Source.

Web Application – Functional Testing Checklist

1. FUNCTIONALITY

1.1 LINKS

1.1.1 Check that the link takes you to the page it said it would.
1.1.2 Ensure to have no orphan pages (a page that has no links to it)
1.1.3 Check all of your links to other websites
1.1.4 Are all referenced web sites or email addresses hyperlinked?

1.1.5 If we have removed some of the pages from our own site, set up a custom 404 page that redirects your visitors to your home page (or a search page) when the user try to access a page that no longer exists.
1.1.6 Check all mailto links and whether it reaches properly

1.2 FORMS

1.2.1 Acceptance of invalid input
1.2.2 Optional versus mandatory fields
1.2.3 Input longer than field allows
1.2.4 Radio buttons
1.2.5 Default values on page load/reload(Also terms and conditions should be disabled)
1.2.6 Is Command Button can be used for HyperLinks and Continue Links ?
1.2.6 Is all the datas inside combo/list box are arranged in chronolgical order?
1.2.7 Are all of the parts of a table or form present? Correctly laid out? Can you confirm that selected texts are in the “right place?
1.2.8 Does a scrollbar appear if required?

1.3 DATA VERIFICATION AND VALIDATION

1.3.1 Is the Privacy Policy clearly defined and available for user access?
1.3.2 At no point of time the system should behave awkwardly when an invalid data is fed
1.3.3 Check to see what happens if a user deletes cookies while in site
1.3.4 Check to see what happens if a user deletes cookies after visiting a site

2. APPLICATION SPECIFIC FUNCTIONAL REQUIREMENTS

2.1 DATA INTEGRATION

2.1.1 Check the maximum field lengths to ensure that there are no truncated characters?
2.1.2 If numeric fields accept negative values can these be stored correctly on the database and does it make sense for the field to accept negative numbers?
2.1.3 If a particular set of data is saved to the database check that each value gets saved fully to the database. (i.e.) Beware of truncation (of strings) and rounding of numeric values.

2.2 DATE FIELD CHECKS

2.2.1 Assure that leap years are validated correctly & do not cause errors/miscalculations.
2.2.2 Assure that Feb. 28, 29, 30 are validated correctly & do not cause errors/ miscalculations.
2.2.3 Is copyright for all the sites includes Yahoo co-branded sites are updated

2.3 NUMERIC FIELDS

2.3.1 Assure that lowest and highest values are handled correctly.
2.3.2 Assure that numeric fields with a blank in position 1 are processed or reported as an error.
2.3.3 Assure that fields with a blank in the last position are processed or reported as an error an error.
2.3.4 Assure that both + and – values are correctly processed.
2.3.5 Assure that division by zero does not occur.
2.3.6 Include value zero in all calculations.
2.3.7 Assure that upper and lower values in ranges are handled correctly. (Using BVA)

2.4 ALPHANUMERIC FIELD CHECKS

2.4.1 Use blank and non-blank data.
2.4.2 Include lowest and highest values.
2.4.3 Include invalid characters & symbols.
2.4.4 Include valid characters.
2.4.5 Include data items with first position blank.
2.4.6 Include data items with last position blank.

Web Application – Interface and Compatibility Checklist

1. INTERFACE AND ERROR HANDLING1.1 SERVER INTERFACE 

1.1.1 Verify that communication is done correctly, web server-application server, application server-database server and vice versa.
1.1.2 Compatibility of server software, hardware, network connections

1.2 EXTERNAL INTERFACE

1.2.1 Have all supported browsers been tested?
1.2.2 Have all error conditions related to external interfaces been tested when external application is unavailable or server inaccessible?

1.3 INTERNAL INTERFACE

1.3.1 If the site uses plug-ins, can the site still be used without them?
1.3.2 Can all linked documents be supported/opened on all platforms (i.e. can Microsoft Word be opened on Solaris)?
1.3.3 Are failures handled if there are errors in download?
1.3.4 Can users use copy/paste functionality?Does it allows in password/CVV/credit card no field?
1.3.5 Are you able to submit unencrypted form data?

1.4 INTERNAL INTERFACE

1.4.1 If the system does crash, are the re-start and recovery mechanisms efficient and reliable?
1.4.2 If we leave the site in the middle of a task does it cancel?
1.4.3 If we lose our Internet connection does the transaction cancel?
1.4.4 Does our solution handle browser crashes?
1.4.5 Does our solution handle network failures between Web site and application servers?
1.4.6 Have you implemented intelligent error handling (from disabling cookies, etc.)?

2. COMPATIBILITY

2.1 BROWSERS

2.1.1 Is the HTML version being used compatible with appropriate browser versions?
2.1.2 Do images display correctly with browsers under test?
2.1.3 Verify the fonts are usable on any of the browsers
2.1.4 Is Java Code/Scripts usable by the browsers under test?
2.1.5 Have you tested Animated GIFs across browsers?

2.2 VIDEO SETTINGS

2.2.1 Screen resolution (check that text and graphic alignment still work, font are readable etc.) like 1024 by 768, 600×800, 640 x 480 pixels etc
2.2.2 Colour depth (256, 16-bit, 32-bit)

2.3 CONNECTION SPEED

2.3.1 Does the site load quickly enough in the viewer’s browser within 8 Seconds?

2.4 PRINTERS

2.4.1 Text and image alignment
2.4.2 Colours of text, foreground and background
2.4.3 Scalability to fit paper size
2.4.4 Tables and borders
2.4.5 Do pages print legibly without cutting off text?

Source.

List with 10 quick attacks for web-based software

  • Overwhelm the software with invalid data
  • Overwhelm the software by pushing beyond boundaries
  • Perform server-process interrupts
  • Try unexpected combination of users and race conditions
  • Try the software from outside the firewall – from a public terminal, in secure mode
  • Have the team bang on a slow-running operation simultaneously. Use JMeter or LoadStorm if you must.
  • Try different browsers – Safari, Opera, FireFox, Internet Explorer 6
  • Look at the business rules the software operates on.
  • Internationalization
  • Common defects

Source: here.

Will come back with my own examples :P.