понедельник, 22 февраля 2010 г.

Top ten secret weapons for agile performance testing by Patrick Kua

Unfortunately there is no video movie available, but there are comments from Mia Ridge who participated in the conference where a presentation was presented:



1. Make performance explicit.
Make it an explicit requirement upfront and throughout the process (as with all non-functional requirements in agile).Agile should bring the painful things forward in the process.

Two ways: non-functional requirements can be dotted onto the corner of the story card for a functional requirement, or give them a story card to themselves, and manage them alongside the stories for the functional requirements. He pointed out that non-functional requirements have a big effect on architecture, so it's important to test assumptions early.

2. One team.

Team dynamics are important - performance testers should be part of the main team. Products shouldn't just be 'thrown over the wall'. Insights from each side help the other. Someone from the audience made a comment about 'designing for testability' - working together makes this possible.

Bring feedback cycles closer together. Often developers have an insight into performance issues from their own experience - testers and developers can work together to triangulate and find performance bottlenecks.

Pair on performance test stories - pair a performance tester and developer (as in pair programming) for faster feedback. Developers will gain testing expertise, so rotate pairs as people's skills develop. E.g. in a team of 12 with 1 tester, rotate once a week or fortnight. This also helps bring performance into focus through the process.

3. Customer driven
Customer as in end user, not necessarily the business stakeholder. Existing users are a great source of requirements from the customers' point of view - identify their existing pain points. Also talk to marketing people and look at usage forecasts.

Use personas to represent different customers or stakeholders. It's also good to create a persona for someone who wants to bring the site down - try the evil hat.

4. Discipline
You need to be as disciplined and rigorous as possible in agile. Good performance testing needs rigour.

They've come up with a formula:
Observe test results - what do you see? Be data driven.
Formulate hypothesis - why is it doing that?
Design an experiment - how can I prove that's what's happening? Lightweight, should be able to run several a day.
Run experiment - take time to gather and examine evidence
Is hypothesis valid? If so -
Change application code

Like all good experiments, you should change only one thing at a time.

Don't panic, stay disciplined.

5. Play performance early
Scheduling around iterative builds makes it more possible. A few tests during build is better than a block at the end. Automate early.

6. Iterate, Don't (Just) Increment
Fishbone structure - iterate and enhance tests as well as development.

Sashimi slicing is another technique. Test once you have an end-to-end slice.

Slice by presentation or slice by scenario.
Use visualisations to help digest and communicate test results. Build them in iterations too. e.g. colour to show number of http requests before get error codes. If slicing by scenario, test by going through a whole scenario for one persona.

7. Automate, automate, automate.

It's an investment for the future, so the amount of automation depends on the lifetime of the project and its strategic importance. This level of discipline means you don't waste time later.

Automated compilation - continuous integration good.
Automated tests
Automated packaging
Automated deployment [yes please - it should be easy to get different builds onto an environment]
Automated test orchestration - playing with scenarios, put load generators through profiles.
Automated analysis
Automated scheduling - part of pipeline. Overnight runs.
Automated result archiving - can check raw output if discover issues later

Why automate? Reproducible and constant; faster feedback; higher productivity.
Can add automated load generation e.g. JMeter, which can also run in distributed agent mode.
Ideally run sanity performance tests for show stoppers at the end of functional tests, then a full overnight test.

8. Continuous performance testing
Build pipeline.
Application level - compilation and test units; functional test; build RPM (or whatever distribution thingy).
Into performance level - 5 minute sanity test; typical day test.

Spot incremental performance degradation - set tests to fail if the percentage increase is too high.

9. Test drive your performance test code
Hold it to the same level of quality as production code. TDD useful. Unit test performance code to fail faster. Classic performance areas to unit test: analysis, presentation, visualisation, information collecting, publishing.

V model of testing - performance testing at top righthand edge of the V.

10. Get feedback.

Core of agile principles.
Visualisations help communicate with stakeholders.
Weekly showcase - here's what we learned and what we changed as a result - show the benefits of on-going performance testing.

Комментариев нет:

Отправка комментария