AI driven expertise & profiling: hype, hope or déjà vu?

May was a busy month. Apart from helping establish then launch a real estate and mortgage business (Bees Homes) I was in Lisboa for Social Now and London for KM Legal UK.

I attended both in the expectation of learning more about the onrush of Artificial Intelligence and its implications for the Knowledge Management profession.

Specifically, I wanted to see how the encouragingly styled Talent and Knowledge Matching / Profiling systems might tackle the challenges of knowledge loss when people depart, of onboarding when people arrive and identifying / ranking expertise that might otherwise be opaque when pulling together teams.

It’s not a new topic: back in the late 90’s I was Business & Strategy Advisor to Sopheon PLC when we acquired Organik (a technology for identifying expertise) and built systems for US Insurers looking to establish the best teams for clients based upon expertise. We never cracked it even though we knew what the issues were (usually motivation)!

Seeking answers at SocialNow Lisboa while Keynote speaker Ellen Trude watches.

Armed with a list of ‘use cases’ I’d worked on with Martin White I set off in search of answers to these questions from both vendors and KM practitioners?

  • Onboarding: A new employee with many years of highly relevant experience joins the firm. How long will it be before their experience is ranked at the same level as their predecessors?
  • Legal: Is the profiling process compatible with the provisions of the General Data Protection Regulation? The thoughts of the Information Commissioner on this are worth a look. Profiling & Automated Decision Making
  • Functionality: Do they offer the ability to present a list of people ranked by expertise?
  • Language: In multinational companies where it is especially difficult to know all the experts, how does the vendor coppe with the fact that documents, meetings and social media traffic will be in local languages?
  • Chinese Walls: How does the application cope with expertise gained on projects that are secure, a common issue in law, finance and R&D where walls need to be erected to prevent commercial information being divulged>
  • Testing: What User Testing is undertaken with a client before signing a contract to verify that the profiling system works?

So, what did I discover? Thierry de Bailllon in his closing Keynote put it very succinctly but with a caveat:

Embrace or die? 88% of technologies already include AI.

Self reinforcing bias?

it’s not Enterprise Social Networks (ESN)!

This Twitter exchange between Ana Neves and Luis Suarez prompted by a question I posed of the Workplace (Facebook at Work) team following their presentation is revealing:

May 12 there’s been a few questions about expertise location 2017 I don’t remember that being the case in previous years #SocialNow

May 12 Well, I think people are starting to understand how critical it is to know who is who within the org beyond just content, right?

Replying to totally! It surprises me it took so long. It’s amazing the role #ESN can have in unveiling that expertise #SocialNow

On the surface the case for ESN is compelling. Yet the majority of vendors at SocialNow focus on information exchange and conversation rather than the capturing and cataloguing of it. One,@mangoappsinc, had a neat tool (they won the “coolest app” prize) with the ability to upgrade comments from threaded discussions and posts to create ranked knowledge resources from the mass of information and conversation.

So, ESN can show who has answered what question, conduct searches across conversations and in many cases act as a project management tool, the new Facebook at Work (Workplace) now allows the creation of documents for example.

Provided the application is linked to HR systems it is possible to retrieve profiles and see what expertise an individual might have. As one vendor (@OrangeTrail showcasing Facebook at Work)) who uses bots to generate responses put it:

‘Questions’ is the key to find experts as people don’t keep profiles updated.

I concur and they are great facilitation platforms though with advanced features that will suffice for many. Yet I left Lisboa though feeling organisations will need to rely on assisted search for some time if they want to take a deep dive into expertise

know what you don’t know

Peer Assist “Problems” for discussion

So onto London and KM Legal UK. An interesting Day One ended with a psuedo Peer Assist in which AI was raised a lot.

One observation (facilitation tip): the session failed to commit the ‘owner’ of the problem to action so as a result the feedback loop to plenary became a series of “we said this.”

Again, as in previous years I felt the focus was on operational tools and techniques which means that KIM Professionals in Legal are more at risk from the onrush of technology.

It reminded me of the issue Librarians faced with the arrival of end user search in the mid 90’s which finished their monopoly of being the people who found stuff in organisations.

Day Two took a deeper dive into technology and its potential impact.

AI in Legal today

This slide sets out where AI is making a difference in Legal.

I tweeted having heard Cliff Fluet’s excellent presentation:

Paralegals beware. AI is coming. Adapt or die?

And I questioned:

How wide is scope of AI? More than Doc Analysis / Creation. Opportunity to broaden knowledge base

As yet no one had focused on expertise and profiling so when one presenter cited the case where a newly arrived CEO asked the Head of HR / Talent Management to let him have profiles / competencies of the staff using their system it got my attention.

I asked whether the results the HR head gave the CEO inferred a level of expertise. It didn’t which got thinking that if the data set is incomplete and the issue of self reinforcing bias is not addressed then over reliance on one source for identifying ‘experts’ is dangerous. Imagine your career prospects if for whatever reason your name wasn’t on the ‘expert’ list given to the CEO?

and finally

So where do I see the state of expertise and profiling systems? Patchy!

Yes there are certainly companies who ‘get it’ but can they do it?

I am indebted here to Martin White who in an excellent report “People and expertise seeking – an overview” summarises the predicament thus:

The most important lesson learned is the need for an expertise location strategy that is linked into HR processes, knowledge management, training, job appraisals and social media development. Finding people with expertise is not a ‘search problem’.  Good search tools can certainly help but without attention being paid to profile quality (even if other types of content are being searched) and a commitment by employees to share their knowledge expertise discovery will not be as successful as anticipated or required.

My takeaways:

  • KIM professionals need a clear strategy (working in partnership with other stakeholders such as HR and IT) and be clear on the questions being solved by any system;
  • They need to be clear what they are getting, what’s missing and how it mitigates the potential for self reinforcing bias when they enter discussions with vendors around automating expertise seeking and profiling;
  • They need to recognise the importance of their role in facilitating the adoption of such systems and accept this is just a part of a portfolio of approaches of identifying, capturing and retaining expertise;
  • They need to be clear what critical knowledge actually is in their organisation and who is likely to have it in order to assess the veracity of the results of any pilot;
  • It doesn’t matter what solution you adopt, if your environment is not conducive to the sharing of expertise and people don’t see the value in it then save the money; and
  • In any event you cannot capture everything people know; we learn and share through stories (failures rather than successes) and those often remain hidden.

Managing networks and Working Out Loud: Collaboration and Knowledge Matchmaking skills

The world is shrinking. At any given moment I know where many of my friends and colleagues are. Technological footprints are heavy and long lasting.

This week for example I see that Arthur Shelley is in Moscow with Ron Young at KM Russia, Donald Clark is in Belfast picking up an award, Phil Hill is getting fit (ter) in Thailand, Patrick Lambe is having breakfast in Lisboa. Gregga Baxter and his wife are supporters of WaterHealth in India.

Through cultivating personal networks I also know what’s happening this week in Khartoum, Tehran, Dubai and Harare. To many that may seem frivolous information; to others (including me) its valuable and if I don’t know then I know a man (or woman) who can. Let me illustrate the issue with a true story.

the art of network management

Many years ago I was charged with setting up the forerunner of a Knowledge Management function for a financial services business in the City of London. It struck me how badly senior officials shared diaries let alone knowledge about clients.

One day I was in the office of the Treasurer of the national oil company of a prosperous Middle East country. As I was about to leave he asked me to stay for the next meeting.

In came four suited bankers. My client took the lead introducing himself and me (as his Advisor). He then asked each one to introduce themselves. And to everyone’s surprise they were from different offices and areas of the same institution. They had all flown down on separate planes to see the same client.

The Treasurer said his diary was open to meetings with the institution but not multiple visits. They lost face not to mention the cost of the travel and opportunity cost.

So knowing what I did I came back to London and, with the support of the CEO, developed and introduced Visit Information Centre (VIC) which showed all visits to our organisation and all meetings outside of it.  Embedded in the day to day workflow the aim was to maximise the valuable time our organisation spent with a client and make sure those in any meeting were briefed on the latest activity. Today this is or should be standard practice; then it involved a shift in mindset.

So fast forward to 12th December 16; its 2pm and I am having an exchange on Facebook with Patrick Lambe about Lisboa where he is spending a week. Concurrently I see that Ana Neves (founder and organisor of SocialNow and “Mrs KM” in Portugal) is online on Skype. I know Ana lives a mere 15 minutes train ride from where Patrick is spending the afternoon. I also know both of them well and believe they would benefit from meeting each other.

Using Messenger I hook them both up and they meet later that afternoon to discuss inter alia an idea I thought both might profit from.

meeting-by-the-tejo

Tea by the Tejo

I coined the phrase “Orchestrated Serendipity” to describe occurences such as this. I have also used the term “making correlations between seemingly unrelated pieces of information”.

In this example I have nothing potential to gain other than knowing that two people I like and respect are now acquainted so my network grows stronger.

Here’s an example of how one thing can lead to another.

an example of ‘Working out Loud’

A few weeks back out of the blue Martin White of Intranet Focus shared a draft white paper on Digital Workplace Governance with myself, James Robertson, Jane McConnell, Sam Marshall and a couple of others. His invitation, which left it up to us as to how we might respond, read:

Colleagues
The attachment is me working out loud on digital workplace governance on a Friday afternoon
Regards
Martin

Our approaches were different. Some came back immediately. Others took their time. Some used comments in Word, others rewrote paragraphs. As Martin said, “the responses always challenge your own thinking.”

I am sure John Stepper (who is widely credited with kicking off the Working out Loud movement) and Ana Silva who is a great proponent of it would be enthused.

Knowledge Matchmaking?

These two exchanges got me thinking about the way I work, the organisations I’ve worked for, the clients I’ve worked with and the networks I am involved in. I have never acted as an introductions broker seeking reward so do organisations and people see value in it?

Previously as a Senior Manager charged with developing new business, my ability to match a need with a solution was prized and rewarded even though the correlation was opaque to my bosses. More often than not the intuition paid off. But does the same apply today in a Knowledge Management environment where logarithms and Artificial Intelligence are making the correlations I used to make?

Perhaps more importantly do people in Knowledge Management have the time, the confidence and the knowledge of the business to be able to put forward ideas and broker connections?

If they do then here’s a few tips:

  1. You have to be in it to win it: if you sit on the sidelines this will never happen.
  2. Be willing to take a risk: yes you might fall flat on your face! But experience tells me that if you go the extra mile people will come back for more.
  3. Be willing to do this without expectation of reward: it’s always difficult to measure the impact in a world of KPI’s. You have to play a long game but be willing to cut if you feel you are being taken for a ride.
  4. Be willing to acknowledge the contribution of others: from personal experience I’ve found there is nothing worse than someone taking what you’ve suggested and packaging it without attribution. A photo is a great way of saying thank you!
  5. Build trust so people are willing to confide in you and trust your judgement: unless you are willing to find out about people and what they do you will never be able to make these connections.
  6. Be clear about why you are making the introduction or sharing Knowledge: I used to be in the cc camp that so many inhabit believing that by informing everyone I was covering all bases. People are too busy and ignore ‘junk mail’.
  7. Develop your internal filtering mechanism: you have to know your business and identify who is going to be a taker vs. a reciprocator.
  8. Respect the contribution people make if you ask for advice: whatever you get back from people is important. They have committed scarce time and each time you ask for a response you are drawing on your reserve of credibility.
  9. Develop a skin as thick as a Rhino: you will be disappointed when others don’t follow your lead and use the contacts or information without acknowledgement. And remember 90% of people online are lurkers so will not go public with their thanks.

And finally

To prove that this is a reciprocal situation. In August I attended an Improvisation event in Oxford. It wasn’t on my radar but Nancy White had posted a comment about it so based on her recommendation I decided to attend: As a Quid pro Quo I wrote up my experiences for the greater KM4Dev community.

If you want good reading on collaboration, Martin and Luis Suarez have been exchanging comments on a fascinating blog post from Luis: “Stop blaming the tools when collaboration fails”.

A book on the sunbed, KM Manager’s critical ‘ates and getting social in Lisboa

It’s been an interesting couple of weeks that began with a vacation in Cabo Verde on an island called Sal (Salt in English-which sort of gives away the terrain). Then a week in Lisboa at the annual SocialNow event, a unique gathering where social enterprise software vendors present their products to an invited evaluation panel representing the management of a fictituous company Cablinc.

a book on the sunbed

Cabo Verde was chosen as the vacation destination for two reasons: it is a former Portuguese colony and my wife (who is Portuguese) and I are trying to visit all of them to see how much of the culture and governance structures remain; and it seemed like a good place to wind down and catch up on book reading.

Without the necessity of a daily commute and the reading time a long journey creates I find the virtual world gets in the way of paperback reading though sitting under a sun umbrella reading a book on KM Strategy might not be everyone’s idea of relaxation on vacation!

Here’s a very abdriged review of my vacation reading list:

  • The Kind Worth Killing: A clever tale, well written with a twist that starts at an airport, the author keeps you Holiday Readingin constant anticipation and it kept my interest throughout.   Great for a 5 hour plane ride.
  • The Girl On The Train: Took a long time to get going (rather like the train system here) switching back and forth among characters.  Ending was well crafted if you have the patience to get there.
  • Relationology: Essentially a business book (101 tips) about managing relationships with stakeholders. Admired the discipline behind the networking approach and how the author has turned his theories into practice for the good of others but felt it devalued the human element of meeting people by making it a very structured process.  Did like suggestions about having groups of mailing lists and being upfront about why you are meeting/calling.
  • Engagement Manifesto: A book that triggered a lot of thought with good tips and approaches. Made the most notes (7 c’s of change, 5 elements of change and ‘The Five Monkeys’ experiment about resistance to change as its “the way things are done around here’).
  • Knowledge Management (KM) Strategy: The book sets out a good, structured and thorough approach and I liked the suggestion that organisations should give primary focus on critical knowledge and strategic knowledge areas when developing their strategy.  I felt though the chapter on KM Technology could have benefited from more visualisation of where the tools fit and what they look like (a picture being worth a thousand words).

KM Manager’s 4 critical ‘ates

While reading the Engagement Manifesto with its 4 of this and 7 of that my thoughts turned to engagement in a KM environment. I’ve been arguing for some time that facilitation is a core competence for all KM Managers. And I think there are 3 others. So my 4 critical ‘ates are:

  • Investigate: Are you putting a buring fire out / solving an immediate business need (operational KM) or is this driven by the vision from the top consisent with the organisation’s stated business direction (Strategic KM)?
  • Negotiate: Up front you need to agree what the scope of your role is and to be tough negotiating what success and hence measures will look like.
  • Facilitate: So much of what a KM Manager does involves facilitation and another sub ‘ate, Navigate. You will become the hub knowing who to go to to ask if you don’t know yourself. You have to facilitate connections, meetings, interactions, events and communities. It requires resilinace and a lot of social skills.
  • Communicate: Senior KM’er’s tell you to devote at least 30% of their time to communicating what you do and getting feedback – its not just about broadcasting it’s about collaborating. Have the KM Elevator pitch always with you. Let all your stakeholders know what you are doing and why.

Which leads me nicely onto Lisboa and the 5th edition of SocialNow.

getting Social in Lisboa

From left, Luis Suarez, Emanuele Quintarelli, Paul Corney, Ana Neves, Jaap Linssen and Marc Wright

5 vendors presented and they were interwoven with a couple of keynotes and presentation from Cablinc ‘consultants’ (see panel alongside) who focused on the business issues facing the organisation setting context for the vendor presentations. I was delighted that Eric Hunter was able to come over from San Diego to sit on the evaluation panel.

Screen Shot 2016-04-19 at 15.34.04

My topic was Knowledge Capture & Retention and ISO 2015. Perhaps surprisingly few in the room were aware of last year’s ISO update which for the first time included this on KM:

A fuller account of the proceedings and the twitter chat can be found at eventifer.

subway_Graphic_2015_1280pxI was really taken by the closing Keynote from Tony Byrne of The Realy Story Group who gave an illuminating talk on the landscape of digital workplace and social enterprise tools and apps. His Technology Vendors Map is well worth a look.

If this is a topic of interest I’d also point you to an excellent article from Dion HInchcliffe on Social Collaboration Tools.

As in 2015 Nooq won the ‘coolest tool’ category. Its a great visualisation tool to show what’s happening in an organisation and sits above enterprise applications.

and finally

This year for the first time there was a day of Masterclasses after the event. I was delighted to have had the opportunity to work alongside Luis Suarez (@elsua) who is a #noemail evangelist.

Screen Shot 2016-04-19 at 16.16.12Luis ran the morning and I ran the afternoon. His method and justification for lowering email usage are compelling and I loved this slide that shows how a social networking platform can be a lot more efficient than using email.

And now back in the UK I have a couple of week’s focusing on the next chapter of the book I am coauthoring with Patricia Eng before travelling back to the Mid East and Lisboa.

A ‘newbie’s guide to Tweet Chat hosting (on Knowledge Capture & Retention)

Screen Shot 2016-03-03 at 17.17.39

I first worked in the City in 1972 as a summer intern in the cheque processing arm of Lloyds Bank Ltd.  We used machines that looked something like this. No typing, just machine minding!

15 years later I was sitting in the machine room of the Marriott Hotel in Jeddah faxing, over an encryted line, a confidential trip memo for my secretary to type up and distribute to selected directors.  Laptops were only just appearing on the market and as for typing, Managers in those days didn’t. If you wanted to communicate confidential information quickly it was the fax.

Fast forward to this afternoon and I am about to host my first TweetChat some 44 years on from my first immersion in technology.

Think about it: I can’t see who I’m talking to; I don’t know who’s ‘listening’; I have little idea whether what I am going to ‘say’ will resonate with the audience: and I have to type at lightening speed. It feels like ‘drinking from the fire hydrant’ to boot!

But there are huge advantages: I can reach a global audience without leaving my Home Office; what I say will have a very long ‘tail’; and it forces me to articulate my thoughts in a very concise way to an audience who may not speak English as their 1st language.

I know from many conversations I’ve had recently that everyone is expected to be up to speed with new technologies and few get trained adequately to do so.

Screen Shot 2016-03-03 at 07.59.06

Here, with grateful thanks to Luis Suarez (@elsua), Ana Neves (@SocialNowEvent) and Ana Aguilar-Corney (@aguilarinteriors) who provided the wise words and tips I show below, is how I went about it.

Set up

  • Use http://www.tchat.io/ to handle the chat. Load that on the browser and forget about everything else.
  • Focus on the tweet chat for the entire time, even if it looks like things may be a bit slow with tweets coming through, don’t go elsewhere. That way you are free of interruptions and focused on the chat.
  • Have a look into the questions of the tweet chat ahead of time, and write some potential answers ahead of time that would fit in tweets, within the 140 character limit. That way when the answers come in you just have to copy and paste and focus on what people tweet for potential responses, faves, RTs. etc. etc.
  • As you see tweets coming through, don’t think about responding to them all. Think about peppering out the interactions: some responses, some RTs, some faves, to balance your interactions without demanding you to type too much, so you can focus on the conversations themselves.
  • Enjoy the tweet chat under the notion you won’t be able to read and respond to everything while the chat lasts and that’s just fine! You can always come back at a later time if you feel you’d need to. Enjoy the flow as if you were reading a fast paced news tracker skimming through and stopping where you feel you can and want to contribute.
  • If you are going to refer people to blog posts or articles make sure you condense the URL’s as you ‘cut and paste’ into your Tweets.
  • Establish a live back channel with the facilitator while you are conducting the chat.
  • Be clear about who is performing what role and ensure someone is producing a Storify of the event that can be circulated later.
  • Don’t be afraid to let the virtual ‘silence’ hang.

Conduct

So armed with the above and a set of thoughts for three questions off I went.

And if you are up for reading an account of how it went go to the Storify Account of the discussion which is here

And finally

The hour (the optimum time) flew by. Armed with the checklist above it was plain sailing.  It did however reinforce the veracity of the ratio I use for physical workshops namely 3-4 x times preparation vs. the length of the event. I spent 3 hours on potential answers and it paid off.

Would I do it again? Yes tomorrow provided there is a clear mandate and set of questions to be addressed.

 

a reluctance to tweet: 10 success factors for virtual teams

It was like throwing a dart into a vacuum

Is how I responded on Twitter to Mark Gould an offsite observer of #KMLegal2015 who bemoaned the lack of online activity by the 100 or so Knowledge & Information Management (KIM) professionals who were attending this year’s Ark Group gathering of the UK KM legal community.

Its baffling: vendors, consultants and indeed KIM practitioners promote the value of social collaboration tools such as Yammer and Jive. Indeed KIM professionals are often at the forefront of efforts to get adoption in their organisation in order to improve collaboration and knowledge sharing. Yet they seem reluctant to ‘walk the talk’ in a public forum.

Perhaps Joanna Goodman got it right when she said:

sessions were quite interactive, so hard to be fully engaged and tweeting

It made me think more about why I tweet at a conference, this is what I posted during a virtual conversation with Luis Suarez a prodigious tweeter (58k to nearly 12k followers):

Why tweet a conference? Expand reach, collaborate, collect and share thoughts ‘on the fly’. Make notes for future blogs.

What really struck me though was the contrast with the Janders Dean Legal Knowledge & Innovation Conference, London #JDKMConf held the week before. That audience made sufficient ‘noise’ that even those who didn’t attend were able to draw conclusions. Here’s what Stephen Sander (The Vue Post) wrote in a witty piece about being a non attendee:

I curated below what I consider to be the best tweets from the Conference. These tweets offer an interesting insight into current themes and issues in legal knowledge, innovation and technology.

Perhaps this is the difference? The Janders Dean event was invitation only – a thought leaders event – whereas KM Legal is an open conference, if you pay up you can go!

Whatever the merits of both, facilitation should be at the core of the KIM professionals competency set and ‘putting stuff out there’ ia good part of that. Too many broadcast rather than engage. Knowledge Management in a comfort zone is not going to change the way a firm works and responds to the significant challenges facing the legal profession which brings me onto why I was there:

Managing Virtual Teams

In December, Martin White and I ran a breakfast breakout event at the RSA entitled The Future for Legal KIM: an outside in perspective’. One of the challenges firms identified as significant but for which they were ill prepared was the management of virtual teams. As a couple of long in the tooth practitioners who have worked across many continents we’d seen a wide range of organisations fail to match their virtual team technology investment with training in how to go about facilitating virtual encounters.

Virtual Teams Presentation StructureOur brief for KM Legal 2015 was therefore entertain the audience, bring the issues to life. Our approach: tell stories and show images.

We divided the presentation into these areas each drawing on events from our knowledge base.

In tackling the culture piece I noted the following:

Let me say right up front: you can’t manage culture just the same as you can’t manage knowledge. In both cases you can create environments in which people are willing to collaborate, share and work towards a shared set of goals.

Many organisations have a set of values and a social contract that underpins the relationship between the firm and employees.

Ultimately a firm is a collection of individuals each with their own reasons for being there. In a virtual team people’s fears, prejudices and behaviours are magnified.

In thinking and rehearsing for the session Martin and I had worked virtually. We learned a lot about clarity of messages and intent behind words and phrase (and we are both English). We (re) discovered the need for a collaboration space with a framework that suited us both.

We discovered a lot more besides, here’s what we shared with the delegates:

Ten virtual team success factors

  1. Virtual teams are the way work gets done: Recognise that virtual teams are going to be increasingly important to any organisation, and ensure that current and potential participants have access to training and mentoring on virtual team management and virtual team meetings.
  2. Set very clear and achievable objectives: Virtual teams should have very clear objectives so that it is possible to set the investment in the team against the outcome and also that team members bring appropriate skills, expertise and authority to take action.
  3. Chose virtual team leaders carefully: Leadership skills that work for physical teams may not be as valuable in a virtual team environment. Other skills are needed and have to be acquired through practice, not just through reading or teaching.
  4. Develop protocols for virtual meetings; Without good team meetings a virtual team is very unlikely to achieve its objectives and so particular care should be taken in developing guidelines for virtual meetings and for facilitating feedback.
  5. Provide team member profiles: Develop good profiles of each team member, taking into account local availability of technology and offices which can be used to take part in virtual meetings (especially in the case of open-plan offices) and language expertise.
  6. Build virtual relationships before putting them to the test: Each team should have an opportunity to meet with other members of the team through an initial virtual meeting where members can introduce themselves and gain experience with the technology being used before the first formal meeting of the team.
  7. Team dynamics can be difficult to manage: Team dynamics of virtual teams can be quite fragile, often depending on a very high level of trust in people they may not have met before. Introducing a new team member into an existing team may mean starting the process of building trust all over again.
  8. Gain consensus on what needs to happen between meetings?: Team members may have different reporting lines, which may impede the overall achievement of objectives. The measure of a virtual team is what it accomplishes between meetings, not how enjoyable the meetings are
  9. “English is our corporate language”: Issues of language and culture need careful consideration but should never be an excuse not to bring specific individuals into a team. There may be a mix of abilities in reading, speaking, understanding and writing in English
  10. Evaluate team and individual performance: The performance of the team and of each member should be carefully evaluated and training and support given where needed.