Archive for March, 2011

March 22, 2011

Exclusive Interview with @DEVOPS_BORAT

great pic of @devops_boratby-@mattokeefe

@DEVOPS_BORAT has exploded onto the DevOps scene as of late, via Twitter. He won Best Cloud Philospher and Best Cloud Tweet at Cloudy Awards 2011. is pleased to share with you an exclusive interview:

@mattokeefe: Congrats on winning multiple Cloudies! Were you able to attend the award ceremony?

@DEVOPS_BORAT: In Kazakhstan we have old saying “If you can lean, you can clean”. Is why I not attend conference but prefer deploy infrastructure on all possible cloud provider.

@mattokeefe: Totally understandable. So where do you work and what is your role?

@DEVOPS_BORAT: I work in small startup in Almaty Kazakhstan. Is part of many startup launch by incubator company with name which is translate as Bird. In Kazakh language is spinoff know as Dropping, so our startup is Bird Dropping #53 finance by venture oil capitalist. We are specialize in social networking in the cloud with emphasis on Human To Android relationship.

In day to day job I have title of Senior Manager of Operation. I manage of myself and of Azamat who has title of Junior Manager of Operation and he only manage himself. We are sufficient manpower for deal with all devops issue in the cloud because we have everything automated. Also we use NoSQL which make it very easy scale, I can not able say at infinite but for practical purpose is infinite.

@mattokeefe: Did you start your career in development or operations? When did you first hear about DevOps?

@DEVOPS_BORAT: Word devops is start with dev then ops. I start career in development of C++ (as small detail, in Kazakh language is pronounce ++C which is more correct). I am happen of agree with Joel Overflow that programmer need learn C/C++ first so they understand pointer. If you not experience null pointer segfault is like you not experience sexytime!

I hear of DevOps in past 2 or 3 years, is coincide with downfall of Agile and rise of cloud and also of Twitter. First reason DevOps was create is because is easier type #devops than #developer #sysadmin but correct name is in actual OpsDev.

For practice DevOps I recommend first follow cloud expert and devops expert on Twitter. Next step is automate bulls shit out of everything.

@mattokeefe: I know what you mean about null pointer segfaults. I’ve seen Java log files full of NullPointerExceptions. When I showed the developers, they said “Oh, those are harmless. You can ignore them.” But they never went away, and I worried that Ops wouldn’t detect a real problem later. Is this something that DevOps can fix?

@DEVOPS_BORAT: In startup if we hear such comment from developer we immediately put them on pager for 1 month. Next time is 2 month and so on. Problem can not be able fix by DevOps. Only way to fix is not use Java in first place. All DevOps rock star are use Scala or Clojure.

@mattokeefe: Wow, you really are hardcore. So tell me, what DevOps tools do you use, and what do you find missing? Are you following the devops-toolchain project?

@DEVOPS_BORAT: Between my personally and Azamat we are try very hard for use all available DevOps tool. Nothing is perfect though so we end up roll our own tool. We tentative call tool Swiss Army Electric Saw, is good for monitor, alert, visual metric, queue, deploy, continuous integration and continuous delivery. Tool is base on node.js so it eliminate disk I/O. We also try hard eliminate network traffic by only allow 56k bandwidth for legacy customer.

I read page of devops-toolchain, I can not able comprehend with limited English. Is philosophic dissertation yes? Is very good if somebody is able get PhD out of it.

@mattokeefe: You guys are very ambitious with tooling. It sounds like you could use more help. If you could hire just one more person for your team, would you choose a developer interested in learning Operations, or an Ops guy looking to learn how to code?

@DEVOPS_BORAT: We are always search for mythical centaur creature 1/2 dev and 1/2 ops. We have business idea of launch RoR Web site for dating of dev and ops. We are hope for ROI in approximate 20 year.

@mattokeefe: Awesome. Are you looking for investors? The DevOps market seems to be heating up, despite Damon Edwards talking about shark jumping and Some Forrester Guy asking for NoOps. What do you think of these remarks?

@DEVOPS_BORAT: We have lot of interest from oil and natural gas baron in Russia. Not need VC dollar from U S and A. As matter of fact region of Almaty Kazakhstan is know as Silicon Camel Hump of Central Asia.

I read blog post of Forrester guy. Content is Noop. In my opinion DevOps is just sign of what is for come. Is going be follow by DevQaOps, then DevUxQaOps, DevUxQaSecOps and in final is pinnacle of Internet Jedi Samurai Jason Calacanis.

@mattokeefe: Well I am blown away by the wisdom that you have imparted so far! I can’t wait to share this with our readers, so I will publish part one of this interview ASAP. Let’s see what sort of questions our readers raise in the comments section, yes? Then I hope that we can have a follow up interview.

Now, just to wrap up part one, how do you celebrate successful DevOps achievements… erm… “happy sexytime” in your country? Do you enjoy pizza and beer, or some other custom?

In startup we have quiet celebration usual. Is involve 2-3 keg of vodka. Is loud first 30 minute then very quiet. Azamat is last to stand. In past our ancestor use of shoot ibex after celebration. In startup we continue tradition by terminate random node in cloud after party, is same thrill of feeling.

@mattokeefe: Oh no, WordPress is down! I hope they didn’t sustain collateral damage from your Chaos Monkey!

“We’re experiencing some problems on and we are in read-only mode at the moment. We’re working hard on restoring full service as soon as possible, but you won’t be able to create or make changes to your site currently.”

Thanks for the interview!
March 8, 2011

DevOps Culture Hacks

Talk by Jesse Robbins (@jesserobbins), Chairman and CEO of OpsCode

Blogged live from DevOpsDays Boston 2011 by @martinjlogan

Tricks for getting DevOps to work in your company, from the technical to the social, taken from experiences at Amazon.

Jesse Robbins was the “Master of Disaster for Amazon”. As an ops guy, he worked multiple 72 hour sessions, took site up time very seriously, and eventually turned into the stereotypical ops guy who said “no” all the time. What’s worse is he discovered that he was proud of it. Jesse started to notice that he was taking site outages personally. Amazon at the time, 2002-2003, was doing standard ops. They deployed in large monolithic fashion which is an absolutely painful process prone to error. Managing such a process and also finding yourself emotionally involved with the work to a high degree is not a productive situation to be in.

A turning point for Jesse in terms of moving from an obstacle in the way of change to someone that really knew how to add value with ops practice stemmed from a battle he got into with the “VP of Awesome” at Amazon. This was the nickname of this particular VP because it seemed that pretty much any highly interesting project at Amazon was under this man’s purview. What happened was that Jesse did not want to let out a piece of software because he knew, for sure, that it would bring the site down. The VP overrode him by saying that the site may go down, but the stock price will go up. So, the software went out, and it brought the site down. Two days of firefighting and the site came back up, and so did the stock price, and so did the volume of orders.

The dev team went on and had a party, they were rewarded for job well done, new and profitable functionality released. At the end of the year, Ops got penalized for the outage! Amazon rewarded development for releasing software and providing value and operations was not a part of that. They were in fact penalized for something that was out of their control.

This of course did not sit well and as a result of this and other similar situations Jesse actually got famous for saying no. Who in their right mind would want to release software and go through that over and over? When the business would put up a sign advertising new functionality that was to be developed, something they were presumably excited about, Jesse would write the word “No” on it.

Operations naturally wanted to protect itself and came up with all kinds of artifice in order to do so. Root cause analysis so that blame could be assigned efficiently. Software freezes that prevented software from being delivered to the site during peak times of the year. This seemed like progress, but clearly was not looking back on it.

(This all sounded fairly familiar to the folks in the room)

On to DevOps

Now to talk about DevOps. We have at least the beginnings of an idea of what to do about the situation described above. Prior to addressing this situation correctly, or at least more correctly, Ops looked at their output as a waste. The best thing they could do was to cost the site $0. Instead we need to be looking at this another way, bringing value through the function of ops. DevOps is about creating a competitive advantage around the things Ops does every day.

Why does the break occur? Historically Ops creates value by reducing change and getting paged when things break. Dev is about value creation and Ops is about protecting that value. This creates a “misalignment of incentives” meaning that different organizations are rewarded for different behaviors. This creates something called local optimization. Knowing these terms will help you talk to MBAs about DevOps!

We have a fundamental misalignment of incentives and in fact a conflict in incentives. Development is exclusively aligned to releasing software and not at all focused on maintaining it. Ops, is the opposite. Each group optimizes locally around this which creates conflict. Operations is focused on minimizing change because that reduces outage where as Dev is entirely focused on maximizing change.

Solving this problem is what DevOps is all about.

The unproductive way of thinking mentioned earlier came about was in an environment with 4000 devs and significantly fewer ops folks. In order to alleviate the problems caused by misaligned incentives and local optimizations were to come up with those punitive changes that are incredibly satisfying to Ops folks but really don’t help solve the problem. Those changes in the form of meetings and review boards that are around to punishing people into releasing the software the way you the Ops person wants. These are the kind of measures that control oriented people gravitate towards, and it feels like progress for them. To be more DevOps don’t try to fight them in this: “Don’t fight stupid, make more awesome”.

One initial thing that changed, that started the real progress, was to align dev and ops in a way that prevented local optimization; putting devs on call for their own software. This started to shift ops from being the people that just dealt with all the problems to people that became experts on all the services that allow the software to run. Ops started to become tier 2, escalation for devs. The way you got there, was to offer devs deployment options and permissions, if they passed some training and were willing to be on call. Initially this caused a fair amount of chaos. Devs had a load of pagers and got messages that confused the heck out of them. There was pain and frustration. This pain and frustration and the fact that devs were now playing with tools in actual production environments, really changed the culture quickly.

Through trial and error, top down fiat, audits, and every carrot and stick approach the formula for this class of organizational change was developed. This what Jesse uses even today to accomplish these changes and what we will concern ourselves with for the rest of this talk.

  • Start small and build on trust and safety.
  • Create champions
  • Use metrics to build confidence
  • Celebrate success
  • Exploit compelling events

Start Small and Build on Trust and Safety

We tend to want to take on the entire org up front. We want to throw everything out, starting at the top and working down. This simply does not work, Jesse had failed multiple times before realizing that this was a failing strategy. Continuous deployment for example seems like something you want to deploy widely. Instead you should start with a small motivated team and build some success there.

Another thing to consider when thinking about starting small and building trust is that when introducing disruptive change in an organization you should lead with questions to garner buy in instead of just telling people that you have the solution and such and such is what to do. Don’t even use the word DevOps, just focus on the problems and get permission to start to change it.

You have to make the experiment of changing things safe. Jesse tells people that he will take 100% of the blame if things go wrong, in exchange for the space needed to make the desired changes. Creating safety is critical in pushing through organizational change. Crucial Conversations by Kerry Patterson is a book that covers this really well and the Jesse thinks is one of the most critical books to read if you want to create organizational change.

Create Champions

“You can accomplish anything you want so long as you don’t require credit or compensation”. It is amazing what you can accomplish when you give away that part of your self that requires recognition. You must shine the spotlight on those in your organization that get it. Get those that are recognizing the need and acting upon what you are pushing forward to talk. Make your boss a champion; this is critical. It is really important that your boss can explain what you are doing and why or at least be able to provide air cover for you.

The second part of this is to give people status; special status. SRE “site reliability engineers” walk around Google with leather bomber jackets. They get hazard pay, they have special parties, they are considered to have elevated status around the organization. When you find your champions do something that makes them stand out. Wikis are quite powerful for this. Write down and explain in very powerful language what these champions do.

At Amazon Jesse created this thing called the call leader program. They trained people to handle high impact events. After a while there evolved a pressure to join. Eventually you become the person that people have to go to in order to get a certain status which gives you personally more organizational power – not the point but helpful in furthering the change you want to further.

Use metrics to build trust

Get as many metrics as you can. Begin to look at them for KPI’s (key performance indicators). These are the things you will use to prove your case. What you are looking for is a story, and a set of metrics to prove you have one. John Allspaw talks about things like MTTR, mean time to recover. These are great for story telling. You want to capture metrics early and use them to tell your story. “Having devs on call will be a great thing for us, and oh, by the way, here is the data that proves it!”

Make sure that you are good at telling your story with the data that you capture. Here Hans Rosling gives a TED talk about how to tell a story with data that Jesse recommends for everyone. This can be used for inspiration. Ideally have your champions tell your story.

Celebrating your Successes

This comes back to “you can accomplish anything you want so long as you don’t require credit or compensation”. Create moments in time where you celebrate the success of the change you created. Have parties when you reduce MTTR by 15 percent. Give people to a moment in time where they recognize the change the created and that the change was good. This gives people a moment in time to look back on and judge progress. This is of critical importance.

Exploit Compelling Events

Compelling events are those big company issues, big or small, it does not actually matter. They are the events that create cultural permission to make important change. An example is the executive mandate toward cloud computing. This is a compelling event that allows you to make a whole bunch of procedural change. Big outages are compelling events, they give you cultural permission to make significant change that would be otherwise impossible in normal times.

When you have a compelling event you don’t encounter resistance, but instead, permission to make large change. If you don’t have such an event, then create it. Jesse had something called “game day” at Amazon. Creating outages to test failure recovery. Non-recovery became a compelling event. Big deployment pushes are examples of compelling events. If you are in the middle of a serious problem with process it is hard to propose chucking it out in flight but if you offer to own the postmortem process you can direct it toward the change you want to make – though start small as indicated by the first point in this list.

The next time you want to create change in your org particularly DevOps change keep in mind:

  • Start small and build on trust and safety.
  • Create champions
  • Use metrics to build confidence
  • Celebrate success
  • Exploit compelling events

What are your thoughts on DevOps culture hacks? Anything to add to the list?

March 8, 2011

DevOpsDays LiveBlog – Cassandra and Puppet

by Dave Connors from Constant Contact

This talk is about how Constant Contact integrated social media into their offering using Cassandra and Puppet. Small businesses look to Constant Contact for help with customer contact. Now social media as part of marketing is really growing and so Constant Contact had to integrate. Social media business rules vs email marketing is quite different but the number one challenge with a social media integration is that the data volume over email is on the order of 10 to 100 times greater.

NoSQL, Puppet, and DevOps practice offers answers on how to accomplish the integration described above rapidly and with low cost. Two million dollars would be the price tag for the integration with their traditional data stores. With NoSQL it is much much cheaper. The second nice thing about NoSQL is that the time to market was reduced. Just the right technology would not have been the solution though, they needed to focus on having a real DevOps culture and practice.

Ops and Dev both face issues in getting the Constant Contact social media integration project done.

  • Data Model – Cassandra is different
  • Monitoring – Old monitoring solution was not suitable
  • Authentication
  • Logging – Lots more data
  • Risk profile
  • Roles and Responsibilities – swapping them around a bit from the traditional approach

This social media project was completed in 3 months. Cassandra/NoSQL and DevOps brought them a lot of advantage in making this possible.

The Dev Perspective

The system architect Jim now speaks about the dev perspective on this project. Cassandra was the tool that was chosen to really underpin the project. It was developed at Facebook and open sourced in 2008. This was incubated at Apache and in use at Digg, Facebook, Twitter, Reddit etc… Cassandra has the following characteristics:

Cassandra is implemented in Java, which does not much matter.

  • It is fault tolerant.
  • It is elastic, meaning you can basically keep adding nodes and it scales more or less linearly as you add nodes.
  • It is durable. Data is automatically replicated to multiple nodes. You can tweak options about consistency and replication.
  • It has a rich data model, not strictly key value. You can actually have some structure to the data if needed.

Some development challenges in working on this project with this technology included:

  • moving target. Cassandra major releases come fast in comparison to DB2 for example.
  • Developer unfamiliarity. Cassandra is not totally trivial to wrap your mind around.
  • Operational procedures. There are not a lot of established best practices out there for dealing with this sort of DB.
  • Reliability concerns. Can you realize the promise of its reliability if you don’t fully understand how to do so.

How this was mitigated/handled for this project

  • Pushing hard on deployment automation – clearly
  • Community involvement. Apache and Cassandra has a very active community for ferreting out best practices.
    Getting into the community is key. Mailing lists and IRC and #cassandra at freenode.
    Contribute back to the community so that you don’t have to maintain your own fork when you find bugs.
  • There is training and consulting available for Cassandra and they used it. There does not exist “one neck to wring” with Cassandra but again you can get paid support and training at DataStax and Constant Contact used them and was happy with it.
  • Lots of monitoring. They put a lot of work into being comprehensive. Munin was used
  • Choosing a good client to Cassandra – Hector was used. (Don’t use Thrift, it is really intended as a driver level client, it does not provide a lot of the things you would want a real application client to do; things like failover and retry).
  • Using switchable modes. Using the relational DB as the system of record as you start to move over to Cassandra.
  • Mirroring is another technique that was employed at the application level. All writes go in parallel to Cassandra and to the relational DB as well. When things fail the RDBMS is the backup.
  • Dialable traffic. Being able to turn down the traffic when things go wrong.

Collaboration was really key in getting this to work. It was a big complex project. We had to have close collaboration and flexible roles. Mark and Jim the two primary dev and ops folks on the project and they had to be flexible. For example they changed the monitoring system that is traditionally used at Constant Contact when it was recognized that the current system would have failed. This is the type of systemic change that would be difficult to do without an environment of collaboration between Dev and Ops. Now that we have covered the dev side, we can talk a bit about ops.

The Ops Perspective

Now Mark will talk about this project from the ops side. Mark is the manager of system automation. He will talk today about how they use Puppet and in general a software tool chain that allows for improved levels of deployment flexibility.

When Mark starts a project, as a system admin, he tries to find the system specifications that will support the system the best. The came up with this machine spec after working with DataStax

3 500gig disks
1 250gig disk
No Swap
Raid Zero Root Partition and Data Storage
32GB Memory

The vendor was not sure if they should order that configuration because there is no internal fault tolerance built into that model. Cassandra however deals with redundancy at the node level though. So the question then became, how many nodes are needed?

  • Cassandra Quorum = 3 (meaning each bit of data needs to ultimately live on three machines)
  • Two data centers
  • Each node can only half the available disk because of RAID
  • ~ 6 TB needed

Ultimately that means 72 nodes which is a fair amount to manage to the level required by the project. Without getting into details about the management it would have been impossible for a human to do it. We wrote a puppet module that handles much of the management of this cluster. Puppet is not the only part of the whole system though. Here is the total tool chain:

Fedora anaconda/kickstart -> Func (for upgrades, puppet module exec’d through func) -> Puppet (for OS and app config) -> Scribe (Facebook’s open source logging framework) -> Nagios (for logging, managed by puppet) -> Munin (for trending)

The tool chain above, really centered around puppet, means that Dev and Ops were able to talk about things in a common language. That language was Puppet. They also started using subversion for their config. Puppet allowed for infrastructure as code.

Operational efficiencies were garnered though using Puppet with Cassandra. Remote logging was a requirement. Cassandra uses log4j natively but resources were not available for remote log4j logging. Ops was able to get Scribe integrated with puppet easily.

Munin is another tool in the stack that allows for JMX trending. It allows critical data points to be identified. With Puppet they could continuously deploy improvements to the trending and analysis tool across the cluster in a uniform way. They did 7 X 92 graphs across the cluster in 5 minutes with Puppet. This gets reused over and over again as more apps get pushed to the cassandra cluster. DataStax provides RPMs for Constant Contact to deploy this software within. Admins in general at Constant Contact must be able to build RPMs. Maven is used to build the RPMs for custom applications.

Traditional Ops vs Today at Constant Contact

  • Infrastructure 4 weeks then to 4 hours to build 72 node today.
  • Development to Deployment 9 months then to 3 months today (for the whole project given comparable projects)
  • Cost millions of dollars then to 150k today.


Q. What was the role of the DBA in this model?
A. DBAs will be the keepers of the data dictionary. They will also be helping with tuning of the actual cluster.

Q. Have you had an opportunity to do version upgrades on a running cluster?
A. Yes, we worked with QA to do a rolling upgrade twice. It worked nicely, no issues. We did a slow roll, sequentially. Cassandra naturally takes care of this with hints.

Q. Both dev and ops roles are writing puppet code. How do you stop them from clobbering each other.
A. We are still working on it, but version control helps a lot. They actually pushed some code into production before it was ready. They expect to be able to treat this puppet code ultimately like any other code.

March 7, 2011

DevOpsDays LiveBlog – DevOps in Government (DoD)

DevOps in government pic of military torpedo boat

the good the bad and the ugly

By Peter Walsh

I want to share with you some of the process that we have implemented at the DOD. We don’t have the silver bullet but we have improved things.

The Ugly

DoD spends more than $32 billion per year on IT systems. In 2008 our software creation and deployment process we went from 38 documents and gates to well over 40. Yay, moving backwards! In the DoD releasing software requires multiple levels of tests and approval, all within distinct organizations that don’t much communicate.

The Bad

Normally you have a program office that pushes problems to solve/projects out to dev contractors, test contractors, test agencies, and ops teams. Each of these groups are quite separate. They all create their own setups, which are distinct, and which basically guarantee inconsistency from one group to another. They manage things by excel, and project. Costs are measured in piles of money, and time is always calendar time, everything is slow.

Another large challenge for DoD systems is this challenge of accreditation. Before anything can be used within the software creation process or delivered it must go through this process. Everything! Hardware, OS, middle ware, applications. This process is implemented by a separate agency. Each of the aforementioned teams goes through this process separately. This process limits flexibility and it restricts change. This is why the government lives on old technology.

Good (all is not lost)

There are 26 projects on a Whitehouse watchlist for improvement and experimentation, some are DoD projects. There is some recognition that we need to be more agile and perhaps Agile. There are a few new programs out there that are trying to enable everyone in the DoD along these lines. Some of the more prominent ones are listed here:

  • is for government open source projects. They look to create community tools and foster a bit of colaboration. The vice chairmen of the cheifs of staff supports this project.
  • Race is a government program for Cloud resources and accreditation assistance.
  • is another project that loks toward improving system provisioning, automated workflows, and test as a service.

Another project within the DoD that has done a lot to support the shorter lifecycles and higher quality delivery is called CONS3RT. We will discuss the approach it takes moving forward here

CONS3RT is a library of all the assets that you have from dev to operational deployment. Assets can be things like servers, databases, monitoring tools, etc… Assets are pulled together into what is called scenario which can be deployed into a cloud space in a repeatable fashion. This allows the different agencies and groups responsible for delivering software as described earlier to work on the same scenarios which eliminates some of the inconsistency between these groups. This also addresses the acredidation requirements. Once something has been accredited then it can be reused in this way between groups with out being reacredited. We realized that accreditation could not be just thrown away in building CONC3RT so it is quite well integrated into what we do. Peter and his group are not trying to tackle everything, but instead to start small and work within the boundaries of what can be changed.

This focus on accreditation is really key also because in the past developers did not want to go through the accreditation process becuase of the restrictions. They developed on their own hardware and then kicked things over the wall to move through the rest of the life cycle. This guaranteed nothing worked for the first dozen times it was kicked over. Now, this is less of a problem. Many other improvements have come out of the use of CONC3RT and this approach in general.

With this sytem reuse also improved. Instead of each organization learning from scratch how to deploy x or y spending piles of money doing it components already placed into CONS3RT can be used. This is something that virtualization and cloud in general can bring to any company.

Time is measured in minutes now a days instead of days and weeks. Things can be deployed with a couple mouse clicks instead of new hardware installation. More resources are also available to people now both in terms of the variety of software they can access as well as the CPU power they can harness through increased utilization. Tallent can be focused because people are not wasting time doing the job of provisioning, accreditation and other coordination activities.

How does software flow through the lifecycle now?

Stage – Developer: New code -> local IDE build -> Jenkins & Maven -> if build good, deploy to integration, if not, back to write code
Stage – Integration: Deployed to Integration -> Debug and test -> if tests pass send to CI if not, delete the artifact and go to dev
Stage – CI: Deploy to CI -> smoke/regression -> if pass -> if enough new -> tag -> send to QA env
Stage – QA: Deploy to QA -> Smoke/regression/manual/exploritory -> add new tests -> tests pass -> new tests push to past env -> tag new bits -> push to production

Most organizations in the government, and in industry, handle the writing of new code, building the code and pushing into production. The stuff in the middle, not so much, many orgs do not have much of a handle on how they are doing those functions. Our pipeline as mentioned above in conjunction with CONC3RT, maven, jenkens, and other tools is really helping us get a handle on it.

Lessons learned

  • Automation is huge.
  • Testing early and often. It is easier to fix the engine when it on the assembly line then when it is on the dealers lot. Simple concept. Virtualization and CI allow you to test often with minimal barrier to doing so.
  • Leanness. This goes back in part to the process stuff. It is a matter of keeping the process simple
  • Flexibility, team trust and commitment. We fall off the wagon sometimes. We need to be able to call eachother out on things so we can keep improving. We need to trust that we will stay committed and on track when thing fail – which they will.
  • Persistence. Stay the course when people push back on you. Managent, peers, and others.
  • Trust, with customers and sponsors.
  • Top cover, someone over the top that can allow you to be persistent and help cover challenges from further above.