Ansible DTAP – Development, testing, acceptance and production

Quis custodiet ipsos custodes?

A majority of Ansible use cases are in application deployment and continuous delivery, a job at which Ansible truly excels. But when using Ansible for such mission critical things, an age-old question might arise:
Who is going to guard to guardians?
In other words, how are we going about continuous delivery and super-cool automated deployments if the Ansible scripts themselves don’t pass the same process?
In my previous post – Testing Ansible I’ve identified 4 different steps in testing our Ansible scripts.
This DTAP process should give an overview and a loosely coupled framework for putting our Ansible code to production.

Development – “Ground zero”

The most important thing about development is to be completely fearless about making errors and to make errors completely and easily reversible.

To achieve that comfort in development, a virtual environment is a must: be it a container, VPS, or a VM… What ever suits the project you are developing. My recommended development environment is Hashicorp’s Vagrant.
Vagrant gives you the comfort of putting up multiple virtual machines in a single environment – properly abstracting the production infrastructure you might have.
When the development is finished, the following tests need to be run:

  1. Syntax test – did we write code or did we write gibberish?
  2. Dry run – are all the prerequisites for the configuration changes present?
  3. Run the scripts – will the script actually run?
  4. Idempotency test – will I make any harm by running the configuration script again on an already configured machine?

It’s important to note that these tests can be fully automated and take almost no time to run – ideal for development.

Testing – “The first trials”

Delayed assertion

The development is finished – great! We’re 1/4 of the way closer to production.
We are left with one important test mentioned in Testing Ansible: Delayed assertion.
Delayed assertion is just you writing more code to accurately test if all conditions required by the feature are met.
After running the smoke tests mentioned in the Development phase and running the Delayed assertion tests, we need to ask the authority to give us clearance for staging.

Authorization for staging

Our code is now swimming with the big fishes – no more development comfort, ad-hoc changes and SSH sessions…

Once we ask the authority for permission to go to staging – we are in the rapids flowing to production, everything from now on is fully automated.
The authority, in our case, is the CI Server.

CI Server’s role in this step is to re-run all the steps done in development, to fix a common cause for failure – developer not testing properly.

The CI Server workflow:

  1. Code is commited to the repository.
  2. Needed VMs/VPSs are spun up exactly as in the development environment
  3. Tests are being run on the machines in the exact way as they should be run on the development environment
  4. If everything is ok – we are ready to get serious.

Staging / Acceptance – “Getting serious”

We have the code, we have the proof that the tests are passing. Now comes staging.
The attitude we have towards the staging environment should be identical to the production environment – otherwise we simply haven’t set the stage well.
The only difference between the staging and the production environment is in the fact that no end-users are using it!
The same tests as in the previous step are run but this time on an exact copy of the production infrastructure.
Depending on the CI tool which you are using, this will be easier or harder to setup, but the ideal workflow should be the following:

  1. Run tests in the staging environment
  2. If the tests fail, mark the build as unsafe and don’t destroy the staging environment
  3. If the tests pass, mark the build as passing and destroy the staging environment


Why don’t we destroy the staging environment when the tests are failing but quickly dispose of it if they’re not?
Simply because we wan’t to have access to the environment which failed to provision normally – to gather data on the failure and to make sure we avoid it in the next build. Marking the build as unsafe in this case simply means that this specific revision CANNOT finish up in the production environment – no excuses.

Production – “The point of no return”

There isn’t much to say about production, I recommend visiting List of religions and spiritual traditions on Wikipedia and picking who to pray to that nothing breaks. Once you stop praying that nothing breaks and start praying that the tests you have written have good coverage you know you’re getting better. Once you stop praying even that the tests are ok, and leave the office immediately after deploying to production, you know that you are a sociopath who just likes to watch the world burn, congratulations!


CI tool – I’m currently experimenting with Go CD which, coming from a short and troublesome experience with Jenkins seems like a nice refreshment.

Anti-concurrent deployments – this is a major issue when you’ve built an automated workflow from development to production, you don’t want people running deployments at the same time, because something will break, and it will break hard. If you can’t setup this kind of control in your CI tool, I recommend Etsy’s PushBot which is an IRC Bot which allows developers to queue in for their turn on deploying.

Military-grade ACLs – you don’t want to trust no-one, not even yourself. Granularize access to certain parts of the workflow wherever, whenever possible. A good practice would be to implement a sharded key shared by multiple members of the team for deploying changes to production, after successfully passing the tests in staging environment.

123 thoughts on “Ansible DTAP – Development, testing, acceptance and production”

  1. I was extremely pleased to discover this web site.
    I want to to thank you for your time just for this fantastic read!!
    I definitely liked every little bit of it and I
    have you saved to fav to look at new things in your web site.

  2. In June 2014-days before the previously declared shutdown day -Microsoft reported it’d the
    GFWL servers to actually shut down, thus games could continue to function generally.

  3. I think that what you typed made a lot of sense.
    But, what about this? what if you typed a catchier title?
    I ain’t suggesting your information is not good., but what
    if you added something that grabbed folk’s attention?
    I mean Ansible DTAP – Development, testing, acceptance
    and production – Lazar's blog is kinda
    vanilla. You could look at Yahoo’s front page and watch how they create article
    headlines to grab viewers interested. You might add a related video or a picture or two to get readers interested about
    what you’ve written. Just my opinion, it could make your posts a little bit more

  4. I am really enjoying the theme/design of your blog. Do you ever
    run into any web browser compatibility issues?

    A couple of my blog visitors have complained about my website not operating correctly in Explorer but looks great in Firefox.
    Do you have any suggestions to help fix this problem?

  5. Actually no matter if someone doesn’t know then its up to
    other viewers that they will assist, so here it takes place.

  6. “Sport Bild”, sagte, dass, obwohl die Heimmannschaft Stuttgart herabgestuft , aber Gross Crowe Izz Bayern scheint die Energie zu einem gewissen Spott zu haben.

    Wolfsburg 1-3 verloren die Herabstufung, um zu
    bestimmen, wegen einer Oberschenkelmuskelverletzung
    Abwesenheit von Konkurrenz Groß Kreutz noch im Gange in Gegenwart eines alternativen Bürste mit einem Gefühl der Gegenwart, schrieb er auf seinem Instagram: “Ich schwöre, : bis zum nächsten Jahr warten, um die zweite Liga Meisterschaft zu bekommen, ich einen neuen Anfang sind wir eine gute Sache ist, auch jetzt verletzt vielleicht Meister Bayern Leistung sicherlich die Bundesliga spannender !!! bekommen würde, aber ich bin ein sicher !!! Dinge, wir kommen wieder !!! ich bin voller Stolz, Leidenschaft und Ruhm !!! wir werden bald in der Bundesliga sein Abschied, und Stuttgart in der gleichen. ”

  7. Nuestro equipo siempre y en toda circunstancia hace que
    estos pedidos web, puesto que tienen una extensa
    gama de productos y unos costes muy razonables. Siempre y en toda circunstancia tenemos un montón de peleas para decidir
    que queremos comer var todos y cada uno de los días, mas pronto decidió que ellen online
    y tener la puerta en un abrir y cerrar de ojos.
    ¿Cómo podría ser de otra forma, utilizar nuestro cada día
    para asegurar a cifras código mínimo, por el hecho
    de que nosotros, sabemos mucho de los ahorros.

    Si vas a emprender, diferentes comidas del planeta todos procuran, no se puede perder.
    Siempre y en toda circunstancia hay una razón para sorprender
    a una orden o la cena para el almuerzo o a fin de que tus amigos y familiares con sabores especiales para una ocasión única.
    Anímate! Pregunta, además de su dirección de mail, la ciudad donde vive, porque entonces qué Especial va a recibir totalmente personalizado.
    Es muy bueno!

  8. Harry – Kane made it clear that he will be very keen to get the opportunity to England is printed pioneer,
    Wayne Rooney change to play midfield retreat will be very beautiful.

    In the just concluded Premier League 2015/16 season, and
    Harry – Kane as Tottenham scored 25 league goals, he also won the Premier League’s top scorer.
    The top scorer in the Premier League came in Harry – after Kane,
    it is Leicester City striker Valdivia, he scored 24 goals.

    And because plagued by injuries throughout the season Rooney
    scored just eight league goals, he is more frequent in the United avant-garde line in the end stage of the season. However, Harry – Kane opinion, Rooney midfield is still able to give the
    team a huge help.

  9. Can I advise you resolve the cellular type of your articles so are theren’t
    countless advertising ads because it turned impossible to pay attention to
    the topic matter distracting me.I needed to give up actually scanning this report about iPad games on my
    real iPad.

  10. magnificent submit, very informative. I wonder why the opposite specialists of this sector don’t
    realize this. You must continue your writing. I am sure, you’ve a huge readers’ base already!

  11. Appreciating the hard work you put into your blog and detailed information you present.
    It’s great to come across a blog every once in a while that isn’t the same outdated
    rehashed information. Great read! I’ve saved your
    site and I’m adding your RSS feeds to my Google account.

  12. Hello there, You have done a great job. I will definitely digg it and personally suggest to my friends.
    I am confident they will be benefited from this web site.

  13. This is a really good tip especially to
    those fresh to the blogosphere. Brief but very accurate information? Thanks
    for sharing this one. A must read article!

  14. In what happens next since it’s a structure followed by different
    activities you’ve then probably thought, – you check out a garage
    where you can spend your earnings on automobiles, paint jobs
    or improvements.

  15. Our Wonder Potential Combat Compromise is in the on-line edition so that it works together with all devices for example: iOS, Android
    & Windows and the method adjustments are performed on our central hosts.

  16. The maximum characters can be united by you for the epic combat that
    may come to a decision the fate of all facts – MIRACLE Potential Fight from all sides of
    the Marvel Universe!

  17. I have been exploring for a bit for any high quality articles or weblog posts in this sort of area .
    Exploring in Yahoo I ultimately stumbled upon this website.
    Studying this info So i am happy to express that I’ve an incredibly excellent
    uncanny feeling I discovered exactly what I needed.
    I most indubitably will make certain to do not forget
    this website and give it a glance on a relentless basis.

  18. Unquestionably imagine that that you stated.
    Your favourite reason appeared to be on the net the simplest
    thing to take into accout of. I say to you, I certainly get irked while other folks
    consider issues that they just do not recognize about. You controlled to hit the nail upon the highest and defined out the whole thing without having
    side effect , other folks could take a signal. Will probably be back to get more.
    Thank you

    Buying a new or used vehicle might be a hard method unless you know what you really are doing. By teaching yourself about car store shopping before you visit the dealership, you can make stuff less difficult yourself. The following tips can help your next shopping journey be a little more satisfying.

    Always provide a technician coupled when looking for a new car. Automobile merchants are well known for selling lemons and you may not need to be their following patient. Whenever you can not get a auto technician to look at autos with you, at least be sure that you have him look at your ultimate selection prior to buying it.

    Know your limits. Before starting buying for your forthcoming automobile or pickup truck, choose how much you can manage to pay out, and adhere to it. Don’t overlook to incorporate desire for your calculations. You can expect to pay out about 20 % as a payment in advance too, so be ready.

    Well before visiting a dealership, know what type of motor vehicle you want. Analysis each one of you alternatives just before buying so that you can determine what works well with your financial allowance and loved ones requirements. Do your homework to discover how much you must pay for any possible automobile.

    Before you sign any contract spend some time to read through every single collection, such as the fine print. If there is anything outlined you do not fully grasp, tend not to indication till you purchase an solution that you comprehend. Unsavory salesmen can make use of a binding agreement to put many charges that were not mentioned.

    In the event you keep the previous guidance at heart next time which you go looking for a auto, you may be more prone to obtain a good offer. Investing in a car lacks to become head ache. Just use the tips out of this write-up and you could receive the auto you need at a great price.

  20. Good day I am so grateful I found your blog, I really found you by mistake, while I was researching on Google for
    something else, Regardless I am here now and would just like to
    say thanks a lot for a incredible post and
    a all round interesting blog (I also love the theme/design),
    I don’t have time to read it all at the moment but I have saved it and
    also added in your RSS feeds, so when I have time I will
    be back to read more, Please do keep up the fantastic b.

  21. Right here is the right website for anyone who wishes to understand this topic.
    You understand a whole lot its almost hard to argue with
    you (not that I personally would want to…HaHa).
    You certainly put a brand new spin on a subject which has been written about for years.
    Great stuff, just excellent!

  22. Fantastic blog you have here but I was curious if you knew
    of any discussion boards that cover the same topics discussed here?

    I’d really like to be a part of community where I
    can get feed-back from other knowledgeable individuals that share the same interest.
    If you have any suggestions, please let me know. Kudos!

  23. Very good info. Lucky me I found your site
    by accident (stumbleupon). I have book-marked it for later!

  24. When I initially commented I appear to have clicked the -Notify me when new comments are added- checkbox and now each time a comment is added I receive 4 emails with the exact
    same comment. Perhaps there is an easy method you are able to remove me from that service?

  25. Hello there I am so thrilled I found your webpage, I really found you by accident, while I
    was looking on Google for something else, Anyways I am here now and would
    just like to say thanks for a incredible post and a all round
    thrilling blog (I also love the theme/design),
    I don’t have time to look over it all at the minute but I have book-marked it and also included your RSS
    feeds, so when I have time I will be back to read much more, Please do
    keep up the awesome work.

  26. Unquestionably believe that which you said. Your favorite justification appeared to be on the web the simplest thing
    to be aware of. I say to you, I certainly get annoyed while
    people think about worries that they plainly don’t know about.
    You managed to hit the nail upon the top and defined
    out the whole thing without having side effect , people could take a signal.
    Will likely be back to get more. Thanks

  27. Thank you, I’ve just been searching for information about this subject for a while and yours is the greatest I have came upon so far.
    But, what concerning the conclusion? Are you certain about
    the source?

  28. A fascinating discussion is definitely worth comment. There’s no doubt that that you need
    to publish more about this topic, it might not be a taboo subject but usually folks don’t
    speak about these topics. To the next! Kind regards!!

  29. I just like the helpful info you provide on your articles.
    I’ll bookmark your weblog and check again here regularly.

    I am fairly certain I will be told plenty of new stuff right here!

    Best of luck for the following!

  30. Pretty element of content. I just stumbled upon your blog and in accession capital to say that I get actually enjoyed account your weblog posts.
    Anyway I’ll be subscribing on your feeds and even I fulfillment you get entry to
    persistently quickly.

  31. I do not write a ton of remarks, but i did a few
    searching and wound up here Ansible DTAP – Development,
    testing, acceptance and production – Lazar's blog.
    And I do have a couple of questions for you if it’s allright.

    Could it be just me or does it seem like some of these responses come across like they are written by brain dead
    individuals? 😛 And, if you are posting at other online sites, I’d like to follow everything fresh you have to post.
    Could you make a list of the complete urls of your shared pages
    like your Facebook page, twitter feed, or linkedin profile?

  32. I blog often and I really appreciate your content.
    This article has really peaked my interest. I’m going to book mark
    your blog and keep checking for new details about once per week.
    I subscribed to your Feed as well.

  33. It’s really a nice and useful piece of information. I’m satisfied that
    you shared this useful information with us. Please stay us up
    to date like this. Thanks for sharing.

  34. Hi blogger, i must say you have very interesting posts
    here. Your website can go viral. You need initial traffic
    boost only. How to get it? Search for: Mertiso’s tips go viral

  35. Thanks , I’ve recently been searching for information about this
    topic for a while and yours is the greatest I’ve came
    upon till now. However, what about the bottom line? Are you certain in regards to the supply?

  36. Definitely imagine that which you said. Your favourite
    justification seemed to be on the internet the easiest thing to remember of.
    I say to you, I definitely get annoyed whilst other folks consider concerns that they plainly don’t realize about.
    You controlled to hit the nail upon the top and
    outlined out the entire thing with no need side effect , folks could take a signal.
    Will likely be back to get more. Thanks

  37. Pretty nice post. I simply stumbled upon your blog and wished to say
    that I have really loved browsing your weblog
    posts. After all I’ll be subscribing in your rss feed and I am hoping you write again very soon!

  38. You are so cool! I don’t think I’ve read through
    something like this before. So wonderful to discover someone with a few genuine thoughts on this issue.
    Really.. many thanks for starting this up. This web site is something that’s needed on the internet, someone with some originality!

  39. I do not know if it’s just me or if everyone else encountering issues with your website.
    It appears as if some of the written text on your
    content are running off the screen. Can someone else please provide feedback and
    let me know if this is happening to them too?
    This might be a problem with my browser because I’ve had this
    happen previously. Thanks

  40. Accordingly, the mechanics associated with promos in Facebook are generally easily manipulated or
    cheated by individuals who have a sizable network of
    friends. Social Search: Social search can be a new term that many of folks involved inside game
    of website marketing are speaking about, these days.
    These reports will likely be accessible to Ad words users within the following weeks; however advertisers (Google’s customers) have to have Ad – Words
    Conversion Tracking enabled.

  41. And, together with the various minerals and nutrients in Himalayan Crystal salt, you will discover calcium:
    a mineral which fortifies as well as whitens your teeth.
    If the region that is affected is rather small, mold remediation can likely be performed by you.

    Salt is certainly the most famous materials all over the world for
    melting ice.

  42. These work best approaches to be assured
    to getting to the top level in the search results. There may be a good amount of SEO guides and blogs on the net that
    tackle top from the line ways of achieve high Google rankings,
    but I’m sure that it is possible to many web owners on the market
    who will be scrambling their supply of high Google rankings.
    One of the most common mistakes we got were title tags incorrectly written.

  43. However, the entire process of discovering the right
    Wii games for kids could be overwhelming for a lot of parents, seeing
    as there are hundreds of choices and dozens more new games being
    included with industry each week. However, in most
    versatile version, the type is same though the expeditions are different.
    This kind of ‘stealth learning’ can be done with just about any domestic activity.

  44. My developer is trying to persuade me to move to .net from
    PHP. I have always disliked the idea because of the costs.

    But he’s tryiong none the less. I’ve been using WordPress on numerous websites for about a year and
    am worried about switching to another platform.
    I have heard good things about Is there a way I can import all my wordpress posts into it?
    Any kind of help would be really appreciated!

  45. I was curious if you ever thought of changing the
    layout of your site? Its very well written; I love what youve got
    to say. But maybe you could a little more in the way of content so people could connect with
    it better. Youve got an awful lot of text for only having one or 2 pictures.
    Maybe you could space it out better?

  46. Great post. I used to be checking continuously this blog and I am inspired!
    Extremely helpful info specially the last section 🙂 I handle
    such information much. I used to be looking for this particular info for a very long time.
    Thanks and best of luck.

  47. Great write up and piece of work i have come across so far, keep it up. i must say it was very much helpful to boost my understanding, will definitely recommend this blog.

  48. I was just looking for this information for a while. After six hours of continuous Googling, at last I got it in your site.

    Thanks for sharing superb information. Your website is so cool. I’m impressed by the details that you’ve on this blog. It reveals how nicely you perceive this subject.

  49. Awesome website you have here but I was curious if you knew of any discussion boards that cover the
    same topics discussed here? I’d really love to be a part of community where I can get feedback from
    other experienced individuals that share the same interest.
    If you have any suggestions, please let me know. Cheers!

  50. It’s actually a great and helpful piece of information. I
    am glad that you just shared this helpful info with us.

    Please stay us informed like this. Thank you for sharing.

  51. I got this web site from my pal who informed me about this website
    and now this time I am visiting this web site
    and reading very informative posts here.

  52. We absolutely love your blog and find the majority of your post’s to be what precisely
    I’m looking for. Would you offer guest writers to write content in your case?

    I wouldn’t mind publishing a post or elaborating on a number of the subjects you write in relation to
    here. Again, awesome blog!

  53. Excellent pieces. Keep posting such kind of information on your site.
    Im really impressed by it.
    Hello there, You have performed an incredible job.
    I will definitely digg it and individually recommend to my friends.
    I am sure they’ll be benefited from this web site.

  54. you are actually a excellent webmaster. The
    website loading pace is incredible. It kind of feels that you are doing any
    distinctive trick. Moreover, The contents are masterwork.
    you have done a excellent activity on this topic!

  55. You actually make it appear so easy together with your presentation however I find
    this matter to be actually something which I feel I’d
    by no means understand. It kind of feels too complicated and extremely wide for
    me. I’m looking ahead on your subsequent submit, I’ll try to
    get the cling of it!

  56. hOur company provides a wide variety of non prescription products. Look at our health website in case you want to to improve your health with a help of health products. Our site offers a wide variety of non prescription products. Look at our health site in case you want to feel better with a help health products. Our company offers a wide variety of health products. Visit our health contributing portal in case you want to feel better. Our company provides generic pills. Visit our health contributing site in case you want to look better. Our company provides a wide variety of non prescription drugs. Look at our health website in case you want to look healthier. Our company offers weight loss products. Look at our health contributing website in case you want to improve your health.
    Our company offers a wide variety of non prescription products. Take a look at our health portal in case you want to to feel healthier with a help health products. Our company offers weight loss products. Take a look at our health contributing website in case you want to look better. Our company provides a wide variety of non prescription drugs. Look at our health website in case you want to strengthen your health with a help health products. Our company offers weight loss products. Visit our health contributing portal in case you want to look better. Our company offers a wide variety of non prescription products. Visit our health site in case you want to strengthen your health with a help of general health products.

  57. Right here is the perfect blog for anybody who really wants to
    find out about this topic. You understand a whole lot its almost hard to argue with you (not that I actually would want to…HaHa).
    You certainly put a fresh spin on a subject which has been discussed for many years.
    Great stuff, just excellent!

  58. hi!,I love your writing very a lot! share we keep in touch extra approximately
    your post on AOL? I require an expert in this area to
    resolve my problem. Maybe that is you! Taking a look
    forward to peer you.

  59. ACLS-PALS-NRP.COM is one of the largest providers of online healthcare certifications and recertifications on the Internet. Being operated by critical care healthcare professionals, we understand the importance of a nurse or a doctor missing a shift because of ACLS, PALS, NRP or any other certification. We make it a priority to get you initially certified or recertified as quickly and efficiently as possible! We understand the need for high-quality and accurate material. All our study materials have been written by medical professionals who have had extensive experience in critical care and have been involved in actual codes and other healthcare management situations.

    Features include:
    • Receive an electronic card in your email as well as a hard copy card in your regular mail.
    • Retake the test up to three times if you don’t pass the first or second time
    • Download and print a card within a short while after passing your test.
    • Email and ask your online instructor questions of areas in the study material you did not understand.
    • No skills checklist required
    • Optional CE credit certificate mailed out with your completion card.
    nrp recertification

  60. Hello, I think your blog might be having browser compatibility issues.
    When I look at your blog site in Opera, it looks fine but when opening
    in Internet Explorer, it has some overlapping. I just wanted to
    give you a quick heads up! Other then that, excellent blog!

  61. Thank you so much for providing individuals with an extraordinarily brilliant opportunity to check tips from this blog.
    It really is very superb plus packed with a great time for me and my office colleagues to
    search your site more than three times per week to see the newest secrets
    you have got. Of course, we are actually amazed for the
    unique advice you serve. Certain two ideas in this article
    are definitely the simplest I have had.

  62. I don’t care about cash, so I have no fear. Don’t hurry and copy others poker moves with out thinking. So some people get but most of them finish up on the dropping side. Every yr, online casinos are providing bonuses.

  63. Hello


    – 100% safe for your site
    – real visitors with unique IPs. No bots, proxies, or datacenters
    – visitors from Search Engine (by keyword)
    – visitors from Social Media Sites (referrals)
    – visitors from any country you want (USA/UK/CA/EU…)
    – very low bounce rate
    – very long visit duration
    – multiple pages visited
    – tractable in google analytics
    – custom URL tracking provided
    – boost ranking in SERP, SEO, profit from CPM

    CLAIM YOUR 24 HOURS FREE TEST HERE=> [email protected]

    Thanks, Kandis Taubman

  64. What’s up to every one, the contents existing at this web page are genuinely remarkable
    for people experience, well, keep up the good work fellows.

  65. Your disposal might be kaput if you discern it leaking, captivating longer to
    triturate, making louder-than-usual noises, or requiring a reset often. (The reset button is like a limit breaker that needs to be
    reset after the disposal shuts off, typically because of a pressure on the motor.) In fact, in the vanguard
    you call the plumber—or throw down hundreds of dollars on a unripe grinder—simply reset the disposal
    by way of important the small (on the whole red) button on the gluteus maximus or discount backside
    of the unit. If this doesn’t achieve, check up on the circuit breaker or come together to make steady it is not tripped.
    If that doesn’t work? It may be period to date back to renege on shopping.

  66. These are examples of things you are someone to know to elevate the chances that you may earn an extent in an industry you like, and which you will have a significant shot at getting some playing time when you receive there.lcd dispay

Leave a Reply

Your email address will not be published. Required fields are marked *