Find the whole Code of Conduct from ThoughtWorks here:https://www.thoughtworks.com/code-of-conduct-germany

Just ping your questions to the chat and we will gather them to make sure we don’t miss anything. :-)

Question: Will you be doomed if state gets somehow lost? Can you recover from a state loss?

you should store your state in a high durability place, such as S3

not exactly but close enough

Terraform gives the possibility to import ressources.

you can import existing infrastructure into a state

terraform import can help

You should backup your state file aswell :-) maybe work with snapshots bevor/after apply

Does terraform give a graphical representation of the infra getting created like for e.g cloud formation designer provides?

But there is no import all yet … it is per resource (at least last I checked)

Question: what happens if resorces were changed outside terraform?

Followup question: What if your state loss happens because a state apply failed and you have an intermediate state?

not every resource can be imported

You should take care of import. If the provider is implemented badly it won't help you very much

Do you persist the state then?

When a state apply fails because AWS rejects the request?

But most of the time they just get destroyed and recreated

the state needs to be persisted in any case. Because if you don’t two different people applying terraform code would recreate resources

On which service does terraform run on?

terraform usually stores backup file while apply

Is there a possibility to snapshot tf-state in a storage-account?

How do changes to resources, which were triggered manually, affect the terraform state?

All major cloud providers, you can even order Dominoes Pizza with Terraform @Rajan

yes

in CI

:-)

Important to mention it doesn’t run all the time … it’s not a service or something … it’s run in a pipeline, makes changes and exits

Recommendation for state version is to enable it on the remote storage eg S3 supports versioning of files

These are the supported systems that Terraform can configure: https://www.terraform.io/docs/providers/index.html

the state in the end is a JSON File. If everything else fails, things can fixed by editing the file and reuploading it, although this is a recipe for pain

terraform cloud is free for state storage as well

My team works with AWS, Github and K8S providers

Indeed there is also Terraform cloud, we can’t use that as our clients have restrictions in place about where sensitive data eg state files can be sotred

tdd is kinda impossible with terraform

terratest is ok to use

but doesn't help much in tdd

Terraform vs. Pulumi…. Which is gaining more traction these days?

terraform

terraform has a bigger offering of providers, pulumi is in very active development as well

pulumi is gaining traction

So how big was the blast radius on your projects outage?

:)

if you are starting from scratch I would encourage to check both and see which one fits your use case better

It depends on how you consider TDD, my team have written tests and then written the terraform to deploy working code to make the tests go green - this is out of scope of this talk but we have run talks on infra testing before and will again

My team have focused on Terraform as it’s a common language and tool set to work with Github, K8S and AWS

interesting

any links Tim?

What does “Go for immutable infra” mean exactly? Other than a DB

No links to hand sorry

@Tim, what does it mean to split state? Is this an outcome of infra modularity?

should we split state for each environment?

YES

immutable infra would be deploying infrastructure that is not changed until the next deployment. For instance, all the networking is based on the TF code reflecting the end state.

Thanks @Mario

I have a question, where do you run terraform code, in azure devops, azure container or azure shell, or local laptop?

Is the concept of pipelines and stages applicable with other tools than Terraforn? E.g. with Ansible?

yes, though the declarative nature of terraform tends to make describing the end state a bit easier

For my team we run all terraform code in pipeline only, we do not run terraform locally at all and have not creds to do so. My team make an active choice to do this.

Quick question: It looks like your terraform code for many services is all in a single “infra” repo - was there any intentional decision to do this instead of putting the terraform scripts alongside the service code inside each services repository? What drove that decision?

Our CI is GoCD running in K8S containers

tim do you use atlantis?

We’ve just started using Atlantis … it’s nice :D

Not yet, we are considering it in the future

Tim, thanks a lot, do you use HachiCorp Vault for secrets or some parts of GoCD for secrets?

I don't like workspaces

Currently we are using K8S “secrets” we will be moving to AWS secrets store “soon” and then in a while we will be standing up vault

What’s the best place to version control infrastructure code, in repo with application code or in separate repo?

My team is building a platform in AWS for a large Thoughtworks client. We are currently getting our MVP up and running for our first team to move to our platform

Plus Locks can only be removed if you have the right permissions.

Did you discuss splitting state even more, e.g. one statefile per service? Why did you decide against it?

monorepo vs mico repos with code?

and last thing, do you use tools like test terratest, kitchen-terraform, validate? or some part of GoCD can help with testing?

I think version control answer is “it depends” and it also depends how tightly linked your code and infra are

Is there anyone who experienced with Jenkins for CI pipeline? How good is Jenkins about Terraform CI pipelining?

My team currently tests with tflint and terraform validate, we have additional in pipeline tests to ensure resources are available and configured correctly.

We are also looking at awsspec for testing

works good

I have 0 Jenkins experience I think that @Mario has much more

I see a lot of azure specific configuration? I was expecting something more platform independent so you can describe the state independent of where it is applied (Azure, AWS, etc)?

Have you looked at terragrunt at all? It can handle splitting and templating of environments. Well worth a look if it fits your needs

any CI tool can be used to be honest, though I’ve seen many centralized jenkins instances that are exceedingly hard to use

why not to split states down to each independent servece?

+1 to terragrunt, can be used effectively to address some terraform pain points

Terragrunt is something that we are looking at next week to fix some of the pain of copy pasted variables

Why not split to each service: There are tons of interdependencies you can create between services and passing variables back and fourth becomes a PITA

I think terragrunt will be obsolete in some time as tf adds those teatures

My team uses both terraform data calls and terraform remote_state as data sources to pass information between small pieces of terraform code

we use remote state too

how "cloud specific" is the resulting terraform code? I'm asking since my team might have to e.g. provision k8s and other services on multiple cloud service providers.

very specific

since resources change by clouds

The k8s terraform will apply to most versions of k8s, but different cloud providers provide different resources as Ashish says

The concepts however are always the same, resources / data objects same templating engine etc

thanks

If you want to create a kubernetes cluster, just take a look at the differences between the terraform resource “azurerm_kubenretes_cluster” and “google_container_cluster” … you have to handle ALL those differences yourself

did you. setup pipeline for terraform right away or after your infra reached a certain state

?

My team went straight to pipeline never ran it locally

State described well here. Also the graph tool https://thorsten-hans.com/terraform-state-demystified

But we where starting with a Greenfield setup and we had the opportunity to make a clear choice of pipelines only

We have sometimes needed to do interactive terraform but only a few times in the last 6 months. We do this from the CI/CDs

agents*

gotcha

We actually tag all our infra with the unique pipeline ID as well so we can trace AWS -> pipeline -> code

Where/how do you publish the various terraform packages? Is it just pushing a docker image to a registry every time?

You can run a terraform registry or just use git but we are not at this stage yet

How do you build the pipeline itself? Also with terraform?

Can we elaborate more on why “the download terraform package” is needed in every stage?

We actually use yaml to define them in git and then have good sutopull them

In Gitlab CI/CD you can use “artifacts” to store the package…

Thanks @Matthias

There is a GoCD terraform provider but it’s a little trickier to automate for us

What if the "test" stage in prod fails?

We've actually created AWS CodePipelines with Terraform, which then later runs Terraform.. works

That’s an interesting question Steffen, and your team should decide if it’s an alert etc

we build pipelines with terraform using aws codepipeline and codebuild

When automating terraform plan/apply in a pipeline (no manual approval), do you put any restrictions on? (i.e. don’t automatically apply if there are any destroy’s in the plan)

@TIm that then looks like dev diverges from prod, so you probably can't sooo easily push a change through

kk, thanks

Currently the Azure DevOps Terraform provider is being developed. So for the moment it is not stable and a lot of resources within Azure DevOps unfortunately can't be terraformed.

Are you running the pipeline from non-master branches? If, what happens there?

We always run from master

haha … chip is implanted

do you run azure rm templates from teraform/terragrunt?

How do you know which version or git commit of the terraform scripts was used to build the current prod?

We tag things with the pipeline ID which is made up of the git short hash + pipeline run count

Like add labels in with the git commit to the resources ?

Thanks @tim

We also do some AWS tricks to have the pipeline ID embedded in the AWS role assume process that we use

thx

For “proper” unit testing with all dependencies in memory, it’s looking compelling on prem by combining Terraform with the golang vCenter vcsim tool where you can replicate a vCenter set of resources locally. Is anyone aware of such simulation tools for any of the public clouds?

There are some AWS stub tools, I’ve never looked at them

What exactly caused your complete infra outage?

👏👏👏

How are you guys managing state migrations? For example, I want to rename a module …

great question!

Gah … enter too early :D

Caused, not cost

I want to rename a module … module.x.something to module.y.something … we’ve always had to do this “by hand” before our pipeline runs with terraform state mv … any hints on how to get this in a pipeline in a "migrations”

tf state mv works well

@Hannes BBasically applying a new resource against a new (wrong) terraform state.

Personday, not Manday ;)

Yeah sure … corrupted state ...

:)

Thanks

so there is a way to provision resources via powershell modules/integration with UI via Azure resource templates (similar to cloudformation on AWS), when you develope infrastrcuture with teraform you are not always have chnages from Azure in modules

Do you have an opinion on terra grunt?

@Sean Mitchell, grep and sed possibly? :D

and I was interested how do you integrate azure rm?

I've done a complete nested module renames

For “proper” unit testing with all dependencies in memory, it’s looking compelling on prem by combining Terraform with the golang vCenter vcsim tool where you can replicate a vCenter set of resources locally. Is anyone aware of such simulation tools for any of the public clouds?

Grep and sed … no :D terraform state mv works fine, but its always outside of our pipeline. We are adding it to our PRs right now to say “Hey, before we merge this PR into environment, do these state moves”

But how to "terraform state.." when you don't have access to the state / infra, because only your CI has @Ashish

It also happened recently when azurerm renamed some resources from like azurerm_hub_dps to azurerm_iothub_dps … terraform wanted to erase and recreate

i did it manually

not in ci

that then, of course, breaks the concept of only CI has access

🙏

yeah

what about rollbacks with terraform?

Get Michaels slides here: https://github.com/michaellihs/meetup-2020-05-07-terraform-pipelines/blob/master/presentation.pdf

but I don't see a good alternative, except that you have a pipeline which allows to do such renaming

Big appreciation for the talk!

👍👍👍

Thanks, very good structured

thank you very much

Thank you

Thanks!!! :)

Thanks

thanks 😊

Awesome!

Thanks!

Thanks!

Thank you so much :)

many thanks

Thanks :)

bye

Thank you!

thank u :)

Great talk! thanks

thanks

Thanks :)

thank you!

thx

I would do the rename via a temp pipeline

Thank you!

Thanks.

Thank you

awesome!! thank a lot TW

Thanks :-)

Thanks for a great talk

Thanks for joining everyone! :)

Thank you guys!

Thank you very much!! I really enjoyed the presentation!

cheerio!

Thank you!

Have to go grab some Pizza now :)

And also thanks to Tim for answering questions in the chat!

3 dots -> save chat

Good question, because there are good questions and answers here :-)