Terraform deploy system
Terraform is an infrastructure-as-code tool that allows you to define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. CiviForm provides Terraform configuration files that allow you to deploy CiviForm on Azure and AWS. Knowledge of Terraform is not required to run them, but reading the high-level Terraform overview may be useful.
You will need some values that are configured outside of CiviForm before you start the setup. Some of the steps are optional, meaning that you can bring up a staging environment and get the app working without them, but they will need to be completed for production setup.
- (Optional) Admin auth client_id, client_secret, and discovery_uri. See setting up Azure AD for an example
- (Optional) Applicant auth client_id, client_secret, and discovery_uri. See setting up the Authentication Providers
- Domain name for your deployment. For example
civiform.mycity.gov
- 1.
- 2.Clone the repo onto the machine you are deploying from. Ideally, this would be a shared instance that multiple people can log onto.
- 3.
- 4.Copy the
civiform_config.example.sh
intociviform_config.sh
and fill out the missing values. You can get a sense of required values depending on your cloud provider by looking at staging-azure or staging-aws configs. - 5.Run
bin/doctor
and install the dependencies. - 6.Run
bin/setup
. What to expect:- Takes 5-10 minutes to run.
- Terraform brings up resources in the cloud (database, network, server, etc).
- Asks confirmation a few times before creating resources, listing everything that will be created.
- Safe to re-run the script if it fails. There is a known issue where
bin/setup
fails on the first run. - The configuration values in
civiform_config.sh
represent the desired state of your CiviForm deployment. Thebin/setup
andbin/deploy
commands work to make your cloud environment match the desired state. If a command fails, your cloud environment may not match the desired state. These commands are safe to retry if they fail. If a command is persistently failing, you can work with our on-call engineer to resolve the issue. Our on-call engineer responds to new issues in the CiviForm issue tracker.
- 1.
- 2.Update the
CIVIFORM_VERSION
value inciviform_config.sh
. - 3.Run
bin/deploy
.
This error can happen when running
bin/setup
for the first time. If you see it, re-run bin/setup
. This is a known Terraform bug.The deploy command is idempotent, so if it fails, try running it again. The setup command can also be re-run, but it resets a lot of variables which are kind of a pain to continually set up when you run it for Azure.
If changes were made upstream, you can change the code in the checkout env, but will need to commit PRs to fix in the main repo.
If you see error like "no such file or directory"
./db-connection: line 2: cloud/azure/bin/lib.sh: No such file or directory
./db-connection: line 21: out::error: command not found
The scripts expect you to be in specific directories. You probably need to
cd
into the checkout directory or the top level directory. If you are running setup
/deploy
/revert
, you will need to be in the top level directory. If you are running a script like db-connection
, you need to be in the checkout directory.This situation can happen when exiting deployment scripts using "Ctrl-C". Terraform acquires a lock every time you run
./bin/deploy
or ./bin/setup
, and releases the lock at the end of the script. This helps to prevent concurrent infrastructure changes. If the deploy process exited outside of Terraform, the lock remains and needs to be force removed in order to run deploy again. To do it, run the following command (assuming you deploy on AWS):terraform -chdir=checkout/cloud/aws/templates/aws_oidc force-unlock $LOCK_ID
Follow official documentation. After your certificate is ready and validated, copy the ARN from your console and put it into the
SSL_CERTIFICATE_ARN
variable in your civiform_config.sh
ARN has format of arn:aws:acm::<user_id>:certificate/<identifier>

Screen Shot 2022-08-10 at 3 50 39 PM
The setup script prompts you to set up Azure AD. There are a few additional steps.
In Azure Portal:
- Go to Azure Active Directory -> App registrations
- Click the tab for "All applications"
- Find your app (staging-dynamic-heron for CiviForm Azure Staging)
Use the menu on the left:
- Authentication: setup the Redirect URI to be what the app expects: https://<custom_hostname>/callback/AdClient.
- You will also need an admin group which creates CiviForm admins
- Token configuration: To allow for CiviForm admins, you need to have Azure AD return the groups claim. Add the security groups claim (you can verify the groups claim is being returned by decoding the base64 token from the token you get back from Azure AD on the website-- if you preserve the log in the Chrome Dev Tool window it should be from https://<custom_hostname>/callback/AdClient)
We support on-demand deployment of the pgadmin web UI to access the CiviForm database. We require explicit IP allow-listing via a list of CIDR blocks. Only these IPs will be able to access pgadmin. The public IP of the host running the web browser used to access the pgadmin web UI (like your work laptop/desktop) must be covered by a block in the list. To detect the public IP of a host running a web browser, visit https://checkip.amazonaws.com.
- 1.Run
bin/run
and pass inpgadmin
for the command. - 2.The deploy tool will auto-detect the public IP of the host it is running on and ask if you would like to add the IP to the allow-list. If you want the deploy tool to wait until pgadmin is available to print out connection information, enter 'y'.
- 3.Enter in a CIDR block that covers the IP of the host that will access the pgadmin web UI. If the host's IP is '127.0.0.1', enter in '127.0.0.1/32' to allow-list just that IP.
- 4.Either accept or reject the allow-list. If you previously chose to add the deploy tool's IP to the allow-list, and then reject the list, the deploy tool's IP will not be automatically added to the list again.
- 5.After accepting a list, Terraform will run to bring up the pgadmin resources. When it asks "Do you want to perform these actions?", enter "yes".
- 6.The deploy tool will attempt to connect to pgadmin every 10 seconds. When a connection is successful, the pgadmin URL and authentication information will be printed. Press Ctrl-c to shortcut this wait.
- 7.Open the pgadmin URL. Log in using the 'login email' and 'login password' printed by the deploy tool.
- 8.Expand the 'CiviForm (1)' item in the left navigation pane. You should be prompted to enter in the password to the database. Enter in the 'database password' printed by the deploy tool.
- 9.Expand the 'Databases (2)' item under the 'CiviForm (1)' item.
- 10.The 'postgres' item under the 'Databases (2)' item is the CiviForm database. Right click on the 'postgres' item and select 'Query Tool' to send commands to the database.
- 1.
cd checkout
- 2.
cloud/azure/bin/db-connection -g sgdev -d civiform-artistic-halibut -v sgdev-civiform-kv
- 1.If on WSL, figure out what the location of the dump file is (possibly
/mnt/c/..
) - 2.
cd checkout
- 3.
cloud/azure/bin/pg-restore -g sgdev -d civiform-artistic-halibut -v sgdev-civiform-kv -f /mnt/c/pg\_dump.dump -b testData.dump
- 1.
cd checkout
- 2.From checkout directory run
cloud/azure/bin/db-connection -d <database_name> -g <resource_group> -v <keyvault>
Note that the database_name does not include the .postgres.database - 3.Wait for the application to say you can run sql (try running
\dt
to make sure you can see the data) - 4.Run the following commands (has to be in order)
DELETE FROM applications;
DELETE FROM applicants;
DELETE FROM accounts;
DELETE FROM programs;
DELETE FROM questions;
DELETE FROM ti_organizations;
DELETE FROM versions;
DELETE FROM versions_programs;
DELETE FROM versions_questions;
Last modified 13d ago