Our Pulumi 101

This documentation is under review

Our infrastructure code resides in the /iac folder of backend projects as a git submodule. It contains a python Pulumi program with some custom abstractions for dealing with cloud resources and credentials.

In this context, a "resource" refers to anything that can be created, edited, and deleted in the cloud provider (AWS in our case) or another service.

Another term you'll often see in this document is "Environment". Here, it refers to an application environment (think dev, staging, prod). In Pulumi, these environments are represented as Stacks.

To begin with, there are three kinds of resources:

• Dedicated: These exist for a single environment. Managed by pulumi
• Shared: These can be shared across environments. Managed by pulumi
• Imported: These can be shared across environments. Not managed by pulumi

Firstly, you need to choose wheter to import the resource or manage it through code. It's almost always preferrable to manage it through code to make replication easy, with some exceptions:

• The resource was already created outside of Pulumi and you want to reference it
• Managing it through code would be way too difficult, and it's not expected to change
• You're trying something out and will define it in code later

Once that's decided, if you chose to not import it, you need to choose whether to share it across environments or not. This is an important distinction, and the first one you should think about when setting up infrastructure. The choices to share or not share certain resources affect their security, cost, and reliability.

Here are some of the reasons why you may want to share a resource:

• It's actually needed in multiple environments (eg. dev and prod need to read from the same bucket, or a certificate would work for both environments)
• It's expensive, and wouldn't really suffer from multiple environments using it (eg. RDS instance with multiple databases)
• It's a resource for internal use, and paying for more than one wouldn't really make sense (eg. A lambda function for notifying slack every once in a while)

This should tell you which kind of resource to use. Now let's dive into some code:

This code won't actually work, it's meant to illustrate how to work with the resources

iac/__main__.py

from resources import Database, Environment, RedisInstance, Application
from lib import SharedStack
import pulumi

project_name = pulumi.project()

shared_rds_instance = Database.shared(
    resource_name='my_shared_rds',
    config=Database.Config(
        **Database.Config.default_kwargs,
        password='abc'
    )
)

imported_redis_instance = RedisInstance.imported(
    resource_name='codeleap-redis',
    arn='arn:something'
)

application = Application.shared(
    name=project_name
)

if not SharedStack.default().is_active:
    env_name = pulumi.get_stack()


    dedicated_env = Environment.dedicated(
        resource_name=f'{env_name}-env',
        application=application,
        variables={
            'DATABASE_URL': shared_rds_instance.url(for_db=f'{env_name}'),
            'REDIS_URL': imported_redis_instance.url()
        }
    )

The snippet above illustrates how to provision resources of the three kinds we saw earlier. Note that dedicated resources are provisioned inside of a conditional. The shared stack is our way of keeping resource references consistent across the individual environment stacks.

Having a shared resource also means that the shared stack must be deployed first, since the dedicated resources won't have anything to reference if it's not (eg. dedicated_env cannot be created without shared_rds_instance existing first). The imported resource does not have this constraint since it's not managed by our code

Configuring resources

There are two ways to connect resources to one another seen in the example above:

• Through values (eg. shared_rds_instance.url())
• Through kwargs (eg. application=application in Environment)

The first method will suffice for most cases. The second is used when the resource receiving the other must perform additional configuration on the received resource for proper function. Rely on autocomplete to choose which one to use.

As seen in the previous section, shared and dedicated resources configuration may be passed through the config keyword argument. Every resource comes with defaults, accessible through it's Config.default_kwargs property.

The config class just extends the arguments of the underlying pulumi resource for the purpose of providing useful defaults. You may consult these arguments on the Pulumi AWS Registry

Always pass resource_name with a descriptive (and unique!) value as well, adding the environment name if it's dedicated.

Creating new resources

If the existing library doesn't have the resource type you need, please open a PR on our pulumi repo with the new resource. Here's a simplified version of the Database resource with some comments on how to do it:

iac/resources/database.py

import pulumi
import pulumi_aws as aws
from lib.resource_config import ResourceConfig
from .base import BaseResource, ResourceType

class Database(BaseResource[aws.rds.InstanceArgs]): ## Subclass BaseResource passing along pulumi's class as a generic


    class Config(ResourceConfig[aws.rds.InstanceArgs]): ## ResourceConfig should receive pulumi's class as a generic as well
        def get_defaults(cls):
            return aws.rds.InstanceArgs( # Define defaults here
                allocated_storage=10, 
                db_name=None,
                engine="postgres",
                engine_version="17.5",
                instance_class=aws.rds.InstanceType.T3_MICRO,
                username="postgres",
                password=None,
                parameter_group_name="default.postgres17",
                skip_final_snapshot=True
            )

    _db:  aws.rds.Instance | aws.rds.AwaitableGetInstanceResult  # This should handle both the possible results of the shared function
  
    @classmethod
    def referenced(self, id:str, sg_index=0):       
        db = aws.rds.get_instance(
            db_instance_identifier=id # Just get it from aws
        )

        return Database(
            _db=db,
            type=ResourceType.IMPORTED # Used for internal logic in BaseResource
        )

 
    
    @classmethod
    def shared(
        cls,
        name:str,
        resource_name:str = "shared-rds",
        config: Config = Config.default
    ):
        
        if cls.is_shared_stack(): # For shared resources, only create them if running in the shared stack
            inst = aws.rds.Instance(
                resource_name,
                identifier=name,
                **config.__dict__
            )
            cls.export_output(resource_name, name)
        else: # Reference them from the previously exported id otherwise
            inst = aws.rds.get_instance(
                db_instance_identifier=cls.get_shared_output(resource_name)
            )
            
        return Database(
            _db=inst,
            type=ResourceType.SHARED  # Used for internal logic in BaseResource
        )

    @property
    def port(self): # Add useful properties !
        return 5432
    
    def url(self, for_db:pulumi.Input[str]|None='', using_password: pulumi.Output[str] = None): # The properties can be functions as well.
        if not for_db:
            for_db = self._db.db_name

        username = self._db.master_username if not self.is_dedicated else self._db.master_username
        
        # You may need to use inputs/outputs to achieve some data formats. See https://www.pulumi.com/docs/iac/concepts/inputs-outputs/
        return pulumi.Output.all( 
            username= username,
            host= self._db.endpoint,
            db_name= for_db, 
            password= using_password or self._db.password
        ).apply(lambda args: (
            'postgres://{username}:{password}@{host}/{db_name}'.format(
                **args
            )
        ))