https://www.piotrswiatek.dev Explore articles and guides on modern, scalable web apps. Learn and stay updated with expert insights on TypeScript, cloud, serverless and architecture. Tue, 22 Jul 2025 12:00:00 GMT https://validator.w3.org/feed/docs/rss2.html https://github.com/jpmonette/feed en https://www.piotrswiatek.dev/images/preview-card.png https://www.piotrswiatek.dev All rights reserved 2026, Piotr Swiatek AWS Serverless Node.js TypeScript System Design Asynchronous architecture GraphQL Onboarding Agile development SDLC DDD Pimp My App https://www.piotrswiatek.dev/articles/event-storming event-storming Tue, 22 Jul 2025 12:00:00 GMT {% messageCard title="Article note" %} This is the first article in the Pimp My App series. The series explores various aspects of modernizing a legacy project. Each post will dive into a different part of the transformation journey. {% br /%} This article focuses on gaining a deep understanding of the project’s domain using a technique called Event Storming. {% /messageCard %} ## The Tale of Tired Codebase It started like so many software projects do — with urgency. A few years ago, the company decided to launch its own e-commerce platform. There wasn’t much time to plan, research or architect. The business team needed something up and running — fast. The goal was clear: **start selling as soon as possible**. A small dev team got to work. They got the initial version running in record time. It did just enough to satisfy early users and gained traction. But then came the first request: *“Can we just add this small feature?”* And then another: *“We need this to go live by the end of the week”*. And another: *“Marketing needs this new flow — it’s top priority”*. There was no time to step back and think about architecture or domain design. No one documented the business logic — it lived in scattered conversations, Slack messages and the minds of a few burned-out developers. Code was copy-pasted, patched and layered on top of itself like sediment. The checkout flow alone had been reworked half a dozen times, with remnants of each version still lurking in the codebase. Today, the system still runs — barely. Everyone knows it’s risky to touch. Developers onboard onto the project with a deep sigh. No one is quite sure how one module relates to another. Every change feels like surgery without a diagnosis. When a bug shows up in production, it’s anyone’s guess what caused it or how to fix it safely. And when a stakeholder says, “Can we just add one more thing?” the engineering team winces. {% twitterPost id="1603711968971558914" /%} So here we are: a product that grew fast, delivered value, and now stands as a monument to years of unspoken decisions. The team knows something has to change. But where do you even start when you don’t understand the system you’ve built? ## Introducing Event Storming This is exactly the kind of situation where Event Storming proves valuable — not as a tool for immediate rewriting or refactoring, but as a way to take the first and most crucial step: **understanding the system**. Event Storming is a collaborative workshop technique designed to uncover how a business process actually works. Instead of starting with code or technical architecture, it starts with business events — things that happen in the domain: `OrderCreated`, `PromoCodeApplied`, `OrderShipped`, etc. In a session, people from different roles — developers, product owners, domain experts, customer support — all get in the same room (or virtual whiteboard) and map out the flow of events that make up the system’s behavior. No diagrams, no UML, no formal structure. Just sticky notes and conversations. The result? A shared understanding of how things really work — not how we think they work, or how they were supposed to work but what’s **actually happening today**. ## Types of Event Storming Sessions Event Storming isn’t a one-size-fits-all technique — it comes in different forms depending on how deep you need to go and what kind of problems you’re trying to solve. Generally, there are three main levels of Event Storming: ### Big Picture Event Storming This level focuses on mapping out an entire business domain at a high level. It’s great for identifying bounded contexts, aligning business and tech and spotting organizational bottlenecks. You’re not interested in every little detail — the goal is understanding how the system behaves as a whole. ### Process-Level *(a.k.a. Design-Level)* Event Storming This is where you zoom in on a specific business process — like “placing an order” or “processing a payment”. The focus is on how things happen step by step, and how information flows between parts of the system. It’s incredibly useful for uncovering domain rules or finding inconsistencies. ### Software Design-Level Event Storming The most granular level, where the goal is to go from domain knowledge to actual implementation details. This level often leads directly into designing aggregates, defining commands, shaping APIs and even laying groundwork for different patterns like Event Sourcing or CQRS. {% hr /%} For our e-commerce case — where we have a system that technically works but is full of unknowns — the most practical place to start is the Process-Level Event Storming. It strikes a balance between high-level clarity and actionable insight. It helps teams make sense of one critical process at a time, identify where the complexity lives, and gradually rebuild understanding from the inside out. In the rest of this article, we’ll focus on Process-Level Event Storming — what it looks like, how to run a session, and why it’s such a powerful tool for untangling messy systems like ours. ## Main Concepts and Recommendations for a Successful Session **1. Start with Domain Events** Always begin by identifying the events that occur in the business process — things that happen and are meaningful to the domain, like "Order Placed," "Payment Confirmed," or "Inventory Reserved." These are the backbone of the session. **2. Use the Ubiquitous Language** Stick to the language of the domain — not tech jargon. Everyone should be able to understand the terms being used. If two people use different words for the same thing, talk about it and pick one. **3. Visualize Everything** Use sticky notes (physical or digital) with distinct colors for different concepts {% link href="#the-color-coded-language-of-event-storming" ariaLabel="Part two of article series" %} (more on that in the next section){% /link %}. **4. Collaborate — No Spectators** Everyone in the room contributes. Developers, designers, product owners, and domain experts should all participate. Silence or dominance from one group usually leads to blind spots. **5. Embrace Imperfection** The goal is discovery, not perfection. It’s okay to be wrong, to make guesses, or to leave sticky notes that say “?” or “needs clarification.” You’re surfacing unknowns — that’s part of the process. **6. Work Left to Right, Chronologically** Events should be placed in the order they happen. This helps build a timeline of the process and reveals cause-effect relationships. **7. Highlight Hotspots and Confusion** If something causes debate, confusion, or disagreement — mark it. These are the valuable parts. They often hide complexity, misunderstanding, or poor design. **8. No Laptops, No Distractions (for in-person)** Focused attention is critical. If remote, use a tool like Miro, MURAL, or FigJam — and ask everyone to stay engaged. **9. Timebox, but Don’t Rush** Give yourself enough time — especially for the first session. Event Storming can feel chaotic at first, but the insights snowball quickly. Don’t rush to “finish”; aim to learn. ## The Art of Curiosity I want to highlight that **curiosity** is one of the most powerful tools you can bring to an Event Storming session. Rather than simply documenting what’s already known, the goal is to uncover gaps, challenge assumptions, and surface hidden complexity within the domain. Asking “why?”, “what happens next?”, or “who decides that?” can reveal critical insights that might otherwise go unnoticed. For example for every event, ask: * *What caused this?* → Leads to a Command. * *What happens next?* → Leads to more Events or Policies. In the next chapter each building block is provided with example guiding questions. ## The Color-Coded Language of Event Storming One of the most powerful aspects of Event Storming is its use of colored sticky notes to represent different elements in a system. Each color has a distinct meaning, and together they bring order to complexity by mapping out the flow of events, decisions, actions, and data across a business process. More than just visual flair, this color system makes cause and effect instantly clear — turning messy domains into collaborative, navigable models. Here are the most common sticky note colors and what they represent: ### 🟠 Domain Event (Orange) **What it is**: Something that happened in the domain and is meaningful to the business. **Why it matters**: Domain events are the backbone of the process. They describe outcomes and help trace the system's behavior over time. They also provide a shared language between tech and business. **Guiding question**: What happened?  ### 🔵 Command (Blue) **What it is**: An instruction to do something — often triggered by a user or system. **Why it matters**: Commands initiate change. They precede domain events and usually represent user intentions or system actions. Every command should ideally lead to one or more events — unless it’s rejected. **Guiding question**: What triggered the event?  ### 🟡 Aggregate / Consistency Boundary (Yellow) **What it is**: A cluster of domain objects that are treated as a single unit for consistency and transactional integrity. **Why it matters**: Aggregates define where decisions are made. They are responsible for handling commands and enforcing invariants. In practice, this is where domain logic lives. **Guiding question**: Who/what decided how to handle the command?  ### 🟣 Policy (Purple) **What it is**: A rule, condition, or automated reaction that kicks off a command when certain events occur. **Why it matters**: Policies help capture business logic and automation. They often live in the background and can be hidden inside code or processes. Making them explicit helps identify where complexity lives. **Guiding question**: What automatically happens next?  ### 🩷 External System / Actor (Pink) **What it is**: Any external service, system, or actor that interacts with your domain but isn’t controlled by it. **Why it matters**: These notes clarify integration points. Identifying external dependencies early helps manage risks, delays, and error handling in the system. **Guiding question**: What external systems interact with our domain?  ### 🟢 Read Model / View / UI (Green) **What it is**: A projection or view of data meant for display — often read-only. **Why it matters**: These notes represent the queries or views that users interact with. They help model what information needs to be available and when — especially important in systems using CQRS. **Guiding question**: What information do users need to see?  {% messageCard type="tip" title="Helpful Tip" %} The screenshots above are provided solely to illustrate examples of color usage. They were intentionally designed to clearly show how different sticky note colors are used. In a real Event Storming session, while you should start with identifying a few Domain Events to get the conversation started, you should avoid focusing exclusively on one type of element. Instead, aim to move from left to right, following the flow of data and decisions through the system, and explore the broader context collaboratively as the model evolves. {% /messageCard %} ## To put this all together Event Storming is not just a modeling technique — it’s a collaborative discovery tool that helps teams untangle complexity, bridge communication gaps, and rediscover the true shape of their systems. At its core, Event Storming is about understanding how things happen in a domain by mapping out domain events, commands, policies, and external interactions using simple, color-coded sticky notes. What sets it apart is its ability to bring technical and non-technical people into the same conversation, using shared language and visual clarity to uncover how a system really works — not how it was imagined or documented. ### Strengths One of its greatest strengths lies in its speed and low barrier to entry. Unlike formal modeling methods, Event Storming doesn’t require specialized tools or upfront structure. It embraces exploration over precision, allowing teams to quickly surface misunderstandings, design flaws, and hidden complexity. It’s especially powerful in legacy systems, where documentation is missing and institutional knowledge has eroded. As shown in example of our e-commerce app, Event Storming makes it possible to reconstruct and clarify even the most chaotic flows, step by step. ### Challenges However, Event Storming isn’t without trade-offs. Its unstructured nature can be overwhelming at first, especially for teams unfamiliar with domain-driven design or those lacking facilitation experience. Sessions can easily go off-track without clear goals or a strong facilitator. Additionally, while Event Storming excels at discovery and communication, it doesn’t provide direct implementation guidance — it needs to be paired with technical design work to turn insights into architecture. ### Summary When used intentionally, Event Storming becomes a lightweight but high-impact practice for teams struggling with unclear business logic, risky changes, or a lack of domain insight. It’s not about drawing pretty diagrams — it’s about creating shared understanding, fostering better decisions, and ultimately building better software.]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek) https://www.piotrswiatek.dev/articles/first-week-commit first-week-commit Mon, 18 Nov 2024 12:55:00 GMT ## *Innovative and Ever-Changing industry* When I asked ChatGPT to describe the IT industry, the first point mentioned was *"Innovative and Ever-Changing"*. But what exactly lies behind this overused phrase? It’s often applied to the tools, technologies, and work methodologies shaping the industry. While that’s accurate, this statement can be expanded further. What’s also innovative and ever-changing are the projects themselves and the overall nature of work in IT. A developer working on the same project for 10 years or more is almost unheard of. Compared to other industries, project rotations in IT are far more dynamic. ## Commit within the first week The title of this post suggests that a new team member should be able to make their first contribution within a week of joining the team. Of course, a week is a general benchmark—some projects might set this timeline at two weeks, while others could aim for just two days. It all depends on the specific context of the project, such as the complexity of the domain, the maturity of the project, and its current state. What’s crucial is that the timeline remains relatively short. Achieving this goal is only possible when various smaller aspects of the development process are well-optimized and standardized at a high level. ## Onboarding and Its Key Components During the onboarding process, it’s not just the new team member who learns; the team and its leaders also gain valuable insights. Observing how the new person navigates the process can reveal areas where the team or project needs improvement. Breaking this process into smaller components is particularly useful. ### Documentation First and foremost, documentation should exist—but more often than not, I’ve encountered the issue of outdated or incomplete docs. It’s also worth ensuring that a consistent notation, like {% link href="https://www.uml.org/" ariaLabel="UML org" %}UML{% /link %} or {% link href="https://c4model.com/" ariaLabel="C4 site" %}C4{% /link %}, is used so that every diagram is clear, unambiguous, and each component easily identifiable. Additionally, having deeper context on major project decisions in the form of {% link href="https://en.wikipedia.org/wiki/Request_for_Comments" ariaLabel="RFC wikipedia" %}RFCs (Request for Comments){% /link %} or {% link href="https://adr.github.io/" ariaLabel="ADR website" %} ADRs (Architecture Decision Records){% /link %} can be extremely helpful. ### Planning A new team member should be able to pick up a small, straightforward ticket that allows them to explore the codebase. Regular refinement sessions, with the entire team present, are critical here. Every ticket should include all necessary information, such as clear acceptance criteria, sufficient context, and a detailed description of changes. If a new hire’s first ticket ends up being overly complex simply because there were no smaller tasks available, it might signal an issue with ticket scoping or inaccurate estimations. ### Implementation I firmly believe every team strives to produce the highest-quality code. However, this goal is often hindered by shifting requirements, time pressure or other factors. A new team member can act as an informal auditor (within reason), bringing a fresh perspective to the existing codebase. They may identify areas where for instance design patterns are missing or where a solution requires refactoring. It’s equally important to understand why these issues exist—for instance, is the review process inadequate? Does the team lack domain or technical knowledge? If so, training may be necessary or other forms of filling up the knowledge gaps. ### Testing It’s unreasonable to expect a new team member to fully test all the pathways their changes might impact. To safeguard against regression, the project should include a robust suite of various tests, such as: * Unit tests * Integration tests * End-to-end (E2E) tests * Smoke tests * Performance tests Metrics like test coverage should be monitored, and periodic stress testing should be considered if necessary. Theoretically stress tests may not always be critical for high-traffic services, this can be offset by having excellent observability in place. ### Deployment Releasing changes should be as automated and safe as possible. My personal recommendation is to use proven patterns like {% link href="https://martinfowler.com/bliki/CanaryRelease.html" ariaLabel="canary release deep dive" %}canary releases{% /link%}, paired with automated mechanisms to prevent faulty changes from being deployed. Ideally, releases should initially target a limited group of users, and any negative deviations from the norm (e.g., increased error rates, higher latencies, or excessive RAM/CPU usage) should trigger an automatic rollback to the last stable version. The ultimate goal is to ensure that the team can deploy changes to their services regularly, while new team members feel confident that even if their changes have issues, they won’t have catastrophic consequences for the team or the project. ## Onboarding as an source of SLOs Furthermore having onboarding classified into smaller segments, the team can establish internal {% link href="https://www.ibm.com/topics/service-level-objective" ariaLabel="IBM SLO deep dive" %}Service Level Objectives (SLOs){% /link%} for each component. For instance, when discussing testing, an example SLO could be maintaining a test coverage level of 90% and having smoke tests and end-to-end (E2E) tests for critical application paths. For deployments, the team might set an objective for rollback times to the last stable version (e.g., no longer than five minutes). Other examples include limiting the time a ticket spends in the review process to 24 or 36 hours. These examples are just starting points. The key is for the team to identify which aspects are most critical to their workflow and create meaningful metrics that ensure these elements are consistently prioritized and well-maintained. ## Summary First and foremost, it’s essential to understand that the speed of onboarding should be a natural outcome of a well-functioning team that excels in every aspect of the {% link href="https://aws.amazon.com/what-is/sdlc/" ariaLabel="AWS SDLC deep dive" %}Software Development Life Cycle (SDLC){% /link%}, rather than a direct goal. The objective isn’t to force someone through onboarding and achieve their first commit within the first week. Instead, it should happen organically. Onboarding, in its various forms, is an inevitable process in any team. It’s crucial to embrace it, as the efficiency of this process can serve as an incredibly valuable indicator of the team’s and project’s overall health.]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek) https://www.piotrswiatek.dev/articles/building-secure-serverless-graphql-api-part-2 building-secure-serverless-graphql-api-part-2 Sat, 06 Jul 2024 12:55:00 GMT {% messageCard title="Article note" %} This is Part 2 of a series on building a secure serverless GraphQL API. While it's not mandatory to read {% link href="/articles/building-secure-serverless-graphql-api-part-1" ariaLabel="Part one of article series" %}the part 1{% /link %}, doing so can help understand the context better. This article delves into security-related aspects, whereas {% link href="/articles/building-secure-serverless-graphql-api-part-1" ariaLabel="Part one of article series" %}the part 1{% /link %} addresses fundamental GraphQL concepts and the process of building one. {% /messageCard %} ## Authentication Authentication is the process of verifying the identity of a user or system. By performing authentication we can add needed context to the requests. AWS AppSync supports several authentication types: * `API_KEY`, * `AWS_LAMBDA`, * `AWS_IAM`, * `OPENID_CONNECT`, * `AMAZON_COGNITO_USER_POOLS`. In case of this application `AMAZON_COGNITO_USER_POLLS` is chosen as the primary authentication type. {% link href="https://aws.amazon.com/cognito/" ariaLabel="Amazon Cognito product page" %}Amazon Cognito{% /link %} is a fully managed identity and user management service. It provides an option to create user pools which are a user directories. Let's start by creating a user pool in serverless framework. Config used for this user pool sets a policy for a password (minimum length of 8) and user attributes. ```yaml UserPool: Type: AWS::Cognito::UserPool Properties: AutoVerifiedAttributes: - email Policies: PasswordPolicy: MinimumLength: 8 RequireLowercase: false RequireNumbers: false RequireUppercase: false RequireSymbols: false UsernameAttributes: - email Schema: - AttributeDataType: String Name: name Required: false Mutable: true ``` With this defined, the next step is linking it to the AppSync API. ```yaml authentication: type: AMAZON_COGNITO_USER_POOLS config: userPoolId: !Ref UserPool ``` ### Secondary authorizer This app is an example of an online store. Some operations, like `placeOrder`, should only be done by admins or logged-in users, while others should only be available on the frontend. To handle this, feature of AppSync can be used that allows having multiple authorizers. In this case, an API key will be used to let the frontend connect to AppSync and run queries. It's a good idea to regularly rotate API keys, so a 1-month expiration time was set for this one. ```yaml additionalAuthentications: - type: API_KEY apiKeys: - name: WebClientApiKey expiresAfter: 1M ``` When both authorizers are defined, the GraphQL schema can be updated to reflect those changes. By default, when multiple authorizers are specified and the schema doesn't use directives to specify which field, query, or mutation should be used by which authorizer, AppSync assumes that the default authorizer will be used for all of them. This means it is not possible to access any of them using secondary authorizers. So in case of queries `getProducts` and `getCategories` should be allowed to be executed by both logged-in and not logged-in users. `getOrders` should only be executed by logged-in users. That's why, under the query declaration, no schema directive is used, and the query defaults to the default authorizer. ```gql type Query { getProducts: [Product!] @aws_api_key @aws_cognito_user_pools getCategories: [Category!] @aws_api_key @aws_cognito_user_pools getOrders: [Order!] } ``` ## Authorization Authorization is the process of determining and granting permissions to a user or system to access specific resources or perform certain actions. It defines what an authenticated user is allowed to do within a system. In case of this application some mutations like `placeOrder` should be allowed to be executed by all logged-in users while others like `createProduct` only be a specified group of users, for example administrators. Authorization can be achieved by first creating a user group in Cognito User Pools and then specifying operations in the GraphQL schema that should only be invoked by members of this group by appending schema directives. Let's start with creating user group ```yaml AdminGroup: Type: AWS::Cognito::UserPoolGroup Properties: GroupName: Admins UserPoolId: !Ref UserPool ``` Let's update graphql schema, bear in mind that name of the group in schema must match name of the group specified in cognito configuration. ```gql type Mutation { placeOrder(input: OrderInput!): Order! createProduct(input: ProductInput!): Product! @aws_cognito_user_pools(cognito_groups: ["Admins"]) createCategory(input: CategoryInput!): Category! @aws_cognito_user_pools(cognito_groups: ["Admins"]) startProcessingOrder(id: ID!): Order! @aws_cognito_user_pools(cognito_groups: ["Admins"]) completeOrder(id: ID!): Order! @aws_cognito_user_pools(cognito_groups: ["Admins"]) } ``` ## Applying least-privilege permissions to IAM roles When creating set of permissions to roles to attached to lambdas or other resources, grant only the ones that are need to perform a task. This way even if something goes wrong there's a smaller chance that unwanted operations will be spread across more resources. Example of limited set of privileges attached to lambda: ```yaml products: handler: src/functions/field/products/products.handler environment: PRODUCT_TABLE: !Ref ProductTable iamRoleStatements: - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt ProductTable.Arn ``` Example of non limited set of privileges attached to lambda: ```yaml products: handler: src/functions/field/products/products.handler environment: PRODUCT_TABLE: !Ref ProductTable iamRoleStatements: - Effect: Allow Action: "*" Resource: "*" ``` ## Mitigating malicious operations ### Limiting query depth GraphQL allows clients to request data in multiple ways, offering various entry points. This flexibility enables the creation of exceptionally large and nested queries, such as the following example where ```gql query NestedQueryExample { getProducts { categories { products { categories { products { categories { # And this goes on and on... } } } } } } } ``` This can be easily fixed in AppSync by specifying `queryDepthLimit` property. ```yaml queryDepthLimit: 3 ``` ### Rate limiting API rate limiting is a technique used to control the number of requests a client can make to an API within a specific time frame. It helps prevent abuse, ensures fair usage, protects against DDoS attacks, and maintains the overall performance and reliability of the API. By setting limits, such as 1000 requests per hour, developers can manage traffic effectively and provide a better experience for all users. This can be achieved by using {% link href="https://aws.amazon.com/waf/" ariaLabel="AWS WAF service page" %}AWS WAF{% /link %} service. AWS WAF (Web Application Firewall) is a security service provided by Amazon Web Services that helps protect web applications from common web exploits and attacks. It allows you to create custom security rules to block or allow specific traffic patterns, such as SQL injection, cross-site scripting (XSS) or enabling **rate limiting**. Example of enabling WAF: ```yaml waf: enabled: true rules: - throttle # limit to 100 requests per 5 minutes period ``` More advanced configuration can also be passed: ```yaml waf: enabled: true rules: - throttle: limit: 200 priority: 10 aggregateKeyType: FORWARDED_IP forwardedIPConfig: headerName: 'X-Forwarded-For' fallbackBehavior: 'MATCH' ``` ## Disabling schema introspection Disabling schema introspection in a production environment is a security best practice often implemented to protect sensitive information about the application's GraphQL schema. Schema introspection allows clients to query the schema itself, revealing details such as data types, field names, and relationships. While useful in development for debugging and client-side tooling, exposing this information in production can potentially aid attackers in understanding the structure and vulnerabilities of the API. By disabling schema introspection, developers reduce the attack surface and mitigate the risk of unauthorized access or exploitation of the GraphQL API. This can be achieved by specifying `introspection` property. ```yaml introspection: false ``` ## Observability and monitoring On the aws docs {% link href="https://aws.amazon.com/compare/the-difference-between-monitoring-and-observability/" ariaLabel="AWS docs on observability and monitoring" %} GraphQL documentation site{% /link %}, such a definition is mentioned: > Monitoring is the process of collecting data and generating reports on different metrics that define system health. Observability is a more investigative approach. It looks closely at distributed system component interactions and data collected by monitoring to find the root cause of issues. It includes activities like trace path analysis, a process that follows the path of a request through the system to identify integration failures. Monitoring collects data on individual components, and observability looks at the distributed system as a whole. Following this definition let's look at ### Observability One of the services that enables to have a observability in a system is an {% link href="https://aws.amazon.com/xray/" ariaLabel="AWS X-Ray service page" %}AWS X-Ray{% /link %}. AWS X-Ray is a service that helps developers analyze and debug applications. It provides an end-to-end view of requests as they travel through different components of an application, helping to identify performance bottlenecks, errors, and dependencies across services. In order to enable X-Ray in AppSync `xrayEnabled` property must be specified. ```yaml xrayEnabled: true ``` When X-Ray we can introspect each trace from aws console.  ### Monitoring Individual components such as Lambda can be monitored with the help of {% link href="https://aws.amazon.com/cloudwatch/" ariaLabel="AWS CloudWatch service page" %}AWS CloudWatch{% /link %}.  ## Conclusions Security is a far more complex subject then what was described here. With that being said guidelines described in this article are a nice starting point in making the apps that you build production ready. {% link href="https://github.com/piotrek12101999/building-secure-serverless-graphql-api-part-2" ariaLabel="Link to github to full code repository" %}As always here's the link to the full code repository{% /link %}.]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek) https://www.piotrswiatek.dev/articles/building-secure-serverless-graphql-api-part-1 building-secure-serverless-graphql-api-part-1 Tue, 02 Apr 2024 12:55:00 GMT {% messageCard title="Article note" %} This is Part 1 of a series on building a secure serverless GraphQL API. In this article, the focus will be on describing key concepts around GraphQL, the issues it aims to solve and implementing a basic GraphQL API using available AWS services. {% link href="/articles/building-secure-serverless-graphql-api-part-2" ariaLabel="Part two of article series" %}Part 2{% /link %} concentrates on ensuring the safety and security aspects of the API. {% /messageCard %} ## What is GraphQL On the official {% link href="https://graphql.org/learn/" ariaLabel="GraphQL documentation site" %} GraphQL documentation site{% /link %}, such a definition is mentioned: > GraphQL is a query language for your API, and a server-side runtime for executing queries using a type system you define for your data. GraphQL isn't tied to any specific database or storage engine and is instead backed by your existing code and data. Let's break down this definition. ### Query language for your API Most query languages, such as SQL, are typically used to query a specific datastore, like a MySQL database. However, GraphQL differs in that it is used to query against an API rather than a specific datastore. This means that in order to fetch data from API you must write a query that selects set of data to be returned. Below sample query is presented where data about dragon is fetched. ```gql query Dragon($id: ID!) { dragon(dragonId: $id) { name specie } } ``` ### Server-side runtime This part refers to the runtime provided by GraphQL, emphasizing that it runs on the server. There are several runtime implementations in languages such as Node.js, Java, Kotlin, C#, and many others. ### GraphQL isn't tied to any specific database or storage engine This means that GraphQL itself does not mandate the use of a particular database or storage technology. With GraphQL, the server defines a schema that describes the types of data available and the operations that can be performed on that data. Clients can then query this schema to request the specific data they need, without being concerned about the underlying data sources. This decoupling between the GraphQL schema and the data sources gives developers flexibility in choosing the appropriate databases or storage engines for their applications. They can use existing databases, microservices, third-party APIs or any other data source that can be accessed programmatically. ## BFF (Backend for Frontends) pattern ### Problem statement Consider an online shopping system that includes both mobile and web applications. Mobile screens are typically smaller than desktop ones. Therefore, when designing the user interface (UI), designers choose to display less data in the mobile version than in the desktop alternative. If the API serving the data is built on the REST architecture and is used by both mobile and desktop applications, the issue of over-fetching often arises. Let's dig deeper into this, consider such a REST API call that display all user orders. `GET /v1/orders` ```json { "orders": [{ "id": "d6b0b4ed-29b0-42d4-8708-e67e7fa01e8e", "purchaserEmail": "user@gmail.com", "status": "CREATED", "products": [{ "id": "cf4df039-3970-4d4c-99b1-b4563339b168", "name": "Basketball", "description": null, "price": "$20.00", "created": "2024-03-23T12:30:00.000Z", "categories": [{ "id": "aa470e5f-2928-45b2-b01f-2ba490737aa7", "name": "Sports" }] }, { "id": "f932995b-7ddd-4c73-b84d-e9d4a09c261a", "name": "Football", "description": "Brand new football, it's the best ball in the whole world", "price": "$25.00", "created": "2024-03-20T10:00:00.000Z", "categories": [{ "id": "aa470e5f-2928-45b2-b01f-2ba490737aa7", "name": "Sports" }] }], "totalPrice": "$45.00", "created": "2024-03-24T12:00:00.000Z" }] } ``` Look at how much data is fetched just to display a single order. On the web application UI, only fields `name`, `status`, `created`, `totalPrice` and a breakdown of the ordered products are displayed. On the other hand on a mobile phone, only the `status`, `totalPrice`, and `created` fields are shown, yet all additional data is still fetched. Such a scenario is referred to as over-fetching, where an excessive amount of data is retrieved. This results in network calls that take a long time to execute (due to transferring unnecessary data) and suboptimal memory management on the client's device. ### Traditional BFF solution Looking at the BFF pattern on the {% link href="https://learn.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends" ariaLabel="Microsoft BFF pattern documentation" %} microsoft docs{% /link %}, such a definition is mentioned: > Create separate backend services to be consumed by specific frontend applications or interfaces. That'd mean that essentially mobile app call would look like: `GET /v1/mobile/orders` ```json { "orders": [{ "id": "d6b0b4ed-29b0-42d4-8708-e67e7fa01e8e", "status": "CREATED", "totalPrice": "$45.00", "created": "2024-03-24T12:00:00.000Z" }] } ``` This can be achieved by either having two separate endpoints `/v1/mobile/orders` and `/v1/desktop/orders` or implementing API Gateway pattern that'll automatically redirect to mobile or desktop service responsible for serving proper dataset based on requests details. Both approaches can quickly become suboptimal when new devices, such as iPads, are supported in the future, requiring yet another service that will have a hybrid dataset between the extensive desktop and compact mobile datasets. ### GraphQL to the rescue! Because in GraphQL each client is responsible for declaring fields that needs to be returned by the server this issue is quickly resolved. On server schema is created that defines all possible fields to fetch and then clients selects a subset of them (or if necessary can access all fields). Need for two separate endpoints or services is no longer valid. Example GraphQL queries calls. ```graphql query MobileOrders { orders { id status totalPrice created } } ``` ```graphql query DesktopOrders { orders { id name status created totalPrice products { id name price categories { id name } } } } ``` {% messageCard type="tip" title="Helpful Tip" %} Yes that does mean that by implementing GraphQL automatically BFF pattern is implemented! {% /messageCard %} {% br /%} {% messageCard type="warning" title="Warning" %} BFF pattern is far more complex subject then simplified version described on this blog (for example under-fetching wasn't described). I highly encourage and recommend reading more about it, for example on this excellent{% link href="https://samnewman.io/patterns/architectural/bff/" ariaLabel="Sam Newman on BFF pattern" %} blog post by Sam Newman{% /link %}. {% /messageCard %} ## Building blocks of GraphQL ### Schema The schema serves as a contract between the server and clients, that defines data types available to query and relationships between them. Schema consists out of queries, mutations and subscriptions. ```gql schema { query: Query mutation: Mutation subscription: Subscription } ``` ### Types Type represents a structure of the data. There are scalar types like `ID`, `String`, `Float` or `Boolean` and complex types like custom objects, `Interface`, `Union` or `Enum`. Type defines the shape of data that can be queried. By default each field in GraphQL is nullable, by appending `!` next to the type definition for the field it's marked as not-null field. ```gql enum OrderStatus { PENDING, PROCESSING, COMPLETED } type Order { id: ID! purchaserEmail: String! status: OrderStatus! totalPrice: String! created: String! } ``` ### Query Queries are used by the clients to fetch data from the server. ```gql # Definition type Query { getOrders: [Order!] } # Client call query OrdersQuery { getOrders { id status } } ``` ### Mutations Mutations are used to modify the data on the server. ```gql input OrderInput { products: [ID!]! } # Definition type Mutation { placeOrder(input: OrderInput!): Order! } # Client call mutation PlaceOrderMutation { placeOrder(input:{ products: ["599256f1-6e03-461e-8cae-bd17ba17c1d3"] }) { id } } ``` ### Resolvers Resolvers are functions responsible for fetching the data associated with a particular field in the schema. Let's take for example `getOrders` query represented in the examples above. For this query we need to specify a resolver that will handle returning orders array. Sometimes a specific type is referencing other custom type. Let's refactor `Order` field to contain array of products. First we define custom `Product` type. ```gql type Product { id: ID! name: String! description: String price: String! created: String! } ``` No we can reference it in `Order` type. ```gql enum OrderStatus { PENDING, PROCESSING, COMPLETED } type Order { id: ID! purchaserEmail: String! status: OrderStatus! products: [Product!] # Reference to custom Product type totalPrice: String! created: String! } # Client call query OrdersQuery { getOrders { id status products { id name } } } ``` In such a case a nested resolver is a useful concept that can help. ## Application goals In this tutorial, an application will be built based on the previously mentioned online shop example. User can see products, categories and his placed orders. There are 3 queries that can be executed: * `getProducts`, * `getCategories`, * `getOrders`. User can place order. Admin can create product or category. Admin can also start processing placed orders and make them completed. For now in this tutorial all mutations are accessible even if user is not an admin. Authentication and authorization of requests is covered in {% link href="/articles/building-secure-serverless-graphql-api-part-2" ariaLabel="Part two of article series" %}part 2{% /link %}. Given mutations can be executed: * `placeOrder`, * `createProduct`, * `createCategory`, * `startProcessingOrder`, * `completeOrder`. ## Application architecture Architecture leverages 3 service: * {% link href="https://aws.amazon.com/pm/appsync/" ariaLabel="AWS AppSync product page" %}AWS AppSync{% /link %}, * {% link href="https://aws.amazon.com/lambda/" ariaLabel="Link to AWS Lambda product page" %}AWS Lambda{% /link %}, * {% link href="https://aws.amazon.com/dynamodb" ariaLabel="Link to Amazon DynamoDB product page" %}Amazon DynamoDB{% /link %}. AWS AppSync is a managed service that simplifies process of creating, deploying and managing GraphQL APIs. In AppSync data sources represent the underlying storage or compute services that resolvers interact with to retrieve or manipulate data. AppSync supports multiple data source: * Amazon DynamoDB, * AWS Lambda, * OpenSearch, * HTTP endpoints, * Amazon EventBridge, * Relational databases * None. My personal preference is to use AWS Lambda as a data source. Among many others, here are a few key benefits of this approach: * Custom Business Logic: In my opinion, it's the easiest way to append any custom business logic to your resolvers. * Easy Testing: It's straightforward to conduct unit testing for Lambda code. * Developer Experience: In the case of other data types, you are either forced to use VTL files or custom AppSync JS runtime. Traditional Lambda, with support for many tools (such as TypeScript in this tutorial), provides an excellent developer experience compared to other options. Each query or mutation is connected to the corresponding data source represented by lambda. Lambda then is responsible for performing (if needed) business logic and writing in case of mutations or reading in case of queries to database represented by DynamoDB table. This flow is represented in architecture diagram below.  In the case of nested fields AppSync is smart enough to automatically detect if the given nested field is queried. If so, it automatically triggers the resolver responsible for the nested field, which in turn executes the underlying data source with AWS Lambda attached to it. This flow is represented in architecture diagram below.  ## Schema Let's begin by designing the schema, which serves as a contract between the server and clients. Based on the fields that we define here, we can later proceed to write resolvers. Let's start by defining types that represent entities within the system. AWS AppSync offers several custom scalars such as `AWSEmail` or `AWSDateTime`, which provide more granular control over the specific data types used within the application. You can find more information about all the available AppSync-specific scalars in the {% link href="https://docs.aws.amazon.com/appsync/latest/devguide/scalars.html" ariaLabel="AppSync scalar types documentation" %}dedicated AWS AppSync page{% /link %}. ```gql type Category { id: ID! name: String! products: [Product!] } type Product { id: ID! name: String! description: String price: String! created: AWSDateTime! categories: [Category!] } enum OrderStatus { PENDING, PROCESSING, COMPLETED } type Order { id: ID! purchaserEmail: AWSEmail! status: OrderStatus! products: [Product!] totalPrice: String! created: AWSDateTime! } ``` {% messageCard type="tip" title="Helpful Tip" %} Many teams choose to make all fields nullable in the schema. This approach enables APIs to return partial data, reduces error propagation, and generally promotes flexibility, resilience, and simplicity in both schema design and client-side development. For the sake of this tutorial, I have decided to make some of the fields non-nullable to emphasize the differences in behavior and notation between nullable and non-nullable fields. {% /messageCard %} With our data types represented in the system it's possible to define queries. Notice how query can accept arguments that are used to customize response based on the user needs. ```gql type Query { getProducts: [Product!] getCategories: [Category!] getOrders(email: AWSEmail!): [Order!] } ``` The next step is to define available mutations. When mutation wants to accept more complex objects as an arguments concept of `input` can come handy. By using inputs we can define objects that can be passed as an argument instead of having lots of arguments that are responsible for passing one property at the time. ```gql enum Currency { USD, GBP } input PriceInput { currency: Currency! amount: Float! } input ProductInput { name: String! description: String price: PriceInput! categories: [ID!]! } input CategoryInput { name: String! products: [ID!]! } input OrderInput { products: [ID!]! purchaserEmail: AWSEmail! } type Mutation { placeOrder(input: OrderInput!): Order! createProduct(input: ProductInput!): Product! createCategory(input: CategoryInput!): Category! startProcessingOrder(id: ID!): Order! completeOrder(id: ID!): Order! } ``` ## Serverless framework setup As a first step let's define serverless plugin. It will be identical to the one described in {% link href="/articles/building-serverless-rest-api-with-lambda#creating-the-serverless-framework-configuration" ariaLabel="Article about building serverless rest api with serverless framework config" %}this article{% /link %}. ## AppSync API definition When schema is defined and ready to be used we can move forward to defining of the AppSync API withing serverless framework. We start with installation of the {% link href="https://github.com/sid88in/serverless-appsync-plugin/tree/master" ariaLabel="AppSync serverless plugin github" %} `serverless-appsync-plugin`{% /link %} that simplifies the process and creates abstraction over raw cloudformation types. All that we need to do is to provide a list of resolvers and data sources and plugin will do the rest. {% messageCard type="warning" title="Warning" %} The AppSync plugin requires an authentication property, which serves as the primary authentication method for accessing the API. Currently, the default `API_KEY` method is utilized to create the API. However, this is further examined and adjusted in {% link href="/articles/building-secure-serverless-graphql-api-part-2" ariaLabel="Part two of article series" %}part 2{% /link %}. {% /messageCard %} ```yaml appSync: name: secure-graphql-api schema: src/api/schema.graphql authentication: type: API_KEY apiKeys: - name: WebClientApiKey expiresAfter: 1M resolvers: Query.getOrders: kind: UNIT dataSource: getOrdersDataSource Mutation.placeOrder: kind: UNIT dataSource: placeOrderDataSource # ...rest of queries and mutations defined in the same way dataSources: getOrdersDataSource: type: AWS_LAMBDA config: functionArn: !GetAtt GetOrdersLambdaFunction.Arn placeOrderDataSource: type: AWS_LAMBDA config: functionArn: !GetAtt PlaceOrderLambdaFunction.Arn # ...rest of data source defined in the same way ``` Nested fields resolvers are defined in a such way: ```yaml appSync: # ... other properties resolvers: # ... other resolvers getProductCategories: type: Product field: categories kind: UNIT dataSource: categoriesDataSource getCategoryProducts: type: Category field: products kind: UNIT dataSource: productsDataSource ``` ## Creating persistance layer For persistance layer similar setup will be used as described in {% link href="/articles/building-serverless-rest-api-with-lambda#creating-persistance-layer" ariaLabel="Article about building serverless rest api with persistance layer config" %}this article{% /link %}. ### DynamoDB indexes Only major difference is in `OrderTable` where concept of GSI is used. Index is used to improve application read performance by allowing for querying the table using alternate attributes. In DynamoDB there are two types of indexes: * Global Secondary Index (GSI) - they can have a different partition and sort key then table's primary key, therefore they are applied to the whole table instead of single partition. * Local Secondary Index (LSI) - they share the same partition key as the table's primary key but has a different sort key, therefore it provides additional query capabilities within the same partition. Although additional indexes provides more querying possibilities and can enhance app performance it comes with some trade-offs: they increase storage and write costs, as DynamoDB maintains additional copies of indexed data. Let's proceed to creation of DynamoDB index ```yaml OrderTable: Type: AWS::DynamoDB::Table Properties: TableName: ${self:custom.resourceSlug}-order-table BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: purchaserEmail AttributeType: S - AttributeName: id AttributeType: S - AttributeName: created AttributeType: S KeySchema: - AttributeName: id KeyType: HASH GlobalSecondaryIndexes: # Here GSI and its properties is defined - IndexName: byEmailAndCreatedDate KeySchema: - AttributeName: purchaserEmail KeyType: HASH - AttributeName: created KeyType: RANGE Projection: ProjectionType: ALL ``` In the code repository query is utilized in such a way. ```ts async findAllByPurchaserEmail( email: Order["purchaserEmail"] ): Promise { const command = new QueryCommand({ TableName: this.tableName, IndexName: "byEmailAndCreatedDate", // Index usage KeyConditionExpression: "purchaserEmail = :purchaserEmail", ExpressionAttributeValues: { ":purchaserEmail": email, }, }); const { Items } = await this.docClient.send(command); return Items as Order[]; } ``` ## Creating resolvers With the API definition and persistance layer in place, we can proceed to create the resolvers. ### TypeScript types generation One of the best aspects of using GraphQL is its excellent developer experience. By defining all the entities and their relationships in the schema, TypeScript types can be generated based on it. Let's start with installing necessary dependencies. ```shell yarn add -D @graphql-codegen/cli @graphql-codegen/typescript ``` After installing dependencies, let's create a configuration file specifying the details of generation. Because AppSync-specific types were used in the schema, additional details need to be appended to the configuration so that it can recognize the properties and determine the end result of generation. ```ts import type { CodegenConfig } from "@graphql-codegen/cli"; const config: CodegenConfig = { overwrite: true, schema: ["src/api/schema.graphql", "src/api/aws.graphql"], generates: { "src/generated/graphql.ts": { plugins: ["typescript"], config: { scalars: { AWSEmail: "string", AWSTimestamp: "string", }, }, }, }, }; export default config; ``` Also additional GraphQL schema file with AWS definitions must be placed in the project. ```gql scalar AWSTime scalar AWSDateTime scalar AWSTimestamp scalar AWSEmail scalar AWSJSON scalar AWSURL scalar AWSPhone scalar AWSIPAddress scalar BigInt scalar Double directive @aws_subscribe(mutations: [String!]!) on FIELD_DEFINITION # Allows transformer libraries to deprecate directive arguments. directive @deprecated(reason: String!) on INPUT_FIELD_DEFINITION | ENUM directive @aws_auth(cognito_groups: [String!]!) on FIELD_DEFINITION directive @aws_api_key on FIELD_DEFINITION | OBJECT directive @aws_iam on FIELD_DEFINITION | OBJECT directive @aws_oidc on FIELD_DEFINITION | OBJECT directive @aws_cognito_user_pools( cognito_groups: [String!] ) on FIELD_DEFINITION | OBJECT ``` Now all it's left to do is to run the generation comment and output file should be placed in `src/generated/graphql.ts` path. ### Mutation implementation As a first step, let's create the function definition in the Serverless Framework. Technically, this could be defined in the AppSync config, but my preference is to separate these two aspects. In the API specification, I focus on API-related details such as name, authentication, resolvers, data sources, and other properties. In the function definitions, I concentrate on Lambda-specific properties like names, environment variables, or attached policies. ```yaml placeOrder: handler: src/functions/mutation/placeOrder/placeOrder.handler environment: ORDER_TABLE: !Ref OrderTable PRODUCT_TABLE: !Ref ProductTable iamRoleStatements: - Effect: Allow Action: - dynamodb:PutItem Resource: !GetAtt OrderTable.Arn - Effect: Allow Action: - dynamodb:GetItem Resource: !GetAtt ProductTable.Arn ``` This resolver is just a TypeScript lambda function that executes necessary business logic and returns `Order` object that has the same structure as defined in schema. ```ts const orderRepository: OrderRepository = new DynamoOrderRepository(); const productRepository: ProductRepository = new DynamoProductRepository(); export const handler: AppSyncResolverHandler< MutationPlaceOrderArgs, Mutation["placeOrder"] > = async (event) => { const { products: productsInput, purchaserEmail } = event.arguments.input; if (productsInput.length === 0) { throw new Error("Order must have at least one product assigned to it"); } const productWriteModels = await productRepository.findTransactionalByIds( productsInput ); const summedAllCurrencies = productWriteModels.reduce>( (acc, { price }) => { if (acc[price.currency]) { return { ...acc, [price.currency]: acc[price.currency] + price.amount, }; } return { ...acc, [price.currency]: price.amount }; }, {} ); const totalPrice = Object.keys(summedAllCurrencies) .map((key) => formatPrice({ currency: key as Currency, amount: summedAllCurrencies[key], }) ) .join(", "); const products: Product[] = productWriteModels.map( ({ price, categories, ...product }) => ({ ...product, // Here we must assert type ourself as the categories property is later on picked up by category field resolver categories: categories as unknown as Category[], price: formatPrice(price), }) ); const order: Order = { id: v4(), created: new Date().toISOString(), products, purchaserEmail, status: OrderStatus.Pending, totalPrice, }; await orderRepository.create(order); return order; }; ``` ### Query implementation Each subsequent resolver follows the same pattern: first, a function is defined in the Serverless framework, and then the actual Lambda code is written to return the appropriate entity. For example this is a `getOrders` query implementation. ```yaml getOrders: handler: src/functions/query/getOrders/getOrders.handler environment: ORDER_TABLE: !Ref OrderTable iamRoleStatements: - Effect: Allow Action: dynamodb:Query Resource: !Sub ${OrderTable.Arn}/index/byEmailAndCreatedDate ``` {% messageCard type="tip" title="Helpful Tip" %} Notice how the created DynamoDB index is utilized by specifying its name when allowing Lambda to query the table using this index `!Sub ${OrderTable.Arn}/index/byEmailAndCreatedDate`. {% /messageCard %} ```ts const orderRepository: OrderRepository = new DynamoOrderRepository(); export const handler: AppSyncResolverHandler< QueryGetOrdersArgs, Query["getOrders"] > = async (event) => { return orderRepository.findAllByPurchaserEmail(event.arguments.email); }; ``` ## Testing Let's proceed to test the application. For this purpose, several methods can be utilized, such as sending a request via Postman. However, for simple cases, I use the built-in AppSync GraphQL playground which automatically uses the generated API key as the authorization method. Let's start with creation of the categories and the products.   Let's fetch all the products and categories at the same using GraphQL superpowers to make this happen in one call.  Let's proceed to placing an order.  Once order is placed, let's fetch my orders.  ## Conclusions First and foremost, this is part 1, covering only API creation. {% link href="/articles/building-secure-serverless-graphql-api-part-2" ariaLabel="Part two of article series" %}Part 2{% /link %} focuses on security aspects. This means that, for now, this API is intentionally made insecure and **is not** production-ready. Apart from that I really do love GraphQL, after fixing some issues like N+1 queries or incomplete support from some development tools it has become my preferred method for building client-facing APIs. For server-to-server communication RPC is still a GOAT but for client calls the ease of selecting the data, tools around it (like {% link href="https://www.npmjs.com/package/@apollo/client" ariaLabel="NPM package of apollo client" %}Apollo Client{% /link %} or {% link href="#typescript-types-generation" ariaLabel="Codegen section in this article" %}codegen{% /link %} used in this tutorial), lots of learning resources about it and backing from major companies makes it a perfect choice. With that being said, this article only covers basic concepts about GraphQL. I highly encourage you to expand your knowledge by following some great blogs about GraphQL, such as: * {% link href="https://graphql.org/blog/" ariaLabel="Official GraphQL blog" %}Official GraphQL blog{% /link %}, * {% link href="https://www.apollographql.com/blog" ariaLabel="GraphQL Apollo blog" %}GraphQL Apollo blog{% /link %}, * {% link href="https://the-guild.dev/blog" ariaLabel="The guild dev Apollo blog" %}The Guild blog{% /link %}. {% link href="https://github.com/piotrek12101999/building-secure-serverless-graphql-api-part-1" ariaLabel="Link to github to full code repository" %}As always here's the link to the full code repository{% /link %}. ]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek) https://www.piotrswiatek.dev/articles/designing-idempotent-serverless-systems designing-idempotent-serverless-systems Tue, 27 Feb 2024 12:55:00 GMT ## Embracing Failures Consider hotel booking system. Booking a stay is a complex operation that "Under the hood" requires many steps to ensure a successful booking: 1. Customer initiates the process by placing request. 2. The hotel validates and confirms the booking request. 3. Room is allocated and reserved for the customer. 4. Payment processing follows to secure the reservation. 5. Confirmation is dispatched to the customer with booking details. Often such a complex operation is decomposed into a controlling process that invokes several smaller services, with each service handling a specific aspect of the entire workflow. Whenever one service or system invokes another, failures are a possibility. They can be caused by many reasons, network issues, database availability, load balancers, system errors and many more. Drawing from this example it is evident that constructing a software system is a complex and challenging process, requiring a series of smaller steps to achieve the desired outcome. Each of this steps can fail from multiple reasons. Given the complexity of even a single operation, not to mention the chain of operations, it's essential to develop systems that acknowledge the possibility of errors and are resilient to them. ### What to do in case of errors Okay, **failures happen**, we get it, so what can we do about it? It's rare that systems fail as a whole. It's more common that they suffer a partial failure when only subset of operation succeeds. The second popular option of failing is a transient failure, which occurs when a request fails for a short period of time. The easiest approach in such a case is just to retry (surprisingly, in most cases, it works really well). Retry mechanism is a default built in mechanism of many AWS services. For example, in the event of a Lambda failure, it will automatically attempt to retry the operation three times by default. For the retry mechanism to function effectively, it's essential to ensure that each operation within the system can be retried without any side effects. ## Distributed Systems Coming back to the example of the hotel booking system, it's likely that this system would be divided into microservices, effectively making it a distributed system. In distributed systems, parts of the system must communicate, a task often accomplished by implementing some form of messaging mechanism. There are many implementations available, each with its own distinct characteristics. Some use cases are more suitable for events (such as {% link href="https://aws.amazon.com/eventbridge/" ariaLabel="Amazon EventBridge product page" %}Amazon EventBridge{% /link %}), while others benefit more from queues (like {% link href="https://aws.amazon.com/sqs/" ariaLabel="Amazon SQS product page" %}Amazon SQS{% /link %}) or implementations outside of AWS, such as {% link href="https://kafka.apache.org/" ariaLabel="Main page of Kafka" %}Kafka{% /link %} or {% link href="https://www.rabbitmq.com/" ariaLabel="Main page of RabbitMQ" %}RabbitMQ{% /link %}. In summary, there are numerous methods for transmitting messages from one part of the system to another. Each of these possibilities offers something unique and also has its own unique drawbacks. However, all of these solutions share one common limitation: {% link href="https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf" ariaLabel="Whitepaper that proofs that exactly-once delivery is impossible" %}it is impossible to achieve exactly-once delivery{% /link %}. There's even this popular joke: {% twitterPost id="632260618599403520" /%} Since there isn't a guarantee that messages will be delivered only once, when designing systems, we must create a system that expects that the same message will be processed multiple times. Therefore, as with the retry strategy, the system must allow each operation to be performed multiple times without causing any side effects. ## What is idempotence Idempotence is a property in computer science and mathematics where an operation can be applied multiple times without changing the result beyond the initial application. In other words, performing the operation once has the same effect as performing it multiple times (no side effects). Here are some examples of idempotence: * {% link href="https://en.wikipedia.org/wiki/Floor_and_ceiling_functions" ariaLabel="Definitions of floor and ceiling" %}Floor and Ceiling functions{% /link %} * {% link href="https://en.wikipedia.org/wiki/Absolute_value" ariaLabel="Definition of absolute" %}Absolute values{% /link %} ### Idempotency with HTTP Request Methods Some HTTP methods are idempotent by default. A safe method is one that does not modify the state of the server, essentially meaning that is a read-only operation. Additionally, a safe method is always idempotent. The following table shows which methods are idempotent and / or safe. {% table %} * Name * Is safe? * Is idempotent? --- * `GET` * ✅ * ✅ --- * `HEAD` * ✅ * ✅ --- * `PUT` * ❌ * ✅ --- * `DELETE` * ❌ * ✅ --- * `POST` * ❌ * ❌ --- * `PATCH` * ❌ * ❌ --- {% /table %} #### GET, HEAD In case of `GET` and `HEAD` requests data is only read and no state modification happens, therefore those operations are both safe and idempotent. #### PUT In case of `PUT` request first invocation updates the resource (for instance updating customer's address). The other requests will basically overwrite the same resource again, therefore not making any change in the state. This operation is idempotent but not safe since there's a write operation happening. #### DELETE In case of `DELETE` request first invocation will delete resource ultimately resulting in `200` (or `204`) response. The other requests will result in `404` response but the state of the resources will not be updated. This operation is idempotent but not safe since there's a write operation happening. {% messageCard title="Important Note" %} When considering idempotence, it's crucial to distinguish between the results (as of response returned to the client) and any side effects it may produce. For example, with a `DELETE` operation, the response returned to the user might vary (such as a `200` response code on the first request and a `404` response code on the second), but the server's state after the initial operation remains unchanged. {% /messageCard %} #### POST In case of `POST` invoking endpoint N amount of times will result in N amount of resources being created each time changing the servers state, therefore this operation is not safe and not idempotent by default. #### PATCH In case of `PATCH` each invocation can potentially result in different outcomes. Let's take as an example updating address then we can send something like `{ "address": "foobar" }`, invoking this N amount of times will not cause any side effects but `PATCH` operation is much more general in a possible ways that updates the resource for instance `{ "op": "move", "from": "/a/b/c", "path": "/a/b/d" }`. This operation is non idempotent therefore `PATCH` is both not idempotent and not safe by default. If you want to read more about how to apply JSON patch, {% link href="https://www.rfc-editor.org/rfc/rfc6902" ariaLabel="JSON patch whitepaper" %}this resource is recommended for further reading{% /link %}. ## Idempotent API Design With that being said, we can now proceed to ensuring idempotency in API design. As previously mentioned, some operations do not require any additional work, such as a `GET` request to the `bookings/{id}` endpoint, as illustrated in the initial hotel booking example. However, in the case of a `POST` request to the bookings endpoint, it is essential to ensure that multiple invocations of the same endpoint (as in a retry strategy) do not result in multiple bookings being placed. There are several solutions to this issue; for instance, it can be assumed that multiple requests with the same body within a short time span may be duplicates of the same request. However, this assumption may not always hold true. Consider a scenario where someone wishes to book multiple rooms in a hotel for a group trip with friends. In this case, they may indeed create multiple requests to book stay at the same hotel with the same properties. ```json { "propertyId": "id", "checkInDate": "2024-02-22T14:00:00.000Z", "checkOutDate": "2024-02-24T11:00:00.000Z" } ``` My preferred approach involves attaching a unique identifier to each call within the request headers. This identifier is generated on the client side. Requests sharing the same client request identifier can be identified as duplicate requests and managed accordingly. Below is presented sequence diagram that visualize this concept.  ## Idempotent Handlers Design In the case of handlers in distributed systems that operate based on pulling messages, this approach can also be utilized. A key requirement for its effectiveness is to embed some form of unique identifier within the message body, facilitating easy identification of the message.  With the theory behind idempotency presented, we can now proceed to creating and designing an example application that demonstrates the concepts discussed in code. ## Prerequisites This is not an entry level article, it's geared towards readers that already created their first serverless applications. If you're new to serverless development, {% link href="/articles/building-serverless-rest-api-with-lambda" ariaLabel="Entry level article that teaches basics of serverless" %} this article is recommend before starting{% /link %}. It provides a step-by-step guide and covers the fundamentals. ## Application goals In this tutorial, we will build an application based on the previously mentioned hotel booking example. Placing a booking will occur by accessing the `POST` `/bookings` route, while accessing the `GET` `/bookings/{id}` route will return booking details. Accepting a booking by the host will happen by accessing the `PUT` `/bookings/{id}/accept` route. In the background, after acceptance, payment will be processed. ## Application architecture The application is divided into two parts. The first part is a synchronous API that utilizes three services: Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. The second part involves asynchronous processing of messages stored in an {% link href="https://aws.amazon.com/sqs/" ariaLabel="Amazon SQS product page" %}Amazon SQS queue{% /link %}. This setup aims to mimic a more complex distributed system where messages are processed asynchronously, in order cover both earlier discussed types of systems (synchronous API exposed to user and asynchronous distributed system that process the messages in the background). Idempotency is handled with the usage of {% link href="https://docs.powertools.aws.dev/lambda/typescript/latest/" ariaLabel="TypeScript AWS Powertools documentation page" %} AWS Powertools{% /link %}, DynamoDB is used as a state store that keeps track of idempotency keys. Diagram of the architecture is presented below.  {% messageCard type="warning" title="Warning" %} The presented architecture is designed purely for educational purposes, and as such, it includes certain simplifications that result in flaws (for example lack of implementation of a Dead Letter Queue (DLQ) for the `paymentProcessor` lambda). {% /messageCard %} {% br/ %} {% messageCard type="tip" title="Helpful Tip" %} In a real-life scenario, I would consider using events instead of messages stored in a queue. This approach would make the system more flexible, as events can be consumed by multiple consumers, rather than just a single recipient. (Of course, Event-Driven versus Message-Driven architecture is a much more complex subject, on which I am planning to write a separate article). {% /messageCard %} ## Creating persistence layer As a state store DynamoDB was chosen because of it's ease of integration with Lambda Powertools. Table was initialized with given config. ```yaml IdempotencyTable: Type: AWS::DynamoDB::Table Properties: TableName: ${self:custom.resourceSlug}-idempotency-table BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: id AttributeType: S KeySchema: - AttributeName: id KeyType: HASH TimeToLiveSpecification: AttributeName: expiration Enabled: true ``` {% messageCard type="tip" title="Helpful Tip" %} Take a look at `TimeToLiveSpecification` property. This is a neat feature of DynamoDB that allows records to be deleted after a specified timestamp. {% link href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html" ariaLabel="TypeScript AWS Powertools documentation page" %} Here is the full documentation and explanation on how it works{% /link %}. {% /messageCard %} With state store table being setup in AWS we can create a TypeScript representation. ```ts import { DynamoDBPersistenceLayer } from "@aws-lambda-powertools/idempotency/dynamodb"; import { BasePersistenceLayer } from "@aws-lambda-powertools/idempotency/persistence"; export const persistenceStore: BasePersistenceLayer = new DynamoDBPersistenceLayer({ tableName: checkForEnv(process.env.IDEMPOTENCY_TABLE), }); ``` {% messageCard title="Important Note" %} Detailed implementation of `BookingTable` and `BookingRepository` is omitted in order to make sure that the focus of article is put on idempotency-related subjects. Please refer to {% link href="#conclusion" ariaLabel="Conclusion section" %} Conclusion section where link to code repository is provided.{% /link %} {% /messageCard %} ## Creating placeBooking lambda function First step is to define a lambda in the yaml file. Notice how lambda is given permission to perform all necessary operation on idempotency state store. ```yaml placeBooking: handler: src/functions/placeBooking/placeBooking.handler environment: BOOKING_TABLE: !Ref BookingTable IDEMPOTENCY_TABLE: !Ref IdempotencyTable events: - http: path: bookings method: POST iamRoleStatements: - Effect: Allow Action: - dynamodb:GetItem - dynamodb:PutItem - dynamodb:UpdateItem - dynamodb:DeleteItem Resource: !GetAtt IdempotencyTable.Arn - Effect: Allow Action: - dynamodb:PutItem Resource: !GetAtt BookingTable.Arn ``` This lambda function is responsible for handling the `POST` operation, which creates a booking request. As previously discussed, `POST` requests are not idempotent by default, so it's necessary to use a unique idempotent identifier to safely distinguish between original and duplicated requests. This is achieved by appending `X-Idempotency-Key` to the HTTP headers. If the key is not appended, the lambda will not proceed with further processing and will return a `400` status code along with an appropriate message. An example of code that accomplishes this is presented below. ```ts if (!event.headers["X-Idempotency-Key"]) { throw new BadRequestException("Missing idempotency key"); } ``` ### Idempotency handler As the first step in creating an idempotent handler, we need to specify the part of the event that will serve as the key. This is achieved by referencing the previously mentioned `X-Idempotency-Key` header and selecting it in the `IdempotencyConfig`. Next, we create a `processingFunction` responsible for implementing the actual business logic. Then, we ensure that this processingFunction is invoked only once by wrapping it with the makeIdempotent higher-order function, and providing the configuration and the previously created persistenceStore as parameters. ```ts import { IdempotencyConfig, makeIdempotent, } from "@aws-lambda-powertools/idempotency"; const config = new IdempotencyConfig({ eventKeyJmesPath: 'headers."X-Idempotency-Key"', }); const bookingRepository: BookingRepository = new DynamoBookingRepository(); const processingFunction = async ({ body }: APIGatewayProxyEvent) => { if (!body) { throw new BadRequestException("Missing body"); } const booking: PartialBooking = JSON.parse(body); const { error } = schema.validate(booking); if (error) { throw new BadRequestException(error.message); } const id = uuid(); await bookingRepository.create({ ...booking, id, status: "REQUESTED", }); return id; }; const placeBooking = makeIdempotent(processingFunction, { persistenceStore, config, }); ``` ### Creation of the actual handler The final step is to combine all of these elements together in the actual handler. It's important to note that the context is provided to the configuration to safeguard the isolated parts of the code outside of the handler. ```ts export const handler: APIGatewayProxyHandler = httpMiddleware( async (event, context) => { if (!event.headers["X-Idempotency-Key"]) { throw new BadRequestException("Missing idempotency key"); } config.registerLambdaContext(context); return placeBooking(event); }, { successCode: HttpStatus.CREATED, } ); ``` ## Creating getBooking, acceptBooking lambda functions Both of the lambdas do not require any additional changes to make them idempotent, as it is guaranteed by the nature of the HTTP verbs used for their definitions (specifically, `GET` and `PUT`). The `acceptBooking` lambda function is particularly interesting because it has two responsibilities: * updating the item in the BookingRepository, * publishing a message to the SQS queue. Code sample below demonstrates how it's achieved. ```ts export const handler: APIGatewayProxyHandler = httpMiddleware(async (event) => { const id = getIdFromPathParams(event); // Checking if booking exists, if not findById will throw 404 error const { amount } = await bookingRepository.findById(id); try { await bookingRepository.updateStatus(id, "ACCEPTED"); return messageQueue.publishMessage({ id, amount }); } catch (err) { // Reverting to initial state in case of error await bookingRepository.updateStatus(id, "REQUESTED"); // Throwing error so can be processed by middleware throw new Error((err as Error).message); } }); ``` ## Creating paymentProcessor lambda function This lambda will be invoked when it pulls a message from the SQS queue, constituting the asynchronous part of the system. As the initial step, it must be defined in the YAML file so that the lambda can be created on the AWS side. Again to the lambda all necessary state store IAM permissions are assigned. What's unique is the events property where we connect created SQS queue with the way that it's invoked. ```yaml paymentProcessor: handler: src/functions/paymentProcessor/paymentProcessor.handler environment: BOOKING_TABLE: !Ref BookingTable IDEMPOTENCY_TABLE: !Ref IdempotencyTable events: - sqs: arn: !GetAtt PaymentProcessingQueue.Arn iamRoleStatements: - Effect: Allow Action: - dynamodb:GetItem - dynamodb:PutItem - dynamodb:UpdateItem - dynamodb:DeleteItem Resource: !GetAtt IdempotencyTable.Arn - Effect: Allow Action: - dynamodb:UpdateItem Resource: !GetAtt BookingTable.Arn ``` ### Idempotency handler The actual lambda implementation is nearly identical to the one presented {% link href="#creating-placebooking-lambda-function" ariaLabel="Link to the previously created lambda" %} earlier{% /link %}. The only difference is that, because each message body contains a unique ID, there's no need to specify a unique property; it will be automatically assumed that the passed argument has a unique parameter. Since this is just a tutorial, third-party integration is mocked. ```ts const config = new IdempotencyConfig({}); const bookingRepository: BookingRepository = new DynamoBookingRepository(); const processingFunction = async ({ id }: PaymentToBeProcessedMessage) => { // Here goes 3rd party payment API service call const paymentProcessedSuccessfully = true; // await paymentService.process(record) await bookingRepository.updateStatus( id, paymentProcessedSuccessfully ? "CONFIRMED" : "PAYMENT_DECLINED" ); return paymentProcessedSuccessfully; }; const handleBookingPayment = makeIdempotent(processingFunction, { persistenceStore, config, }); export const handler: SQSHandler = async (event, context) => { config.registerLambdaContext(context); for (const record of event.Records) { await handleBookingPayment(JSON.parse(record.body)); } }; ``` ## Deploying and testing Let's start deployment process to `test` environment by running `yarn deploy:test` command (feel free to use `npm` or `pnpm`).  Let's first try placing the booking request, notice how returned id doesn't change when `X-Idempotency-Key` is the same, it's a proof that lambda detects that the request is duplicated and shouldn't be processed second time but rather same result returned. {% video src="/assets/designing-idempotent-serverless-systems/place-booking.mov" /%} Let's now try to get placed booking to make sure that it was saved in the database.  Final test is to try to accept the booking and see whether it was asynchronously processed by `paymentProcessor` function. {% video src="/assets/designing-idempotent-serverless-systems/confirmation.mov" /%} ## Conclusion In conclusion, constructing idempotent handlers is essential for creating reliable systems that are resilient to unexpected errors and retries. However, it's important to note that this tutorial serves as a general orientation and does not delve into all the hidden complexities of ensuring idempotency (such as late arriving requests from multiple clients). Additionally, the system built here is not intended to be implemented "as is" in a production environment; rather, it serves as a simple showcase of the available tools that make handling this case easier. I hope that this tutorial was helpful! {% link href="https://github.com/piotrek12101999/designing-idempotent-serverless-systems" ariaLabel="Link to github to full code repository" %}Link to the full code repository is here{% /link %}. ]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek) https://www.piotrswiatek.dev/articles/building-serverless-rest-api-with-lambda building-serverless-rest-api-with-lambda Fri, 16 Feb 2024 12:55:00 GMT ## Prerequisites Before diving into the guide it's crucial to ensure that you have access to the {% link href="https://aws.amazon.com/free" ariaLabel="Link to aws account starting page" %}AWS account{% /link %}. This tutorial doesn't cover setting it up or configuring AWS credentials. Links to appropriate resources for these tasks are provided below for your convenience: - {% link href="https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html" ariaLabel="Read more about how to create AWS account" %} creating AWS account{% /link %}, - {% link href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html" ariaLabel="Read more about how to manage access keys for IAM users" %} managing access keys for IAM users{% /link %}, - {% link href="https://docs.aws.amazon.com/en_us/serverless-application-model/latest/developerguide/serverless-getting-started-set-up-credentials.html" ariaLabel="Read more about how to set up AWS credential" %} setting up AWS credentials{% /link %}. ## Application goals In this tutorial, we'll build an application enabling clients to create and read posts. Adding posts will be achieved by accessing the POST route, while accessing the GET route will allow clients to read existing posts. ## Application architecture Architecture leverages 3 services: - {% link href="https://aws.amazon.com/api-gateway/" ariaLabel="Link to Amazon API Gateway product page" %}Amazon API Gateway{% /link %}, - {% link href="https://aws.amazon.com/lambda/" ariaLabel="Link to AWS Lambda product page" %}AWS Lambda{% /link %}, - {% link href="https://aws.amazon.com/dynamodb" ariaLabel="Link to Amazon DynamoDB product page" %}Amazon DynamoDB{% /link %}. API Gateway acts as a gateway for incoming HTTP requests, providing a managed endpoint that can be used by clients of API. AWS Lambda functions are then triggered by these API requests. DynamoDB serves as a managed NoSQL database that is used for storing data. All of those services are managed by AWS, this means that AWS is responsible for provisioning, maintenance, and operational tasks. Diagram of the architecture is presented below.  ## Infrastructure as Code Infrastructure as Code (IaC) is the process of defining your infrastructure in code. This approach simplifies the process of provisioning and managing resources. This guide demonstrates usage of that process with the usage of the {% link href="https://www.serverless.com/" ariaLabel="Link to Serverless Framework index page" %}Serverless Framework{% /link %} as a CLI tool. It simplifies the development and deployment by defining resources in YAML files in a straightforward manner. It also provides simple deployments of infrastructure with a single command. ## Creating the Serverless Framework configuration In order to configure the Serverless Framework configuration file must be placed in the root directory of your project. File should be named `serverless.yml`. This file is a central component of a serverless application. It serves as a blueprint for defining various aspects of the app. Below is presented serverless configuration file used for in this application. ```yaml service: serverless-api plugins: - serverless-webpack - serverless-iam-roles-per-function package: individually: true custom: webpack: webpackConfig: "webpack.config.js" includeModules: false packager: "yarn" configFile: ${file(.env.${opt:stage, 'dev'}.yml):} resourceSlug: serverless-api-${self:custom.configFile.STAGE} provider: name: aws runtime: nodejs20.x architecture: arm64 stage: ${self:custom.configFile.STAGE} region: ${self:custom.configFile.REGION} environment: STAGE: ${self:custom.configFile.STAGE} functions: - ${file(src/functions/functions.yml)} resources: - ${file(src/database/database.yml)} ``` ### Plugins First section of config includes plugins. Plugins are extensions that provide additional functionality and features beyond the default functionalities provided by the Serverless Framework. In case of this application two plugins are used: * {% link href="https://www.serverless.com/plugins/serverless-webpack" ariaLabel="Read more about serverless webpack plugin" %}`serverless-webpack`{% /link %} - this plugin is used to support TypeScript in our codebase. From a technical standpoint, while the {% link href="https://www.serverless.com/plugins/serverless-plugin-typescript" ariaLabel="Read more about serverless typescript plugin" %}`serverless-plugin-typescript`{% /link %} is an option, utilizing the webpack plugin provides greater control over the appearance of the final output. Given the significance of cold starts in serverless applications, this level of control becomes particularly crucial. In order to make this plugin work `webpack.config.js` file must be place in a root directory of the application. {% link href="https://github.com/piotrek12101999/building-serverless-rest-api-with-lambda/blob/main/webpack.config.js" ariaLabel="Custom webpack plugin on github" %}Here we provide the webpack configuration{% /link %} used in this application, which emphasizes code minification and excludes packages already available in the Lambda environment. However, feel free to customize it according to your specific requirements. * {% link href="https://www.serverless.com/plugins/serverless-iam-roles-per-function" ariaLabel="Read more about serverless iam roles per function" %}`serverless-iam-roles-per-function`{% /link %} - this plugin is used to easily define IAM permissions for each lambda. Read more in dedicated {% link href="#aws-iam-execution-role" ariaLabel="Section in this article dedicated to AWS IAM execution roles" %}AWS IAM execution roles section{% /link %}. ### Custom The `custom` property in the config allows to create and define custom configuration and variables. In case of this application necessary information for `serverless-webpack` plugin are defined there. Additionally, we specify the path to the configuration file containing environment variables, along with the `resourceSlug property`, which ensures that our resources are prefixed with this slug upon creation, simplifying searching them in AWS console. Below `.env.dev.yml` file is presented containing environment variables. ```yaml REGION: eu-central-1 STAGE: dev ``` {% messageCard type="tip" title="Helpful Tip" %} When specifying the region for resource deployment, it's advisable to differentiate between development and production environments. This practice aids in isolating environments in the event of regional downtime or assists in the disaster recovery process. While the decision-making process for enterprise applications may be more complex (leading to having the same region for both development and production environments). For purposes of this application, we've opted for `eu-central-1` for the `dev` stage and `eu-west-1` for the `prod` stage. {% /messageCard %} ### Provider section The `provider` section specifies the settings and configurations specific to the chosen cloud provider. For this tutorial, AWS is selected as the provider, and the latest Node.js runtime (version 20) is utilized for Lambda functions. These functions run on ARM based AWS Graviton processors, which enhance performance and offer cost savings, {% link href="https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/" ariaLabel="AWS report on Graviton processors usage with Lambda" %} as per AWS reports{% /link %}. Furthermore, the deployment region and stage are also configured within this section (reusing the values from config file mentioned above). ### Referencing other files To enhance readability and adhere to the separation of concerns principle in the Serverless configuration file, the definitions of Lambda functions and DynamoDB resources are stored in separate files. This is accomplished using the `${file(path)}` function. ## Creating persistance layer To kick off, our initial task involves setting up a DynamoDB table using the Serverless Framework. Let's proceed by defining the DynamoDB table configuration that will be generated by the Serverless Framework (the actual service responsible for creating the resources is {% link href="https://aws.amazon.com/cloudformation/" ariaLabel="Link to AWS CloudFormation" %}CloudFormation{% /link %}. Serverless uses it under the hood by transforming serverless files into a CloudFormation template, which is a more intricate representation). ```yaml PostTable: Type: AWS::DynamoDB::Table Properties: TableName: ${self:custom.resourceSlug}-post-table BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: slug AttributeType: S KeySchema: - AttributeName: slug KeyType: HASH ``` {% messageCard title="Important Note" %} In the `TableName` property, we've utilized the variable `${self:custom.resourceSlug}`. The variable was initialized within the `custom` property located in the `serverless.yml file`. To learn more about it, {% link href="#custom" ariaLabel="Custom section in this article" %}refer back to the "Custom" section {% /link %}. {% /messageCard %} ### Repository pattern In this guide, we use the repository pattern, a design pattern that enables the separation of logic related to data retrieval from the actual retrieval process itself. I highly encourage to {% link href="https://learn.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/infrastructure-persistence-layer-design" ariaLabel="Repository pattern on microsoft documentation page" %}check out this Microsoft documentation page{% /link %} and read more about it. First we'll create `Post` interface that represents the data model. ```ts export interface Post { slug: string; title: string; content: string; createDate: string; } ``` We'll now craft the `PostRepository` interface, outlining operations specific to the persistence layer. ```ts export interface PostRepository { create(post: Post): Promise; findAll(): Promise; } ``` Now, we'll proceed to build the `DynamoPostRepository` class. This class implements the `PostRepository` interface, concentrating on executing methods defined by the repository. It conceals the intricacies of data retrieval and writing associated with DynamoDB as internal implementation details. For this tutorial, we utilize the latest AWS SDK version 3, which introduces modularized packages, enabling streamlined access to SDK functionalities. {% messageCard title="Important Note" %} In this tutorial, the `findAll()` method implementation utilizes the `scan` operation, which retrieves all items from the table by scanning the entire dataset. In production-ready applications, it's advisable to avoid using `scan` and instead opt for the `query` operation. By doing so, you can optimize performance and reduce costs, as query can help save additional reads. {% /messageCard %} {% br /%} {% messageCard type="warning" title="Warning" %} It's important to note that if the stored data exceeds 1MB, this read operation will not return all results. DynamoDB will instead provide a `LastEvaluatedKey` parameter. This parameter enables you to implement pagination, effectively resolving the issues at hand. {% /messageCard %} ```ts function checkForEnv(variable: string | undefined) { if (!variable) { throw new Error("Missing env variable"); } return variable; }; export class DynamoPostRepository implements PostRepository { private readonly client = new DynamoDBClient(); private readonly docClient = DynamoDBDocumentClient.from(this.client); private readonly tableName = checkForEnv(process.env.POST_TABLE); async findAll(): Promise { const command = new ScanCommand({ TableName: this.tableName, }); const { Items } = await this.docClient.send(command); return (Items || []) as Post[]; } async create(post: Post): Promise { const command = new PutCommand({ TableName: this.tableName, Item: post, ConditionExpression: "attribute_not_exists(slug)", }); await this.docClient.send(command); } } ``` ## Creating getPosts lambda function Once more, we initiate the creation process by crafting a `yaml` definition file. This file contains details regarding the function that will be provisioned by the Serverless Framework. ```yaml getPosts: handler: src/functions/getPosts/getPosts.handler events: - http: path: posts method: GET environment: POST_TABLE: !Ref PostTable iamRoleStatements: - Effect: Allow Action: - dynamodb:Scan Resource: !GetAtt PostTable.Arn ``` ### Amazon API Gateway integration Integration with API Gateway is established through the utilization of `event` properties. By specifying lambda invocation via HTTP events, API Gateway resources are provisioned accordingly. The `path` and `method` properties, as their names imply, define the URL structure of the API and the supported {% link href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods" ariaLabel="Read more about HTTP request methods" %}HTTP request methods{% /link %}, respectively. ### AWS IAM execution role Each AWS resource in order to perform actions on other resources and AWS services must be granted execution role. In the role we specify policies that lists exactly what kind of operations resource can execute on other resources / AWS services. In case of this tutorial serverless plugin `serverless-iam-roles-per-function` allow us to define in easy way list of permissions that will be assigned to execution role created for lambda. Permissions are defined in `iamRoleStatements` property. In case of `getPosts` lambda scan operation is allowed to be performed on `PostTable` resource. {% link href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html" ariaLabel="Lambda execution roles documentation page" %}Here is more information about execution roles{% /link %}. {% messageCard type="tip" title="Helpful Tip" %} You can checkout lambda execution role by inspecting lambda in AWS console and navigating to configuration -> permissions tab. {% /messageCard %}  It's worth noting that besides permission that is defined in our codebase, lambda by default is granted permission to create logs in {% link href="https://aws.amazon.com/cloudwatch/" ariaLabel="Link to Amazon CloudWatch product page" %}Amazon CloudWatch{% /link %} service. ### Intrinsic function reference Take a look how we reference earlier created `PostTable`. For this we use: * {% link href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html" ariaLabel="Link to !Ref function documentation page" %}!Ref{% /link %} allows referencing of other resources, * {% link href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html" ariaLabel="Link to !GetAtt function documentation page" %}!GetAtt{% /link %} allows getting attributes of other resource, all attributes are always specified in the resource documentation ({% link href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html#aws-resource-dynamodb-table-return-values" ariaLabel="Link to dynamodb return values" %}example{% /link %}). ### getPosts lambda handler `handler` function serves as an API Gateway proxy handler. The code itself is straightforward, primarily focusing on retrieving all posts from the posts repository using the findAll method. It then returns these posts in a stringified format with a success status code. This function relies on a single dependency, namely `postRepository`. Notice that this dependency is defined above the handler body, enabling AWS to cache it in between Lambda invocations. ```ts const postRepository: PostRepository = new DynamoPostRepository(); export const handler: APIGatewayProxyHandler = async () => { const posts = await postRepository.findAll(); return { statusCode: 200, body: JSON.stringify(posts), }; }; ``` ## Creating createPosts lambda function Process of creating function responsible for creation of posts is analogical to the one presented above. Only key differences will be pointed out. Below is presented code responsible for creating lambda resource in AWS. ```yaml createPost: handler: src/functions/createPost/createPost.handler events: - http: path: posts method: post environment: POST_TABLE: !Ref PostTable iamRoleStatements: - Effect: Allow Action: - dynamodb:PutItem Resource: !GetAtt PostTable.Arn ``` ### Creating createPosts lambda handler Handler itself is also really similar to the one presented above. Below is code of the handler. ```ts const postRepository: PostRepository = new DynamoPostRepository(); const getBadRequestResponse = (message: string) => { return { statusCode: 400, body: message, }; }; export const handler: APIGatewayProxyHandler = async ({ body }) => { if (!body) { return getBadRequestResponse("Missing body"); } const post: Post = JSON.parse(body); const { error } = schema.validate(post); if (error) { return getBadRequestResponse(error.message); } try { const result = await postRepository.create(post); return { statusCode: 200, body: JSON.stringify(result), }; } catch (err) { if ((err as Error).name === "ConditionalCheckFailedException") { return getBadRequestResponse("Article with this slug already exists"); } return { statusCode: 500, body: "Internal server error", }; } }; ``` I decided to use `joi` as the input validator, feel free to use a different validator if joi doesn't suit your needs. Below presented is the code representing `schema` object used in the handler. ```ts export const schema = joi .object({ slug: joi.string().required(), title: joi.string().required(), content: joi.string().required(), createDate: joi.string().isoDate().required(), }) .required(); ``` ## Deploying and testing With our code and configurations prepared, we're ready to deploy the application and conduct testing. Deployment is initiated by executing the sls deploy command. To simplify the deployment process, several scripts have been created within the package.json file. ```json "scripts": { "deploy:test": "sls deploy --stage=dev", "deploy:prod": "sls deploy --stage=prod" } ``` Let's start deployment process to `test` environment by running `yarn deploy:test` command (feel free to use `npm` or `pnpm`).  We can test deployed api by using {% link href="https://www.postman.com/" ariaLabel="Link to Postman app page" %}Postman{% /link %}. Let's start by trying to create post.  As observed, the post has been successfully created and should now be stored in DynamoDB. Let's proceed to fetch all posts.  Success 🚀🎉! Application works. It correctly stores data in database and then read them and expose to client by the RESTful API. ## Conclusion In conclusion, building REST API with use of serverless offers a powerful and scalable solution for modern app development. Throughout this article, we've explored the basics of the Serverless Framework. Learned how to set up Lambda functions and TypeScript. We stored our data in DynamoDB and exposed functionalities by creating REST API with usage of Amazon API Gateway. With that being said there are still many areas that were not covered for sake of simplicity of this tutorial. Some main areas of improvements are: * pagination of results, * tests of the code (unit, integration, E2E), * logging inside of Lambda functions, * more scalable and optimized DynamoDB queries, * authentication of API, * leveraging built in Amazon API Gateway data validators (I'd still suggest to keep `joi` or other validator for more advanced validation), * and many more. {% link href="https://github.com/piotrek12101999/building-serverless-rest-api-with-lambda" ariaLabel="Link to github to full code repository" %}Link to the full code repository is here{% /link %}. ]]> piotr.swiatek.1999@gmail.com (Piotr Swiatek)