Optimizing Large-Scale Data Exports: FastAPI, GraphQL, and AWS S3

Introduction

Struggling with slow, inefficient file downloads in your application? Generating and handling large file downloads can be a nightmare—UI freezing, server overload, and endless waiting. But what if you could make it blazing fast, scalable, and real-time? In this blog, we’ll walk you through an event-driven, modern approach using FastAPI, GraphQL, AWS SQS, and GraphQL Subscriptions to handle massive datasets (5M+ rows) effortlessly!

Challenges with Large File Downloads

1. UI Freezing Due to Large File Generation

When a file is generated synchronously on the backend, the request remains open, blocking other operations.

2. Server Overload & Scalability Issues

Large files stored on the backend can consume significant resources.
Traditional polling methods increase unnecessary API calls and degrade performance.

3. Inefficient Polling for File Status

Repeated polling increases server load and slows down response times.
No real-time updates for file readiness.

4. Scaling Background Processing Efficiently

Using traditional Redis Queue (RQ Worker) or Celery may require manual infrastructure scaling.
A serverless, event-driven approach provides better scalability and cost efficiency.

5. Efficiently Handling Large Datasets (5+ Million Rows)

Querying a large dataset at once can overwhelm memory and slow down performance.
Efficient batch processing and streaming are required to handle such datasets.

Industry-Standard Optimized Approach

We solve these challenges using:

FastAPI for high-performance API processing.
AWS SQS and Lambda for serverless background task processing.
AWS S3 Signed URLs for scalable file storage and secure downloads.
GraphQL Subscriptions (Hasura/Apollo) for real-time status updates.
Chunked File Transfers for optimized large file downloads.
GraphQL for efficient data fetching and batch processing.

Solution Overview

Tech Stack Used:

Frontend: Vue.js (with Quasar Framework) + Apollo GraphQL Client
Backend: FastAPI (Python) + GraphQL + GraphQL Subscriptions (Hasura/Apollo)
Task Queue: AWS SQS + AWS Lambda
Storage: AWS S3 with Signed URLs
Event-Driven Processing: AWS Lambda + EventBridge

Workflow Breakdown:

User requests file generation → Vue.js sends a request to the backend.
Backend fetches data from GraphQL in batches and publishes tasks to AWS SQS → Tasks get queued in SQS.
AWS Lambda listens for SQS messages → Processes batched database results, generates the file incrementally, and uploads it to AWS S3.
File status updates are sent via GraphQL Subscriptions → Real-time updates without polling.
Vue.js listens to GraphQL Subscription → UI updates instantly when the file is ready.
Download via AWS S3 Signed URL → User downloads the file securely from AWS.

Backend Implementation (FastAPI + GraphQL + AWS SQS + Lambda + GraphQL)

1. Install Dependencies


pip install fastapi boto3 requests gql pandas xlsxwriter

2. FastAPI Backend (app.py)


from fastapi import FastAPI
import boto3
import json
import time
import requests

app = FastAPI()

# AWS Configuration
SQS_QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/your-account-id/your-queue"
S3_BUCKET_NAME = "your-bucket-name"
AWS_REGION = "us-east-1"

sqs = boto3.client("sqs", region_name=AWS_REGION)
s3_client = boto3.client("s3", region_name=AWS_REGION)

GRAPHQL_ENDPOINT = "https://your-graphql-api.com/graphql"

@app.post("/generate-file")
def generate_file():
    """Handles file generation by querying GraphQL in batches and sending tasks to AWS SQS."""
    task_id = str(int(time.time()))
    query = """
    query GetLargeTableData($offset: Int!, $limit: Int!) {
      largeTable(offset: $offset, limit: $limit) {
        id
        column1
        column2
      }
    }
    """
    batch_size = 100000
    offset = 0

    while True:
        response = requests.post(GRAPHQL_ENDPOINT, json={"query": query, "variables": {"offset": offset, "limit": batch_size}})
        data = response.json().get("data", {}).get("largeTable", [])
        
        if not data:
            break
        
        message = json.dumps({"task_id": task_id, "data": data})
        sqs.send_message(QueueUrl=SQS_QUEUE_URL, MessageBody=message)
        offset += batch_size

    print(f"Task {task_id} successfully queued!")
    return {"task_id": task_id}

Frontend Implementation (Vue.js + Apollo GraphQL Client)

1. Install Dependencies


npm install @vue/apollo-composable graphql apollo-client

2. Vue Component for File Generation & Download


<template>
  <div>
    <q-btn label="Generate File" @click="generateFile" />
    <q-spinner v-if="loading" color="primary" />
    <a v-if="downloadUrl" :href="downloadUrl" download>
      <q-btn label="Download File" color="green" />
    </a>
  </div>
</template>

<script>
import { useSubscription } from '@vue/apollo-composable';
import gql from 'graphql-tag';

export default {
  data() {
    return {
      taskId: null,
      loading: false,
      downloadUrl: null,
    };
  },
  methods: {
    async generateFile() {
      this.loading = true;
      const response = await fetch("/generate-file", { method: "POST" });
      const data = await response.json();
      this.taskId = data.task_id;
      this.listenForUpdates();
    },
    listenForUpdates() {
      const { result, loading, error } = useSubscription(gql`
        subscription OnFileGenerated($taskId: String!) {
          fileStatusUpdated(task_id: $taskId) {
            download_url
          }
        }
      `, { taskId: this.taskId });
      
      watchEffect(() => {
        if (error.value) {
          console.error("GraphQL Subscription Error:", error.value);
          this.loading = false;
        }
        if (!loading.value && result.value) {
          this.downloadUrl = result.value.fileStatusUpdated.download_url;
          this.loading = false;
        }
      });
    }
  }
};
</script>

Final Thoughts

This FastAPI + GraphQL + AWS SQS + Lambda + GraphQL Subscriptions + S3 Signed URLs approach ensures a serverless, real-time, scalable solution for large file downloads. By leveraging efficient batch processing for querying 5M+ rows, event-driven processing, and secure cloud storage, we eliminate performance bottlenecks and improve the user experience.

Ready to transform your file processing experience? If you're working with millions of rows and need a fast, scalable, and serverless solution, this is your answer! Try it out and let us know how it works for you!

About Author

Vinay Chaudhari

Introduction

Challenges with Large File Downloads

1. UI Freezing Due to Large File Generation

2. Server Overload & Scalability Issues

3. Inefficient Polling for File Status

4. Scaling Background Processing Efficiently

5. Efficiently Handling Large Datasets (5+ Million Rows)

Industry-Standard Optimized Approach

Solution Overview

Tech Stack Used:

Workflow Breakdown:

Backend Implementation (FastAPI + GraphQL + AWS SQS + Lambda + GraphQL)

1. Install Dependencies

Frontend Implementation (Vue.js + Apollo GraphQL Client)

1. Install Dependencies

2. Vue Component for File Generation & Download

Final Thoughts

See What Our Clients Say