From Pandas to ClickHouse: The Evolution of Our Data Analytics Journey - Incentius Blog Home - Incentius - Enable Digital Transformation [![Logo](/img_astro/incentius-logo.svg)](/) Solutions About [Case Studies](/portfolio/) [Blogs](/blog-posts/) [Career](http://aurochs.freshteam.com/jobs) Open main menu ![Menu icon](/img_astro/menu1.png) [Get In Touch](/contact/) Solutions [![](/img_astro/generative-black.svg) ![](/img_astro/brain-green.svg) Voice AI](/voice-ai/) [![](/img_astro/product-development-black.svg) ![](/img_astro/product-development.svg) Product Development](/product-engineering/) [![](/img_astro/generative-black.svg) ![](/img_astro/brain-green.svg) Generative AI](/generative-ai/) [![](/img_astro/data-black.svg) ![](/img_astro/database-green.svg) Data Engineering](/data-engineering/) [![](/img_astro/mvp-black.svg) ![](/img_astro/rocket-green.svg) Data & Analytics](/data-analytics/) [![](/img_astro/cloud-black.svg) ![](/img_astro/computing-green.svg) DevOps & CloudOps](/devops-and-cloudops/) About Us [![](/img_astro/product-black.svg) ![](/img_astro/product-green.svg) About Us](/about-us/) [![](/img_astro/team_black.svg) ![](/img_astro/team_green.svg) Team](/team/) Solutions ![](/img_astro/dropdown-arr.svg) [ ![](/img_astro/generative-black.svg) Voice AI ](/voice-ai/)[ ![](/img_astro/product-development-black.svg) Product Development ](/product-engineering/)[ ![](/img_astro/generative-black.svg) Generative AI ](/generative-ai/)[ ![](/img_astro/data-black.svg) Data Engineering ](/data-engineering/)[ ![](/img_astro/mvp-black.svg) Data & Analytics ](/data-analytics/)[ ![](/img_astro/cloud-black.svg) DevOps & CloudOps ](/devops-and-cloudops/) [Blogs](/blog-posts/) About ![](/img_astro/dropdown-arr.svg) [ ![](/img_astro/product-black.svg) About Us ](/about-us/)[ ![](/img_astro/team_black.svg) Team ](/team/) [Case Studies](/portfolio/) [Career](http://aurochs.freshteam.com/jobs) [Get In Touch](/contact/) From Pandas to ClickHouse: The Evolution of Our Data Analytics Journey Chetan Patel 1. Oct 28, 2024 2. 4 min read ![](/_astro/image_6.width-1000_2leMCm.webp) At Incentius, data has always been at the heart of what we do. We’ve built our business around providing insightful, data-driven solutions to our clients. Over the years, as we scaled our operations, our reliance on tools like Pandas helped us manage and analyze data effectively—until it didn’t. The turning point came when our data grew faster than our infrastructure could handle. What was once a seamless process started showing cracks. It became clear that the tool we had relied on so heavily for data manipulation—Pandas—was struggling to keep pace. And that’s when the idea of shifting to ClickHouse began to take root. But this wasn’t just about switching from one tool to another; it was the story of a fundamental transformation in how we approached data analytics at scale. ### The Early Days: Pandas, Our First Love When we first adopted Pandas, it felt like we had unlocked the perfect solution. The flexibility, the powerful data frames, and the ease with which we could manipulate small to mid-sized datasets—it was a game-changer. Our team of data engineers and analysts loved it for its simplicity. And, for a long time, it served us well. But then something happened. Our datasets began to grow, and so did the complexity of our queries. We went from handling thousands of rows to millions, and then, in what seemed like no time at all, billions. The once seamless operations we had with Pandas turned into long waits for processes to complete, or worse, system crashes. We found ourselves asking: **How can we keep scaling without compromising performance?** ### The Bottleneck: Pandas in a Big Data World At first, we tried to optimize Pandas in every way possible. We ran computations on smaller chunks of data, tried parallel processing techniques, and even moved to bigger and more expensive machines to support the growing memory requirements. But these were short-term fixes for a long-term challenge. Pandas was designed to load data into memory, which, for our growing datasets stored on S3, was becoming a major bottleneck. We realized that as the data we were handling continued to scale, our tools needed to scale with it. Pandas, for all its strengths, wasn’t designed for this new world of distributed, high-performance data analytics. That’s when we started exploring alternatives—and found ClickHouse. ### Enter ClickHouse: A New Frontier in Data Analytics We didn’t immediately jump into using ClickHouse. Like any good story, there was a journey of discovery, a few moments of doubt, and ultimately, a realization that this was the solution we needed. ClickHouse came onto our radar because of its reputation for handling real-time, high-performance analytics. It was built to thrive in environments like ours—where datasets are huge, queries are complex, and the need for speed is paramount. We started small, running a few test queries on ClickHouse to see how it would perform against Pandas. The results were staggering. Where Pandas took minutes, sometimes hours, to process data, ClickHouse completed the same tasks in seconds. The first time we ran a complex aggregation on ClickHouse and saw the results in the blink of an eye, we knew we were onto something. ### The Turning Point: Scaling Without Limits Transitioning from Pandas to ClickHouse wasn’t just about better performance; it was about rethinking how we managed our entire data pipeline. Here’s what changed: 1. **Handling Larger Datasets with Ease**: ClickHouse’s columnar storage model meant we could now work with datasets that would’ve been impossible to manage in Pandas. Instead of loading everything into memory, ClickHouse processed data directly from S3, allowing us to scale infinitely without worrying about memory limits. 2. **Real-Time Insights, Faster Than Ever**: One of the biggest challenges we faced with Pandas was the time it took to generate real-time reports. With ClickHouse, real-time analytics became just that—**real-time**. We could now offer clients up-to-the-minute insights on their data, something that would’ve taken hours in our previous setup. 3. **Distributed Processing, Maximum Efficiency**: ClickHouse’s ability to distribute queries across multiple nodes unlocked a new level of efficiency. We were no longer constrained by the limitations of a single machine. We could now process billions of rows of data across multiple servers, achieving performance that was unimaginable with Pandas. 4. **Seamless S3 Integration**: One of the most powerful features of ClickHouse is its seamless integration with S3. This eliminated the need for us to move data between different storage systems or perform complex ETL processes just to analyze it. We could query data directly where it was stored, saving time, money, and resources. ### From Challenge to Opportunity: What This Means for Our Future Looking back, the decision to transition from Pandas to ClickHouse was more than just a technical upgrade—it was a turning point in how we think about data. The challenges we faced with Pandas forced us to push the boundaries and explore new technologies. ClickHouse wasn’t just a replacement; it became the foundation for a more scalable, robust, and future-proof data infrastructure. Now, instead of being bogged down by the limitations of in-memory processing, we’re able to take on projects that involve massive datasets with confidence. Our clients benefit from faster insights, more reliable data processing, and a system that’s built to grow with them. ### Conclusion: The Evolution Continues The move to ClickHouse wasn’t the end of our story; it was just the next chapter. As we continue to evolve and scale, we’re constantly looking for ways to push the envelope, to find new tools and technologies that allow us to deliver even greater value to our clients. The lesson we learned from this transition is simple: **As the world of data evolves, so must we.** Our journey from Pandas to ClickHouse is a testament to that philosophy—an evolution driven by necessity, but one that has opened the door to endless possibilities. And with ClickHouse powering our analytics, the possibilities are truly endless. About Author ![](/_astro/chetan_piucv.width-300_ZK3Vor.webp) Chetan Patel ![](/img_astro/thumb.png) ### See What Our Clients Say ![](/img_astro/left-chevron_1.svg) ![](/img_astro/left-chevron_2.svg) ![](/img_astro/mind-gap-crop.svg) Mindgap Incentius has been a fantastic partner for us. Their strong expertise in technology helped deliver some complex solutions for our customers within challenging timelines. Specific call out to Sujeet and his team who developed custom sales analytics dashboards in SFDC for a SoCal based healthcare diagnostics client of ours. Their professionalism, expertise, and flexibility to adjust to client needs were greatly appreciated. MindGap is excited to continue to work with Incentius and add value to our customers. Samik Banerjee Founder & CEO ![](/img_astro/trusted-by-businesses/worldatwork_coloured.webp) World at Work Having worked so closely for half a year on our website project, I wanted to thank Incentius for all your fantastic work and efforts that helped us deliver a truly valuable experience to our WorldatWork members. I am in awe of the skills, passion, patience, and above all, the ownership that you brought to this project every day! I do not say this lightly, but we would not have been able to deliver a flawless product, but for you. I am sure you'll help many organizations and projects as your skills and professionalism are truly amazing. Shantanu Bayaskar Senior Project Manager ![](/img_astro/trusted-by-businesses/gogla_coloured.webp) Gogla It was a pleasure working with Incentius to build a data collection platform for the off-grid solar sector in India. It is rare to find a team with a combination of good understanding of business as well as great technological know-how. Incentius team has this perfect combination, especially their technical expertise is much appreciated. We had a fantastic time working with their expert team, especially with Amit. Viraj gada Gogla ![](/img_astro/trusted-by-businesses/humblex_coloured.webp) Humblx Choosing Incentius to work with is one of the decisions we are extremely happy with. It's been a pleasure working with their team. They have been tremendously helpful and efficient through the intense development cycle that we went through recently. The team at Incentius is truly agile and open to a discussion in regards to making tweaks and adding features that may add value to the overall solution. We found them willing to go the extra mile for us and it felt like working with someone who rooted for us to win. Samir Dayal Singh CEO Humblx ![](/img_astro/mind-gap-crop.svg) Transportation & Logistics Consulting Organization Incentius is very flexible and accommodating to our specific needs as an organization. In a world where approaches and strategies are constantly changing, it is invaluable to have an outsourcer who is able to adjust quickly to shifts in the business environment. Transportation & Logistics Consulting Organization Consultant ![]() Mudraksh & McShaw Incentius was instrumental in bringing the visualization aspect into our investment and trading business. They helped us organize our trading algorithms processing framework, review our backtests and analyze results in an efficient, visual manner. Priyank Dutt Dwivedi Mudraksh & McShaw Advisory ![](/img_astro/mind-gap-crop.svg) Leading Healthcare Consulting Organization The Incentius resource was highly motivated and developed a complex forecasting model with minimal supervision. He was thorough with quality checks and kept on top of multiple changes. Leading Healthcare Consulting Organization Sr. Principal ![](/img_astro/mind-gap-crop.svg) US Fortune 100 Telecommunications Company The Incentius resource was highly motivated and developed a complex forecasting model with minimal supervision. He was thorough with quality checks and kept on top of multiple changes. Incentive Compensation Sr. Director Most Read [ ![](/_astro/ChatGPT_Image_Apr_2_2025_12_06_35_PM.width-1000_ZUChn3.webp) Optimizing Large-Scale Data Exports: FastAPI, GraphQL, and AWS S3 Struggling with slow, inefficient file downloads in your application? Generating and handling large file downloads can be a nightmare—UI freezing, server overload, and endless waiting. ![](/_astro/Vinay_Chaudhari.width-300_1QPJnB.webp) Vinay Chaudhari 1. Apr 08, 2025 2. 4 min read ](/blog-posts/optimizing-large-scale-data-exports-fastapi-graphql-and-aws-s3/)[ ![](/_astro/Screenshot_2025-04-02_at_12.09.33_PM.width-1000_ZyU6y1.webp) How to Build a Web Search Agent with Phidata: A Step-by-Step Guide Phidata is an **open-source framework** for building AI agents with capabilities like memory, tool use, and reasoning​ It’s model-agnostic and supports various large language models (LLMs), allowing developers to turn any LLM (like OpenAI GPT or local models) into a functional **search agent**. ![](/_astro/Kartik_Puri.width-300_1YNCyp.webp) Kartik Puri 1. Apr 02, 2025 2. 4 min read ](/blog-posts/how-to-build-a-web-search-agent-with-phidata-a-step-by-step-guide/)[ ![](/_astro/e20f5335-cbf6-414c-a096-7d2849109cff.width-1000_15DfBS.webp) Migrating to Soketi: Improving Our Real- Time Communication Architecture Our journey with real-time data communication has evolved significantly. Initially using Flask-SocketIO with Vue, we faced several challenges that led us to seek a more robust solution. ![](/_astro/pratik.width-300_Z2vB9eu.webp) Pratik Patel 1. Mar 17, 2025 2. 4 min read ](/blog-posts/migrating-to-soketi-improving-our-real-time-communication-architecture/)[ ![](/_astro/startup-employee-looking-business-charts-usin.width-1000_2knrAp.webp) Understanding AI Agents: A Comprehensive Guide to AI Agentic Workflows In the rapidly evolving landscape of artificial intelligence, AI Agents have emerged as a transformative technology that represents far more than just industry jargon. ![](/_astro/chetan_piucv.width-300_ZK3Vor.webp) Chetan Patel 1. Feb 28, 2025 2. 4 min read ](/blog-posts/understanding-ai-agents-a-comprehensive-guide-to-ai-agentic-workflows/) ## We use cookies We use cookies to improve site. Some cookies are necessary for our website and services to function properly. Other cookies are optional and help personalize your experience, including advertising and analytics.You can consent to all cookies or decline all optional cookies. Without a selection, our default cookie settings will apply. I agree I decline [![logo](/img_astro/logo.svg)](/) Incentius is a technology and professional services company providing analytics, operations and product engineering services. [![facebook](/img_astro/facebook.svg)](https://www.facebook.com/Incentius/)[![twitter](/img_astro/twitter.svg) ](https://x.com/incentius/)[![linkedin](/img_astro/linkedin.svg)](https://www.linkedin.com/company/incentius/posts/?feedView=all) ![location](/img_astro/pin.svg) Office No.201B, 2nd Floor, Beta-1, Giga Space IT Park, Viman Nagar, Pune - 411014, Maharashtra, India ![email](/img_astro/message-icon.svg) info@incentius.com ![phone](/img_astro/telephone.svg) +1 (302) 415 3208 ![](/img_astro/secure-footer.svg) ![](/img_astro/secure-footer-mobile.svg) * Solutions * [Voice AI](/voice-ai/) * [Generative AI](/generative-ai/) * [Data Engineering](/data-engineering/) * [Product Development](/product-engineering/) * [Data & Analytics](/data-analytics/) * [DevOps & CloudOps](/devops-and-cloudops/) * [UI/UX Design](/ui-ux-design/) * Recent Blog * [Medallion Architecture in Practice: Six Layers, One Source of Truth](/blog-posts/medallion-architecture-in-practice-six-layers-one-source-of-truth/) * ["Document later" is how teams say "document never"](/blog-posts/document-later-is-how-teams-say-document-never/) * [Your Business & AI - What It Actually Means for You](/blog-posts/your-business-ai-what-it-actually-means-for-you/) * [How Internal Agent-Built Software Reduced Our SaaS Dependency](/blog-posts/how-internal-agent-built-software-reduced-our-saas-dependency/) * [MarkItDown: A Powerful Utility for File Conversion to Markdown](/blog-posts/markitdown-a-powerful-utility-for-file-conversion-to-markdown/) * [Python 3.14: A Game-Changing Release for Modern Development](/blog-posts/python-314-a-game-changing-release-for-modern-development/) * [Building a RESTful Blog API with Flask and Swagger: A Complete Guide to Streamlined API Development](/blog-posts/building-a-restful-blog-api-with-flask-and-swagger-a-complete-guide-to-streamlined-api-development/) * [Tailwind CSS 4.0 vs 3.x: What Changed and How We Implemented It](/blog-posts/tailwind-css-40-vs-3x-what-changed-and-how-we-implemented-it/) * [The Future of AI: What to Expect in the Next 5 Years](/blog-posts/the-future-of-ai-what-to-expect-in-the-next-5-years/) * Menu * [About](/about-us/) * [Blog](/blog-posts/) * [Career](http://aurochs.freshteam.com/jobs) * [Case Studies](/portfolio/) * [For Startups](/startups/) * [Let's Talk](/contact/) © Copyright 2025 Incentius [Request Site](/) [Terms and Conditions](/) [Privacy Policy](/privacy-policy/)