<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Crafting Software Diary]]></title><description><![CDATA[Crafting Software Diary]]></description><link>https://blogs.lampham.dev</link><generator>RSS for Node</generator><lastBuildDate>Fri, 17 Apr 2026 13:51:37 GMT</lastBuildDate><atom:link href="https://blogs.lampham.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Beyond Big-O: How Hardware Shapes Code Performance.]]></title><description><![CDATA[In programming, code performance tends to be tighten to algorithm. We often think of code complexity as the primary factor for code speed. The rule of thumb is:

The fewer operations a program performs, the faster it runs.

While this principle is va...]]></description><link>https://blogs.lampham.dev/beyond-big-o-how-hardware-shapes-code-performance</link><guid isPermaLink="true">https://blogs.lampham.dev/beyond-big-o-how-hardware-shapes-code-performance</guid><category><![CDATA[memory-management]]></category><category><![CDATA[performance]]></category><category><![CDATA[algorithms]]></category><category><![CDATA[Computer Science]]></category><category><![CDATA[Programming Blogs]]></category><dc:creator><![CDATA[Lam PHAM]]></dc:creator><pubDate>Wed, 02 Jul 2025 13:18:43 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1751467981324/364ba41f-b81c-4838-b6b6-b50d2807af00.jpeg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In programming, code performance tends to be tighten to algorithm. We often think of code complexity as the primary factor for code speed. The rule of thumb is:</p>
<blockquote>
<p>The fewer operations a program performs, the faster it runs.</p>
</blockquote>
<p>While this principle is valid, it overlooks a critical aspect: performance comparisons only make sense when the environment remains the same. And by <em>environment</em>, I mean the hardware: CPU, GPU, RAM etc— because ultimately, code is just instructions, what executes tasks is the hardware. Code doesn't just define <em>what</em> tasks to perform, but can also tell the hardware <em>how</em> to do it.</p>
<p>Yet, most developers focus heavily on minimizing the <em>what</em>—reducing the number of operations—while completely ignoring the <em>how</em>. The reason is understandable: optimizing the how requires a solid understanding of hardware-level behavior, and as <em>software</em> engineers, we often:</p>
<ul>
<li><p>Tend to prioritize high-level features and functionality.</p>
</li>
<li><p>Lack familiarity with low-level hardware operations.</p>
</li>
<li><p>May not even realize how much hardware-aware coding can boost performance.</p>
</li>
</ul>
<p>In this article, we'll see how a deeper understanding of hardware can turn even small code changes into significant performance gains.</p>
<h1 id="heading-memory-reading">Memory reading</h1>
<p>Let’s look at a simple example.</p>
<p>Given this array of length <strong>2 073 600</strong> (<strong>1920×1080)</strong>:</p>
<blockquote>
<p>Note: this example only works with arrays, not linked lists—  we’ll explore why later in the section.</p>
</blockquote>
<pre><code class="lang-kotlin"><span class="hljs-keyword">private</span> <span class="hljs-keyword">const</span> <span class="hljs-keyword">val</span> COLUMN = <span class="hljs-number">1920</span>
<span class="hljs-keyword">private</span> <span class="hljs-keyword">const</span> <span class="hljs-keyword">val</span> ROW = <span class="hljs-number">1080</span>
<span class="hljs-keyword">private</span> <span class="hljs-keyword">val</span> length = COLUMN * ROW
<span class="hljs-keyword">private</span> <span class="hljs-keyword">val</span> list = ArrayList&lt;<span class="hljs-built_in">Int</span>&gt;()
<span class="hljs-keyword">for</span> (i <span class="hljs-keyword">in</span> <span class="hljs-number">0</span> until length) {
    list.add(i)
}
</code></pre>
<p>Now, let’s write a program that iterates through the entire array and processes each element. For this, we are going to study 2 approaches:</p>
<ul>
<li><p>The first method iterates through items sequentially: <em>0, 1, 2, 3, 4 …</em></p>
</li>
<li><p>The second method iterates in a leapfrogging pattern: <em>0, 1920, 3840, 7680, …, 1, 1921, 3841, ...</em></p>
</li>
</ul>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Sequential</td><td>Leapfrogging</td></tr>
</thead>
<tbody>
<tr>
<td>0, 1, 2, 3, 4 …</td><td>0, 1920, 3840, 7680, …, 1, 1921, 3841, ...</td></tr>
<tr>
<td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751395194092/d3fd9cd5-c415-4537-a837-b777fbd6e2ce.png" /></td><td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751395183902/762de7a5-5f48-4b56-939c-5767650f6c5d.png" /></td></tr>
</tbody>
</table>
</div><p>These 2 approaches have the same complexity, perform exactly the same number of operation, the only difference is the order in which the items are accessed.</p>
<p>Before reading further, take a moment to think: what could possibly make one method faster than the other?</p>
<p>...</p>
<p>Alright, let’s reveal the answer. Here’s the benchmark comparison of the two methods:</p>
<blockquote>
<p><strong>Note:</strong> The benchmark is done with kotlin-benchmark library, in the Mode.AverageTime</p>
</blockquote>
<pre><code class="lang-kotlin">Benchmark                        Mode  Cnt  Score   Error  Units
Benchmark.sequentialIteration    avgt    <span class="hljs-number">5</span>  <span class="hljs-number">0.952</span> ± <span class="hljs-number">0.003</span>  ms/op
Benchmark.leapfroggingIteration  avgt    <span class="hljs-number">5</span>  <span class="hljs-number">3.517</span> ± <span class="hljs-number">0.015</span>  ms/op
</code></pre>
<p>The benchmark shows that the sequential approach completes an operation in approximately <strong>0.952 millisecond</strong> , while the leapfrogging approach takes about <strong>3.517 milliseconds</strong>. In other words, sequential reading is roughly <strong>3.7 times faster</strong> (this number is not consistent as it differs depending on the hardware and the data structure used) than leapfrogging reading 🔥🔥.</p>
<p>Wondering why?</p>
<p>To understand the reason, we’ll examine the data reading function—since the only difference between the two methods lies in the <strong>order</strong> in which data is read.</p>
<p>Although memory architecture can vary across CPU designs, all systems generally include RAM and multiple layers of cache. When a program requests data, the CPU first checks the cache, if the data isn't there, cache fetches it from RAM, which is significantly slower. However, if future data requests hit previously fetched blocks, those slow RAM accesses can be avoided.</p>
<p>In short, the fewer memory fetches from RAM, the faster the program runs.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751073021900/0a6583b1-8de1-410a-884c-adc1837ccee4.png" alt class="image--center mx-auto" /></p>
<p>Below is what happens in the two above programs:</p>
<div class="hn-table">
<table>
<thead>
<tr>
<td>Sequential</td><td>Leapfrogging</td></tr>
</thead>
<tbody>
<tr>
<td>0, 1, 2, 3, 4 …</td><td>0, 1920, 3840, 7680, …, 1, 1921, 3841, ...</td></tr>
<tr>
<td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751395194092/d3fd9cd5-c415-4537-a837-b777fbd6e2ce.png" /></td><td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751395183902/762de7a5-5f48-4b56-939c-5767650f6c5d.png" /></td></tr>
<tr>
<td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751382894720/c30ed581-2bd3-441f-8cdc-ddebc7971dad.gif" /></td><td><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1751395148903/1a91d715-05b2-42b0-8f5e-052f87e1b86f.gif" /></td></tr>
</tbody>
</table>
</div><p>With the same amount of data requested, sequential reading typically requires far fewer RAM fetches than leapfrogging, making it significantly more efficient.</p>
<blockquote>
<p><strong>Important note:</strong> As mentioned earlier, arrays are the best fit for sequential reading because they store data in <strong>contiguous</strong> memory blocks. <strong>Non-contiguous</strong> structures (like LinkedLists) don’t offer the same performance benefits.</p>
</blockquote>
<p>Best-practice takeaway: Always aim to reduce the number of RAM fetches. In case of contiguous storing, sequential reading is the most efficient approach. This could also be optimized further by knowing the capacity of cache— newly fetched data is loaded into the cache, while older data is evicted to make room—and tailoring the data reading pattern accordingly.</p>
<p>By understanding how cache fetching works, you can significantly improve your program’s performance with just a tiny code change.</p>
<h1 id="heading-numpy-vectorization-operations">Numpy vectorization operations</h1>
<p>I have recently started exploring some Machine Learning basics. While studying the Linear Regression model (with multiple features), I came across a problem that involves multiplying two arrays element-wise:</p>
<p>$$
\mathbf{w} = [w_1, w_2, \ldots, w_n],\quad \mathbf{x} = [x_1, x_2, \ldots, x_n]
$$</p><p>$$
\mathbf{w} \cdot \mathbf{x} = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n
$$</p><p>From a programming perspective, this is a straightforward problem. The easiest and most intuitive approach is to use a <em>for loop</em> to multiply each pair of elements and then sum all the products.</p>
<pre><code class="lang-python">f = <span class="hljs-number">0</span>
<span class="hljs-keyword">for</span> i <span class="hljs-keyword">in</span> range(<span class="hljs-number">0</span>, n):
    f = f + w[i] * x[i]
</code></pre>
<p>To me, iterating over every element and doing the math seemed inevitable. Therefore, I believed the <em>for loop</em> was the most efficient solution and that no better alternative existed. And then, I got introduced to the <em>Numpy</em> library with its built-in <code>dot()</code> operation:</p>
<pre><code class="lang-python">f = np.dot(w,x)
</code></pre>
<p>At first, I assumed the <code>dot()</code> function was just a fancy wrapper around a <em>for loop</em>—until the instructor made this bold claim:</p>
<blockquote>
<p>Numpy's dot function can be hundreds or even thousands of times faster than a for-loop.</p>
</blockquote>
<p>Wait what? 😮😮 Like how?</p>
<p>Out of curiosity, I looked it up and discovered how it was made possible. It turns out that the <em>Numpy</em> library uses a variety of sophisticated techniques that take advantage of modern hardware capabilities to maximize performance when working with massive numerical data.</p>
<h2 id="heading-pre-compiled-code">Pre-compiled code</h2>
<p>Even though <em>Numpy</em> is a python library, the vectorization methods are pre-compiled C (or sometimes Fortran) code. This compiled code runs much faster than a Python <em>for-loop</em>— where each iteration is interpreted at runtime.</p>
<h2 id="heading-vectorization-simd">Vectorization (SIMD)</h2>
<p>In ML context, data is usually vectorized, meaning it is transformed into numerical vectors that models can process efficiently. Vectorization allows <strong>computations on entire vectors at once</strong>, leveraging parallel processing capabilities of modern CPUs and GPUs. This is enabled thanks to <strong>Single Instruction, Multiple Data </strong> (SIMD), a technique where one instruction performs operations on multiple data points <strong>simultaneously</strong> within a single CPU cycle. For example, in a dot product operation, all multiplications \( w_i x_i \) are performed in parallel, and their results are summed. This results in significantly <strong>faster computations </strong> and <strong>simpler code</strong>.</p>
<h2 id="heading-efficient-memory-management">Efficient Memory Management</h2>
<h3 id="heading-memory-layout">Memory layout</h3>
<p>Numpy's data structures—primarily the <em>ndarray</em>—store data in <strong>contiguous memory</strong> blocks. As analyzed in the previous section, this memory layout allows for much faster reading and processing of large datasets.</p>
<h3 id="heading-cache-optimization">Cache optimization</h3>
<p>We've seen how utilizing the cache can improve code performance. NumPy uses <strong>blocking/tile algorithms</strong> to maximize <strong>data locality</strong> and <strong>cache utilization</strong>. The idea is to divide computations into smaller, more manageable chunks called <em>blocks</em> or <em>tiles</em>. Each block is processed entirely before moving on to the next, helping ensure that the necessary data stays in the cache for as long as possible.</p>
<h2 id="heading-parallelism-and-multi-core-utilization">Parallelism and Multi-Core Utilization</h2>
<p>In addition to parallel data processing <strong>within a single CPU core</strong>, Numpy also supports parallelism across multiple cores, benefiting from the same optimization techniques used for single-core performance.</p>
<h2 id="heading-and-more">And more...</h2>
<p>There are still many other algorithms tailored to specific hardware, memory designs, and device architectures. All of them contribute to creating a sophisticated solution for processing large datasets.</p>
<h1 id="heading-takeaways">Takeaways</h1>
<ul>
<li><strong>Algorithm complexity isn't everything</strong>: Reducing the number of operations helps, but it's only effective if the hardware environment stays consistent.</li>
<li><strong>Hardware matters</strong>: A solid understanding of fundamental hardware operations—such as data access, cache behavior, memory management, and CPU architecture—can help you design significantly more efficient solutions.</li>
<li><strong>Code optimization has limits, thus hardware matters even more</strong>: At some point, algorithmic improvements reach their theoretical minimum. Hardware, on the other hand, continues to evolve. Major tech companies invest heavily in better processors, memory, and accelerators—giving you new tools to push performance further.</li>
<li><strong>Hybrid optimization is essential</strong>: For large-scale or performance-critical tasks, neither algorithmic efficiency nor hardware tuning alone is sufficient. Combining both—writing efficient code and aligning it with hardware behavior—yields the best results.</li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Optimizing Large Android Project Builds : When Recommended Tunings Aren't Enough 
- Part 2]]></title><description><![CDATA[In the previous part, we discuss about why solely following standard advice often isn’t enough to boost your build performance. In this part, we will explore strategies to really unlock your builds— while also keeping them consistently optimized and ...]]></description><link>https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-2</link><guid isPermaLink="true">https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-2</guid><category><![CDATA[Android]]></category><category><![CDATA[android app development]]></category><category><![CDATA[gradle]]></category><category><![CDATA[gradle profiler]]></category><category><![CDATA[Build tool]]></category><category><![CDATA[jvm]]></category><dc:creator><![CDATA[Lam PHAM]]></dc:creator><pubDate>Fri, 25 Apr 2025 09:02:25 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1745361747508/01c27aa2-5a41-4814-b4a1-c1d2d86cdbef.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><a target="_blank" href="https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-1">In the previous part</a>, we discuss about why solely following standard advice often isn’t enough to boost your build performance. In this part, we will explore strategies to really unlock your builds— while also keeping them consistently optimized and ensuring no regression silently sneak back in.</p>
<p><a target="_blank" href="https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-1">Link to Part 1</a></p>
<h1 id="heading-keys-to-improving-build-performance">Keys to improving build performance</h1>
<h2 id="heading-understand-the-tools">Understand the tools</h2>
<p>If you have read through the previous part, you are likely noticed a common pattern behind why simply enabling the recommended features fall short: developers often do not understand <em>how</em> the tools work. They tends to understand <em>what</em> these features do, but not <em>how</em> they do it. And that <em>how</em> is essential when it comes to troubleshooting build issues and getting the most out of each feature.</p>
<p>At the end of the day, improving build time ultimately comes down to making ways for the existing features to work at their best. There’s no need to introduce new techniques but adjusting the system so that the Gradle features can do what they are designed to do. Therefore, understanding the tools deeply is the inevitable path to the success.</p>
<p>Take Build Cache as an example, its concept is straightforward, not new and relatively easy to understand: it skips task execution if there are no changes between builds. Most developers would easily grasp the idea, or the <em>what</em>. However, imagine having a problem where a second build is launched right after the first one without modifying any thing, and yet some tasks still get executed. Knowing the <em>what</em> might help you spot that something’s off, but it will not guide you how to investigate and fix it. This is where understanding the <em>how</em> truly matters. Moreover, knowing the concept input-output of Build Cache helps ensure you do not introduce regression while extending your build scripts.</p>
<p>In Android ecosystem, understanding the tools starts with understanding <strong>Gradle</strong> itself and its build mechanisms: Gradle Daemon, Build Cache, incremental build, parallelism and so on. Next comes the <strong>Android Gradle Plugin</strong> (a.k.a AGP), another major tool in managing Android-specific builds. A solid grasp of AGP helps you navigate platform specific concerns like build types, variants, minification and others. Beyond that, it’s also essential to understand the <strong>Programming Languages</strong> involved and their compilation processes. Knowing how Java and Kotlin handle compilation—especially features like <em>compilation avoidance</em> and the differences between the two—allows you to write production code that’s much more build-friendly. Lastly, familiarity with the underlying build environment—<strong>the JVM</strong>—can be a great advantage when it comes to optimizing build memory usage or tuning other performance-related aspects.</p>
<h2 id="heading-understand-your-project">Understand your project</h2>
<p>To choose the right tools or setups, knowing your project’s characteristics and what it needs is everything. Get inspired by solutions from other projects, but always remember that your project is different.</p>
<p>For instance, both Android build and Java servers run on JVM, but they are optimized for different goals. A Java server JVM is tuned for runtime performance, focusing on high throughput, low latency, and efficient resource management (memory, CPU) to handle concurrent requests and long-running processes.. On the other hand, Android builds are short-lived processes that prioritize build speed, consistency and toolchain stability.</p>
<p>As a result, a GC optimized for a server JVM may not work for an Android build. Dalvik, the Android runtime, is a great example for this distinction, as it is designed specifically for mobile application runtime environment with different prioritizes.</p>
<p>Another example, if your project is structured as a large single module with a high volume of unit tests, <a target="_blank" href="https://docs.gradle.org/current/userguide/performance.html#a_run_tests_in_parallel"><em>enabling parallel test</em></a> is critical and helps significantly speed up your testing task.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745656412286/f29fbcbf-f99a-4727-b446-a062ec15c8cc.png" alt="Parallel test execution" class="image--center mx-auto" /></p>
<p>However, if your project is already structured as a multi-module setup, this option generally won’t provide much additional benefit when running all unit tests across the entire project—and in some situation, it may even backfire the build with the cost of spawning extra processes.</p>
<p><em>Note that there are still cases where this feature is quite useful even in a multi-module project—I will show an example of that in a later section.</em></p>
<h2 id="heading-understand-build-environment">Understand build environment</h2>
<p>So far, we have seen that tools’ efficiency can vary much depending on projects. Another key factor that contributes significantly to the effectiveness of these solutions is the build environment.</p>
<p>Building in local is different than building in CI, or in Android context, building a debug version is different than building a release one.</p>
<p>Locally, incremental builds and build cache are incredibly efficient because cache are stored on your machine and you probably have enough space to retain them over time. Furthermore, the differences between consecutive builds are often minimal, this allows JVM to optimize itself and improve performance across builds. That said, building in local is often faster and you have the flexibility to fine-tune them based on your work machine’s characteristics.</p>
<p>In contrast, building on CI is a bit trickier. For instance, if you are using ephemeral agents, you don’t have a persistent storage on agents to retain cache data. As a result, incremental build and build cache can’t function as straightforwardly as they do in local environments. In addition, the JVM’s ability to optimize itself over successive runs is completely lost.</p>
<p><em>Note: Actually, while there is no way to preserve incremental builds, you can always be able to apply build cache in your CI setup. I will talk about this in more detail in the next section.</em></p>
<p>Secondly, CI agents often come with hardware limitations. If unfortunately you aren’t given a well-provisioned setup, it will be challenging to adjust the builds to run effectively on these constrained agents.</p>
<p>With that, knowing the characteristics and limitations of your build environment is important— because even the best tools can fall short if the environment isn’t set up to support them.</p>
<h2 id="heading-do-proper-benchmarks-and-monitoring">Do proper benchmarks and monitoring</h2>
<p>If there is not one-size-fits-all solution, and the effectiveness of build tools depends heavily on the project and environment, how can we ensure that a solution actually works well for your use cases?</p>
<p>Well, benchmark it!</p>
<p>Only benchmarking can validate the efficiency of a solution. There’s no theory that can guarantee a specific Garbage Collector would fit your system well— benchmarking is a reliable way to finding out.</p>
<p>Besides, setting up a build metrics monitoring system is just as important. Similar to benchmarking, it help approve a solution’s outcomes. However, benchmarking is a tricky business— it requires deep expertise as it is not always easy to accurately simulating a real-world system. If any part of the setup doesn’t reflect your actual production scenario, the benchmark result may be misleading. On the other hand, a monitoring system captures data from real system usage, making it far more trustworthy and 100% reflective of how your builds truly perform. The only tradeoff of a monitoring system is it takes time to gather data, while benchmarking gives quicker insights.</p>
<p>Beyond validation, a monitoring system also offers other benefits: it provides a holistic view about build heath, helps detect bottlenecks or spot areas for potential improvement.</p>
<p>Ultimately, benchmarking and monitoring provide the data-driven foundation needed to validate any build solution—without them, applying new tools should be done with caution.</p>
<h1 id="heading-real-world-scenarios">Real world scenarios</h1>
<p>Enough with the theory, let’s dive into a series of practical solutions I’ve applied, as a part of my team, while optimizing a large-scale Android project with over 500 modules.</p>
<p>Keep in mind that these solutions are based on the aforementioned underlying tunings that are already enabled by default.</p>
<h2 id="heading-remote-build-cache">Remote build cache 😎</h2>
<p>Our main project is backed by a huge volume of unit tests, which counts around 75k tests. We run the unit test CI pipeline as a mandatory check for every commit in Pull Requests (PR). In order for the full testsuite to complete, it takes almost one hour and a half🔥🔥. This is certainly an unacceptable amount of time for a single PR check.</p>
<p>As explained earlier, having incremental build and build cache enabled doesn’t cut any second of CI build time. Therefore, we had to deploy a <strong>remote build cache</strong> system. The core idea remains the same as local build cache, the only difference compared to a local setup is that the cache is stored remotely instead of on local machines. We use S3 as the remote storage to store cache and share it across the builds. This dramatically reduced our unit test execution time from a <strong>consistent one hour and a half to a range from several minutes to 40 minutes</strong>🚀🚀— depends on how many tests are skipped because of the cache.</p>
<p><em>While unit test was the original reason we set up this remote build cache, all other builds have benefited as well, leading to a significant overall reduction in build times.</em></p>
<p>Note: We don’t share this remote cache with local builds for a couple of reasons:</p>
<ul>
<li><p>Local builds are already well supported by a rich set of local cache.</p>
</li>
<li><p>Local environments tend to have more noisy or inconsistent builds, and pushing cache from these could degrade the overall quality of the shared remote cache.</p>
</li>
<li><p>While having little benefits for local builds, interacting with remote cache does cost additional money.</p>
</li>
</ul>
<p>You can find details on how to enable remote build cache <a target="_blank" href="https://docs.gradle.org/current/userguide/build_cache.html#sec:build_cache_configure_remote">here</a>, or consider using <a target="_blank" href="https://gradle.com/develocity/">Develocity</a> solution to help setup and manage it more easily.</p>
<h2 id="heading-addressing-build-cache-misses">Addressing build cache misses 😎</h2>
<p>After enabling remote build cache, we still observed an odd behavior where all unit tests in a particular module—the application module specifically— are re-executed even when no changes are introduced in the new builds.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745532030393/480f1ee1-6e6a-4ca9-b542-219fec3a1cdc.png" alt class="image--center mx-auto" /></p>
<p>This is when we learned about <em>build cache misses</em> and how they were affecting our builds. We manually diagnosed build cache by utilizing the <code>-Dorg.gradle.caching.debug=true</code> flag, as suggested <a target="_blank" href="https://docs.gradle.org/current/userguide/build_cache_debugging.html#helpful_data_for_diagnosing_a_cache_miss">here</a>. While this is a quite time-consuming and challenging approach, we will show a much simpler alternative a bit later. Through this manual debugging process, we identified 2 libraries as the root cause: NewRelic and Crashlytics. If you are interested in the details, I opened 2 tickets to both repositories: <a target="_blank" href="https://github.com/newrelic/newrelic-android-agent/issues/168">Newrelic</a> and <a target="_blank" href="https://github.com/firebase/firebase-android-sdk/issues/6000">Crashlytics</a>. Unfortunately, neither library has provided a proper fix for the issues so far. We had to implement our own workaround for the Newrelic case, but the Crashlytics issue remains unresolved. Luckily, the Crashlytics problem only affect release builds, which represent a smaller portion of the builds in our workflow.</p>
<h2 id="heading-develocity">Develocity 😎</h2>
<p>Up until that point, we had been receiving frequent complaints about local builds hanging for dozens of minutes, even half an hour. Unfortunately, we had no visibility on what happened on other people’ machines. Given that we were actively working on improving our build system— including migrating the tools, troubleshooting build issues, optimizing some build pipelines etc— it became clear that we were missing a proper monitoring system to support these works. This is when we decided to run a POC with <a target="_blank" href="https://gradle.com/develocity/">Develocity</a>, the tool that could address all our needs.</p>
<p>Develocity, formerly known as Gradle Enterprise, is a tool that helps monitor all your builds, either CI or local.</p>
<p>This tool firstly opened <a target="_blank" href="https://github.com/gradle/develocity-build-validation-scripts">a ways simpler approach</a> to debug build cache missing issue. By utilizing this, we discovered numerous other cache misses across different scenarios and resolved them, which ultimately generate an approximate <strong>30%</strong>🚀🚀 reduction in build times across all pipelines.</p>
<p>In general, we use Develocity to:</p>
<ul>
<li><p><strong>Monitor builds</strong> and to get a holistic view about system’s heath: both in local and in CI.</p>
</li>
<li><p><strong>Improve unit tests speed as well as quality</strong> with <em>Predictive Test Selection</em> and <em>Flaky Test</em> features.</p>
</li>
<li><p><strong>Investigate issues</strong> more effectively. For example, by looking at some of the longest local builds, we found out that the hanging build issue often comes from dependencies fetching— usually due to people not being connected to the company VPN, since some dependencies are hosted in our internal artifactories.</p>
</li>
<li><p><strong>Identify areas for improvement</strong>. I have a great example about this that I will talk about in very shortly.</p>
</li>
<li><p><strong>Validate our solutions</strong>. With comprehensive build metrics, it's easy to back up changes with clear data—whether it's through simple charts or key numbers.</p>
</li>
</ul>
<h2 id="heading-build-regression-pipeline">Build regression pipeline 😎</h2>
<p>The example above shows how impactful a cache miss is to the build. That’s why it’s important not only to resolve existing cache misses, but also prevent new ones from getting introduced into the project. Investigating cache misses is time-consuming and we definitely don’t want to do it regularly. Hence, having recurring jobs to run the <a target="_blank" href="https://github.com/gradle/develocity-build-validation-scripts">build validation scripts</a> will help automatically detect cache misses early, ensuring the quality and reliability of our build cache over time.</p>
<p><a target="_blank" href="https://gradle.com/blog/monitoring-build-performance-at-scale/">Here is an example how Netflix did it</a>.</p>
<h2 id="heading-parallel-unit-tests">Parallel unit tests 😎</h2>
<p>In the previous section, I mentioned that <a target="_blank" href="https://docs.gradle.org/current/userguide/performance.html#a_run_tests_in_parallel"><em>parallel</em> <strong><em>test</em></strong> <em>execution</em></a> isn’t beneficial in a multi-module project. This is because when running unit tests across the entire project, each module runs its tests in a separate task. And given that the <a target="_blank" href="https://docs.gradle.org/current/userguide/performance.html#sec:parallel_execution"><em>parallel</em> <strong><em>task</em></strong> <em>execution</em></a> is already enabled, those test tasks will naturally run in parallel anyway.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745535790108/d1d7ec43-306d-4c97-a793-321d32c4807a.png" alt class="image--center mx-auto" /></p>
<p>However, there are still cases where parallel test execution is actually useful. The chart below—taken from the Develocity dashboard— roughly shows what our unit test pipeline looked like at the time:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745535596411/a929b059-0b97-480b-b832-70c748c75a87.png" alt class="image--center mx-auto" /></p>
<p>It is not hard to spot the problem in this chart. The pipeline starts off strong, utilizing all 6 cores in parallel. But as the build progresses, most modules finish quickly—except for two large ones: <code>:payment</code> and <code>:search</code>. These two tasks continue running on just two cores while the remaining cores sit idle.</p>
<p>What a waste of resources!</p>
<p>The solution was to break these 2 heavy modules into smaller chunks— which means we <em>only</em> enabled <em>parallel test execution</em> for <code>:payment</code> and <code>:search</code>. This allows their tests to run across more cores— in other words, in parallel. This way, we could utilize all the available resources to significantly speed up the build.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745657592136/6fb10d1b-00d3-4019-882f-2cfa712ab1b5.png" alt="Parallel test execution for :payment and :search" class="image--center mx-auto" /></p>
<p>Here’s what the build looked like after applying the fix:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745536174306/d13d920b-747d-4d71-b1dd-101b348b6103.png" alt class="image--center mx-auto" /></p>
<p>This piece of work has improved <strong>40%</strong>🚀🚀 of the unit test builds involving these 2 modules— excluding cases where the results were already cached. This is a great example of how understanding your project—and having proper monitoring in place—can lead to meaningful improvements in your builds.</p>
<h2 id="heading-garbage-collector-benchmarking">Garbage Collector benchmarking 🤔</h2>
<p><a target="_blank" href="https://developer.android.com/build/optimize-your-build#experiment-with-the-jvm-parallel-garbage-collector">This suggestion</a> from Google also caught our attention— they recommended trying the Parallel GC. We decided to do a benchmark and compare between G1 GC and ParallelGC.</p>
<p>We used <a target="_blank" href="https://github.com/gradle/gradle-profiler">gradle-profiler</a> and setup a CI pipeline to run the benchmark over night (for a benchmark, we run with fresh builds, without cache, so it often take hours for the benchmark to finish).</p>
<p>Here’s an example showing how the Parallel GC performed on one of our pipelines compared to G1 GC:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745538402146/d0df071b-5a76-42f2-9bc6-95e265330efb.png" alt class="image--center mx-auto" /></p>
<p>Based on the results, <strong>we didn’t find enough evidence to justify switching to Parallel GC</strong> 😞😞. So we continue to stick with G1 GC for now.</p>
<h1 id="heading-takeaways">Takeaways</h1>
<p>There’s no shortcut to optimizing build performance. While Gradle’s default options can help improve 70% of yous builds, the remaining 30% will drag them down if you don’t know how to navigate it. The essentials to overcoming build issues is a deep understanding about the ecosystem:</p>
<ul>
<li><p>Learn to effectively use major tools in the Android build system: <em>Gradle, AGP, Languages (Java/Kotlin), JVM</em></p>
</li>
<li><p>Understand your project from a holistic perspective. This includes setting up proper metrics and monitoring to consistently track the system’s health and spot regression early.</p>
</li>
<li><p>Always experiment and benchmark the impact of a feature on your project before applying them.</p>
</li>
</ul>
<p>Consistent, informed iteration is what leads to meaningful and sustainable performance gains.</p>
<h1 id="heading-resources">Resources</h1>
<p>For those who want to learn more about Gradle and AGP, I strongly recommend this book:</p>
<ul>
<li><a target="_blank" href="https://www.amazon.com/Extending-Android-Builds-Pragmatic-Gradle-ebook/dp/B0CXMZBZL6">Extending Android Builds: Pragmatic Gradle and AGP Skills with Kotlin</a></li>
</ul>
]]></content:encoded></item><item><title><![CDATA[Optimizing Large Android Project Builds : When Recommended Tunings Aren't Enough - Part 1]]></title><description><![CDATA[Link to Part 2
Overview
In large Android projects—or any Gradle-based project—build speed is often a pain point. This is understandable because building a JVM-based application involves multiple steps, including compiling code and resources, minifyin...]]></description><link>https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-1</link><guid isPermaLink="true">https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-1</guid><category><![CDATA[Android]]></category><category><![CDATA[gradle]]></category><category><![CDATA[Build tool]]></category><category><![CDATA[android app development]]></category><category><![CDATA[android apps]]></category><category><![CDATA[jvm]]></category><category><![CDATA[gradle build tool]]></category><dc:creator><![CDATA[Lam PHAM]]></dc:creator><pubDate>Tue, 22 Apr 2025 22:36:56 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1745361550378/42d91e21-9814-4fd3-b2de-3563aab7fa9d.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><a target="_blank" href="https://blogs.lampham.dev/optimizing-large-android-project-builds-when-recommended-tunings-arent-enough-part-2">Link to Part 2</a></p>
<h1 id="heading-overview">Overview</h1>
<p>In large Android projects—or any Gradle-based project—build speed is often a pain point. This is understandable because building a JVM-based application involves multiple steps, including compiling code and resources, minifying and optimizing the output, and assembling everything into the final artifact. These steps are inherently time-consuming, especially in large codebases with hundreds or even thousands of modules.</p>
<p>Gradle is the go-to build tool for these types of project. Over years, it has developed numerous of genius solutions to optimize build performance. Yet, even with all these features enabled, many app engineers still find themselves frustrated by sluggish build times. This is often where Gradle takes the blame—criticized for being overly complex without addressing developers’ expectation.</p>
<p>In this Part 1 of a 2-part series, we’ll explore why, despite benefiting all Gradle’s optimizations, your builds might still feel inefficient. And from there, in Part 2, we will dive into what we could do to actually improve your builds even further.</p>
<h1 id="heading-why-recommended-tunings-dont-often-help">Why recommended tunings don’t often help?</h1>
<p>When searching for ways to optimize a Gradle-based build— whether for Android project or in general— you will often come across these typical suggestions, commonly found in Gradle’s official documentation or shared through articles, blog posts, and conference talks:</p>
<ul>
<li><p>Update to the latest tools (Gradle, AGP, JDK, Java/Kotlin and so on) version.</p>
</li>
<li><p>Enable build cache: <code>org.gradle.caching=true</code></p>
</li>
<li><p>Use incremental build.</p>
</li>
<li><p>Use compilation avoidance.</p>
</li>
<li><p>Enable parallel execution: <code>org.gradle.parallel=true</code></p>
</li>
<li><p>Enable parallel test.</p>
<p>  <img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1745656088146/9e1e90a9-38d0-4bf3-8bf4-96285315aaa9.png" alt class="image--center mx-auto" /></p>
</li>
<li><p>Enable Gradle daemon: <code>org.gradle.daemon=true</code></p>
</li>
<li><p>Increase heap size: <code>org.gradle.jvmargs=-Xmx...M</code></p>
</li>
<li><p>Enable some specific Garbarge Collectors, for i.e: parallelGC <code>org.gradle.jvmargs=-XX:+UseParallelGC</code></p>
</li>
<li><p>Fine-tune JVM options</p>
</li>
<li><p>and so on</p>
</li>
</ul>
<p>These solutions look appealing— one single line and your builds are supposed to be supercharged. What more could we ask for? Here comes the very common mistake: many developers do no more than appending these lines into there <code>gradle.properties</code> file and expect a dramatic speedup. The hard truth is, while these options are genuinely effective, we won't often see the build performance change much.</p>
<h2 id="heading-just-because-you-enabled-it-doesnt-mean-you-made-a-difference">Just because you enabled it doesn’t mean you made a difference</h2>
<p><strong>In new versions of Gradle,</strong> <strong>most of the above options are enabled by default</strong>. In other word, explicitly adding them— without any deeper tuning— usually <strong>doesn’t</strong> make any changes. That’s one of the main reasons why your build time often remains unchanged.</p>
<p>And when you think about it, it makes perfect sense. If simply pasting a few lines could boost your builds, wouldn’t Gradle just enable them out of the box?</p>
<p>The only benefit of adding these lines is to give you the confidence that you haven’t missed anything. Ever found yourself hitting <code>CTRL + C</code> dozens of times before finally pressing <code>CTRL + V</code>? Yeah, it’s kind of like that.</p>
<h2 id="heading-one-size-doesnt-fit-all">One size doesn’t fit all</h2>
<p>The majority of Gradle solutions will have an impact in many types of projects. However, some are tailored for specific systems, based on factors like project size, system requirements (e.g, memory efficiency, low latency, high throughput etc.), team size or other considerations.</p>
<p>Let’s take a look at the Gradle official documentation, <a target="_blank" href="https://docs.gradle.org/current/userguide/performance.html#run_the_compiler_as_a_separate_process">it recommends running Java compilation in a separate process</a>. However, a crucial information is tucked away at the very end of the section:</p>
<blockquote>
<p>Forking compilation rarely impacts the performance of small projects. But you should consider it if a single task compiles more than a thousand source files together.</p>
</blockquote>
<p>Or in case of the ParallelGC, Google suggests <a target="_blank" href="https://developer.android.com/build/optimize-your-build#experiment-with-the-jvm-parallel-garbage-collector"><strong><em>experimenting and testing</em></strong></a> with the JVM parallel garbage collector. The emphasis on "experiment" is important—GC performance can vary greatly depending on the specific project. That’s why Google recommends testing it first. However, this nuance is often overlooked, and people tend to misinterpret the advice as simply “use the JVM parallel garbage collector.”</p>
<p>When desperately looking for a quick solution, who really has the patience to read all the way through—especially when a shiny code snippet, seemingly the holy grail to all your problems, has already got your focus?</p>
<h2 id="heading-tuning-isnt-about-maxing-out-settings">Tuning isn’t about maxing out settings</h2>
<p>I’ve seen many blogs advising to allocate more heap size— which is completely a great suggestion, except they often specify a <em>magic value</em> (usually 4gb) and don’t give any further explanation.</p>
<pre><code class="lang-bash">org.gradle.jvmargs=-Xmx4g
</code></pre>
<p>Gradle puts 2gb in its recommended <a target="_blank" href="https://docs.gradle.org/current/userguide/performance.html#increase_the_heap_size">code snippet</a>.</p>
<p>In a regular JVM, bigger heap could indeed help reduce garbage collection pause-time, increase application throughput, reduce latency and everything. However, for a given system, increasing the heap size eventually stops yielding benefits. It's also important to keep in mind that device memory sets an upper limit.</p>
<p>Therefore, an efficient max heap size really depends on many factors: the project size, the libraries used (e.g, <a target="_blank" href="https://github.com/robolectric/robolectric">Robolectric</a> is famous for being memory-hungry), the device memory, the environment: either it’s a local build where you don’t want to sacrifice all your device’s memory to the builds, because you still want your browser to play music— or a CI build in a Linux agent where you can allocate all available memory, but you will have to watch out for the <a target="_blank" href="https://neo4j.com/developer/kb/linux-out-of-memory-killer/">OOM Killer</a>, which can terminate your processes at any time.</p>
<p>That said, allocating too little memory can slow down the build process, while giving too much doesn’t always make an impact— and in certain conditions, it could backfire the builds.</p>
<p>The most important thing is that these articles, blogs, talks shouldn’t be the primary source for defining performance metrics in your own system.</p>
<p><em>Notes: You may assume that hitting the upper limit (the device memory capacity) is rare. However, in practice, the max heap size is often well below from the total memory the OS allocates to your build at any given time. For instance, on a machine of 32GB of RAM, you may think it’s safe to set the max heap size to 16GB. However, that heap only represents the heap of the main process— typically the Gradle daemon. Your builds may also spin up other processes, such as Kotlin Daemons or Gradle Workers, which collectively can push the peak memory usage much higher and close to the upper bound 32GB.</em></p>
<h2 id="heading-not-all-build-flags-are-plug-and-playsome-need-your-code-to-meet-specific-criteria">Not all build flags are plug-and-play—some need your code to meet specific criteria.</h2>
<p>Gradle build cache, one of the Gradle greatest features, functions based on a task’s inputs and outputs. If inputs stay unchanged, the task is skipped and cached outputs are reused. However, if you create a task without any inputs or outputs, or inputs are different between builds even though there’s no updates, build cache won’t work.</p>
<p>You may say you never create any custom Gradle tasks in your project. Well, your libraries certainly do!</p>
<p>The same principles apply for some other features such as compilation avoidance (this depends mostly on the languages like Groovy/Java/Kotlin, not Gradle) where you must write your production code in a way that the compiler could skip recompiling as much code as possible.</p>
<p>Therefore, enabling these options without crafting your production and build code won’t yield fruitful results.</p>
<h2 id="heading-wrong-expectation">Wrong expectation</h2>
<p>Sometimes, you just have high expectations for what these options could deliver.</p>
<p>Configuration Cache is another piece of engineering by Gradle. As of the time of writing, it’s still in active development, and not yet enabled by default. Its fundamental is similar to Build Cache, but it targets the configuration phase. If you're hoping it will drastically cut down your build time, like Build Cache does, you're likely to be disappointed.</p>
<p>In general, configuration phase takes up only a small portion of the Gradle build lifecycle. This duration varies based on the complexity of the project and the tasks you invoke, however, it typically ranges from hundreds millisecond— for small projects, to seconds even a few minutes—for large ones. Hence, compared to the execution phase—which can take dozens of minutes or even hours, improvement from configuration caching is often hard to notice.</p>
<p>This is not to say Configuration Cache isn’t worthwhile. It is definitely valuable, especially in large project builds. However, setting wrong expectations would lead to disappointment and misunderstandings about the feature.</p>
<h1 id="heading-part-1-wrapping-up">Part 1 wrapping up</h1>
<p>All the previous points are not meant to deny the great benefits of the recommended tunings. Simply enabling them (which, by the way, are often on by default) could help optimize 70% of your builds. However, the remaining 30% tends to be the most complex and requires a deeper, more targeted approach to truly improve. In small projects, this part is often negligible and has little impact on developers’ productivity. The real problem arises as the projects grow larger and larger.</p>
<p>In part 2, we will dive into some key strategies to tackle that remaining 30%, backed by some real-world examples.</p>
<p><em>Disclaimer: This 70-30 split is not based on any hard analysis or data. It's just a metaphor, inspired from</em> <a target="_blank" href="https://addyo.substack.com/p/the-70-problem-hard-truths-about"><em>this blog about AI-assisted coding tools</em></a> <em>to convey the idea.</em></p>
]]></content:encoded></item></channel></rss>