KAFKA-18422 Adjust Kafka client upgrade path section (#19119)

This patch adds a section about upgrading clients to the upgrade notes.

Reviewers: Ismael Juma <ismael@juma.me.uk>, David Jacot <djacot@confluent.io>

docs/upgrade.html

@@ -19,8 +19,10 @@
 
 <script id="upgrade-template" type="text/x-handlebars-template">
 
-<h4><a id="upgrade_4_1_0" href="#upgrade_4_1_0">Upgrading to 4.1.0 from any version 3.3.x through 4.0.x</a></h4>
+<h4><a id="upgrade_4_1_0" href="#upgrade_4_1_0">Upgrading to 4.1.0</a></h4>
-    <h5><a id="upgrade_410_notable" href="#upgrade_410_notable">Notable changes in 4.1.0</a></h5>
+
+<h5><a id="upgrade_4_1_0" href="#upgrade_4_1_0">Upgrading Servers to 4.1.0 from any version 3.3.x through 4.0.x</a></h5>
+    <h6><a id="upgrade_410_notable" href="#upgrade_410_notable">Notable changes in 4.1.0</a></h6>
         <ul>
             <li><b>Producer</b>
                 <ul>
@@ -29,7 +31,27 @@
                 </ul>
             </li>
         </ul>
-<h4><a id="upgrade_4_0_0" href="#upgrade_4_0_0">Upgrading to 4.0.0 from any version 3.3.x through 3.9.x</a></h4>
+
+<h4><a id="upgrade_4_0_0" href="#upgrade_4_0_0">Upgrading to 4.0.0</a></h4>
+
+<h5><a id="upgrade_clients_4_0_0" href="#upgrade_clients_4_0_0">Upgrading Clients to 4.0.0</a></h5>
+
+<p><b>For a rolling upgrade:</b></p>
+
+<ol>
+    <li>Upgrade the clients one at a time: shut down the client, update the code, and restart it.</li>
+    <li>Clients (including Streams and Connect) must be on version 2.1 or higher before upgrading to 4.0. Many deprecated APIs were removed in Kafka 4.0.
+        For more information about the compatibility, please refer to the <a href="/{{version}}/documentation/compatibility.html">compatibility matrix</a>
+        or <a href="https://cwiki.apache.org/confluence/x/y4kgF">KIP-1124</a>.</li>
+</ol>
+
+<h6><a id="upgrade_clients_400_notable" href="#upgrade_clients_400_notable">Notable changes in 4.0.0</a></h6>
+
+<ul>
+    <li>Please see notable changes in the server section.</li>
+</ul>
+
+<h5><a id="upgrade_4_0_0" href="#upgrade_4_0_0">Upgrading Servers to 4.0.0 from any version 3.3.x through 3.9.x</a></h5>
 
 <p>Note: Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed. As such, <b>broker upgrades to 4.0.0 (and higher) require KRaft mode and
     the software and metadata versions must be at least 3.3.x</b> (the first version when KRaft mode was deemed production ready). For clusters in KRaft mode
@@ -52,12 +74,9 @@
         Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java">MetadataVersion</a>
         has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_4_0_IV1(23, "4.0", "IV1", true)</code> means this version has metadata changes).
         Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li>
-    <li>For the Kafka client upgrade path, note that many deprecated APIs were removed in Kafka 4.0. Additionally, upgrading directly to 4.x from certain versions is not feasible.
-        For more information, please refer to <a href="https://cwiki.apache.org/confluence/x/y4kgF">KIP-1124</a>.
-    </li>
 </ol>
 
-    <h5><a id="upgrade_400_notable" href="#upgrade_400_notable">Notable changes in 4.0.0</a></h5>
+    <h5><a id="upgrade_servers_400_notable" href="#upgrade_servers_400_notable">Notable changes in 4.0.0</a></h5>
     <ul>
         <li>
             Old protocol API versions have been removed. Users should ensure brokers are version 2.1 or higher before upgrading Java clients