{"id":3584,"date":"2026-11-02T09:00:28","date_gmt":"2026-11-02T16:00:28","guid":{"rendered":"https:\/\/yyc.ninja\/?p=3584"},"modified":"2026-11-02T09:00:28","modified_gmt":"2026-11-02T16:00:28","slug":"when-storage-is-the-bottleneck-capacity-lessons-lakehouse-kubernetes","status":"publish","type":"post","link":"https:\/\/yyc.ninja\/index.php\/2026\/11\/02\/when-storage-is-the-bottleneck-capacity-lessons-lakehouse-kubernetes\/","title":{"rendered":"When Storage Is the Bottleneck: Capacity Lessons from a 28-Application Lakehouse on Kubernetes"},"content":{"rendered":"<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1200\" height=\"630\" src=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover.png\" alt=\"When Storage Is the Bottleneck: Capacity Lessons from a 28-Application Lakehouse on Kubernetes\" class=\"wp-image-3583\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" srcset=\"https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover.png 1200w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-300x158.png 300w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-1024x538.png 1024w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-768x403.png 768w, https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-100x53.png 100w\" \/><\/figure>\n<p class=\"wp-block-paragraph\">Capacity planning for Kubernetes usually starts with CPU and memory. Storage latency tends to get less attention, partly because well-provisioned clusters rarely expose what happens when it runs out. So I ran an exercise to expose it: deploy a full data platform on storage that I knew was far too slow for it, then document exactly how the platform failed and what it took to make it work.<\/p>\n<p class=\"wp-block-paragraph\">This was never a production design. A platform of this size belongs on SSD or NVMe, and the limits of the hardware were clear before the first install. The value was in finding out how those limits would surface, because undersized storage rarely announces itself. It failed in ways that looked nothing like a storage problem.<\/p>\n<p class=\"wp-block-paragraph\">The test platform was a full open-source lakehouse on a three-node Kubernetes cluster, deployed through Argo CD as 28 applications and about 110 pods:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Storage and table format:<\/strong> SeaweedFS (S3 API) and Nessie (Iceberg catalog)<\/li>\n<li><strong>Query and processing:<\/strong> Trino, Spark, Kafka, NiFi and Airflow<\/li>\n<li><strong>Analytics and data science:<\/strong> Superset, JupyterHub and MLflow<\/li>\n<li><strong>Metadata, features and serving:<\/strong> Gravitino, Feast and KServe<\/li>\n<li><strong>AI tooling and observability:<\/strong> WrenAI, Langfuse and Opik<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">By design, every node sat on the same HDD-backed storage. That made seek time, not bandwidth, the scarce resource, and it amplified behaviour that faster hardware would hide. The same pattern appears on any platform with slow or contended persistent storage, including small nodes and burst-limited network volumes, so the lessons below apply well beyond this cluster.<\/p>\n<h3 class=\"wp-block-heading\">Slow startup looks like failure to Kubernetes<\/h3>\n<p class=\"wp-block-paragraph\">The first rollout produced no useful error messages. It produced restarts:<\/p>\n<ul class=\"wp-block-list\">\n<li>Airflow&#8217;s API server looped on exit code 143.<\/li>\n<li>Nessie entered <code>CrashLoopBackOff<\/code>.<\/li>\n<li>Trino&#8217;s coordinator restarted every two minutes.<\/li>\n<li>Langfuse&#8217;s database came up without its user.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">These had a single cause. Helm chart defaults assume an application starts within a minute or two, and on seek-bound storage that assumption fails:<\/p>\n<ul class=\"wp-block-list\">\n<li>A JVM loading thousands of class files needs 3 to 5 minutes.<\/li>\n<li>Airflow needs about 5 minutes to import its Python dependencies.<\/li>\n<li>PostgreSQL&#8217;s first <code>initdb<\/code> takes several minutes.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">The kubelet sees a failed probe and kills the container. The next attempt is just as slow, so the loop never resolves itself.<\/p>\n<p class=\"wp-block-paragraph\">Restart loops are an inconvenience. The serious cases are the ones that corrupt state:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Interrupted migrations.<\/strong> A liveness probe killed Langfuse&#8217;s web pod partway through about 450 Prisma migrations. One migration stayed marked as failed, and the next start refused to run.<\/li>\n<li><strong>Partial initialization.<\/strong> Airflow&#8217;s metadata database restarted between <code>initdb<\/code> and setting the password. On the next start, the Bitnami entrypoint found a data directory and skipped initialization, leaving a user with no password.<\/li>\n<li><strong>False success.<\/strong> Opik&#8217;s migration wrapper printed &#8220;Database migrations completed successfully&#8221; after failing with <code>Connection refused<\/code>, because MySQL was still in a 14-minute first start.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">The remediation was systematic:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Add a <code>startupProbe<\/code> to every stateful workload,<\/strong> budgeted from measurements on the actual hardware: 10 to 20 minutes for JVMs and databases. A startup probe holds off liveness checks until the application has started once, so a slow first boot no longer reads as a hang.<\/li>\n<li><strong>Patch charts that hard-code their probes.<\/strong> Opik&#8217;s MySQL sub-chart does not expose them, so a Kustomize patch in Argo CD adds one.<\/li>\n<li><strong>Protect first initialization.<\/strong> A database should never run its first init on an overloaded node. If init is interrupted, delete the empty volume and restart on a quiet node.<\/li>\n<li><strong>Verify init containers after every install<\/strong> with <code>kubectl logs &lt;pod&gt; -c &lt;init-container&gt;<\/code>. A successful pod status does not prove a successful migration.<\/li>\n<\/ul>\n<p class=\"wp-block-paragraph\">The trade-off is deliberate. A genuinely hung Langfuse is now detected after 20 minutes rather than 2. With these startup times, that is the correct setting: slower detection is far cheaper than corrupted state.<\/p>\n<h3 class=\"wp-block-heading\">Shared storage couples unrelated workloads<\/h3>\n<p class=\"wp-block-paragraph\">In a typical cloud cluster, each node has independent disks. Here, every pod on every node competed for the same disk head, which changed several operational rules:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>Parallel work is slower than serial work.<\/strong> Installing five Helm releases at once, or pre-pulling 20 images per node concurrently, made the disk thrash between unrelated files. One 50 MB image took 30 minutes to pull. Deployment automation now installs one release at a time and pulls images sequentially.<\/li>\n<li><strong>Node restarts carry real risk.<\/strong> Restarting the Kubernetes node agent on a busy worker took 9 minutes. After 5 minutes in <code>NotReady<\/code>, Kubernetes evicted every pod on that node, and the remaining nodes absorbed the load all at once. The control plane, which was not tainted against workloads, reached 99% memory. The incident cost about an hour. The rule now is to drain the node or raise the eviction timeout before restarting an agent under I\/O load, and to taint control plane nodes.<\/li>\n<li><strong>Kubelet timeouts stop holding.<\/strong> The default runtime request timeout is 2 minutes. Unpacking a 1 GB image took longer, so each retry collided with the attempt still in progress and failed with <code>failed to reserve container name<\/code>. Recovery meant scaling heavy applications to zero, letting the core services settle, and returning workloads one at a time.<\/li>\n<li><strong>Diagnostics add load too.<\/strong> A <code>du<\/code> scan of the image store ran for an hour after its remote timeout expired, because a timeout on the calling side does not stop the process on the node. It pushed one worker to 74% iowait. On a contended node, use <code>df<\/code> or the kubelet stats API, and apply <code>timeout<\/code> on the node itself.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Two &#8220;slow disk&#8221; incidents that were not the disk<\/h3>\n<p class=\"wp-block-paragraph\">Slow storage becomes the default explanation for everything, which makes it easy to misdiagnose other faults. Two of the worst incidents had different root causes.<\/p>\n<p class=\"wp-block-paragraph\"><strong>inotify exhaustion.<\/strong> Dozens of pods sat in <code>ContainerCreating<\/code> while image pulls showed 0 MB\/s, which looked like I\/O starvation. The containerd log showed the real cause: <code>failed to create inotify fd: too many open files<\/code>. The common Linux default of 128 inotify instances per user is not enough for a node running around 50 pods. Two sysctl settings fixed it:<\/p>\n<pre><code class=\"language-ini\">fs.inotify.max_user_instances = 8192\nfs.inotify.max_user_watches = 1048576\n<\/code><\/pre>\n<p class=\"wp-block-paragraph\"><strong>Memory pressure disguised as I\/O.<\/strong> Image pulls stalled for more than 10 minutes with the network idle and workers at 60% to 91% iowait. The Grafana Alloy DaemonSet was sitting at its 512 MiB limit. Under that pressure, the kernel kept dropping the pages of the agent&#8217;s own binary and reading them back from disk. This came to only about 5 MB\/s of random reads per node, but with more than 900,000 major page faults it was devastating on a seek-bound disk. Raising the limit to 1 GiB immediately dropped iowait from 60% to 27%.<\/p>\n<p class=\"wp-block-paragraph\">Before attributing a problem to storage, check <code>vmstat<\/code> (<code>si<\/code> and <code>so<\/code>), major faults (<code>pgmajfault<\/code> in <code>\/proc\/vmstat<\/code>) and the containerd log.<\/p>\n<h3 class=\"wp-block-heading\">The highest-use fix: a small SSD tier<\/h3>\n<p class=\"wp-block-paragraph\">One measurement defined the problem: <strong>a synced write took 400 to 590 ms on the HDD.<\/strong> Databases perform at least one synced write per commit, so PostgreSQL, MySQL, etcd, RocksDB and ClickHouse metadata were all constrained by the same operation.<\/p>\n<p class=\"wp-block-paragraph\">Replacing all of the storage would have solved the problem and ended the exercise. The more useful question was how little fast storage it would take to change the outcome:<\/p>\n<ol class=\"wp-block-list\">\n<li><strong>etcd first.<\/strong> The control plane was at 83% to 94% iowait and the API server kept dropping out. Moving etcd&#8217;s data directory to an 8 GB SSD volume resolved this on its own.<\/li>\n<li><strong>Databases and metadata next.<\/strong> Each worker received a 12 GB SSD volume, and 20 of the 25 PersistentVolumeClaims moved onto it, covering every database, catalog and metadata store. Bulk data, including S3 objects, Kafka logs, NiFi repositories and notebook homes, stayed on the HDD.<\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\">Synced write latency dropped to <strong>3.4 to 8.3 ms, roughly 70 to 118 times faster.<\/strong><\/p>\n<p class=\"wp-block-paragraph\">Three decisions kept the change cheap:<\/p>\n<ul class=\"wp-block-list\">\n<li><strong>No migration.<\/strong> local-path volumes are directories, so a symlink moved each one without a new StorageClass or any change visible to the pod.<\/li>\n<li><strong>Selective placement.<\/strong> Only latency-sensitive data moved, and it fit in 24 GB across the workers.<\/li>\n<li><strong>Retained safety margin.<\/strong> The longer probes stayed in place in case of future contention.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Memory requests are a budget, not a measurement<\/h3>\n<p class=\"wp-block-paragraph\">Late in the rollout, the workers reached 99% and 94% of allocatable memory requests while actual use sat near 77%. Nothing new could schedule, including a single notebook. Chart defaults were the cause:<\/p>\n<table>\n<thead>\n<tr>\n<th>Component<\/th>\n<th>Requested<\/th>\n<th>Used<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>kserve-controller<\/td>\n<td>600 MiB<\/td>\n<td>54 MiB<\/td>\n<\/tr>\n<tr>\n<td>spark-operator controller<\/td>\n<td>512 MiB<\/td>\n<td>34 MiB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p class=\"wp-block-paragraph\">Reducing requests to measured use plus a margin, with limits unchanged, recovered about 1.2 GB per node.<\/p>\n<p class=\"wp-block-paragraph\">This applies to components that idle, not to everything. JVMs whose heap equals their request, such as Trino and NiFi, should be left alone. Spark needed the opposite treatment: its executor minimum is about 450 MiB of heap plus overhead, so the smoke test required a 640 MiB executor. Real jobs need more headroom than that.<\/p>\n<h3 class=\"wp-block-heading\">Disk capacity needs planning too<\/h3>\n<p class=\"wp-block-paragraph\">Container images alone reached 53 GB on one worker. The kubelet starts evicting pods below about 10 GB free, and each Kubernetes upgrade pulls about 1 GB of new images per node, so headroom disappears quickly.<\/p>\n<ul class=\"wp-block-list\">\n<li>Worker disks were expanded online from 100 GB to 160 GB.<\/li>\n<li>The control plane disk could not grow in place because the OS installer had put the swap partition after the root partition. The fix was to remove the swap and extended partitions, grow root, and switch to a swap file.<\/li>\n<li>JupyterHub&#8217;s 2.35 GB image takes 30 minutes to pull and was among the first to be garbage-collected once a disk passed 85%. A completed pod that references the image keeps it on the node, which avoids that pull.<\/li>\n<\/ul>\n<h3 class=\"wp-block-heading\">Recommendations<\/h3>\n<ol class=\"wp-block-list\">\n<li><strong>Measure synced-write latency before deploying anything,<\/strong> using <code>fio --fsync=1<\/code> or <code>pg_test_fsync<\/code>. Above 50 ms, databases will suffer.<\/li>\n<li><strong>Set probe budgets from measurements, not chart defaults,<\/strong> and add a <code>startupProbe<\/code> to everything stateful.<\/li>\n<li><strong>Serialize disk-heavy operations:<\/strong> installs, image pulls and restarts.<\/li>\n<li><strong>Treat requests as a scheduling budget<\/strong> and right-size them from <code>kubectl top<\/code>.<\/li>\n<li><strong>Put etcd and databases on SSD first.<\/strong> Even a very small tier gives the highest return on a limited budget.<\/li>\n<li><strong>Build node tuning into a golden image<\/strong> so that inotify limits, <code>max_map_count<\/code> and swap settings are in place before the first workload arrives.<\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\">The exercise confirmed what the hardware suggested from the start: this storage was the wrong choice for this platform. What it added was a detailed picture of how that mismatch shows up in practice, as probe failures, corrupted state, evictions and misleading symptoms, rather than as a clear storage alarm. It also showed how far measurement can go. A small amount of well-placed fast storage and a handful of measured settings turned an unstable deployment into a working 28-application platform, with no change to the compute.<\/p>\n<p class=\"wp-block-paragraph\">Those failure patterns are the real takeaway. They are much cheaper to learn in a deliberate test than during a production incident.<\/p>\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Get the next one in your inbox<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.<\/p>\n\n\n\n\t\t\t<form id=\"sib_signup_form_2\" method=\"post\" class=\"sib_signup_form\">\n\t\t\t\t<div class=\"sib_loader\" style=\"display:none;\"><img\n\t\t\t\t\t\t\tsrc=\"https:\/\/yyc.ninja\/wp-includes\/images\/spinner.gif\" alt=\"loader\"><\/div>\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_action\" value=\"subscribe_form_submit\">\n\t\t\t\t<input type=\"hidden\" name=\"sib_form_id\" value=\"2\">\n                <input type=\"hidden\" name=\"sib_form_alert_notice\" value=\"Please fill out all required fields.\">\n                <input type=\"hidden\" name=\"sib_form_invalid_email_notice\" value=\"Please enter a valid email address.\">\n                <input type=\"hidden\" name=\"sib_security\" value=\"ddf681221a\">\n\t\t\t\t<div class=\"sib_signup_box_inside_2\">\n\t\t\t\t\t<div style=\"\/*display:none*\/\" class=\"sib_msg_disp\">\n\t\t\t\t\t<\/div>\n                    \t\t\t\t\t<p class=\"sib-email-area\">\n    <input type=\"email\" class=\"sib-email-area\" name=\"email\" placeholder=\"you@example.com\" required=\"required\">\n<\/p>\n<p class=\"sib-FIRSTNAME-area\">\n    <input type=\"text\" class=\"sib-FIRSTNAME-area\" name=\"FIRSTNAME\" placeholder=\"First name (optional)\">\n<\/p>\n<input type=\"hidden\" class=\"sib-SOURCE_DOMAIN-area\" name=\"SOURCE_DOMAIN\" value=\"yyc.ninja\">\n<input type=\"hidden\" class=\"sib-SIGNUP_SOURCE-area\" name=\"SIGNUP_SOURCE\" value=\"post\">\n<p>\n    <input type=\"submit\" class=\"sib-default-btn\" value=\"Join the Ninja Dispatch\">\n<\/p>\n<p class=\"ninja-fineprint\">No spam. Unsubscribe anytime.<\/p>\t\t\t\t<\/div>\n\t\t\t<\/form>\n\t\t\t<style>\n\t\t\t\tform#sib_signup_form_2{background:#192a3d;border-radius:10px;padding:16px;font-family:inherit}form#sib_signup_form_2 input[type=text],form#sib_signup_form_2 input[type=email],form#sib_signup_form_2 select{width:100%;box-sizing:border-box;margin:0 0 10px 0;padding:10px 12px;border:1px solid #33465c;border-radius:8px;background:#0f1c29;color:#e8eef5;font-size:14px}form#sib_signup_form_2 input::placeholder{color:#8a9bb0}form#sib_signup_form_2 .sib-default-btn{width:100%;margin:2px 0 0 0;padding:11px 14px;background:#e63946;color:#ffffff;border:0;border-radius:8px;font-size:15px;font-weight:600;cursor:pointer}form#sib_signup_form_2 .sib-default-btn:hover{background:#d02f3c}form#sib_signup_form_2 p{margin:0 0 10px 0}form#sib_signup_form_2 .ninja-fineprint{margin:8px 0 0;font-size:11px;color:#8a9bb0;text-align:center}form#sib_signup_form_2 p.sib-alert-message {\n    padding: 6px 12px;\n    margin-bottom: 20px;\n    border: 1px solid transparent;\n    border-radius: 4px;\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n}\nform#sib_signup_form_2 p.sib-alert-message-error {\n    background-color: #f2dede;\n    border-color: #ebccd1;\n    color: #a94442;\n}\nform#sib_signup_form_2 p.sib-alert-message-success {\n    background-color: #dff0d8;\n    border-color: #d6e9c6;\n    color: #3c763d;\n}\nform#sib_signup_form_2 p.sib-alert-message-warning {\n    background-color: #fcf8e3;\n    border-color: #faebcc;\n    color: #8a6d3b;\n}\n\t\t\t<\/style>\n\t\t\t\n","protected":false},"excerpt":{"rendered":"<p>Capacity planning for Kubernetes usually starts with CPU and memory. Storage latency tends to get less attention, partly because well-provisioned clusters rarely expose what happens when it runs out. So I ran an exercise to expose it: deploy a full data platform on storage that I knew was far too\u2026<\/p>\n","protected":false},"author":1,"featured_media":3583,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[44,46,45],"class_list":["post-3584","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-how-to","tag-data-engineering","tag-mlops","tag-streaming"],"featured_image_urls_v2":{"full":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover.png",1200,630,false],"thumbnail":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-150x150.png",150,150,true],"medium":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-300x158.png",300,158,true],"medium_large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-768x403.png",768,403,true],"large":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-1024x538.png",1024,538,true],"RoboGalleryMansoryImagesCenter":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-600x630.png",600,630,true],"RoboGalleryPreload":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover-100x53.png",100,53,true],"1536x1536":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover.png",1200,630,false],"2048x2048":["https:\/\/yyc.ninja\/wp-content\/uploads\/2026\/11\/cg-when-storage-is-the-bottleneck-capacity-lessons-lakehouse-ku-cover.png",1200,630,false]},"post_excerpt_stackable_v2":"<p>Capacity planning for Kubernetes usually starts with CPU and memory. Storage latency tends to get less attention, partly because well-provisioned clusters rarely expose what happens when it runs out. So I ran an exercise to expose it: deploy a full data platform on storage that I knew was far too\u2026<\/p>\n","category_list_v2":"<a href=\"https:\/\/yyc.ninja\/index.php\/category\/how-to\/\" rel=\"category tag\">How To Ninja<\/a>","author_info_v2":{"name":"YYC Ninja","url":"https:\/\/yyc.ninja\/index.php\/author\/admin\/"},"comments_num_v2":"0 comments","_links":{"self":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3584","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/comments?post=3584"}],"version-history":[{"count":1,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3584\/revisions"}],"predecessor-version":[{"id":3585,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/posts\/3584\/revisions\/3585"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media\/3583"}],"wp:attachment":[{"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/media?parent=3584"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/categories?post=3584"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yyc.ninja\/index.php\/wp-json\/wp\/v2\/tags?post=3584"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}