Agentic Web AEO Enhancements & CSP Security Policies — August 24, 2026

Today, I tackled several critical enhancements for the agentic web, focusing on AEO improvements, security policies, and user experience. I spent a solid five hours working through a series of eleven commits that not only improved functionality but also addressed some existing issues.

One of the standout features I shipped was the expanded '/llms-full.txt'. This new addition provides a complete public portal directory, linking to various LLMs with a detailed cost/metro/industry URL matrix. Given that agents increasingly probe this endpoint first, it was crucial to ensure that it was comprehensive and easily accessible. This feature was a significant improvement in the agentic SEO landscape, making it easier for users to find relevant information quickly.

I also addressed some security concerns by implementing a stricter Content Security Policy (CSP). I allowed specific hosts like 'static.cloudflareinsights.com' for script sources, which improves the reliability of the Real User Monitoring (RUM) tools I have in place. However, this wasn't without its challenges. The 'first-enforce-day' console showed blocked requests that needed to be debugged, highlighting the importance of thorough testing before deploying CSP changes.

Moreover, I made significant adjustments to the sitemap. Previously, I was stamping 'lastmod=now' on static routes, which inadvertently taught Google to distrust the 'lastModified' field where it was genuinely applicable. I stopped this practice, ensuring that evergreen routes now omit 'lastModified' timestamps, which should help with better indexing and trustworthiness in search results.

One of the more frustrating bugs I encountered was related to the 404 page. My initial implementation redirected all dead or mistyped URLs to a catch-all page, which inadvertently created soft 404s that cluttered Google Search Console reports. After some troubleshooting, I built a real 404 page that returns an honest status code. This change not only improves the user experience—guiding humans to a map or home page—but also provides crawlers with accurate status information.

In terms of performance and security, I removed the 'X-Powered-By' header to eliminate unnecessary framework fingerprints. I also adjusted caching settings for marketing assets, ensuring that they were cached effectively without being re-fetched unnecessarily. This was a minor tweak that, while it seemed trivial, could lead to significant performance improvements over time.

I also made some changes to the authentication flow. Previously, the '/api/auth/me' endpoint would return a 200 status with a user object of {user:null} for anonymous users, which led to red console errors on every page load. I fixed this by returning a 401 status instead, providing a clearer signal for the front-end useAuth hook. This not only cleaned up the console but also improved the developer experience when debugging.

Overall, today was a blend of enhancements and fixes that contribute to building a more robust agentic web platform. I find it fascinating to see how each small change can ripple through the system, improving both performance and user experience. It’s a constant balancing act between innovation and stability, especially when I’m doing this solo with Claude Code in VS Code.

As I continue to build toward my goal of a one-man-show company with a billion-dollar valuation, these incremental improvements are vital. They not only enhance the product but also serve as a solid foundation for future features. It’s rewarding to see the project evolve, even if it comes with its share of challenges and time-consuming debugging sessions. Each commit is a step closer to realizing my vision for the agentic web, and I’m excited to see where this journey takes me next.