Portal SEO Enhancements & Bot Detection Improvements — May 26, 2026

Today, I set out to enhance the SEO capabilities of my portal while improving bot detection features. After a long 8-hour day, I successfully shipped 13 commits, including 11 features and improvements, alongside 2 crucial fixes. The main goal was to make the portal more accessible to crawlers and LLMs, thereby enhancing its visibility on the agentic web.

One of the standout features I implemented was the introduction of a dedicated crawler tracking system. Now, users can see which crawlers and LLMs have visited their portals, displayed prominently below the portal cover image. This not only provides valuable insights into traffic but also creates opportunities for better optimization. I expanded the bot allow-list to include over 50 different crawlers, ensuring that we cater to a wide variety of bots, including those from major search engines and AI platforms.

In addition to crawler tracking, I introduced 3D Seen-by pills, which visually represent the bots that have interacted with the portal. This enhancement not only adds a layer of interactivity but also makes the data more digestible for users. Users will now find the 'Seen by' label above the bot pills, allowing for a clearer understanding of which bots are engaging with their content.

However, the day wasn’t without its challenges. I encountered a bug in the bot detection logic where the TelegramBot was not being prioritized correctly due to a conflict with the Twitterbot detection. This issue was particularly tricky as it required diving into the user-agent strings to ensure that the detection matched correctly. After some back-and-forth debugging, I was able to resolve this by adjusting the order of the bot matches, ensuring that the TelegramBot would be correctly recognized before the Twitterbot. This fix ate up a chunk of my time but was crucial for ensuring accurate bot detection.

Another significant enhancement was the public LLM-callable POST API for asking questions about portals. This feature allows external LLMs to make requests and receive structured responses, which is a big step toward making the portal more interactive and user-friendly. I implemented rate limits to prevent abuse and included comprehensive JSON-LD documentation to help developers understand how to interact with this new API endpoint. The documentation was an additional layer of work that I initially underestimated but ultimately felt was essential for usability.

On the SEO front, I focused on creating a dedicated sitemap for the portal, which users had requested. This involved generating a new /portals/sitemap.xml that lists every public portal URL, complete with updated timestamps for better indexing. I also added nofollow attributes on event portals to control how search engines interact with those pages. These SEO improvements are foundational for ensuring that our content can be discovered by both traditional search engines and AI systems.

I also made some user experience tweaks on the home page. I increased the number of orbs displayed in the SpaceLevelsSection and adjusted their opacity to create a more vibrant and engaging visual experience. Users had reported that sections below the hero looked flat, so these adjustments aimed to bring more life and color to the interface.

Reflecting on today, I’m proud of the progress I made, especially considering that I did this all solo with Claude Code in VS Code. It’s a testament to how powerful AI can be as a partner in this journey. These enhancements not only push the boundaries of what the portal can do but also inch me closer to my long-term goal of building a one-man-show company with a billion-dollar valuation. Each commit is a step toward that vision, and I’m excited to continue this journey of building in public, sharing insights, and learning through the process.