Graphite

What are the top ChatGPT Ads advertisers doing differently? Last week we hosted a webinar on ChatGPT Ads with Harel and Liat from Similarweb and the great Yoav Rubin from monday.com. Here's the playbook I'm seeing on the top advertisers run: ## On targeting (this is what the image in this post is about) - IMO the most interesting insight: they found ways to target new intents that are unique to ChatGPT, ie Generative Intent. - Their campaigns are built around intent, not keywords. Eg. monday.com splits into three buckets: tool seekers (shopping for a specific tool, the closest thing to search, this can include branded terms), problem seekers (exploring a problem, before thinking about a tool), chat actions (mid-task, "build me a quarterly plan"). - The top accounts all target roughly these 3 groups of intent: a. Generative intent or "actions", this is very often pre-problem. Eg. "Paste your ChatGPT draft. Pick a template." (Resume.io), "Paste Your Notes. Get a Deck." (Gamma) b. Problem aware: "Track KPIs without the spreadsheet" (monday.com), "End Your Meeting Fatigue" (RingCentral) c. Solution aware: this covers the classic high intent prompts ("I need project management tool") but also own brand and competitor brands. In the visual, we mapped real ads from top advertisers to the categories that the ChatGPT team used in their paper that describes how people use it. ## On Creative - They run a lot of creative variety. The top 50 advertisers run a median of 191 distinct ads! (vs ~10 ads for advertisers that are only getting few impressions). - They write unique copy for the channel. Very few ChatGPT headlines match the same advertiser's Google Search headline. - They all lead with the action, not the category or the brand. Eg. monday.com's headlines start with a verb so you see what you can do and what you get: "Build a live project dashboard", not "#1 project management software". ## On Landing pages - They have a unified flow that connects copy, image and landing page, all hinting at the same action. If the ad says "build a dashboard", the page very clearly healps the user to build a dashboard. - They have dedicated landing pages that help users move their chat from ChatGPT to their product. Eg. Resume.io runs dedicated landing experiences built for the conversational context (follow Steen who has been posting some good insights) - This is also not a surprise given that we know how landing pages are a strong targetin signal as well -- If you want to learn more, lmk and I'll send you the recording and maybe a skill we are building that has our playbook built in :)

How Long Does It Take To Impact AEO? * I have previously said that while SEO takes time (3+ months usually), optimizing for AI can happen faster (days). This is partially true. * Many AI answers do not use search and do not have citations (10%-80% of answers have citations depending on how you decide which prompts to focus on). Answers without search and citations rely entirely on the underlying model's parametric knowledge. * According to our analysis of public disclosures from LLM companies, it takes at least 2 to 5 months for AEO initiatives to affect parametric knowledge. Why Does It Take So Long? * New models are released every few months. Further, new models have a "training data cutoff" which ranges from a few weeks to a few months old. So, a model released today isn't trained on data released yesterday. There is a time delay to train these new models. Why This Matters We recommend complementing strategies based on citations with parametric knowledge optimization (PKO): * Most AEO strategies focus on citations and ignore the model’s parametric knowledge, the information stored in the model’s weights. * But not all responses use search, and even when they do, the model’s parametric knowledge can shape both its fan-out queries and how it uses search results to answer. * Therefore, we should also focus on PKO, which aims to improve how models describe and recommend a brand by influencing the training set. Measurement * Rather than only measuring weekly and looking for incremental trends, also look for large discrete jumps among different model versions. * When you see a change in a new model, check our paper or the LLM to see the training data cutoff and only draw conclusions about changes you made prior to the training data cutoff. Anything after necessarily cannot have an effect until a newer model has it in its training data. * Use APIs with web search disabled to ensure responses are based only on parametric knowledge. Collect at least fifty responses per prompt. This process is needed only once per pinned model version, since the response distribution will not change as long as the prompts, system instructions, and sampling settings remain fixed. Question Has anyone experimented with ways to optimize the parametric knowledge? Paper link in the comments. + Gregory Druck Who is the primary author on this paper.

It takes at least 2 to 5 months (depending on the model) for published content or new mentions to affect the model’s parametric knowledge, whereas new content can be cited (via search) in days. * New models released by OpenAI, Anthropic, and Google are typically trained on data that is several months old. * The latest date when pages are included in the training set is called the knowledge cutoff. * We use the term knowledge cutoff gap to mean the amount of time between the knowledge cutoff and the release date. * The knowledge cutoff gap is the minimum time-to-impact for newly published content. The attached shows knowledge cutoff gaps for models released since 2022. Why is this important? We recommend complementing strategies based on citations with parametric knowledge optimization (PKO): * Most AEO strategies focus on citations and ignore the model’s parametric knowledge, the information stored in the model’s weights. * But not all responses use search, and even when they do, the model’s parametric knowledge can shape both its fan-out queries and how it uses search results to answer. * Therefore, we should also focus on PKO, which aims to improve how models describe and recommend a brand by influencing the training set. Recommendations: * Be aware of knowledge cutoffs and gaps. Do not expect impact from parametric knowledge optimization in two weeks when the knowledge cutoff gap is five months. * Do not expect impact in a specific model when changes were made after the knowledge cutoff for that model. * When new models are released, supplement standard prompt tracking with parametric knowledge visibility tracking. Use APIs with web search disabled to ensure responses are based only on parametric knowledge. Collect at least fifty responses per prompt. This process is only needed once per pinned model version, as the response distribution will not change as long as the prompts, system instructions, and sampling settings remain fixed. Paper link in the comments.

Parametric Knowledge Optimization & AEO LLM vs. LLM + Search * LLM - LLMs are trained on billions of documents to build their models. Parametric knowledge is the knowledge an LLM learns during training. * LLM + Search - Sometimes, LLMs also perform search or other tasks to add necessary context. For example, if you ask what the weather is, an LLM won't know, but it can retrieve live weather information via search to help answer this question. AEO Is Not Just Citation Optimization * Most AEO strategies aim to improve a brand’s visibility, position, and sentiment in AI answers by growing its citation share or earning mentions in cited third-party sources. * Most AEO strategies ignore the LLM and its parametric knowledge, the information stored in the model’s weights. Many AI Answers Don't Use Search * Depending on the data source, search is used between 10%-80%. For many entity queries (best X for Y), search is used often. However, many other prompts don't use search. Similarweb estimates that roughly only 10% of ChatGPT answers have citations. Winning Citations Isn't Enough * Even when search is used, being cited is not enough. We present evidence that a brand with low visibility in the LLM without search rarely appears even when it's cited with search (11% of the time). Brands that appear often in the LLM without search appear almost always with search. * So, much of what drives which brands are mentioned is the parametric knowledge of the underlying model (even when search is used). Parametric Knowledge Optimization * For AI answers without search, optimization of search and citations has little to no effect. Instead, you need to optimize the parametric knowledge. * We need a new set of AEO strategies to optimize for the parametric knowledge. Measuring Parametric Knowledge Is Hard * Unlike citations, we cannot directly access the inputs to the underlying model. * New models are launched every few months, and there is a cutoff to the training data. So, a change made today will take several months to have an effect. Parametric Knowledge Influences Fan-Out Queries * Many AI answers use fan-out queries to gather information. Many of these fan-out queries are site:SITE.com queries, such as -- site:nike.com best running shoes -- as a fan-out. * These heavily affect AI answers. The underlying model drives these fan-out queries. To win these answers, you need to optimize the parametric knowledge.

How To Gather Real AI Prompts * There is much focus on how to optimize brands for particular prompts in AI. However, most people guess what prompts they should optimize for. This can call the entire strategy into question. * Outside of Similarweb, in our opinion, there are no reliable data sources that show users' real prompts. Existing tools are guessing based on dubious underlying data. * We propose a way to gather real user prompts in AI at scale for free. * Guide: https://lnkd.in/epTWRPDh How Did You Hear About Us? * Attribution in AI is also hard. One solution is to ask users to self-report the channel primarily responsible for their conversion. This is often done by asking “How Did You Hear About Us?” * Self-reporting can also be extended to gather more information, namely, real user prompts. What Prompt Did You Use? We built a new conversion funnel at Graphite that asks the following questions. 1. How Did You Hear About Us? (channel) 2. What LLM Did You Use (product) 3. What prompt did you enter (prompt) This now tells you not only whether a user entered via AI, but also which LLM they used and the prompt they entered. Results * We now see 30 % of inbound leads come from AI. 52% came from ChatGPT and 40% from Claude. * We receive 10x more referral traffic from ChatGPT than Claude, while the % of leads that self-report Claude is almost as high as ChatGPT. This suggests conversions via Claude are particularly underreported using last-touch, far more than ChatGPT. * AI leads tend to be less qualified than other channels like referrals. * Only 21% of leads that self-report as coming from AI appear as referral traffic from AI. The rest came from search (49%) and direct (30%).

ChatGPT attributed organic sessions in GA4 fell ~45% across a number of sites we track This is a bug on OpenAI's side and I expect them to fix it very soon. ## What happened - Starting Oct 9, all sites with meaningful AI Assistant traffic in GA4 dropped (from 22-57%) while Google and Direct have stayed roughly flat or grew. The first dip shows on Oct 8, the big step on Oct 9. This coincides with a few of their big releases last week. - In short, what happened is that ChatGPT started opening links with noreferrer and without utm_source=chatgpt.com. With no source, GA4 will file the visit as Direct, instead of as AI Assistant. ## What were seeing now - Sources panel and shopping links still carry the UTM. Inline links don't. The referrer was a desktop web signal. 95% of mobile ChatGPT visits relied on the UTM alone, so app traffic is most exposed - Tomek Rudzki from Peec AI also reported that with the new model ChatGPT now links straight to sites ~4-5x more, esp to homepages, and those links are untagged ## How to account for it in reporting - Annotate Oct 8 - ChatGPT numbers across that date aren't and until things are fixed aren't reliable - Look at ChatGPT and homepage direct together. For example, on one of the sites I looked at homepage direct traffic rose 17% (while Google was flat) w/ all new visitors - Watch new-user direct on pages that got ChatGPT traffic before - ChatGPT Ads clicks with your own UTMs should be unaffected (see my other guide on this) - I would use this opportunity to push for an attribution survey if you don't have one (ask "how did you hear about us" post sign up).